There does not seem to be a clear-cut way to determine when a paladin's actions have become egregious enough to justify them breaking their oath. Do a lot of minor slights eventually add up and cause an oath to be broken? Do they get some sort of warning like losing their laying on of hands ability?
You're right. There is no clear cut way. That's by design. How this works varies a lot by playstyle:
Some groups have a solid understanding of ethics and jurisprudence and use some limited form of said understanding adjudicate alignment in D&D. Others use their full understanding of ethics, which cumulates for them into a D&D compatible system, to perform that adjudication. These groups are very similar except that the first group tends to have less issues adjudicating alignment in difficult situations where the latter group discovers holes in their beliefs about morality. In both of these cases alignment is adjudicated according to the chosen moral/ethical system, and what constitutes oath-breaking will be adjudicated similarly according to the related principles of jurisprudence. As moral relativism becomes, unfortunately, increasingly popular, objective alignment becomes less and less so, especially amongst simulationist groups, and so these two methods of alignment adjudication are unlikely.
Some groups think alignment is stupid, meaningless, and entirely subjective. These people are very vocal. They likely also think that oathbreaking is similarly problematic. Like alignment-based powers in previous editions, in these groups if you're going to use anything that's based off alignment or alignment-like systems (like the paladin class) you should first check with your DM to find out 1) if it's allowed and 2) what it means in the context of the campaign. One common solution in the past in these kinds of groups has been just to waive the alignment restriction entirely, for the Paladin class.
Some groups think that obviously alignment is objective, because it's in the rules, but they just don't understand it yet. These groups will post difficult alignment questions as they come up on online fora. The outcome of alignment based actions in such groups will be fickle and likely a point of tension.
Some groups try and adjudicate this according to the original inspiration for the alignment system, which is a set of novels written by Michael Moorcock. This is hard because the novels are novel-y and don't explain exactly what Law and Chaos are in cut and dry terms. Such groups would probably not adjudicate the oath the same way they do alignment, because Jurisprudence is fundamentally a principle of Balance, not Law, in that system, and there's almost nothing to go on for how something like the 5e paladin oath should be dealt with.
In any case, if you are not the DM, ask your DM how the metaphysics of ethics and jurisprudence are related in their game. If they are related, seek to focus more on the adjudication of ethics than jurisprudence as the ethical component is probably dominant in most GM adjudications. If they are not related, ask about Jurisprudence in general, and the adjudication of your class abilities in particular, if necessary, focusing on what an Oath means, what it means to break it, and how reparations might be made if the oath is indeed broken.
If you are the DM, I strongly recommend you consciously decide what philosophical system of justice you will use in your campaign, as all of the other options amount to unconsciously using some system and putting some introspective thought into this choice before you start running the game is likely to make these kinds of decisions much easier and much faster during the game sessions, since you will spend much less time figuring out how to approach each specific case.
If you want a place to start looking at different approaches to the philosophy of oaths, I recommend this paper.
First and foremost, the only clear-cut answer one might obtain on this subject would be from the DM running the specific campaign in question. The classification of actions as lawful, chaotic, good, or evil and the relative magnitude of the action relative to the classification (i.e., the action is evil, but just how evil?) is based entirely on the ethos of the particular religion, culture, details of the outer plane which is the source of the alignment, etc. That said, detailed knowledge of these things is generally not necessary.
More important is understanding why ‘falling from grace’ is included in the rules in the first place. Like most other character restrictions and limitations in the rules, this one is included to help enforce balance between the character classes. Paladins are granted a number of special abilities that are of significant advantage. Immunity to fear and disease alone are high order advantages. These advantages are offset by a number of equally strict limitations such as the single lawful good alignment. One might fancy a Paladin views (or should view) strict discretion and application of great power to be as much a measure of ‘power’ as the effects of its use.
The methodology behind the mechanics, however, must always be balanced against the reason for playing the game in the first place; to have a fun and meaningful game play experience. The Paladin’s ‘fall from grace’ restriction/consequence is purposefully left vague and is intended more as a threat or failsafe than as a rule to be implemented. That said, use it as such. Certainly actions deemed as utterly evil should be met with immediate expulsion from the ranks of the chosen. However, assuming the player is attempting to stay in the spirit of the game, such blasphemy will be very rare. Most potentially ‘damning’ acts fall into a gray area of lesser sin. Continuing onward with the assumption of a well-intentioned player, a good dm should ‘guide’ the paladin in such a way as to bring clarity to the ‘correct’ path of behavior when deviation has or is about to occur.
There are several typical methods to impart this ‘wisdom’ upon the character and these are often most effective when used in combination. Providing a simple verbal warning of a sort is the more kind of these tactics. This may be a direct ‘dm to player’ dialogue or something a bit more subtle and ‘in-game’. “You are suddenly blinded by an intense light. A moment later the glare from the noon sun striking the surface of the lake ahead subsides with the passing of a cloud, but you can’t seem to shake the feeling you have been given a sign…” Alternatively, a post-action penance of some sort may be mandated by a high authority local theocrat, or demanded from a brief encounter with an ‘angel’ or other minor servant of the cause. A quest to find some sacred artifact and a hefty tithe to the local order will often curb even the most ignorant of players. The severity should fit the score so to speak, but a few of these instances are usually enough to make the point. Ultimately, the actual act of stripping the character of Paladin status should only be undertaken when multiple warnings and/or penalties are ignored or as a last resort of sorts in any case.
My opinion is that, for the most part. it is when they break an obvious part of their oath (like using poison for example) or willing commit a truly evil act with no good end.
Though on specific part of oaths. I feel that there are exceptions that a specific paladin can have. I'll use an example I had from a character
Scandal: The character had a flaw/compulsion to be a dracophile from his dragon blood. So I said that to avoid falling from favor and becoming shunned by the people he protected, he could lie about that part of his personality.
I also said that as long as their act wasn't grievous and for a truly good end, he could avoid almost any provocation (for example: a Paladin that kills the corrupt king of a corrupt land wouldn't fall from grace)
To sum up: I focused on the idea that justice is to some extent more important than their oaths.