Contingency takes place after its trigger, so the trigger you suggested wouldn't prevent damage from meteor swarm
From the description of the contingency spell:
The contingent spell takes effect immediately after the circumstance is met for the first time, whether or not you want it to, and then contingency ends.
As you can see, contingency occurs after the triggering circumstance (the same way readied actions do). As a result, if you declare the triggering circumstance as "when someone casts a meteor swarm in an area that includes me", the contingent banishment spell will only take effect after meteor swarm finishes being cast.
Since the duration of meteor swarm is instantaneous, there's no extra step between the spell being successfully cast and the meteors being brought down (and doing damage). Thus, the spell would only trigger after you'd already been hit by the spell.
It doesn't matter since the timing doesn't work, but even if you found a way to cast banishment on yourself before the meteor swarm hit you, it wouldn't help; since you would be incapacitated the moment you finished casting the spell, the spell would instantly end and you'd immediately be un-banished, as Jeremy Crawford confirms here. (...Assuming you're on your native plane, of course.)
So even if you managed to find a way to banish yourself before the meteor swarm went off, you'd find yourself back where you were before anything else happened.