I want to make a melee weapon that, if you score a critical hit, casts banishment on the target. I don't mind it being of a high rarity, but I don't want it to be broken. I wasn't planning on making this effect limited to once per day or even limited by charges (although if it turns out that this weapon is broken without it, I will consider charges). The spell save DC of 15 is also arbitrary at the moment, based on a very rare weapon.
Weapon (any melee weapon), ??? (requires attunement)
When you roll a 20 on an attack roll made with this weapon, you cast banishment on the target you hit if the target is a creature (save DC 15). The weapon maintains concentration on the spell, which can only be interrupted if you roll another 20 on an attack roll made with this weapon before the spell's duration has elapsed.
(In other words, with respect to that last sentence in my magic item's description, if you crit twice within a minute, then the first target reappears again, i.e. isn't banished)
It's only a 4th level spell, which you cannot cast by choice but rather is cast on random chance, so I'm not sure what the rarity of this effect will be. If it's super-legendary or something, I can introduce charges to limit it and bring the rarity back down to something balanced, but if it's actually not as bad as I'm thinking, then I might make it a +1 or +2 weapon as well to bump it up to very rare (which is roughly what I'm aiming for).
So, what is the rarity of a melee weapon that casts banishment every time you crit?