Maybe not quite balanced, but a fantastic idea!
First of all, I love the concept of this spell. It's an idea whose effect logically matches its flavor description, and is an interesting way to majorly inconvenience a primarily spellcasting creature.
There are a couple of features I am confused by here. For example, I wonder if a character casts a spell with a longer casting time than one action, do they have to make this check on each of their turns? But these details are relatively minor. Again, it's a great idea.
That being said, there are a few reasons I consider this spell unbalanced.
It's probably too low level (Should be 6th)
This spell is in a rather rare category of "suck with no save" spells. The target is guaranteed to be influenced by this spell, and while it's possible that someone under its effects could successfully cast all of their spells, even the possibility that they could waste their turn at any given time is rather daunting.
One of the rare other examples of such spells is Otto's Irresistible Dance (PHB, p. 264).
Casting Time: 1 action
Range: 30 feet
Duration: Concentration, up to 1 minute
Choose one creature that you can see within range... Creatures that can’t be charmed are immune to this spell.
A dancing creature must use all its movement to dance without leaving its space and has disadvantage on Dexterity saving throws and attack rolls. While the target is affected by this spell, other creatures have advantage on attack rolls against it. As an action, a dancing creature makes a Wisdom saving throw to regain control of itself. On a successful save, the spell ends.
In a lot of ways, your proposed spell is similar to the above spell. Both spells mainly influence one broad category of characters (your spell influences caster's while Otto's influences martial characters), both give an additional d20 to rolls that could cause it to fail (disadvantage in Otto's case, concentration in yours), and both have no saving throw.
Naturally there are some differences between the two spells. For example, your spell doesn't influence the defense of the target, while Otto's does (by giving advantage on attacks against the target and imposing disadvantage on Dex saving throws). Also, your spell doesn't influence the target's mobility, but it also doesn't give any opportunity for the target to end the spell early (other than causing the caster to stop concentrating). On the other hand, your spell will cause the target to waste their entire turn for failing one "concentration check", while most higher level targets get multiple attacks and could counter the disadvantage by gaining advantage.
It's debatable whether the differences are equivalent, but I think they're very close. All things considered, I'd say that this spell is pretty much equivalent to Otto's Irresistible Dance. But that begs the question why it is a full level below it.
Is one level such a big deal? (Yes)
The difference between a 5th and 6th level spell slot is very significant. Not only is it available two levels earlier(Level 9 vs Level 11), but 5th level spell slots can be used twice as often for most of a caster's career (as full casters only gain a second 6th level spell slot at level 19). So I'd recommend bumping this spell up to 6th level, or lowering its power somehow (such as by giving the target the option to spend an action to make a Wisdom save to attempt to end the spell). This is especially significant as you're considering giving this spell to Sorcerers and Warlocks. These two classes can get a large number of 5th level spell slots (through Sorcery Points or short rests respectively).
Other than that, I think this spell is great. It fills a significant hole in the current system, for a spell to inconvenience casters between Counterspell and Antimagic Field. It's clever, fun, and worthwhile. Bump it up to a 6th level spell, and you should be all set.