Using Opposed Ability Checks / Contests
Let's examine two cases of trying to do this: one with the Grappler Feat and one without.
With the Grappler Feat
If the Barbarian has the Grappler Feat, the grapple does more than simply reduce speed to zero. It allows a chance to restrain the caster with a subsequent action. The pinned/restrained condition does not explicitly say that it prevents spell casting ... but let's work with what we have.
... a Strength (Athletics) check contested by the target’s Strength (Athletics) or Dexterity (Acrobatics) check (the target chooses the ability to use). If you succeed, you subject the target to the grappled condition
The significant grappled condition is: A grappled creature’s speed becomes 0, and it can’t benefit from any bonus to its speed. As you note, that by itself is no prevention to spell casting (unless a spell requires a movement component).
Escaping a Grapple
A grappled creature can use its action to escape. To do so, it must succeed on a Strength (Athletics) or Dexterity (Acrobatics) check contested by your Strength (Athletics) check.
Each turn of grapple is a contest: grappled creature can to get free, which might include casting a spell to do so.
Action Economy Note: this attempt to escape may preclude using an action to cast a spell, but any bonus action or reaction spell casting is still available should the caster try to escape using an action.
Using a subsequent action ... with the feat
• You can use your action to try to pin a creature grappled by you. To do so, make another grapple check. If you succeed, you and the creature are both restrained until the grapple ends.
Restrained Condition from Appendix A: (You both are restrained)
• A restrained creature’s speed becomes 0, and it can’t benefit from any bonus to its speed.
• Attack rolls against the creature have advantage, and the creature’s attack rolls have disadvantage.
• The creature has disadvantage on Dexterity saving throws.
No disadvantage on attempt to get out of the grapple. On subsequent rounds you can use non-weapon attacks (advantage for attacks against caster are canceled out by disadvantage for attacks on the caster).
Note: If your party cleric wants to cast Sacred Flame at this caster, the target has disadvantage on the Dex save ... it's a single target cantrip. But that's not what you are trying to accomplish.
Even with the feat, the DM has to make a ruling.
Recommended Ruling: require an additional contested check to stop the caster from trying to cast during each turn. The caster is resisting with a will. (And likely trying to break the grapple). There is no easy button, the enemy fights back.
Note: Spells with only a verbal component are not going to be stopped. For example, a Command to "grovel" that the Barbarian fails his save on should break the grapple.
Without the Grappler Feat
Reduce the target to speed zero, no explicit constraints otherwise, and of course each turn the caster can try to break grapple. So far, this kind of detailed restraint, while familiar in a cinematic sense from TV and movies, doesn't have explicit rules support with a simple Grapple. Since the Barbarian has not paid the price for the feat, simple grapple should be less effective than grapple with feat.
Rulings over Rules
I recommend that if you want to add this feature or capability, and there's no feat, the Barbarian has to attempt a second Grapple after a successful grapple, each turn/round, with *disadvantage. (Otherwise, why the Grappler feat?). Likewise, the caster isn't restrained. I'd also rule that the Barbarian can make no other attacks when trying this advanced grapple without the feat.
Disadvantage reflects how much harder it is to control a resistant opponent, but leaves open a chance for some heroics or unique happenings during the game.
Inspiration, Advantage and Disadvantage
Does the Barbarian have an inspiration point, previously awarded? The DMG pages 240-241 introduces the use of inspiration:
Awarding Inspiration is an effective way to encourage roleplaying and risk taking ... the character can have no more than one Inspiration at a time.
The lesser case of a bardic inspiration point helping a given roll isn't as powerful as DM awarded inspiration, which gives advantage, but it can't hurt to add a few points to the die roll.
You usually gain advantage or disadvantage through the use of special abilities, actions, or spells. Inspiration can also give a character advantage. The GM can also decide that circumstances influence a roll in one direction or the other and grant advantage or impose disadvantage as a result.
This is the kind of scenario where, as a GM, I see the Inspiration rules as providing that "one special time" that the player with an inspiration point burns that one precious point and gets advantage, to cancel disadvantage when trying to pin/restrain the caster. It is not guaranteed to work, but it can tip the odds in the Barbarian's favor for that first round.
If circumstances cause a roll to have both advantage and disadvantage, you are considered to have neither of them, and you roll one d20.
If the player has not earned an inspiration point, it's less likely that the cool thing he's trying to do will work. If he's earned that inspiration point, there's a better chance he pulls it off.
Inspiration aside, and depending upon circumstances, you can always apply advantage or disadvantage for a given attempt if a circumstance suggests to you that it makes sense. (For either the barbarian, or for the spell caster trying not to be controlled like this). That's one of those judgment things that you'll rule on in each situation as it arises.
No "Easy Button"
Don't forget: every turn, the caster gets that chance to escape the grapple.