They should prevent, but there are no rules for this.
As far as I have read there are no indications that conditions as restrained and grappled affect spellcasting at all. The problem is that there are evidence that "proves" that those conditions should affect spellcasting, particularly the restrained condition. One of these rules is Armor Proficiency (PHB 144, emphasis mine). Armor proficiency stipulate that, without proficiency, the caster is unable to cast spell in an armor. In PHB 201 there is a small section that goes a little deeper in the reasons why using an armor that you are not proficiency interfere with spellcasting. From this segment we can see the implications of wearing armor and hampering spellcasting.
Armor Proficiency: Anyone can put on a suit of armor or strap a shield to an arm. Only those proficient in the armor’s use know how to wear it effectively, however. If you wear armor that you lack proficiency with, you have disadvantage on any ability check, saving throw, or attack roll that involves Strength or Dexterity, and you can’t cast spells.
Casting in Armor: Because of the mental focus and precise gestures required for spellcasting, you must be proficient with the armor you are wearing to cast a spell. You are otherwise too distracted and physically hampered by your armor for spellcasting.
Since being too distracted and physically hampered is a reason to nullify spellcasting, other conditions that physically hamper the caster would, and should, also hamper spellcasting (game balance aside). The closest and easiest condition that apply hampering conditions similar to the lack of armor proficiency is the restrained condition (emphasis mine).
Restrained (PHB 292)
- Attack rolls against the creature have advantage, and the creature’s attack rolls have disadvantage.
- The creature has disadvantage on Dexterity saving throws.
As you can read from the quotations, some effects are similar between armor proficiency and the restrained condition. Also, as a note, it seems that it require at least some level of dexterity to be able to cast spells, but that is never touched in D&D for balance factors. Thus, the evidence points that, at least the restrained condition, should hamper spellcasting.
There are some illogical things in the rules that make you wonder why a particular feature like wearing armor hampers spellcasting, but other things much more invasive as being restrained by a creature do not hamper spellcasting at all. But, if the condition is frequent enough, this would defeat the purpose of spellcasters, since they are too easy to shoot down.
What to do?
Do these condition should stop spellcasting? If the situations occurs too often, no, it should not be applied unless the players want the challenge.
Is it an oversight form the authors? Most likely no, since they removed any vestige of the cumbersome 3.X arcane failure by armor for a more simple and manageable "or you have it or you don't, but nothing in between" mechanic, they might have foreseen these situations. The moment you add more rules like this, more and more situations are going to appear that would stop spellcasting and more things to track down, and that seems out of the spirit of 5e.
As a DM
All the logic points out that it should hamper spellcasting, if you have a vision for this there are some rules that support the theory. Just remember, if a change seems to powerful, you have all the right to revert your decision.