Spending resources (Feats, Ability Points, etc.) to defend against this will likely cripple your character. My suggestion is to play your characters strengths, not its weaknesses.
If you absolutely must have some way of defending against it or making it less effective, you have options, but they're costly.
Run
If you get hit, get out of dodgeDodge. You should never engage when you're at a disadvantage. Invest options that will allow you to safely retreat.
Use it first
If you can identify which enemies are likely to be able to cast it, use it against them first. Invest in having a good Initiative so you can insure you can get it off before they do.
Be a better fighter
If you're formidable without spellcasting, it won't effect you as much.
Play a Bard
Bards can basically defend against anything. It's quite ridiculous in my opinion. If you're clever you'll always go first, always have ample defenses, and won't be crippled if you do get hit.