- Until you can't follow the normal grapple rules, you follow the normal grapple rules (PH 155-6). Your foe provokes an attack of opportunity from you. You take the attack of opportunity, make the touch attack roll to grab, don't provoke an attack of opportunity yourself because of your feat Improved Grapple (PH 95-6), succeeed on the touch attack roll to grab, and then stop because you can't take the free action that's required to make the opposed grapple check.
The foe has been grabbed (and released because you couldn't start the grapple), not grappled, and has taken no damage, so according to the RAW, the foe continues casting his spell without penalty.
- Were I DMing, I'd probably house rule that such an interruption mandates a Concentration skill check (DC 10) from the foe, but only because you've devoted resources to being able to make an off-turn unarmed touch attack that doesn't provoke attacks of opportunity, and that's a fairly hefty investment... but you've still just touched a caster while casting, and although that's annoying, it's not like you're a storm-tossed ship or a galloping horse or an earthquake; instead you've essentially hit him for 0 damage. I'd even let you force multiple Concentration skill checks if you've iterative attacks (on your turn versus a foe casting a spell with a 1-round or longer casting time) or if the creature provokes multiple attacks of opportunity.
- You can get all the way up to making an opposed grapple check versus an opponent who provokes an attack of opportunity by attempting a grapple, but you can't make the opposed grapple check--and thefrefore won't be able to do more than grab and release him--unless it's your turn or you have an off-turn free action available from another source; it's still a free action to make the opposed grapple check to get the hold.
Free Actions Can't Be Taken When It's Not Your Turn
Of course, a creature can take a free action to speak when it's not the creature's turn because the game says, "In general, speaking is a free action that you can perform even when it isn't you turn," but even then the DM's totally allowed to say no if he wants.
However, in all other cases, "Free actions don’t take any time at all, though there may be limits to the number of free actions you can perform in a turn," says Free Actions on PH 144. According to the Glossary a turn is "[t]he point in the round at which you take your action(s). On your turn, you may perform one or more actions, as dictated by your current circumstances" (PH 314). There's an argument that the Glossary's definition of free action, which reads, in part, that "one or more such [free] actions can be performed in conjunction with actions of other types," means that if a creature can perform even one off-turn non-free action (or sometimes, the argument runs, performing a free action just doing something different from another free action), the creature can also perform a DM-limited number of free actions. Technically, this means combining an immediate action with free actions (one type of action and another type of action) is a thing, but combining free actions with attacks of opportunity isn't; attacks of opportunity aren't actions. (They're not under any header on Table 8-2: Actions in Combat on PH 141 nor are they given an action type in Attacks of Opportunity on PH 137).
"You can perform one or more free actions while taking another action normally," says PH 139. Broadly, what normally means in the context of D&D is wide open for debate, but here, given the rest of the text, I take normally to mean on your turn--if, for example, speaking off-turn were normal and not the exception the PH needn't've pointed it out. Thus mute or gagged creatures aren't at a combat disadvantage because they can't combine speaking with other free actions, and yelling, "Hey, Scooby!" doesn't somehow also allow a creature to drop to the floor when it's not the creature's turn. Although that's funny.
The Problem with Immediate Actions
The PH wasn't written with immediate actions in mind; creature's turns are supposed to be discrete--the only off-turn things creatures're supposed to do according to the PH are speak (if the DM allows it) and take 1 or more attacks of opportunity. The immediate action rules changed that, adding an other type of action that interacts oddly with free actions. According to the RAW, if a creature has an immediate action available, it can take that immediate action and combine it with free actions. That means, for example, a creature can cast the spell ruin delver's fortune [trans] (SpC 178) and drop to the floor off-turn.
This Makes Some Folks Very Angry
A creature can take free actions beyond speaking off-turn is a hill some folks are willing to die on. I don't know why. I bring this up because if the DM allows creatures to take off-turn free actions besides speaking, and you raise this issue, it's possible you'll be shouted down, drawn and quartered, or even expelled from the gaming group. It's that contentious, and folks will bleed over it. If the DM allows off-turn free actions, and you're okay with it, I strongly suggest you roll with it rather than bringing up the arguments I've just made. It's just safer that way. Further, so you know, I have no horse in this race (seriously, whether or not free actions can be taken off-turn in D&D 3.5 is not that important to me in the grand scheme of things)--I am merely putting forth the rules as I see them, so if you think them totally wrong-headed, that's okay.