If you are holding the charge on a touch spell, will getting hit by melee, ranged attack, or spell dissipate the spell you're holding?
If it's a melee attack, does the attacker feel the effects of the spell you are holding?
The attacker is assumed to have avoided your hand, which is holding a charge of magical energy. You still hold the charge, he is not affected with it. Even if he Grapples you, he’s assumed to be holding your hand away, just as he’d be assumed to be holding a dagger away.
In general, you have to actively try to touch someone to trigger it. It requires an attack action on your part, an attack roll, and so on. In a Grapple, you have to win a Grapple check to get an attack.
A free, retributive effect like you propose is quite rare in 3.x, and usually quite weak. To be able to do it with just any touch spell is potentially overpowered and opens up shenanigans that are probably inappropriate.
The spell-storing property can, in Pathfinder, be applied to armor to do something similar, but
It’s a special +1-equivalent property that has to be explicitly built into the armor
Even then it’s limited to 3rd-level or lower touch-attack spells that are cast into the armor ahead of time, and in Pathfinder, at least, triggering the discharge requires an immediate action (in 3.5 the property was written before immediate actions were a thing and it wasn’t updated, but I’d recommend adding that houserule).
I would assume that when someone is holding a charge for a touch attack, the touch attack wouldn't do anything to an attacker normally. The only circumstance in which it might cause something is if they touched the hand holding the charge, or you touched them. An attacker with a weapon would most certainly not discharge the touch attack. The only time a touch attack's charge is dispelled is if the hand which holds the charge (or item or weapon in the case of channeling) actually touches a valid target.
Though that does bring up an interesting idea... Channeling a touch spell into your armor or shield as a defense, possibly even sending it through an opponent's weapon... Will have to look into the rules for channeling touch spells through items...
There is no actual difference between "I touch you" and "you touch me."
The rules say the spell discharges even if you unintentionally touch something. This means the caster can't actually control the discharge of the spell, they can only attempt to create the proper circumstances to get the spell to discharge on it's own.
Other answers talk about the hand, but the rules do not even mention the hand, neither directly nor by implication. The rules do not prevent you from making touch attacks with feet, knees, elbows, or any other body part.
A touch attack just represents an intentional attempt to contact the opponent, who naturally will avoid it, even if they had no clue why you were attempting to touch them (quite simply, if we are fighting and I try to poke you with my finger, you won't let me because you know my goal is to defeat you, therefore, anything I want to do is most likely a bad thing for you. This concept is an instinctual response.).
However, if the opponent has no idea you have a touch spell ready, they will not avoid touching you in a way determined by them (I.E. trying to punch you), even though they will avoid letting you touch them in they way you desire.
A touch spell only requires contact between caster and target. There is no actual difference in that contact from how the contact came about because either the two are in contact, or not. It is a binary situation.
Now, if the caster controlled the discharge of the spell, then one might argue they would need to ready an action to discharge it if touched, but since it is absolutely clear that the caster does not control the discharge, it both allows the retributive use but also makes the spell less useful because you can't do other stuff until the spell is discharged or dismissed.