The rules for grapple say that starting a grapple provokes an AoO. Does this still apply if the attacking creature is invisible (invisibility spell)?
Your character might be subject to Greater Invisibility spell, that does not end when he attacks. In this case, he would not lose it. Now, that would create an awkward situation - the target knows its being grappled, but can't see who is grappling. Concealment rules come to help:
You can't execute an attack of opportunity against an opponent with total concealment, even if you know what square or squares the opponent occupies.
By RAW that's how it works. By RAI as well - Attacks of Opportunity represent split-second attacks against an opponent who momentarily lowered his guard to perform some reckless action, executed just before this action happens. Since your character is invisible during that time period, you still can't use an AoO.
However, if you just have regular Invisibility, grappling counts as a direct attack in Invisibility spell description
The spell ends if the subject attacks any creature. For purposes of this spell, an attack includes any spell targeting a foe or whose area or effect includes a foe. Exactly who is a foe depends on the invisible character's perceptions. Actions directed at unattended objects do not break the spell. Causing harm indirectly is not an attack. Thus, an invisible being can open doors, talk, eat, climb stairs, summon monsters and have them attack, cut the ropes holding a rope bridge while enemies are on the bridge, remotely trigger traps, open a portcullis to release attack dogs, and so forth. If the subject attacks directly, however, it immediately becomes visible along with all its gear.
Source, emphasis mine
When executing a grapple action, which is a combat maneuver, the PC is causing harm directly. Upon execution, the grappler becomes immediately visible and is subject to AoO normally. However, that only happens after the action is complete, not upon its declaration. It does not have to be successful, but it has to be an attempt - thus still not allowing an AoO from the target.