The Invisibility spell states:
If the subject attacks directly, however, it immediately becomes visible
Who is the attacker when a familiar delivers a touch spell? The familiar, the spellcaster or both?
The Familiar is the Attacker
From the PFSRD:
This ability does not specify that the familiar counts as its master for the purposes of making the attack, just that it delivers the touch spell with the same parameters that the master would use. As a result, the default assumption - that the familiar is another creature and is treated as such - takes effect.