The rules are a bit unclear, but technically, the familiar retains invisibility when delivering a touch spell.
Absent further clarification, the most reasonable conclusion is that the caster—not the familiar—makes the attack roll, deals damage, or casts the spell when a familiar "delivers the touch".
However, this interpretation might feel dissatisfying and may not align with RAI. Be sure to discuss this with your DM to determine what works best for your game.
Further Considerations
There are two principal situations to consider mechanically when deciding which interpretation works best for your table. Let's call them the Help Exploit and the Reaction Exploit.
The Help Exploit
This is the (RAW) interpretation where the invisible familiar retains invisibility because the player character (PC) is considered the attacker. Under this interpretation, the PC readies the desired spell and sets the trigger to: "My familiar helps me hit my target." The attack roll is made with advantage (from the familiar's Help action), and the familiar retains invisibility.
The Reaction Exploit
This is the (RAI?) interpretation where the invisible familiar loses invisibility because the familiar is treated as the attacker. In this scenario, the PC readies the desired spell and sets the trigger to: "My familiar touches my target." The attack roll is made with advantage because the familiar is an unseen attacker. The familiar "attacks" by using its reaction, loses invisibility, and then immediately uses its action to turn invisible again.
Both interpretations result in a similar outcome through different action economies. However, it’s important to discuss these options with your group to ensure everyone agrees on the preferred approach.