This just takes a little thought and planning.
First let's look at the part of the Find Familiar spell that applies here:
While your familiar is within 100 feet of you, you can communicate with it telepathically. Additionally, as an action, you can see through your familiar's eyes and hear what it hears until the start of your next turn, gaining the benefits of any Special senses that the familiar has. During this time, you are deaf and blind with regard to your own senses.
Since you are deaf and blind, you will not be seeing her eyes, but your familiar is still at risk. If you are using a familiar with Blindsight (bat), neither of you are at risk.
Remember though, using your familiar this way takes an action, so, as a caster, you would only be able to cast a spell with a casting time of Bonus Action. This only lasts until the beginning of your next turn so you would have to use another action to repeat with same limitations.
Haste will only work with this if your DM views using your familiar as Use an Object. Fairly useless as a combat tactic unless you're an Arcane trickster rogue that can take advantage of the action limitations of Haste.
This can be useful in other ways though. Say for instance your party walks into an area with several life like stone statues. If you believe they are victims of a Medusa or basilisk, then it's time to break out the bat and do some scouting.