If I'm a Pact of the Chain warlock and I cast hex on my familiar, can I use the Relentless Hex eldritch invocation to teleport next to my familiar as a bonus action for the duration of the hex spell?
Yes, you can
Answering the question asked in the body, the text of the Hex spell does not require that the target be an enemy:
You place a curse on a creature that you can see within range
so you can target any creature that you can see. The text of Relentless Hex simply requires:
As a bonus action, you can magically teleport up to 30 feet to an unoccupied space you can see within 5 feet of the target cursed by your hex spell or by a warlock feature of yours [...] you must be able to see the cursed target.
So you can Hex your familiar, and you can then use your Relentless Hex invocation to teleport within 5 feet of it so long as you can see it, the space you wish to teleport to, and both are within 30' of you.