When I run the game, I generally see the difference between Invocation for Effect and Compel as the difference between The World At Large and The PCs.
Invocation for Effect generally makes something happen. I use Invocation for Effect on NPCs, as seen here, and on the environment.
Compels generally make a PC act or fail to act. I've used Compels both ways, depending on the situation. I find that they work just as well in a positive sense, "You have to chase the car with your girlfriend in it," is a fine Compel.
You'll note that in both cases, whatever it is takes place without a roll. They both create events, not just modifiers for events.
In your situation, I would call what happened a Compel, just because it applied to a PC. The NPC was due a free Tag, which deprived the PC of his FATE point, so I think you did the right thing...up to a point.
Since it was a Compel (in my book, because it was applied to a PC), the player should have had a chance to pay a FATE point to avoid it. No roll to resist, the PC already failed that roll when the Aspect was applied. But I find that player agency is too important to just steamroll - if the player had been willing to pay up, he could have fought that first suggestion. If the vamp wanted to try again, to push harder, as it were and to pay the FATE required, that would have been acceptable, too. And the player then has the choice to give in (and take the FATE point) or pay again to keep resisting.
The "Compel Auction" from SotC no longer exists in DFRPG, and I find that I don't miss it - you can use multiple rounds of pushing (as above) to get the same effect without the added bidding.