I tried to cast abjure enemy on a vampire as it turned to mist form to hold it still. Does it still count as a targetable creature in this case?
Shapechanger. If the vampire isn’t in sunlight or running water, it can use its action to polymorph into a Tiny bat or a Medium cloud of mist, or back into its true form.
Misty Escape. When it drops to 0 hit points outside its resting place, the vampire transforms into a cloud of mist (as in the Shapechanger trait) instead of falling unconscious, provided that it isn’t in sunlight or running water
A vampire in mist form is still a vampire and thus still a creature. All other targeting rules would apply the same as with any other creature.
The mist ability of the vampire is just a shapechange ability. It is mechanically no different than the vampire turning into a bat or any other creature for the sake of targeting. And a vampire in any of those shapes are very obviously creatures as well.
More importantly, nothing in the effect description for the vampire's shapechange ability says that it changes its creature status, and that would be a very major (and extremely odd) effect and thus would have to be written.
Abjure enemy specifically - Yes
Abjure Enemy. As an action, you present your holy symbol and speak a prayer of denunciation, using your Channel Divinity. Choose one creature within 60 feet of you that you can see. That creature must make a Wisdom saving throw, unless it is immune to being frightened. Fiends and undead have disadvantage on this saving throw.
On a failed save, the creature is frightened for 1 minute or until it takes any damage. While frightened, the creature’s speed is 0, and it can’t benefit from any bonus to its speed.
As long as you can see the vampire in its mist form then you can target it with abjure enemy.