Yes, a vampire soul can be trapped and if it is released and the body is in the range, it returns as a vampire, if not, the soul is released. This can happen because the spell describes positive and negative recognition:
In a group of life forces, the caster can sense a difference of four
or more levels/Hit Dice and can determine whether a life force is
positive or negative energy.
But if the jar is broken, and the spirit have not a body to return, it departs:
If the receptacle is destroyed while the caster's life force occupies
it, the caster is irrevocably slain.
But we have to make some considerations here, if the caster was a living mage and his body is dead, the situation is the described above. We can assume the same for living beings possessed, with the body dead.
However, a vampire is not a living being, its body is already dead, so if the pcs cannot destroy it fully (with a 'Desintegrate' spell, for example), the negative soul returns and begins to regenerate or assume cloud form if the HP is 0 or lower.
Finally, only the caster can possess another body, his victims can not.
P.S.: For the sake of curiosity, your question raises a LOT of another questions, like: Which powers the caster has in the vampire body? A released vampire soul goes to a caotic evil outer plane? Returns to the Negative Plane? And what if it was a lawful good person in life?
P.P.S.: Some people use 3e to fill the void of descriptions of 2e, so it is useful to add a comment: In 3e the spell description clarifies that the caster can only attack positive life forces, what excludes vampires and silence most of the questions of my first p.s.