This seems awfully powerful, being fully permanent with no way to dispel, unlike True Polymorph or Shapechange, once you get past the 24h window. So if you can succeed at getting your soul into a more powerful and durable body (e.g. a high level barbarian), that's a huge power boost. For example, Grog Strongjaw of critical role had 298 HP as a Barbarian 17 / Fighter 3, on the strength of a good Con score and some lucky rolls for d12 hit dice on level up. And Str 17 / Dex 15 / Con 22 (without magic items) are pretty damn solid stats for a Wizard to inherit, as they replace the Int/Wis/Cha dump stats.
Even with Wish, Magic Jar, or True Polymorph, 5e doesn't normally have any way to permanently increase your physical stats without any caveat or vulnerability, apart from limited ASIs. (Cassie's answer made the same point, along with a good suggestion that dumping the old body is a good practice if you're evil.)
Your 1 minute cast time is a good way to make it unusable during combat, but even that and the required 24 hour concentration don't make it unusable for a body-snatcher to pick off a single member of another party whose body they want to upgrade to.
(Picking off a member of an opposing force, from stealth for assassination reasons, is harder, as the victim's soul does end up in a body nearby and can maybe rejoin their friends. But it might be a body with physical stats not useful to their class, e.g. a rogue with low dex. And perhaps you could cast from hiding with your old body tied up, and a henchman ready to stab it on your signal.)
As far as pulling this off against a hostile opponent, you can do this on anyone you capture and take back to your stronghold. You can cast for a minute while they're chained up. (On success, just Misty Step out.)
Or try it as an ambush from hiding, or in a crowded room with a diversion. The limited range of 30ft makes it hard to pull off, but you only have to get lucky once with stealth or trickery. If you're detected while casting, just try to escape until you can try again. With some non-magical means to obscure sight (a puff of smoke), you can probably teleport away without getting counterspelled.
If you can find a way to block sound and be hidden while casting, there's no sign of threat until the spell goes off. And not even then if your acting skills are up to the task.
If the friends of a victim don't realize what happened, you can wander off. Or otherwise escape them to complete the 24h concentration in private, where you're not in danger of losing concentration. (Perhaps have a Greater Restoration lined up for the all-nighter you're going to need to pull to maintain concentration.)
As soon as you're out of sight of anyone that could counterspell you, Dimension Door or Teleport away. Perhaps to the entrance of a Magnificent Mansion you cast earlier, where you can wait out most of the 24 hours. (You can't cast any spells with a cast-time greater than 1 action while you maintain concentration. You could prep something like a Glyph of Warding to put up a mansion when you arrive though, and concentrate for you.)
If the victim was asleep or passed out drunk, they might end up asleep in your old body. Otherwise perhaps arrange some restraints and magical silence (via glyph of warding for after you cast it?) so they can't yell to alert their friends. Or just plan to teleport away before they can raise the alarm and people can react. Worst case you lose concentration on Soul Swap, in which case you're back in your original body to try to escape and pull the same trick another day.
Even with a Cha save that you accidentally left out, this seems stronger for assassination and similar things than any existing 5e spells. But possibly not; there is an unwilling-teleport spell, Scatter (6th) in XgE, which is 1 action, 30 ft range, which would let you teleport someone into a cage on a cart (up to 120ft) on a failed Wis save. Fill the cage with whatever death traps you like, and drive away with fast horses.
And Vortex Warp (2nd) from SCC can teleport an unwilling target on a failed Con save from up to 90ft away. (Several of the other Strixhaven spells seem very powerful, too, like Silvery Barbs making save-or-suck spells such as this much easier to land.)
But those don't have the ability to steal the body and leave the target with a permanent downgrade, e.g. a barbarian with low Str or a rogue with low Dex.
As far as short term impersonating someone, Dominate Person only has 1 minute duration. But with this, you could potentially be the king for days, if you can spirit away or dispose of the body containing the actual king's soul.
Perhaps True Resurrection (9th) and/or Reincarnate (5th) can create a body with the original stats for that soul, if you cast it as if you don't have any of the body so True Res makes a new one. So the victim could get their physical stats back, instead of being left with whatever wimpy body the evil wizard swapped into before swapping with them.
But Revivify, Raise Dead, and Resurrection (7th) should just put the soul back into the current body, with its physical stats and HP.
The requirement to have a former possession might be easy, if you can hire a local pickpocket to take something from your target. Or a beggar to ask for something. Presumably just gold wouldn't count because it's essentially fungible (one specific piece isn't special), so the ruse might be a little more involved. But could be as simple as being cold and asking for something warm to put on. Or maybe a bandage on someone injured. Or maybe even a sandwich or part of their lunch if a hungry beggar can convince them to part with food. Again, these depend on your target being Good or otherwise generous.
Perhaps the standard should be a prized possession, or a valued possession, with the implication that they actually care about it. Something with some metaphysical connection to them. Not like the sandwich they bought earlier or supplies like bandages or blankets.
(If we're talking Critical Role characters, where do we draw the line with doughnuts for Jester Lavore? I think doughnuts and cupcakes as a general concept are deeply important to her. But any specific doughnut she shared with someone isn't strongly tied to her identity. OTOH, doughnuts in general are metaphysically tied to her identity and personality, so one from her could maybe work as a material component if you want it to narratively work that way.)
So the details of the component requirement might be important for DMs to balance the spell. You might want to include some guidelines, or leave it open for DMs to interpret as they way. But I'd suggest some word like "important former possession", not just "former possession". If the possession has to be something that the character would miss (if they knew about it being missing), they'd be on alert if it was taken, making it harder to cast this from across the room in a bar or in the market while they're shopping.
But if they have a home you can burgle while they're out, they might not know about a prized possession (like memento from a friend). Might not even notice it missing even though it would still count, so this isn't a strong limitation in all cases. Still it's a hoop you have to jump through to pull this off, if the item must be one they care about.
Having villains with access to this spell in your world is more scary than than other 9th level spells. They can fully be anyone, with no remaining sign of magic or way to dispel. Even an antimagic field won't revert them.
And the victim is gone, perhaps caught in a body that's about to die. (cf. The Corpsetaker in the Dresden Files novels, although that evil wizard could body-swap very quickly. The 1-minute cast time is a very good idea to prevent this being combat-usable.)
Spell lists: Druid?
Druid doesn't seem a good fit for the usual themes of that class, like preserving the natural order of things.
Reincarnate is one of the few spells Druids get that messes around with souls and bodies. But reincarnate is almost fundamentally different from soul swap, being about rebirth, which is part of the cycle of life. Body-hopping your way to immortality and power, or just to play tradesies with your friends or lover(s), feels way different.
Of course a specific NPC druid that's "gone bad" could narratively have reason to explore ways to use natural magic for unnatural effects, and perhaps end up being able to do this, so could have it if you want. Otherwise I don't think it's a good fit, even for druids that intend to use it for non-evil purposes.
Clerics do stuff with souls like putting them back into dead bodies, and there are evil deities. So I could see having this on the cleric spell list. Some clerics would of course not take it, for the same reason they wouldn't take Animate Dead which is also on the cleric list.
By comparison, Magic Jar and Clone are only on the wizard spell list. (Not sorc, bard, or even surprisingly even warlock).
You could imagine a druid version that's narratively like a Shapechange on 2 targets, if you want to find a way for Druids to use a spell like this. But that would change the bodies, leaving the souls at the same coordinates, instead of for example transferring a victim into a body already chained up or caged.