I'm playing a wizard and I really want to trap someone's soul. I've thought about casting glyph of warding using magic jar, but I don't want them to be able to escape. Is there any way to do this without them being able to escape? I want the classic "soul trapped in an object" feel.
RAW: there is a spell called "Trap the Soul" listed on page 211 of the PHB. It is a Wizard spell. Oddly enough, there isn't a spell description to accompany this spell.
RAI: it appears that there was at one time the intention to have a soul trapping spell, but it never made it to print. The function was folded into the "Imprisonment" spell.
See page 252 of the PHB for details of the Imprisonment spell.
The Minimus Containment feature, one of five options, seems to meet your criterion of
I wanted the classic "soul trapped in an object" feel.
It shrinks the target to about one inch in height and it describes them as being stuck inside of a gem.
As to "not being able to escape" that will always depend on if this entity / person / demon has any friends who could free said entity. If you keep the gem / prison on your person, then you'd have to be overcome before this entity could be freed.
You'd want the same hope of some chance of being freed if it were cast upon you, I suspect.
In Magic and in D&D, as in war, nothing is final.
Xanathar's Guide to Everything may have filled this gap
If you check out @Vylix's answer, the Soul Cage spell from Xanathar's fills some of the requirements.
The soul cage spell description states:
This spell snatches the soul of a humanoid as it dies and traps it inside the tiny cage you use for the material component. A stolen soul remains inside the cage until the spell ends or until you destroy the cage, which ends the spell. While you have a soul inside the cage, you can exploit it in any of the ways described below. You can use a trapped soul up to six times. Once you exploit a soul for the sixth time, it is released, and the spell ends. While a soul is trapped, the dead humanoid it came from can't be revived.
This spell is also an 8th-level spell, which is probably the revised final form of trap the soul which was removed from the PHB. It is available in the Xanathar's Guide to Everything book (p. 165).
The Monster Manual (p203) describes the process by which a wizard voluntarily does this to transform into a lich:
A lich is created by an arcane ritual that traps the wizard's soul within a phylactery. Doing so binds the soul to the mortal world, preventing it from traveling to the Outer Planes after death. A phylactery is traditionally an amulet in the shape of a small box, but it can take the form of any item possessing an interior space into which arcane sigils of naming, binding, immortality, and dark magic are scribed in silver.
With its phylactery prepared, the future lich drinks a potion of transformation-a vile concoction of poison mixed with the blood of a sentient creature whose soul is sacrificed to the phylactery. The wizard falls dead, then rises as a lich as its soul is drawn into the phylactery, where it forever remains.
This is not described as a spell that can be cast by players in typical spell-book format, nor does it say it can be cast "offensively" against others, but it does mean that a very similar ritual to what you want already exists within the 5E rules (and world). This might give you a better starting point for adapting it to the needs of your campaign that starting from scratch (and it might make it more acceptable to players/the DM if it already exists).
The components and effects are already listed; you'd need to decide upon:
- Whether the the ritual can be used on someone else against their will.
- If so, how (eg. can they just be immobilised nearby? Asleep? Knocked out? etc.). Do they get some sort of chance to escape the spell's effects (eg. a saving throw, or several?)
- A casting time.
- Possible consequences of failure.
- If relevant, whether the spell can be reversed.
Note that there are further implications of this ritual:
Soul Sacrifices. A lich must periodically feed souls to its phylactery to sustain the magic preserving its body and consciousness. It does this using the imprisonment spell. Instead of choosing one of the normal options of the spell, the lich uses the spell to magically trap the target's body and soul inside its phylactery.
The Ring of Mind Shielding is an option as well.
This magical item from the DMG has a caveat to it; if the ring does not already contain a soul, upon death its wearer's soul is trapped inside it.
Without knowing your goal or other criteria, this is a viable option in addition to the other answers here.