There are lots of mechanisms by which PCs can choose to enforce contracts they get NPCs to agree to. Ultimately all of these will come down to either enforcing the terms of a contract themselves or getting some outside party to agree to do it, though. A sufficiently powerful being in D&D will almost always be able to get out of any contract it makes or is subject to, just as the PCs are so able to bypass attempts to enforce contracts they have signed should they so wish, given sufficient magical resources.
Some such methods include:
- Using spells that don't require concentration and can trigger automatically upon breach of contract
- For example, geas is such a spell.
- Using magic items that enforce contracts
- None spring to mind in official material, but there are plenty of homebrew items replicating e.g. 3.5's Contract of Nepthas (Complete Arcane 148)-- magical paper for writing contracts on.
- Using other spells or magical abilities or the threat thereof to enforce the terms of a contract
- For example, fireball is such a spell.
- Using the service of a powerful creature, organization, or other entity, with access to magical abilities, to enforce the contract.
- For example, the Kolyarut (see the Mordenkainen's Tome of Foes entry for the Marut).
- Using lesser creatures to enforce the contract via magic
- For example, placing a glyph of warding on a hostage or their dwelling place or valuables of some kind, or using dominate monster to force a slave to enforce the contract for you
Generally, anything that is itself or can be affected by magic can be used to magically enforce the terms of a contract to some extent, since literally anything can be used to enforce the terms of a contract but not everything can be construed as doing so 'magically'.