I am looking for a spell or wondrous item, that would allow to trap the victim inside a limited area and then it slowly shrinks, crushing the victim?
I was hoping to find some answers in epic level handbook or book of vile darkness, but there was nothing like that.
As long as it's not homebrew, other non-dnd d20 systems and supplements are fine.
Yes, there is:
This spell, found in the Spell Compendium, has the following as part of the descriptive text:
This spell creates a bizarre, but useful, machine that you can command to perform any simple, physical task that can be described in twenty-five words or less. You can order the machine to perform the same task over and over, but you can't change the task. You must specify the task when you cast the spell. The machine always acts on your turn in the initiative order. (It can act during the turn you cast the spell.)
However, it has a weight limit, so you may have to... ah... adjust your target to fit.
Note that the Greater version of the spell has a higher weight limit, greater strength, and does more damage.
The spell also appears in the FR setting.
You might use a command phrase like: "Constrain, trap, hold, prevent from escaping, and slowly crush this target over the next hour." The machine would then presumably use it's limbs to execute said actions.
Hungry Gizzard from the Spell Compendium creates a gizzard which traps (Reflex save) the target and then tries to digest it, dealing crushing and a bit of acid damage every turn. The target is trapped in a limited area and slowly crushed, so it might work, if the flavour suits you.
Wall of Stone (PH) allows trapping creatures, which could be followed by a casting of Stone Shape (PH) to release the ceiling part of the wall. A 2 inch thickness (CL 8) one square section would weigh over 600 pounds, assuming a density of about 150 lb/ft^3, so it would take at least 18 strength to be able to lift it even a little bit, and a weaker character should be trapped and, eventually, crushed through the application of gravity.
Other alternatives, which fit the request less well, but may be helpful as references for making custom magic items and such:
Bigby's Crushing Hand (PH) grapples the target and crushes it. It limits movement, but does not really provide a "limited area".
If you already have the area in question in the form of a trap, and you just need to teleport the target into it, then spells like Dimensional Shuffle (PH2), Baleful Transposition (SC) or Greater Plane Shift (SC) might work as well.
Sarcophagus of Stone (SC) or Entomb (Fr) trap the target, but provide no crushing, instead letting it suffocate.
Imprisonment (PH) or Trap the Soul (PH) trap permanently in suspended animation.
