Enlarge/Reduce, lvl 2 PHB p.237
Reduce. The target's size is halved in all dimensions, and its weight is reduced to one-eighth of normal. This reduction decreases its size by one category - from Medium to Small, for example. Until the spell ends, the target also has disadvantage on Strength checks and Strength saving throws. The target's weapons also shrink to match its new size. While these weapons are reduced, the target's attacks with them deal 1d4 less damage (this can't reduce the damage below 1).
So RAW there is no guidance as to what happens, so it is down to the DM to adjudicate. For guidance I would look at spells that damage the target if they somehow negatively interact with an object when a spell completes.
Meld into Stone, lvl 3 PHB p.259
Minor physical damage to the stone doesn’t harm you, but its partial destruction or a change in its shape (to the extent that you no longer fit within it) expels you and deals 6d6 bludgeoning damage to you.
Etherealness, lvl 7 PHB p.238
When the spell ends, you immediately return to the plane you originated from in the spot you currently occupy. If you occupy the same spot as a solid object or creature when this happens, you are immediately shunted to the nearest unoccupied space that you can occupy and take force damage equal to twice the number of feet you are moved.
Dimension Door, lvl 4, PHB p.233
If you would arrive in a place already occupied by an object or a creature, you and any creature traveling with you each take 4d6 force damage, and the spell fails to teleport you.
So being forcibly shunted out of a block of stone at lvl 3 does 6d6 damage, being displaced 10' due to an object in the way after plane shifting at lvl 7 does 20 force damage (roughly the average of 6d6), and the bounce back from a teleport effect with an object in the way for a level 4 does 4d6.
Not that obvious perhaps but these give a rough scale. As a DM I would impose the following: if the person reduced was put into a Small hole in a block of stone and the hole Stone Shaped closed so properly encased in stone, it seems appropriate that the maximum damage for the spell would be 6d6 bludgeoning as they burst out of the stone. I would impose the same for a lump of metal. A bottle might be 1d6 slashing as the bottle smashes, 2d6 for a wooden crate or a thick glass bottle or a metal flask. An iron cage or metal bound chest might be 3d6. A stone sarcophagus might me 4d6. An iron coffin might be 5d6.
A better way to do it is probably that it does 2d6 bludgeoning damage to the creature as they burst out of the container. This is simple and spell level related.
All very subjective and not at all RAW, but to me these suggestions seem in scale with other spell effects. If you think this suggestion does too little damage, I always try to remember that science and/or common sense has no real place here as it is magic that "explains" what happens, and also it is only a 2nd level spell so the magic is relatively weak. After all if you could kill 10th level fighters this way then it would be THE spell to get, along with a metal chest on wheels.
