Twice this week, I have faced a situation where Polymorph was going to be cast on a creature that had swallowed a PC. As a DM, I ruled that if the polymorphed form was smaller, the swallowed PC would burst out, causing damage to the creature. As a player, my DM ruled that the spell would simply fail.
Are there any rules on what should happen in this scenario? The texts of Polymorph and the swallow ability of various monsters don't seem to address this situation.
(Related, but for 4e)