The rules assume you already know how walls work
I've looked everywhere I can think of, and as far as I can tell there's nothing in the rules explaining that you can't walk through walls.1 Despite this, I've never met or heard of a DM that allows their players to walk through walls. Walls are commonly understood by everyone to be barriers to movement, in addition to providing total cover. This includes both voluntary movement, such as walking and flying; as well as involuntary movement, such as being pushed, pulled, or carried; but generally not teleportation and similar effects that allow one to arrive at their destination without movement.
(It might seem obvious or even a bit patronizing to go through all of this, but it's important to establish it as a baseline that we all agree on, since as far as I can tell "walls block movement" isn't actually an explicit rule. Also, JC has stated in a tweet that walls "work the way we expect them to, except for when the rules say otherwise.")
Magical pushing still causes physical movement
So if we take it as a given that walls are impassible, the question becomes whether there is anything special about a push effect caused by a spell or magical effect, compared to something like a mundane shove attack. In general, I would say there is no difference. Just because the thing that launched the target backward is magic doesn't make the movement itself magical. Even if you were to pick up the target with a Telekinesis spell and try to carry them through the wall, it wouldn't work, whether the wall belongs to a castle or a Forcecage spell. Regardless of the force doing the pushing, the target is being physically moved, so any barrier to physical movement will block them.
You need the right "magical means" to escape a Forcecage
The Forcecage spell says:
A creature inside the cage can't leave it by nonmagical means.
This establishes that you can't escape the cage by hacking your way through the wall with a axe, picking a lock, using a crowbar to separate the bars, etc. However, it doesn't mean that any magic spell or effect will get you out of the cage. Even when you're using magic, the wall of the cage is still a wall. You're going to need a spell that can get you through a wall. Teleportation is one way, subject to the limitations described in the spell. Disintegrate can also punch a hole in the cage for you. However, Passwall isn't going to help you, because the cage isn't "wooden, plaster, or stone", and Etherealness won't get you out either, since the cage extends into the Border Ethereal. There may be other spells that can get you out, but the point is, a spell will only allow you to escape the cage if it helps you pass through a wall made of magical force.
1The closest I can find to an explicit statement that walls are impassible is this rule from the grid variant rules:
Corners. Diagonal movement can’t cross the corner of a wall, large tree, or other terrain feature that fills its space.
There is also this excerpt from the DMG about movement in the border ethereal plane:
[...] solid objects on the overlapped plane don’t hamper the movement of a creature in the Border Ethereal.
Both of these assume that walls are impassible, but don't say so on their own.
