The spell would fail
Any teleportation spell needs to be aimed at a valid target, just like any other spell. In our case, a "valid target" is an empty space that respects the spell's conditions.
In the case of a demiplane or any other plane limited in size, the area "outside" of those boundaries does not exist. Because of that, there is no empty space (or, for that matter, any place at all) that could be targeted by the spell. Just like if you tried using a spell on a target that is not valid, your spell will simply fail to take effect.
In a practical situation, your DM would probably tell you beforehand so you don't waste a spell slot, though.
So, do I take force damage?
The spell Dimension Door states the following :
If you would arrive in a place already occupied by an object or a creature, you and any creature traveling with you each take 4d6 force damage, and the spell fails to teleport you.
If you do try to cast the spell outside of a demiplane, you do not target a place that's occupied by "an object or a creature". So even if you decided to cast the spell, according to the rules as written, it shouldn't inflict the force damage.
What about a bag of holding ?
The inside part of a bag of holding is considered as a separate plane, very similar to the demiplane we just talked about. This means you shouldn't be able to leave it with a spell that does not authorize cross-plane travel.
Do note that even if you did use such a spell, you'd have to target the plane where the bag is located, and not the space beyond the "demiplane's" boundaries. The plane is not inside the bag, the bag's entrance is merely acting as a portal.
The same is true for a portable hole, since it's even more obvious that it acts as a portal to another plane, rather than a magically-larger container.
And if I target through the opening?
It's up to the DM and how they want to work with gates/portals. Dale M's answer and the questions linked to that answer cover this topic very well.