The spell prevents it from being opened, not from taking damage.
"You and the creatures you designate when you cast this spell can open the object normally.
The clarification arrives in this part of the spell description:
You can also set a password that, when spoken within 5 feet of the object, suppresses this spell for 1 minute. Otherwise, it is impassable until it is broken1 or the spell is dispelled or
suppressed.
It, means the object that the spell was cast upon. The meaning of "broken" can mean an object taking damage until it is broken1.
There is more than one way to skin a cat. Will you rely on your strength? You can, and check your DC against this enhanced barrier.
Or, you can work smarter, not harder, and use a tool. Consider what the simplest tool, the lever, does for increasing the effective force you apply to something.
Example: On p. 247 of the DMG, a small chest which is "resilient" has 3d6 hit points. If it is made of wood, you'd expect the AC to be 15, if made of Adamantine, and AC of 23. If it is protected with an arcane lock, someone can still hit it (with a weapon) enough times to smash it and render any lock, magical or mundane, moot. What that does to the contents of the chest is another matter.
Doors and other structural items will have hit points as the DM assigns, and AC per the table on page 246.
The DM may choose to waive the "autohit with a 20" on objects (which I would as DM) so that particularly strong objects take a lot more work to break via brute force.
1 Breaking something and attacking it to deal damage are separate things in D&D 5e. One is a Strength check to force it open / snap it in two / cave it in / whatever. It succeeds or fails in one roll. The other takes normal hitting with a weapon and dealing HP damage until it runs out of HP, at which point it becomes broken. Credit to @SevenSidedDie and @Theo Brinkman