According to RAW, objects in 5e can explicitly be attacked, regardless of whether they are worn or not, and its not up to the gm as not only do spells specify whether they target objects or not, but its baked into the rules of attacking. This is the entry for making an attack:
Whether you’re striking with a melee weapon, firing a weapon at range, or Making an Attack roll as part of a spell, an Attack has a
simple structure.
Choose a target. Pick a target within your attack’s range: a creature,
an object, or a location.
There is also no general restriction about worn or carried objects. Some spells and abilities do however have specific restrictions.
Firebolt for example includes a caveat for attacking specifically items that are worn or carried
You hurl a mote of fire at a creature or object within range. Make a
ranged spell Attack against the target. On a hit, the target takes
1d10 fire damage. A flammable object hit by this spell ignites if it
isn't being worn or carried.
If you couldn't attack worn or carried objects, firebolt wouldn't specify what happens when you do
What is up to the DM is deciding how much ac and hp objects have, what resistances vulnerabilities and immunities they have if any ( a piece of paper would be vulnerable to fire and a stone wall resistant or immune to it for example)
So in the example you give players could simply attack the sword. According to the rules on object hp (https://roll20.net/compendium/dnd5e/Objects#content) it should have about 5 hp (2d4 as it is tiny) and an ac of 19 if its made of steel, as well as resistance to all damage like all magic items. If you want it to be more durable, say its resilient and has maximum hp (aka 8) and that its made of mithral for an ac of 21.