Everything on your character sheet that you have not explicitly dropped is considered "worn or carried".
We can look to the Arcane Trickster rogue's Mage Hand Legerdemain feature for a little bit of guidance on this:
You can retrieve one object from a container worn or carried by another creature.
Considering the intent of this feature (mage hand gaining the ability to steal items from people), anything a character has should be considered worn or carried. Because otherwise, the regular mage hand would be able to steal from any such containers that were not considered worn or carried.
Under the "Strength" section of chapter 7 in the PHB, we also find this under the Lifting and Carrying rules:
Carrying Capacity. Your carrying capacity is your Strength score multiplied by 15. This is the weight (in pounds) that you can carry, which is high enough that most characters don’t usually have to worry about it.
This claims that everything contributing to a character's encumbrance is "carried" – which explicitly exempts everything that has a listed weight, and implicitly exempts everything without from being ignited by standard fire damage.
This is a simplification in 5e: earlier editions found that subjecting worn/carried objects to burning from spells would either bog down the game with too many rolls or make fire magic too powerful with respect to other damage types. Plus, players hated it.