Only items that say so are magic items
An item only is a magic item when the rules say so. When do they say so? Either if or if is explicitly described as magical, or it is listed as a magic item in the Dungeon Masters Guide, or another published source (like Xanathar's Guide to Everything or Tasha's Cauldron of Everything).
These are the only two ways the rules tell us an item is magical:
Either the item's description says it is magic. This is simple.
Or, it is listed in a section or list of magic items. Context matters, as Jeremey Crawford explains in this tweet. Not every item must repeat explictly that it is a magic item, if it is listed in a section that enumerates magic items. Thomas' answer gives explicit quotations for the DMG and other published lists.
Any items that do not fullfil one of these two criteria are not magic items, because the rules do not say they are, even if they are relevant to the process of casting spells or if they cause unusal effects. Neither of these properties is sufficient to make an item a magic item. If they were, the rules would need to tell us so which they do not.
There are no secret rules. If an item does not somehow say it is magical, it is not. This accepted principle is explained here in detail with multiple references.
Would it be reasonable to expect the rules to state that an item is not a magic item when it is not? No: there is an infinite number of thing that are not. Would they also need to confirm that a magic item is not a creature, is not a spirit, is not a substance, is not a spell, is not an ability, is not a hoovercraft? And that each of these things is not one of the others? Of course not. So, while we might want to see a more explicit defintion of what makes a magic item a magic item in the rules, such a definition is not included. Instead they tell us explicitly when an item is magical.
Adventuring gear table
There is a table of adventuring gear on page 150 of the PHB, that lists nearly entirely mundane items. The only magic item on the table is the potion of healing. All others are mundane. If you find an item on this table, other than potion of healing, you have a shortcut to know it is a mundane item.
Potion of healing. The healing potion is a magic item. It is both listed in the DMG as a magic item, and its description explicitly describes it as a "magical red fluid". It is also the only single item in the table that is using italics font. (All other italics use if for group headings, such as Ammunition, or Holy Symbols.
Holy Water: Holy Water is created by a ceremony and has special effects on undead, but it is not described itself as magical, nor is it listed as a magic item in the DMG. It therefore is a mundane item.
Spellbook: Normal Spellbooks as given on the equipment list are likewise mundane items. There are magic spellbooks like the Arcane Grimoire from Tasha's, but the normal one isn't.
Spell Casting Foci: Spell casting foci, whether arcane or druidic are likewise not magical. There are many items like wands, staffs or rods in the DMG that are magic items, but the normal ones are not.
Healer's Kit: the salves in the healer's kit are not described as magical. This is also mentioned explicitly in the Sage Advise compendium: "using a nonmagical item, such as a healer’s kit".
Special substances: these include Acid, Antitoxin, Alchemist's Fire, and Basic Poison. While their chemical composition has effects that are similar to those of spells, they are all mundane in nature. None of them are described as magical, and none of them are listed in the DMG as magic items.