Would an evil mage be able to spam fireballs in a combat encounter or would they run out of slots to use eventually?
Yes, NPC spellcasters (usually) use spell slots
This is covered by the explanation of monster attributes and abilities at the beginning of the Monster Manual. Spellcasting is covered on page 10 (emphasis mine):
A monster with the Spellcasting class feature has a spellcaster level and spell slots, which it uses to cast its spells of 1st level and higher (as explained in the Player's Handbook). The spellcaster level is also used for any cantrips included in the feature.
In other words, they cast spells just like the PCs do, and their description will say which spells they have and how many slots they have. (For example, the Archmage NPC (MM, p. 342) is an 18th-level spellcaster with a list of spells and slots, e.g., they have four 1st-level slots.)
Innate casters are the exception
The “usually” covers the exceptions that don't use a Spellcasting feature. These are usually more monstrous critters that don't usually get called “NPCs”.
For example, a Pixie (p. 253) doesn't cast spells like a class and has no level, and has the Innate Spellcasting instead of the Spellcasting feature. Instead of casting using spell slots, they have “At will” and “1/day each” abilities that list the specific spells they can do on those terms. See the Monster Manual page 11, “Limited Usage” for details on these abilities.
NPCs built as PCs:
These would have the exact same requirements and restrictions as a PC of equivalent level and class.
NPCs built as Statblocks, such as in the Monster Manual:
These usually have an indication, under a header such as "Innate Spellcasting", with the spells they know and the number they can cast per day. Alternatively, they may have a spells known chart, with a number of slots available for that spell level.
In general, Monsters and NPCs are built with limitations to spellcasting, although some may have Spell like abilities or Actions that function as a spell which they may use an unlimited number of times per day.
The Evil Mage statblock from the Starter Set does have limited spells per day, as outlined in it's Spellcasting section.
Building a Mage NPC:
If you were planning to build a Mage NPC, it is up to you how you assign abilities, spells, and attribute points. When designing a spellcasting NPC however, you must take special care to properly calculate the challenge they will represent due to the spells they have access to, in addition to their at will abilities.
The NPCs in the back of the MM have spell slots listed along side the typical spells they have available. Example from the Archmage on MM p342
- 2nd level (3 slots): detect thoughts, mirror image, misty step
- 3rd level (3 slots): counterspell, fly, lightning bolt
- 4th level (3 slots): banishment, fire shield, stoneskin
- 5th level (3 slots): cone of cold, scrying, wall of force
- 6th level (1 slot): globe of invulnerability
They follow the same rules as the PCs if they cast a spell of a given level they utilize a slot for said spell. Exceptions would be if the entry indicated that it would not cost a slot to cast, Catrips for example or innate casting with usages per day. Admittedly, since the NPCs are mostly humanoids of playable types you won't find many of those.
Spell casting NPCs in the back of the MM, PHB, or VGtM list the level of the spells, number of spell slots, and known spells. Monstrous creatures often have 1/day, 3/day, or at will spells listed in their spell blocks.
You can also chose to build NPCs with PC levels, in which case you can follow the PC rules for spell casting.
Of course, being the DM, you can chose to make an NPC with a stronger or weaker spell casting regimen, based on if you want them to feel stronger or weaker than the players (be careful about significantly changing the number of high levels spells).
Additionally, you might consider allowing different rules for non-combat spell casting. For example, in my homebrew world, most clerics are not fighting types. Sure there are adventurers bestowed with power from the Gods like the cleric class, but most "clerics" my PC's meet aren't NPC's with PC levels, they're NPCs who can cast any non-combat cleric spell "at will" (takes them like 1-5 minutes to cast a 1 action spell though).
By the same token, most wizards can't cast while simultaneously avoiding arrows, swords, and enemy fireballs. Most probably couldn't even cast fireball without consulting a spell scroll (even if they understand the concepts behind it). They can, however, enchant items, teleport people, and cast other "non-combat" spells at a much higher frequency (and potentially faster) than their combat oriented colleagues.
A good comparison to real life is comparing a member of the Army Corp of Engineers (ACE) to an Army Sapper. Sappers are specialized in a few military applications of engineering (bridge building, explosives sizing and use, etc...), and can preform those tasks quickly and effectively in high pressure situations. They are less skilled at long term, large, complex engineering projects. An ACE, on the other hand, is more versatile in their known subject matters, and can better build more complex designs. They would be next to useless in a high pressure/low time situation though (in comparison to the Sapper who has military training).