You can...maybe have an artificer subclass in your third-party setting. Things are made more complex because that specific class is not covered by the OGL. You could, however, obfuscate the class name in a way that someone else would be able to figure out what you were referring to.
For example, let's say that you're a well-known dungeon master who made use of copyrighted names for various entities, such as Vecna, Pelor, Tharizdun, and others, but did so within your own homebrew setting. When it comes time to publish your setting guide, what do you do?
If you're Matthew Mercer, creating the Tal'dorei setting guide1, you change the names, add some flavor text for how these entities behave in your setting, and call it a day. Nobody is going to doubt for even a second that "The Dawnfather", who has the exact same domain and overall lore as Pelor, and who is referred to as Pelor within much of the recorded campaign, is just Pelor with the serial numbers filed off.
Given that game rules are not protected by copyright (but specific words are), I suspect that artificer could see a similar treatment; a lawyer would probably be necessary to be certain of how you're wording things, but it's definitely doable.
Subclasses for any class contained in the basic ruleset, however, are fair game. In the same Tal'dorei setting guide, there are new subclasses for cleric, barbarian, sorcerer, and monk-- all explicitly referring to their original class. Because the base classes are covered by the OGL, you have free reign2 to publish new subclasses to fit into that framework.
1 Explorer's Guide to Wildemount is an official Wizards of the Coast product, but Critical Role: Tal'dorei Campaign Setting is very much not. It features the usual obfuscation methods employed by non-official material, such as being labeled with a vague "5e" indicator without any references to D&D directly. Of course, this is a big shiny book from a team of professionals who definitely had lawyers on hand to make sure they stayed on the right side of the law, and I suspect that WoTC is aware that it would look very bad for them to sue someone who has played no small part in their ridiculous profits of late.
2 Sort of. There are tons of legal disclaimers throughout the setting guide, and Matthew Mercer and the Critical Role team have quite a bit more access to WoTC than your average bear. I want to make it very clear that the answer to your question is "Yes, you can publish custom subclasses, but maybe not for this specific class", but also, "This is a complicated situation, and the one example I can think of where it's been done was from a team that had a lot of resources which you almost certainly do not have access to".