It’s quite common.
Banning high-end (and low-end) material is a very common practice. It’s massively more effective and sensible than, say, banning books X, Y, and Z when your goal is to have a certain power level; it gets right to the heart of the issue you’re looking at.
In my experience, however, it’s better to just ban Tier-1 characters. A single level of cleric, while very good, does not make someone a Tier-1 character, but it does enable a lot of other options (Travel Devotion, various Divine Feat options, paying feat taxes efficiently, etc.). Single-level, or few-level, dips in archivist, artificer, druid, or wizard are less frequently desirable, but I can think of arguments for each.
By the same token, it’s pretty common to want players to avoid Tier-5 characters, but sometimes judiciously using a Tier-5 class is appropriate – a couple levels of paladin for Divine Grace, a level of monk for the feats, whatever.
So yes, tell the person playing the cleric that the class is too powerful for the game you’re DMing, and maybe you’ll allow minimal use of it. Tell the player who thinks monk is a good idea that it’s not, and that you need characters to have more power than monk offers in order to be able to make your life easier. A level, to get a punch of relevant feats – fine, if you must. But only if you’re grafting that onto something that could really use it, and be competent.
And it does sound kind of like some of your players are in a place where even allowing minimal usage may cause problems, just because they do seem liable to try to weasel their way into getting more out of you. At least the first time, it may be best to just say “no,” though again I do recommend in general to allow judicious, minimal use of classes outside of the desired tier-band, whatever it is.
Because you are absolutely right – DMing for split-tier parties is extremely difficult and frustrating. In fact, avoiding that situation is exactly the stated purpose of the tier system. See JaronK’s tier list for classes, in the intro spoiler:
Thus, this system is created for the following purposes:
- To help DMs judge what should be allowed and what shouldn't in their games. It may sound cheesy when the Fighter player wants to be a Half Minotaur Water Orc, but if the rest of his party is Druid, Cloistered Cleric, Archivist, and Artificer, then maybe you should allow that to balance things out. However, if the player is asking to be allowed to be a Venerable White Dragonspawn Dragonwrought Kobold Sorcerer and the rest of the party is a Monk, a Fighter, and a Rogue, maybe you shouldn't let that fly.
(emphasis mine)
The reason anyone cares about tiers at all is to attempt to recognize and address that situation. You are intended to look at the tiers, pick a certain tier or tier-band, and use that as a guide to keeping all PCs on roughly equal footing so that DMing becomes more manageable. So yes, this is “OK,” at least insofar as common wisdom is concerned – it’s actually the whole idea!
- By the way, if your players are looking for acceptable alternatives to favorite classes, this answer may be a helpful jumping-off point.