OK, since the tags have changed from system-agnostic to dnd-3.5e, the answer to this question has changed. I’m going to state right upfront that this answer is fairly elitist: unfortunately, that is what Wizards deemed best for its game system. Dungeons & Dragons 3.5 was designed to be an “ivory tower,” where those who know the tricks have all the power. It was designed this way from Day 1, and no amount of book banning will fix it. You have to know the system, including a huge array of supplements, and where the traps and problems are, in order to use it well.
So if this answer is elitist (and it is), it’s not because I think this is a good thing: it’s because that’s how Wizards designed the game. I actually detest this fact of 3.5, and therefore try to help systems that attempt to fix it.
You can play it without doing so, of course. But I argue that there are many better systems if that is what you wish to do. Because of the way 3.5 was designed, it has many, serious flaws, which are only worth accepting if you want to deal with the huge amount of material available.
Note that none of this answer makes sense in any system other than 3.5. It is only because of the exact nature of 3.5’s specific flaws that this answer makes any sense.
To begin,
Of course, DMs should not allow books they do not have access to
Or are unfamiliar with. This should go without saying. My answer is explicitly ignoring this reason for banning a book. I’m talking about DMing a book you have and have read, for any other reason.
Other than that, banning books wholesale in D&D 3.5 is a bad idea
There is no book in Dungeons & Dragons 3.5 that is entirely bad. Even Serpent Kingdoms and Book of Exalted Deeds have some few things worth using that aren’t going to break the game. Savage Species is the best argument available, but there are things in that book that I consider good for the game, and moreover Savage Species isn’t really a 3.5 book to begin with (it was going to be a spin-off product, that got cancelled and turned into a “3.25” book just before the release of 3.5).
A multitude of option is 3.5’s only real strong point
Moreover, Dungeons & Dragons 3.5 really does not have a lot going for it, and I say this as someone who plays it more than any other system. If you are not taking advantage of its singular shining strength, it’s incredible array of available options, the system has very little to offer. There are better systems, whose only real drawback compared to 3.5 is the relative lack of material.
Furthermore, the overwhelming majority of 3.5 material is extremely generic. It is a system that welcomes adaptation and reuse of material in novel ways. Sure, that book might specifically be in Eberron, a word that does not even exist in your setting: that doesn’t mean that the Educated feat ought to be unavailable (a lack of academia in your setting might, but then other options in the book may be appropriate and we’re now talking about specifics, which is fine). Feeling hog-tied by the default fluff of various options is unnecessarily limiting on DMs and players: refluffing is an excellent way to make unique characters.
And DMs should not be alone in adapting material. DMs should have the final say on whether or not a given adaptation is valid, but player’s should be encouraged to get involved in the world, flesh out its various nooks and crannies to find a place for their character.
Banning supplements for balance makes no sense:
The worst balance is in Core
Without a doubt, the most wildly imbalanced book ever published by Wizards of the Coast for 3.5 is the Player’s Handbook. The Cleric, Druid, and Wizard were far-and-away the most powerful classes published originally, and after all the dozens and dozens of supplements, they still are. They’ve been joined by the Archivist (Heroes of Horror) and Artificer (Eberron Campaign Setting), they’ve been somewhat improved by supplements (Divine Metamagic, Planar Shepherd, celerity), but ultimately 60% of the most powerful, broken classes in the game are found in the PHB alone, and they derive nearly all of their power from the spells and abilities in that book.
Meanwhile, the Fighter, Monk, and Paladin struggle mightily in Core, and while they have, as Core classes, received tons and tons of supplemental support, they remain among the weakest classes in the game. The only ones generally weaker are the classes based on these three (Complete Warrior’s Samurai is a Fighter with pre-selected feats, Complete Adventurer’s Ninja is a Monk with some stealthy features and a super-nerfed version of Sneak Attack).
So in one book, you have eleven classes: 3 are super-powerful, and can literally dominate entire campaigns alone, and 3 are pitifully weak, struggling to survive even when working together. The remaining 5 tend to be close to one extreme or the other (Sorcerer is almost as powerful as Cleric, Druid, and Wizard; Barbarian, Ranger, and Rogue are nearly as weak as Fighter, etc.)
Balance improved as time went on
The later a book is, the more likely things are to be tightly balanced.
Complete Champion (published 4 years after the PHB) finally provided easy ways to move and full-attack at the same time. Tome of Battle finally gave martial/mundane types options and versatility, and toned down on how hard not full-attacking would cripple martial characters. Between these two and some help from having a wealth of options, a well-made martial character can keep up with a spellcaster who doesn’t try too hard. It’s not great balance, but it’s a whole lot better.
On the flip-side Player’s Handbook II and Tome of Magic finally involved some reasonable spellcasters in the Beguiler and Binder, and even some outright weak magic types in the Dragon Shaman, Healer, and Shadowcaster, who could play nicely with the originally underpowered classes.
Banning books to stop the munchkin makes no sense
Because, again, the worst content is in Core. A competent optimizer dead-set on ruining your game needs nothing more than the Player’s Handbook. The solution isn’t to ban anything – except maybe that player, if he can’t be made to play nice. Don’t try to set rules to keep people you don’t trust in line: just don’t play with people you don’t trust. The game is going to fail anyway if there is no trust.
Banning books to help new players makes sense, but is not necessary
Newbies usually find their own comfort zone.
Some will simply stick with the easily-available Core on their own, and should be helped to avoid all of Core’s traps (seriously, Core is awful about those). This sort of player will probably need a lot of help, so you’ll be in for a fair bit of work. You may actually do very well in directing them to some later books – it is much, much harder to fall into a trap with Tome of Battle than it is with the Player’s Handbook. A Warlock or a Warblade are easy to make, and to a certain extent “self-optimize,” which allows the player to focus on that which he finds fun and interesting instead of agonizing over power levels (or, more realistically, making you agonize over them).
Others will seek help online, and put together something using a bunch of books – make sure they’ve actually read what all of those options do, and certainly read the thing online yourself to make sure you agree with the conclusions made about rules interactions and accept the level of power that the concoction creates. This involves, of course, some work also, but I’d argue it involves less: you have to read whatever guide was used, but those are put together for the sake of making it easy to follow. (do try to avoid dndwiki though, it’s astoundingly consistent in its awful-ness. There are diamonds in the rough there, I’m sure, but I have never seen one)
Do Ban Specific Content
There are lots of things in various books that are poorly thought out, just don’t fit, and so on. These should be banned. That does not mean banning everything else in the book along with it. For example, the Artificer is phenomenally powerful, and relies on (and creates!) Eberron’s magitech society. Definitely not appropriate for, say, a campaign in a savage prehistoric world. It makes sense to ban that. The Warforged, too, for the same reason, though they could easily be golems or elementals, and has much better balance than the Artificer.
There is no reason to allow the kobold ex-paladin to call on Pazuzu, or for Sarrukh to exist, in your custom setting. The shadowcraft mage should be limited. The fleshraker should not be available as an Animal Companion. Et cetera and so on. There are absolutely important limits a DM should place on the material available to his players. They just don’t come in convenient “Overpowered Book,” “Underpowered Book,” and “Just-Right Book.”
Yes, this means you, the DM, need a lot of system mastery
Ultimately, yes, DMing 3.5 successfully requires a fair amount of knowledge of the system. The system was designed to reward system mastery from the ground up. The books are littered with (apparently intentional) traps, and equally littered with (probably unintentional) routes to extreme heights of power. And not all of them involve hairbrained combinations of vaguely-written rules. Contingency and greater teleport and shapechange are very simple spells: they just dominate the game. You, as a DM, have to know about these things if you’re looking to have much success with 3.5.
In my opinion, all of the above highlights some very serious problems with 3.5. If you do not want to know about all of this material, the solution isn’t to limit yourself to Core – Core is the worst of them, literally – it’s to find a new system that suits your preferences better. There are many.