The notion that you cannot simultaneously optimize and roleplay is known as the Stormwind Fallacy, that is, widely recognized (and named!) as an inaccurate statement. If anyone tells you that you cannot do both, you should ignore them. Their narrow-minded insults are not worth considering.
The Stormwind Fallacy is often extended to the statement that roleplaying and optimization are mutually independent, that is, how much you roleplay has zero effect on how much you optimize, and vice versa. This is a harder statement to make, and becomes a question of definition.
What is optimization?
The big question for whether the “stronger” form of the Stormwind fallacy is, in fact, fallacious is what you mean by “optimizing.” Within D&D 3.5, it is possible to become literally capable of doing anything, instantaneously, at level 1. Powers dramatically exceeding those attributed to the gods in the setting. The particular tricks that enables this have themselves been optimized for executing the tricks as quickly as possible, since the only time such a character is vulnerable is before they have completed it.
This is what is known as “theoretical optimization.” Theoretical because it is never intended to be used in a game; there would be no point. TO, as it’s known, is its own separate game from D&D, for all it is based on D&D’s rules. It’s a game of playing with, abusing, and stretching those rules and seeing how badly you mess things up. Pun-pun, described above, gets infinite or as-large-as-he-likes stats for everything, every ability ever published, and arguably can start making up his own. That’s the extreme. The TO game, itself, is often played with various “houserules” – restrictions on what you can do to make it a little more interesting. Commonly, no infinite loops, for example.
But this has little and less to do with D&D 3.5. It’s a different game, and is not related to how you play D&D. Optimization for a game of D&D, rather than theoretical optimization as its own separate game, is sometimes known as “practical optimization,” but more frequently just simply optimization. This distinction is very important.
Theoretical optimization limits roleplaying. Pun-pun can never be challenged in any meaningful way, and while you certainly can roleplay a god-like being like Pun-pun, and roleplay him well, he is still just one character; if you define optimizing as literally doing everything you can, the logical conclusion is to always play Pun-pun. Wouldn’t, I’d argue, automatically mean you’re roleplaying badly, but it does have an effect.
“Real” optimization: optimizing for constraints
But practical optimization is optimizing for a purpose. Within engineering, optimization problems are defined by their constraints: the reason why there is an optimal answer is because constraints prevent you from going beyond whatever that optimal answer is. So what constraints do we have?
There are two: the constraints applied by your table and group (gentlemen’s agreements, houserules, expected power levels, setting and themes, and so on), and the constraints applied by your character.
Group and game constraints
You should see the expectations of the group you play with and the game you are playing as constraints on what your character can do. Even if your character would be better using some third-party sci-fi book’s futuristic weaponry, you can’t use them if you’re playing in a high-fantasy game with people using only official material. Even if a series of tricks would mean your wizard has phenomenal cosmic powers, you can’t use them if you’re playing a game of swords and sorcery where magic is rare and evil and real heroes use swords (note: the D&D 3.5 system is, in my opinion, an atrocious choice for such games; the system is kind of built on assuming that wizards do have phenomenal cosmic powers).
So that’s one constraint. That’s more about playing the actual game than it is about roleplaying; it’s about showing up to the same table as everyone else and joining with them on the quest. Without that, you’re playing different games and the friction from that will cause problems – and if you are the one ignoring the constraints, you are in the wrong, and best case scenario you have to retire the character and start over; worst case scenario you’re kicked out of the group.
Character constraints
The other constraints are based on your character, and are much more tightly tied to roleplaying. Your character should have goals and should have talents – and all “talent” really means is innate interest and tendency. There are certain things your character wants to get done, and there are certain methods that your character gravitates towards for solving his problems.
In Dungeons and Dragons, goals typically involve delving into a dungeon and slaying a dragon – that is, going on some epic quest of danger and being a hero that through wit, ingenuity, and more than a little combat skill, overcomes those dangers, completes the quest, kills or otherwise defeats the Big Bad Evil Guy, and saves the world/the girl/the MacGuffin. There are sub-goals and side-goals and maybe that’s not what the character originally set out to do, but ultimately this is what the game is about and what the system is designed to support.
There are, of course, variations you can apply (maybe you are the Big Bad Evil Guy, so you kill the hero and overcome the forces of good, and take over the world or whatever), but straying too far from the formula tends to cause various problems (e.g. a pacifist is a problem, because the system has very detailed rules for combat, which you’re no longer using, while having very simplistic rules for non-combat that tend to be an unsatisfying focus; a system that is better at non-combat situations would be a better choice for that character).
But the point is, ultimately your character is going to have to face danger, and almost certainly is going to have to use force to overcome at least parts of that danger.
So how is your character going to do that? Will he enroll in an arcane university, unraveling the secrets of the universe? Maybe that was in itself his original goal, just to learn magic for magic’s sake, and then plot happened and now he has to face danger. That’s fine. But maybe instead your character will take up her father’s sword and set out to save her kidnapped little brother, and never consider wasting time in a school. Maybe your character grew up on the streets, and was making a simple “honest” living as a pickpocket when he found himself where he shouldn’t have been, was caught overhearing something he shouldn’t have, or stole the wrong thing, and now has an evil cult hounding him and his desperate attempts just to get them to leave him alone lead to him leading the resistance against their plans for world domination.
Each of these characters is good at different things. Your character is going to be good at one of these, or something else entirely. But the point is, your character has a real problem to solve, and he or she is going to want to solve it as well as they can. They are going to want to use the skills they are good at and then get as good as they can be at them. That’s just how heroes react to having critically important quests of great danger thrust upon them.
So whatever it is that you do, and whatever it is that you want to achieve, you are a hero (or a great villain), and you’re going to want to do it right. In-character, you are going to optimize your skills. And this is the reason why there is no need for any conflict between roleplaying and practical optimization. People don’t consciously go into danger with suboptimal gear or skills, when they’re fully aware of other options that are just better and cost the same thing.
So this is how you should optimize: you are making your character the best that they can be. They exist in a world (remember the group and game constraints), they have their own interests and talents, so best is defined differently for them than it would be for another character in another game, but nonetheless, within those constraints, optimizing is making that character the best he could be.
And that’s good for roleplaying, too. When you want to roleplay a master swordsman, it’s important that you actually have strong swording skills. If the dude who just picked up a sword because he needed something is as good, or better than your master swordsman, you have a problem that’s going to negatively impact your roleplaying. And unfortunately, within D&D 3.5, that situation is all too likely to occur.
Conclusion
D&D is about heroic fantasy, and heroes react to dangerous quests by meeting the challenge, training and equipping themselves for the quest, and being prepared. In other words, the choices you make to optimize your character, provided that they fall within the constraints of the group and game, as well as of the character him- or herself, are very likely the choices that your character would have made, as well. Thus, there is a lack of friction between roleplaying and optimization: both want the same thing, so long as you are cognizant of the constraints.
