I have a similar problem with a few friends who partake of a weekly pathfinder game I run. Two of them know very well how to min-max and power-game but choose not to for flavor reasons (or at least limit their ludicrousness), and two others don't know how not to min-max.
To give a quick example, when I ask the two who restrain themselves a bit what kind of character they want to play I get simple and oft mundane answers: Thief who's down on his luck and needs to find some work, or Cleric attempting to be a missionary in unfamiliar lands. When I ask the other two the same question they start in with "what books are allowed, any special races, can I have this feat printed in X third party resource?". And if I say "no, not class... just what do you have in mind for a character... like background... or story?" I get sighs and terse responses saying "I can't come up with a story or background for a character that doesn't exist yet... I need stats and abilities and I'll make up story for why I have them!"
In my opinion the latter two in my group prioritize rules and structure over the story or narrative feel of the campaign entirely. To me it seems a bit backwards to "make up story for abilities" because I learned that in DnD I should "take classes/abilities that fit my character concept".
Now, back to the OP problem... It seems like you might be experiencing the same kind of thing in your game. Your players aren't treating the game you are running as a story or narrative experience where they are the main characters, they are treating it like a hack-n-slash board game where optimized characters and munchkin-combo tactics are, for lack of a better way to put it, how they win. What this all boils down to is that there is no such thing as "complex-but-balanced rules" to solve this problem. If you crunch the numbers and factor in as much data as you can, there will always be a rule combination or specific build that is "optimal" enough to turn a "fair fight" as run by a competent DM into "charge/attack/win.
Aside from attempting to change the mood or mindset of your players with the many well-thought-out options above, which do not require a rule-book, I would suggest taking the vitality/wounds alternate rules and modifying them to your liking. Doing so has produced a much more dangerous environment in my game and has resulted in much more "cautious" play which requires a bit of planning and tactical prowess on the part of my players. The wounds system can be accompanied by permanent injury rules or something similar, to add some gravity to the combat in a game. I find that when players slip into a mindset where being stabbed by a giant with a great-sword "isn't a big deal because they still have half their HP" they need a wake-up call. That wake up call comes in the form of ability score penalties, speed-hindrances and decreased combat effectiveness that is a "burden" upon the others to fix or heal. Social pressure kicks in and on a sub-conscious level players start worrying about getting "hurt" because it will slow things down, which translates into something resembling "actual concern for their character's well-being".
The ultimate goal should be striking a balance between challenge and fun, without players feeling resentment or pitching a fit when their choice of action results in loss or death. The second part is harder because it relies more on the maturity and emotional state of the player. I can assure you it is far easier when dealing with players that put some kind of investment into the personality and behavior of their characters, if they always have the same "charge/attack/kill" response in all situations they may not be able to handle it when things go wrong. They will blame the rules, or the situation the DM put them in, or just flat out say "this is unfair".
I could be entirely wrong with my interpretation of your question or perhaps the motivation behind wanting to change your players' approach. I sort of read the overall problem being that "your players don't seem to appreciate the danger of a situation at all" and "how can I get them to have some real concern for their well-being so they don't just kick in the door on every occasion?"