I can't see any good reason for someone to use Total Defense when they have Combat Expertise. If I understand correctly, using Combat Expertise the character (assuming a BAB of 4 or higher) could have a +4 dodge bonus to armor class, while receiving a -4 penalty to all attacks. Using Total Defense, that same character would receive the same bonus to AC, but would have to give up all attacks.
In theory, a character with a BAB lower than +4 would see a larger AC bonus from Total Defense than from Combat Expertise. Once BAB +4 is attained, Combat Expertise is strictly superior.
However, both options are exceedingly poor. Situations dire enough to give up offensive capability completely, or severely hamper it, should generally be responded to by retreating, not trying to “turtle.” That maneuver is only appropriate when cornered and help will arrive very soon (1 round, maybe 2). Since this is situation every character should be actively trying to avoid as much as possible, Total Defense is superior to Combat Expertise simply because you don’t need to have planned on getting into that situation in order to use it. Because you should be avoiding that situation as much as you can, spending a feat on a situation that occurs rarely and you are actively avoiding is unwise. The attacks you make at a −5 penalty are just not worth a feat, nor is the +1 to AC.
If you are concerned about this situation, and want to spend resources on handling it, it is far, far better to invest in a “panic button” that can get you out of such a situation. Anklets of translocation provide two swift-action, 10-ft. teleports a day for a measly 1,400 gp. That’s worth way less than a feat, and will go a long way to making sure you never have to turtle in the first place.
Unfortunately, that argument is moot in a lot of cases because characters are forced to acquire Combat Expertise because of its frequent use as a feat tax. Once you’ve got it (and BAB +4 or more), there’s no reason not to use it if you find yourself in a situation where you have to turtle. You should still have panic buttons, and should still use them first, of course.
It's almost always better to use Combat Expertise in this situation.
There are a few third-party classes, and I think a few PrCs, that get special benefits when using total defense. Those characters might have a tougher decision to make. In normal play, though, if you need the AC, it's usually better to go ahead and take your attack(s) with the penalty than to sacrifice them.
(This assumes you're going to either use total defense or use Combat Expertise, and doesn't take options like don't use either defensive option into account.)
There is a synergy between the Tumble skill and the Fight Defensively and Total Defense actions. If you have more than 5 ranks in Tumble, you get:
- +1 Dodge AC when fighting defensively
- +2 Dodge AC when using Total Defense
As a result, with a meager 5 ranks in Tumble, achievable by level 2, you get a total of +6 Dodge AC when using Total Defense.
At this point, it becomes:
- Total Defense: +6 Dodge AC, no retaliating
- Combat Expertise: +5 Dodge AC, -5 Attack Roll (damned by the cap, but already a heavy penalty)
It is also important to note that Combat Expertise is capped by BAB:
- a 3/4 BAB character (such as a Rogue) obtains +5 BAB at level 7th
- a 1/2 BAB character (such as a Wizard) obtains +5 BAB at level 10th
For those characters, Combat Expertise only comes into full effect very late compared to the synergy bonus of Tumble (if in class skill).
EDIT: As @Wyrmwood pointed and I missed:
assuming a BAB of 4 or higher
Then I will consider it as so from now on.
I believe the main advantage of Total Defense over Combat Expertise and Fighting Defensively is the possibility of a safer approach or crossing a threatened area generating AoO before your attack, specially with 5 ranks in Tumble.
Total Defense is clearly a better option for those that need the extra AC now and are not threatening anyone and/or can't attack. Even more to those that also are in the threatening area of an enemy that you can't hit melee from your position (bigger enemy or reach weapons).
Imagine these scenarios:
• Level 9: Two Athach surprised the party in a closed space and you (a Rogue) are alone at the other side, with no way out except step between/alongside them. It would be hard to tumble all that threatening area at this level, unless you have enough speed and confidence. However, you may use your standard action to get a +6 dodge bonus to AC and try to pass alive against the 2 AoO. A level 9 Rogue, for example, usually have an AC between 21 to 25 at level 9, with +6 she goes between 27 to 31 against the +16 to hit of the athaches (+12 in a full attack). You may not have enough speed to cross this huge area, but maybe enough to reach to cross one and stay close to the rest of the party (at least close enough for the cleric reach with Revivify).
• Level 6: An already hurt human Scout must cross the battle to reach the cleric to receive healing, but with 5 thugs on his way, even with his 40ft.(12m) speed, quick tumble while avoiding 5 AdO is too much (DC 33). Thanks to his Mobility feat (normal feat for as Scout at level 4 bonus feat) he already has +4 to AC against AdO for crossing this death corridor, using the Total Defense standard action, the aforementioned scout has +10 extra dodge bonus to his AC to escape his predicament and reach the healing mach..., I mean, the cleric.
If you change to people that can't tumble his way out and have Combat Expertise (Fighters, Crusaders etc), both scenario are still true, but they may have a reach weapon that allow them to use combat expertise from relative safety, but everything goes down when you give longspears (or spiked chains if you are really mean) to the ogres or fighting a Gargantuan Monstrous Centipede, basically anything with more reach than you.