Do magical armor enhancement bonuses increase the maximum DEX mod allowed for an armor?
If so, awesome! If not, are there other ways to optimize AC through a DEX driven character?
The magical enhancement bonuses on armor do not affect the Maximum Dexterity Bonus (MDB) the armor has.
There are a couple of ways to get a better MDB.
Special materials for example. Darkleaf and Mithril both allow another +2 to MDB for armors made from those substances.
Feats which somewhat link off DEX such as Two Weapon Defense but they have prerequisites. There is also Dodge, and Low Profile, though you need to be a small character (or smaller) for that last one.
You could also look at Combat Expertise which will trade off Attack bonuses for Dodge bonuses. Fighting defensively also works with the trade of sacrificed Attack for Dodge bonuses. Not strictly speaking just for Dexterity high characters... more for BAB high characters.
I won't re-iterate the other (high-quality) answer, but I will note a few additional things to keep in mind:
Armor training lets you add another +4 to maximum dex, as well as reducing Armor Check penalty by as much. It also lets you wear Medium and Heavy armor without speed reduction at levels 3 and 7 respectively.
For example, the Celestial Armor has a maximum Dexterity bonus of +8. It also starts off with a 6 base armor bonus and +3 Enhancement. It's an excellent choice for characters having/expecting to get high Dexterity bonuses. If you have 20+ Dexterity, it's strictly superior to Full Plate. 24+ Dexterity will make it superior to Mithral Full Plate.
Given sufficient Dex, it will beat everything under normal circumstances (though if you have 40 or more Dexterity, armors like Ceremonial Silk without Dex cap can outpace it.)
I found the RPGBot guide on this to be some excellent reading. He also does some of the math, if you're into that sort of thing.
unfortunately, NO.
If you want to boost your ac with a DEX based character,
The first thing to know is
If you are a fighter, you can take any armor with a good max dex, make it mithral, you should get a medium armor with +11 max dex or something like that, you can event take a heavy armor (tatami-do modify it to get -1 AC +2 Max dex nimble to get a +11 Max dex bonus in Heavy armor !!! - at lvl 15) If you will not reach +11 dex take a o-yoroi (+10) or a full plate (+9)
Then if you do not need to be in melee, fly, burrow, take cover!
If you have too much dex, it's bracer of armor, but never forget that some circumstance make you loose you dex bonus, like feint.
then everything that is not an armor (shield, amulet of natural armor, ring of protection, inherent natural armor (race, or polymorph effect), ioun stone dusty rose prism
use polymorph any object to become tiny humanoid ;) or something with natural armor (also humanoid).
be careful when investing feats, like
+1 to AC is weak (for a feat), and the max dex bonus only apply on AC, not other skills or hit bonus.
Without feats the max armor bonus for light armors is around 16, 17 for medium and 18 for heavy (max dex, + enhancement bonus, and special material)
You can also use an armor with no max dex, but then the bonus from the armor is only +6 (1 (+5 enhancement)) which is less than a bracer of armor who can go to +8.
so if you dex bonus is higher than +10 the best (for ac) is the bracer of armor (but it is very expensive) and do nothing else (like spell storing), always funny when the barbarian want to full round and become staggered (don't work on pounce)