Is there a way within the rules of Dungeons and Dragons 3.5 Edition to increase the enhancement bonus on specific magic weapons. For example if my character had a Mace of Terror (a +2 heavy mace with some extra magical abilities) would there be any way for it to be improved so that it would be a +3 heavy mace with the same magical capabilities?
Yes, according to the d20SRD:
A creator can add new magical abilities to a magic item with no restrictions. The cost to do this is the same as if the item was not magical. Thus, a +1 longsword can be made into a +2 vorpal longsword, with the cost to create it being equal to that of a +2 vorpal sword minus the cost of a +1 sword.
The Mace of Terror is a +2 heavy mace with a unique special ability that neither improves weapon accuracy or damage nor imitates any available non-unique weapon special ability (like returning or spell storing).
The cost to improve the enhancement bonus to +3 would be the same as for any +2 weapon: 10,000 gold.
It depends on your DM, but I'd rule yes.
Sometimes, determining how much of the cost of a specific weapon or piece of armor (shields included) comes from its unique properties is very hard, so some DMs just say that the weapon can't be crafted in any other variant than the one presented (with the added rationale that if they wanted you to be able to craft different versions of it, the developers would have provided the cost for the specific effect in the usual weapon enhancements table/chapter).
Anyway (proving negatives is a hard thing to do, so be aware that I might be wrong) I've never seen any manual snippet preventing people from upgrading those weapons/armors/shields.
A +2 weapon costs 8000 gp on top of its being perfect (300 gp) and the price of the heavy mace (12). The mace of terror costs 30240 gp more.
Now, 38240 is not a multiple of 2000×(any squared number) so I'd rule this property costing a flat 30240 gp. While I don't think such a property (3 times per day only, with a low will save) is wort that much, I'm going to use this value in my calculations and determine that the +3 equivalent costs 10000 gp more, for a grand total of 48552 gp.
Improving the mace to the next level would be done, as it is usual for upgrading magic weapons, by asking someone who has the appropriate weapon enchanting feat and access to someone casting the required magic weapon spell every day for 100 days.