Ars Magica meets your requirements, though it is not horribly "similiar" to 3.5
I am something of an Ars fanboy here, but let me address your requirements in turn.
Ars Magica 5th edition is a game of Magi in mythic europe... by default. However, the system and world supports many other genres and types of play, so long as they centre around an excellent magical system.
Battles are fast and fluid. I have found it extremely easy to make ad-hoc encounters when I was running Ars Magica, though my group was rather pacifistic. There don't seem to be any over-powered combat rules that are not representations of real life. (Clearly a group is pretty effective against a lone individual). Moreover, the injury system has a lovely gritty feel that is quite hackable, as wound recovery times are weeks and months, but can be tuned to your specific narrative needs.
"Supports fully-customizable spells by default" You won't find a system better than this than ArM. By combining the five techniques (verbs) and ten forms (nouns), you have over fifty approaches to making one's will manifest in the world. Each combination provides guidelines for magi making their own spells, both in the heat of the moment and as a matter of careful research. Each section also includes a set of sample spells, that cover the basic approaches of the art. This, too, can be tuned to what you want, with non-magic spellcasting provided for in case players want to cavort with fae, devils, or angels.
ArM5 supports high-fantasy and heroic play out of the box, as you can create scenarios where heroes can triumph (instead of dying horribly. My players were cautious for a reason.) It also supports the other sub-genres of fantasy and, without too much of a stretch, can support some of the essential questions of science-fiction (at least if you take Brin's definition) It also supports an amazingly rich philosophical basis, if you're into that sort of thing.
For what you're looking for, I recommend Ars Magica, though it has a very different game philosophy than D&D.
For completeness' sake, however, there are some significant differences:
- Combat, by default, is lethal. It has death-spirals enabled by default. So the moment you're hit with even a light wound, it becomes far easier for an opponent to tag you again... and again. On the other hand, if all the opponent is doing is light wounds, you can take an arbitrary number of them. There are no "hit points" per se. This will come as something as a shock to D&D players.
- It's not balanced. Ars Magica is a simulation of magic in mythic europe. While the setting is eminently portable to basically any magic-using setting, players are expected to have multiple characters: their magi, and assorted companions and grogs. At the same time, there exist options for "heroic" non-magi to exist at the same power level. 5th Edition is a lot better at this. It is a bit of a stretch though, and if your characters aren't interested in exploring magic by default, a discerning selection of sourcebooks will be necessary. I'm happy to help (as I own basically all of them.)
- It's really really easy to get bogged down in details. My game happily supported a full calculation of expenses for the covenant. However, not all players like playing with budgets in their RPGs. Make sure you figure out what activities your players like to do and make sure the game focuses on those. Don't just add sub-systems from one of the other sourcebooks because they look fun. On the other hand, if your players like a certain kind of details, it's quite easy to enable as detailed a simulation of that specific aspect of the world as you like. Up to and including traditional army versus army combat, financial wrangling, philosophical debates in a university, and a theological conflict between a Bishop and a Rabbi.
- When they say that a particular subsystem takes a long time to implement, they're not lying. While it's certainly possible to simulate your characters' life by figuring out what they've done every season since they turned 5 years old, be prepared to write off 40 hours to that activity. It's probably not wise to allow this "extremely complex and overly detailed character creation" by default.
Informational resources
- Complete, free, legal versions of 4th Edition core book here. While 5th edition makes a number of significant tweaks, the fundamental mechanics and setting are fundamentally the same.
- Fourth edition "Jump start" scenario here. The rules briefing on page 13 will give you a feel for the system. While 5th edition changes some subtle (but important) aspects, the core philosophy is explored well. Unfortunately, this booklet does not describe the combat mechanics in any appreciable detail. If you find you like the feel of the rest of the rules, I recommend asking a question of how combat works and/or pinging me privately in chat for a sample combat.