What I'm looking for is a system designed around unintended targets,
backfires, and unexpected additional effects.
This brings automatically to mind the Dead and Wild magic areas in the Forgotten Realms Campaign setting.
The main idea, is that all magic worldwide is woven like a weave and attached to the goddess of magic in the setting (Mystra). Some holes, (whole areas or so small like a 5' spaces) exist in the weave, and there any spell cast can have unpredicted effects. If the magic in the area is totally obsolete, these are the dead magic areas. No casting is allowed here, neither it is possible to cast arcane magic at all.
Wild magic areas are a little more interesting, as the casting of any spell will either backfire, non be cast at all, cast normally, thrown on a random target, or be miscast in other ways, (funny, interesting but also dangerous ways). You actually roll a d% to choose from a table with possible outcomes for casting in such areas.
You will find extended rules and references in both the Forgotten Realms campaign setting book and also the supplement book Magic of Faerun. But beware, both of the books are 3.0 edition and they will need some minimum conversion before becoming totally playable and not gamebreaking in 3.5 edition. Look for the book Player's Guide to Faerun, as not only it is a useful book, but it's the only supplement written in 3.5 and it is used to convert characters, spells, feats etc from 3.0 to 3.5.