First, make sure your DM understands this is a houserule.
Show him the actual rules regarding this situation in the Player’s Handbook. @DuckTapeal has quoted the relevant section from the System Reference Document, which is official and accurate, but for maximum effectiveness go back to the book.
In addition to that line in the Magic section, the Armor and Weapon Proficiency sections of Bard, Sorcerer, and Wizard all have lines akin to “wearing medium or heavy armor or using a shield incurs a chance of arcane spell failure if the spell in question has a somatic component (most do),” (taken from Bard on the SRD; Sorcerer and Wizard specify all armor).
You are correct that the quoted line about multiclass Bards refers only to spells from other arcane spellcasters. I.e. a Bard/Wizard does not have to worry about light armors’ ASF with his Bard spells, but does for his Wizard spells. Fighter, as a class with no spells, arcane or otherwise, doesn't even begin to come into play there.
So try to make sure the official rule is clear to him. This may not convince him of anything, but it is an important first step.
Second, discuss social contract and expectations.
It sounds like you have already been playing, and he is only now raising this point. Making major houserules like this in the middle of a game is something I have a really strong objection to; if you do as well, that is worth bringing up.
Furthermore, bring up the fact that this is quite weak, is not being done to be a munchkin or anything but literally to take a pretty weak tactic for roleplaying’s sake. Tell him that you expect a certain amount of trust between DM and player; if he really thinks you are trying to mess up his game or something like that, you have more serious issues than multiclassing Bard and Fighter.
Another thing I would bring up is the idea that I expect a good DM to work with players to execute character archetypes in ways that are amenable to the campaign and setting, and to then challenge those characters in ways that are interesting and meaningful to the character. In other words, I expect a certain amount of personal cooperation between DM and players. Forcing you to not play the character you want to play is not being a good DM, in my mind.
Third, show him just how easy it is to ignore this houserule
It’s very easy to cast spells in armor without paying any attention to Arcane Spell Failure. I wrote a pretty long answer on that subject.
I’ve listed the most relevant ones for you below. Note that if he’s houseruling how non-Somatic spells work with respect to Arcane Spell Failure, he could easily houserule any or all of these, or may not even have to depending on how many books you have available. That doesn’t make this irrelevant, however. The goal isn’t necessarily to use any of these (though actually I totally do recommend that because restricting yourself to non-Somatic spells is an extremely weak choice), but to demonstrate that what you want to do is an expected and normal part of the system, and to demonstrate how large a houserule this potentially is.
In short, I don’t think Arcane Spell Failure is ever meant to be rolled, you’re merely supposed to have to expend resources eliminating it. If he actually wants you to roll it, he does not understand the system very well. Even a 5% Arcane Spell Failure chance is not viable; sooner or later, and probably sooner rather than later, it means failing to cast a crucial spell when you do not have time to wait a round to cast it again. Under no circumstances should you merely accept it.
The authors of the books seem to have understood this, at least by the end of 3.5’s publishing run. All the ways of mitigating or ignoring ASF have costs associated with them, because it is those costs – not the chance of failure that you should never accept – that are the real balancing measure here. Your DM is not making a small change here, he’s making a huge one. This should reinforce your argument about the social contract. He is overturning a lot of material that expects Arcane Spell Failure to work differently.
Divine Bard Variant from Unearthed Arcana
Unearthed Arcana offers the divine bard variant, which does not have to worry about Arcane Spell Failure at all because his spells aren’t Arcane to begin with. You do suffer from alignment restrictions (pretty strict ones, too), and Divine Bard uses Wisdom for Spellcasting instead of Charisma, but you also get to add a bunch of spells to your list.
Dungeonscape Armored Mage Alternate Class Feature
Dungeonscape allows multiclass Fighter/spellcasters to ignore the Arcane Spell Failure of their armor, regardless of weight, so long as the spell’s level is less than or equal to their Fighter level plus one. This is a pretty bad choice but depending on how many Fighter levels you already have it may be no worse than what you already have.
Complete Arcane’s Battle Caster Feat
Battle Caster from Complete Arcane allows you to ignore Arcane Spell Failure for heavier armors than usual, if you already could for armors of a certain weight. Normally a bad option as it doesn’t give proficiency and has a high feat cost, for a Fighter/spellcaster who already has proficiency and a number of bonus feats, it could work very simply.
Just having 0% Arcane Spell Failure
A mithral breastplate is Light armor, and therefore you, as a Bard, can cast spells in it without paying any heed to Arcane Spell Failure. If you make it a feycraft mithral breastplate and add Thistledown Padding, it also has 0% Arcane Spell Failure anyway.
A +1 twilight feycraft mithral breastplate or a +1 mithral breastplate with Thistledown Padding also has 0% Arcane Spell Failure.
A +1 twilight fey-and-gith-craft mithral full plate with Thistledown Padding has 0% Arcane Spell Failure as well. This is the hardest one to convince a DM of, since having both Feycraft and Githcraft is dubious. Personally, I think this is a really interesting potential plot hook; I wouldn’t just let a player pick one up at a store, but a short side-quest for getting this armor made, convincing a gith armorer and a fey armorer to work together, could be quite fun.
Mithral is Core, twilight is in the Magic Item Compendium or Player’s Handbook II, feycraft and githcraft are in Dungeon Master’s Handbook II, and Thistledown Padding is in Races of the Wild. If any of these are unavailable to you, a single level in Spellsword in Complete Warrior reduces Arcane Spell Failure by 10%, and can replace any two of the last three, or twilight.
The Runesmith from Races of Stone
I have no idea if your character happens to be a Dwarf; with the Charisma penalty, probably not. However, the point of this isn’t to actually be one, it’s to show your DM a class that requires Heavy Armor Proficiency and grants, as one of its major and explicit benefits, the ability to remove Somatic components from all spells so he never has to worry about Arcane Spell Failure. It may help your case.
Fourth, be prepared to either live with it or leave
You may not be successful in your negotiations. In fact, I suspect you won’t be: your DM seems to be extremely narrow-minded and utterly unwilling to show any flexibility on his preconceived preferences, regardless of any argument you might have to make. Furthermore, he seems more than willing to abuse his DM position to enforce his preferences on his players. That is a bad situation.
Your DM is 100% wrong on all points, and furthermore sounds like he has absolutely no interest whatsoever in having any conversation with a player who wants to do anything outside his own extremely-narrow preferences. I'd not play a single session more with him. Not worth it. I believe he has badly gone past the limits of Rule 0.