Probably the easiest option is to have the summoner already have his minions out and ready when the party gets there. Why would someone who is traditionally weak in a one-on-one confrontation wait until the confrontation actually happens? He'd want to have the upper hand going in. Also what the others said about having the summoner out of sight. Invisibility works, so do slightly more insane options. Maybe as the party enters the room, he's perched up on the support beams or a ceiling fixture. Maybe he's phase-shifted just enough that they can't hit him.
He could also be masquerading as something else, especially if he has the feats that let him cast unnoticed. This works doubly well if there's some obvious target set up as the patsy. Perhaps some lieutenant commanding the minions. Maybe just some unfortunate schmo who gets to wear the impressive robes and then get whacked by the party. The actual summoner could be posing as an assistant, a servant, even another monster.
If your party's high enough level to handle it, consider a multi-classed summoner. The party evidently goes after the summoner first because they recognize him as a summoner. But if he's got a couple of levels of something else, he won't be immediately identifiable. Especially if those couple levels are a combat class and/or it's a monstrous humanoid like an ogre or bugbear. As long as they don't see the summoning go down or figure out where it's coming from, there would be no reason to assume the one that looks like a fighter is actually the summoner. The additional class would probably also give it some more soak potential and thus more staying power.
While these all directly address the summoner part of the equation, you could also do away with the summoner entirely, or use it as bait. Instead of a summoner and a horde of minions, maybe it's a squad of a couple of toughs and some nasty little bitey things. Or maybe while they're busy giving the unfortunate summoner a steel enema, a bard or another type of mage starts screwing with their stats, or laying down the mind control. Or a cleric or a druid could do some horrible, horrible things to them. The players have identified the game as Summoning Central, and have identified the summoners as the priority target, and I can't say that I fault their logic. So change it up a little. Surprise them. They always take out the summoners because there is always a summoner to take out. That's the thing about players. If the encounters are predictable, they'll learn how to demolish them in the most efficient way possible.