How many minions (in combat) is too many?
Whatever works for your table
This is going to vary from table to table, and even session to session. There isn't a hard number, you're just going to have to work it out.
Now, we're talking about combat minions, that need to act in combat. As Darth Pseudonym pointed out, keep 'em out of combat and it doesn't matter.
Of course, anyone can have minions, but casters seem particularly prone. Anyone can hire some mooks, but casters can acquire legions of beholden followers, and really, why not?
You probably need to talk about it
You probably need to talk with your DM, and maybe other players.
Gameplay problems with minions
Minions can slow down combat for everyone, can be hard to run, can overpower a character, and can centerstage a player.
Almost always these things are going to make the game less fun for other players.
Slowing down combat/can be hard to run
Just managing to-hits, and damage, and tracking hit points, effects, position, and so on can slow down gameplay.
I've found this can be partially mitigated by being more organized: writing down specific to-hits, keeping special track of hit points, etc. You can even average hits and damage to speed things up even further.
The DM can have minions go on the same initiative as their master, and can essentially leave them out of attacks.
But no matter what, extra minions slows things down.
Overpowering a character/centerstaging a player
Minions do not necessarily overpower a character, because they often don't hit all that hard. But even if they don't hit all that hard, they can take lots of other actions. And this ends up taking up time, no matter how efficient you are.
One thing you can do is limit minions to roughly the amount of actions other players get. If the party's fighter gets 3 attacks, then balance the minions to that.
What we did
I'm the player referred to in the original post. For several battles, my wizard had a familiar, a shield guardian, 3 bound elementals, and a sim.
In talking with the other players, it just wasn't as much fun. Not always, but sometimes, even often.
We've solved the problem, more or less. We didn't make hard-and-fast rules, but we do a handful of things:
Mostly, the minions stay out of combat, or semi- out of combat. For instance, the shield guardian takes up a defensive position and doesn't attack. I have a bound elemental spirit of air who usually stays out of combat and provides eye-in-the-sky recon on the group chat (telepathic bond).
We keep the numbers limited. Teleport helps with this, since there's a creature limit, but there are ways around that, but one way or another we keep the minion count low. I watch gameplay, and if it seems like any other players are getting overshadowed, I decrease minion activity.
When they fight, they're extremely efficient. If I can't say what they do and the effect in like one sentence, they don't go that turn. If they're fighting, I'm prepared with exactly the formulas needed, so no looking things up. If there is any hesitation whatsoever, I just have them do nothing.
The DM kills them. Minions are expensive. I go through them like tissue. It takes days and spell slots to set things up. The DM makes me account for time, and then doesn't give us enough.
We give them other things to do. Often the minions are busy either out of combat, or back home, or running some errand.
They fight their own battles. We've had the minions fight some monsters off-stage, so they help absorb some of the monsters, but don't take game time.
We keep them in reserve. The other players are amenable that there may be big fights where more minions are useful. In that case, they
Change the character. After discussion with the DM, we changed my wizards subclass. She was a conjuration wizard, and without conjured creatures (or less conjured creatures) in combat, I felt like the character was losing out on the subclass capstone. The DM and I were talking, and the DM said, if it isn't fun, or if something else would be more fun, change it. So my wizard become a scribe wizard. The DM was like, justify it however you want, and I made up some in-game stuff that felt right.
Talk about it. We have an after-session talk after every session, and we text and voice chat between sessions. We rarely talk about minion issues, but everyone feels like if they have an issue, we can talk about it.