Do you know Tucker's Kobolds? Kobolds are 1/4th CR per. In an open fight, you pit a group of them against 1-st level characters in open field. But 20 Kobolds pushing burning walls in tunnels that are using pikes, alchemist fire, and blowguns are deadly way beyond their challenge rating of 5, even to a party that is way over level for that. The difference? Tactics and strategy simply don't appear on the stat block at all.
The main problem is Action Economy. Many weak attackers, even if each one is easily dispatched, risk a few good lucky attacks. If you have 1 target with 10 Hit Dice, that's one enemy's worth of attacks on the group. The group with its many attacks has a much higher chance to crit - and due to statistics, to hit in general. A subset of this problem is things that Paralyze, Stun, Petrify and Dominate the PCs and can, among other things, impact the action economy a lot.
Groups sponge attacks. 10 targets with one Hit Dice each do hand out more attacks. They also swallow more attacks. Assume each PC does about one and a half Hit Dice in damage. That means the 10 HD monster takes about 7 hits to down. But without area attacks, 10 enemies require 10 hits, as there is no overflow between targets.
Terrain can totally change the encounter. Remember tucker's kobolds. They use moving walls. They get half to full cover from the wall, but their attacks don't suffer from that. However, the wall, as a piece of terrain does not impact the CR calculation at all. Thus imbalancing the results of it.
Likewise, Surprise and focus fire can change the whole event too: Tucker's Kobolds consistently gain the surprise round. They are many, and some of their attacks can and will be lucky. If they focus their attacks on the physically less sturdy ones in their surprise round, they have a good chance to down the cleric or wizard - who usually are required to keep the party in the fight or do AOE damage. Such a focussed surprise attack will shift the action economy in their favor even harder than their sheer numbers do by taking out the AoE, Buff and Healing capacity of the team.
Special group advantages
And then there are things like Pack Tactics and Swarm that make such groups punch even more above their weight class.