Low CR monsters pose a significant threat, even to higher level PCs.
Due to the nature of bounded accuracy in DnD 5e, high level PCs are unlikely to be invulnerable to the attacks of any monster no matter the CR. In fact, an encounter with a larger number of lower CR creatures will be more dangerous than an encounter with a smaller number of higher CR creatures even if the two encounters have the same properly calculated encounter CR (as described in the DMG pg 81, "Creating a Combat ENcounter).
Given many attacks against a player, attacks will certainly get through. Every round these creatures stay alive is that many more chances for more attacks to get through. It doesn't take many poor PC rolls or great monster rolls before things look grim. Additionally, single-target control spells and abilities of any level won't be very effective, and it's not uncommon for players to be hesitant to use their beefier area spells against creatures they don't consider individual threats.
It all amounts to low CR creatures being lethal. I would caution you against using them too liberally, add two or three too many to an encounter and your players may rapidly be overwhelmed.
Party: 4 PCs at level 5
Encounter A: 18 x Bandit (CR 1/8) = CR 5
25xp multiplied by 18 creatures, total multiplied by 4 due to number of creatures being over 15 = 1800xp adjusted
This encounter has a total HP pool of 198.
Each round, this encounter gets 18 melee attacks at +3 to hit dealing 1d6+1 damage each, or 18 ranged attacks at +3 to hit dealing 1d8+1 damage each.
Let's say they all attack the tank of the party, wearing +1 Plate and a shield with an AC 21. They hit on an 18 or better, dealing an average of 5.5 damage * 18 = 99. On average 14.85 damage (0.15*99) gets dealt to the player each round.
Encounter B: 1 x Drow Elite Warrior (CR 5) = CR 5 = 1800xp
This encounter has a total HP pool of 71.
Each round, this encounter gets 2 melee attacks at +7 to hit dealing 4d6+4 each.
Let's say it's attacking the same tanky PC. It hits on a 14 or better and deals an average of 18 damage per hit * 2 = 36. On average, 12.6 (0.35*36) damage gets through.
- Encounter A: 198 HP, 14.85 dmg/r
- Encounter B: 71 HP, 12.6 dmg/r
On top of these numbers, consider the other points mentioned above such as the ease of disabling the one enemy vs 18 and the sheer amount of damage a party can dish out in a round.
Even though this is certainly not the best example, I think it illustrates how lethal low CR monsters can be. Hopefully my math was correct, but even if it isn't I think the factual numbers speak for themselves.
Down the line, I suspect you will find the opposite to be a bigger problem: How do I make individual bad guys a significant threat without threatening insta-killing PCs?
A solution that I have begun employing, taken from 4e and suggested by Matt Colville, is to take those low-CR creatures, give them only 1HP each, and add them to "boss fight" encounters. This way they still pose the same threat with their multitude of attacks, but can more consistently be removed from the fight by players (less bad-luck-TPKs, as I see it), and allows for the actual boss to get a round or two to shine in before he inevitably gets his person Held, grappled, Divine Smite, Hunter's Mark, Sneak Attack, crit into the dust.