Given a dice pool mechanic where a number of six-sided dice equal to Stat + Skill are rolled, and the number of 4s, 5s, and 6s is counted:
An increase in the size of the pool increases the absolute variance. Is there a mechanism that helps decrease the variance as the pool increases? The only one I've seen is replacing dice in the pool with automatic successes.
I think the answer of Lexible is mathematically sound, I'd like to add some examples of how other existing games already do this and what you might take from it:
Games like Shadowrun, where the dice mechanic works almost exactly as you said (with the exception of only 5 and 6 counting), you can buy successes instead of rolling. Rolling has a wide variance (for example 9 dice will give you between 0 and 9 successes and it only gets more variant when you roll against somebody who might do good or bad themselves). So statistically, 3 dice give you one success. The rulebook allows you to not roll and instead take one success for every 4 full dice you would ave rolled. That way it's the player who can opt for less randomness, although at a price because over all rolls statistically he would have done better with rolling.
Many games use a fixed base value. Taking Shadowrun as an example again, in older editions the initiative was determined by taking the raw value (let say 8) and adding as many dice rolls with the success chance for the game. That makes for 8-16 successes for skill 8, but 6-12 successes for skill 6. Even on their worst roll, people with a high skill would be better than people with a low skill and average roll. Less variance.
So the solution will be to have a static part and a dynamic part. It's up to you to balance this to your liking. On the extreme sides, where either part is zero, you end up with no dice rolling at all, or checks where you have many dice and a huge variance or luck factor. A fun game is probably somewhere in the middle.
Subtract the distribution mean from your roll. If using N dice as you describe, each die contributes 0.5 towards the distribution mean of the whole pool, so the distribution mean is N/2. The resulting distribution will have a mean of zero. Example: Roll 5d6 to get 2,2,4,5,5. So three successes minus the distribution mean, or 3 – 2.5 = o.5.
Divide the result from step 1 by the sample distribution standard deviation, or in your case by the square root of N/4. So for five dice that's the square-root of 1.25, or about 1.12. The resulting distribution will have a variance of one. For example: From the result in step 1, we would get 0.5/1.12, or about 0.446.
Say you want a target variance of A, you simply multiply the result of step 2 by the square-root of A, where A is greater than or equal to zero. For example: You (for whatever reason) want your distribution's variance to equal four, so the result from step 2 times square-root of four is about 0.446 * 2 = 0.892.
Finally, you want a greater number of dice to give a greater average result, so add your original distribution mean back in so that the mean of a size N dice pool is still N/2. For example: Adding the distribution mean for five dice to the result of step 3 gives 2.5 + 0.892 = 3.392.
Let \$n\$ be the number of dice or number of successes. The standard deviation of a success-counting dice pool is proportional to \$\sqrt{n}\$, so the standard deviation of the number of successes indeed increases with the size of the pool.
However, if we are willing to abandon the assumption that the number of dice/successes has a linear interpretation, then there is a solution that does not require any change to the rolling mechanics. Suppose that instead the pool size and required number of successes are quadratic proportional to some parameter---call it \$\ell\$ for "level".
$$n \propto \ell^2$$
In other words, instead of considering 1 die, 2 dice, 3 dice, ... to be equally spaced, we would consider 1 die, 4 dice, 9 dice, ... to be equally spaced. Then, we have:
$$\ell \propto \sqrt{n}$$
$$\frac{d\ell}{dn} \propto \frac{1}{\sqrt{n}}$$
Therefore, a (small) deviation in the number of successes \$n\$ produces a change in level \$\ell\$ that is proportional to \$\sqrt{n}\$ times smaller. This cancels out the increase with in standard deviation exactly, so that the "standard deviation" in terms of level \$\ell\$ is constant.
One potential downside of this approach is that the pool size may grow to an physically unwieldy size more quickly. Quickly-growing pool sizes may also favor increasing the success threshold to 5+ or even 6 to avoid excessive separation between levels.
This also allows us to look at the increasing standard deviation in another way: as diminishing returns of the value of each additional die (relative to e.g. bonuses in a d20 + modifier system). Some designers may see this as a plus; for example, they may want beginners to improve faster, or for it to be more difficult to gain an overwhelming advantage through stacking up a large pool.
