I think the answer of Lexible is mathematically sound, I'd like to add some examples of how other existing games already do this and what you might take from it:
Games like Shadowrun, where the dice mechanic works almost exactly as you said (with the exception of only 5 and 6 counting), you can buy successes instead of rolling. Rolling has a wide variance (for example 9 dice will give you between 0 and 9 successes and it only gets more variant when you roll against somebody who might do good or bad themselves). So statistically, 3 dice give you one success. The rulebook allows you to not roll and instead take one success for every 4 full dice you would ave rolled. That way it's the player who can opt for less randomness, although at a price because over all rolls statistically he would have done better with rolling.
Many games use a fixed base value. Taking Shadowrun as an example again, in older editions the initiative was determined by taking the raw value (let say 8) and adding as many dice rolls with the success chance for the game. That makes for 8-16 successes for skill 8, but 6-12 successes for skill 6. Even on their worst roll, people with a high skill would be better than people with a low skill and average roll. Less variance.
So the solution will be to have a static part and a dynamic part. It's up to you to balance this to your liking. On the extreme sides, where either part is zero, you end up with no dice rolling at all, or checks where you have many dice and a huge variance or luck factor. A fun game is probably somewhere in the middle.