Ok there are a number of factors, in what you are seeing.
Sampling Bias
One is sampling bias. You have been looking at char-ops threads.
Your sample is not all RPG players, only those who choose to talk about it on the internet. You have also likely looked only at systems you are interested in/have heard of. It is not a representative sample.
Further, since this is just a general impression, rather than a intentional collected figures analysed with statistics, you are vulnerable to a number of subconscious human perception faults like confirmation bias.
People online can only enjoy theoretical Char-Op with concrete numbers
A reason so much of this shows up online is: Theoretical CharOp is Fun. People enjoy optimising characters with no intent to play them. Things like Pun-Pun, The Omniscificer, The Omnicaster, were never intended for play -- they were mental exercises. It is fun to build characters, and play with numbers. Even if you are not doing the extremes of cheese, building say "the best character that can buff my friends without actual magic" is a fun way to spend a afternoon -- even if you never play it. But to do this building, and to have this fun, there needs to be an assumption of non-randomness. This explains alot of cases where class progression is assumed.
Design Change
There is a actual tendency in many RPGs to take randomness out of the character creation/levelling process. For several reasons:
- Rolled stats are fair, when everyone does it, but not balanced.
- The guy who rolls excellently for stats, will just be plain better than others.
- It doesn't let you build the character you want (depending on implementation)
- In some systems, this can be extreme, like the guy who's concept was "burly dumb, but honest warrior", who rolls really well for everything, might end up more a better lier than the conman.
- Sure he doesn't have to use it, but it will be there on his sheet forever. He might feel the need to explain when he got so good at lying -- that is going to change his backstory.
- Some would argue you should build the concept after the stats, but if the player had a concept they are excited about lined up, why not let them play it?
- The play who rolls really poorly, is going to have a bad start to the game.
- It is not fun, it leaves a bad taste of "Everything would be different if I had just rolled better"
- Not a great way to start a game.
- Bad or Good luck on stats affects you more now than at any other point.
- If rolling well on a stat gives you +4 to all rolls made using it, then this is the same as always getting lucky on every roll that uses it.
- Dramatic failing a Roll to climb might make you fall once, dramatic failing to roll that determines your skill level at climbing means you will never manage to climb. (Depending on the system this might be rectifiable, in more or less time.).
Similarly, for class progression. It is about putting the agency in Player's hands.
When it comes to class progression, if the players can let the GM know that "my long term goal for this character is to have him become a dude who is all about becoming friends with ghosts." at the start of the campaign, and the GM thinks this concept is cool, then he knows to put a NPC who can teach the players about ghosts in the campaign.
While this is becoming more common [citation needed], it is by no means ubiquitous across game systems or play groups.