Typically, the players would be expected to make any rolls for their own character. That includes damage rolls.
The Player's Handbook is generally written from the perspective of the 'agent' responsible for the character or creature in question.
For example, under Attack Rolls (pg 194) it says:
When you make an attack, your attack roll determines whether the
attack hits or misses. To make an attack roll, roll a d20 and add the
"You", in this case, is whoever is controlling the character that is attacking. And then, under Damage Rolls (pg 196) it says:
Each weapon, spell, and harmful monster ability specifies the damage
it deals. You roll the damage die or dice, and add any modifiers,
and apply the damage to your target.
It's clear that the "you" or "your" for both the attack and damage is referring to the same person. And for a PC, that person is going to be the person playing that character.
If it's a monster or NPC attacking, then "you" refers to whoever is dictating the actions of that monster or NPC - typically the DM.
In the Dungeon Master's guide there is also a chapter on Running the Game. It even says here, under "Rolling Attacks and Damage" (pg 235 of the DMG):
Players are accustomed to rolling an attack first and then a damage
roll. If the players make attack rolls and damage rolls at the same
time, the action moves a little faster round the table.
This pretty much spells it out that players are expected to make their own attack rolls and damage rolls. It even suggests that the players make the two rolls at the same time to speed things up (the very reason you stated that you rolled the damage).
Outside of damage rolls, there is a situation where a DM may make a roll for the player. This is actually spelled out just above the "Rolling Attacks and Damage" section. And this is if you, the DM, don't want the player to know how good their check is (the example given is if a player wants to make an Wisdom(Insight) check to see if the baroness is charmed; if the player rolled this, the dice result would give away a clue - so the DM may roll in secret so that the player doesn't know whether a negative result is due to a low dice roll or because the baroness isn't charmed!).
But the above is an exception - done mainly to maintain an air of uncertainty.
Now...it's your game and you're the DM. And if you still want to roll the damage dice then that is up to you. But hopefully its clear that the rules do assume that it is the players that typically do this. And personally I think if you take too much away from your players, it might make them feel like you're virtually playing the game on your own!