The Sage Advice Compendium Document answers this
In the section on Sorcerer class features and the section on Spell Attacks the following three things are asked and answered (emphasis mine):
Q. When casting a spell that affects multiple targets, such as scorching ray or eldritch blast , do I fire one ray or beam, determine the result, and fire again? Or do I have to choose all the targets before making any attack rolls?
A. Even though the duration of each of these spells is instantaneous, you choose the targets and resolve the attacks consecutively, not all at once. If you want, you can declare all your targets before making any attacks, but you would still roll separately for each attack (and damage, if appropriate).
[...]
Q. Elemental Affinity improves one damage roll of a spell, not multiple rolls? So with scorching ray, I don’t add my Charisma modifier to each ray that hits?
A. That’s correct. Elemental Affinity benefits one damage roll per casting of a spell, even if the spell allows more than one roll. So, for example, the feature improves one of the rays of a scorching ray spell or one of the beams of an eldritch blast spell.
Q. Can Empowered Spell affect all the rays of a scorching ray spell, or just one?
A. A sorcerer’s Empowered Spell could affect more than one ray of a scorching ray, abiding by the feature’s die limit. For instance, if you create three rays with the spell and you have a +3 Charisma modifier, you could reroll one of the damage dice for each ray, or two of the damage dice for one ray and one of the damage dice for another one.
All of these quotes establish that scorching ray involves multiple damage rolls