Yes, the caster would be affected
I've thought about this some more, and I think the answer lies not in the description of the creatures' various gaze attacks, but in the text of scrying itself. From PHB, under the description of the Scrying spell:
On a failed save, the spell creates an invisible sensor within 10 feet
of the target. You can see and hear through the sensor as if you
were there. The sensor moves with the target, remaining within 10
feet of it for the duration. A creature that can see invisible objects
sees the sensor as a luminous orb about the size of your fist.
The line "as if you were there" would seem to answer any questions regarding range- for purposes of sensory effects, it is as though the viewer were present. That means the scryer can be affected by sight and sonic based attacks as long as the scrying sensor is within range, regardless of the actual distance to the caster. This makes intuitive sense as well as matching RAW (and also makes scrying a little riskier, which is awesome)
This is both good and bad for the scryer
Note that, while this does put the scryer at risk of becoming charmed, confused, frightened or turned to stone by a target they scry on, it also resolves any confusion about special senses and range, so a scryer with Truesight or Darkvision could use those senses, even though physically they are far out of range.