Continuous effects from the same source doesn't stack, but immediate effects happen and are resolved instantly, so stacking doesn't apply to them.
If the spell's effect is simply that "the subject takes 1d6 fire damage per round", and you cast it twice on someone in consecutive rounds, they won't take any more damage in the second round than they already would to begin with, the same way that "target is afraid for 10 rounds" doesn't stack; they'd just suffer for 11 rounds rather than ten.
If the spell's effect is that "the target takes 1d6 fire damage immediately, and also takes 1d6 fire damage for the next two rounds" (for example), when cast on them a second time they would suffer the immediate 1d6 fire damage when you cast and continue to suffer the 1d6 ongoing damage in the same round. On subsequent rounds they'd suffer the normal non-stacking 1d6 fire damage from the ongoing effect.
The wording of Ray of Flame (assuming the version of the spell I have found from google is the one you're using):
A burning ray shoots out at the target from your upturned palm. The sound of a crackling fire follows the ray’s path. You must succeed on a ranged touch attack with the ray to strike a target. If your attack is successful, the ray deals 1d6 points of fire damage per two caster levels (maximum 5d6). The target must also make a Reflex save or catch fire, taking 1d6 points of fire damage each round until the flames are put out (requiring a DC 15 Reflex save; see Catching on Fire, DMG 303).
Your attack does an instant Xd6 fire damage, so that part will always happen and isn't subject to any rules about stacking. The secondary effect of this spell isn't ongoing magical fire damage, it's that you literally just catch fire, and then suffer the effect of being on fire as per the normal rules for being on fire taken from the DMG:
Characters at risk of catching fire are allowed a DC 15 Reflex save to avoid this fate. If a character’s clothes or hair catch fire, he takes 1d6 points of damage immediately. In each subsequent round, the burning character must make another Reflex saving throw. Failure means he takes another 1d6 points of damage that round. Success means that the fire has gone out. (That is, once he succeeds on his saving throw, he’s no longer on fire.)
The ongoing damage isn't actually a magic effect subject to rules about how spells can stack, it's just a result of being on fire in a totally mundane way. As it happens though, you can't be on fire twice, you're either on fire or you're not; so if you cast the spell at someone who's already on fire, the secondary effect does nothing.
