Both games often fail to make distinctions between lethal damage and other kinds of damage, the games assuming the reader just somehow knows. But, fortunately, lethal damage is the most common kind of damage, so it's safest for the reader to assume that when an effect says it deals damage or adds to damage, the effect means it deals or adds to lethal damage. Anyway, here's a little help.
The Rules Compendium for Dungeons and Dragons 3.5 offers some insight…
First, the Rules Compendium says, "Any spell that requires an attack roll is weaponlike" (132). Then it continues, saying
Weaponlike spells that deal some form of damage—lethal damage, nonlethal damage, ability damage, or ability drain—or that bestow negative levels can threaten critical hits, can be used with precision damage…, and can be used with favored enemy damage bonuses.1 Since a weaponlike spell isn’t actually a weapon, Strength modifiers on damage rolls and magical effects that increase weapon damage don’t increase damage from a weaponlike spell unless the spell’s description says otherwise. (136)
(Emphasis mine.) This means, for example, the 1st-level Sor/Wiz spell chill touch [necro] (PH 209) typically threatens a critical hit on a natural 20 and, if confirmed, deals 2d6 points of damage, and, if the defender fails the Fortitude saving throw, the defender's dealt 2 points of Strength damage. Also, for example, this means a rogue/wizard that successfully hits a typical flat-footed foe with the ray created by the 4th-level Sor/Wiz spell enervation [necro] (PH 226), in addition to bestowing the negative levels, the rogue/wizard deals his sneak attack damage as negative energy damage (see Rules Compendium 136). But even if the mighty wizard has Strength 18, he still only deals 1d6 points of electricity damage per caster level (maximum 5d6) with the 1st-level Sor/Wiz spell shocking grasp [evoc] (PH 279) unless using an unarmed unarmed strike to deliver what's normally a touch attack (PH 141-2).
…But the spell shadow arrow is complicated
Then there's the weird corner case of the 4th-level ranger spell shadow arrow [necro] (Champions of Ruin 34). The spell shadow arrow, on its surface, doesn't seem like a weaponlike spell. Its target entry is a masterwork arrow or a masterwork bolt, for instance, not a creature, but the spell's description does say, in part, that
Make a ranged touch attack with the missile instead of a normal ranged attack. Instead of dealing normal damage, a shadow arrow deals 1d6 points of Strength damage. The arrow or bolt must be fired during the same round the spell is cast, or the magic dissipates and is lost. The projectile is destroyed even if it misses its target. (Champions of Ruin 34)
(Emphasis mine.) This DM would rule that since there's no other way to use the spell's effect except by making with it an attack roll, the spell is, in fact, a weaponlike spell and follows the rules above, including no Strength modifiers on the (ability) damage roll and, for example, no increase in the (ability) damage from magical effects due to the weapon firing it nor spells cast on it.
The alternative to this is, perhaps, a reading like It creates a magic arrow that's fired and that means the spell itself is not weaponlike; the arrow's just ammunition fired from a weapon and 'Instead of dealing normal damage' means instead of the arrow dealing normal arrow damage not instead of the attack dealing normal damage! If the DM agrees then doesn't immediately ban the spell, the DM may allow any damage modifiers that would apply to a normal arrow's damage to apply to the shadow arrow's ability damage. Doing so would trivialize many encounters and probably relegate the spell to the too-dangerous-to-use pile (because if the PCs use it, the antagonists will, too).
(An example: A level 9 mystic ranger on round 1 casts the 1st-level ranger spell hunter's mercy [div] (SpC 117) followed by the 1st-level ranger spell hunter's eye [div] (PH2 114-5) modified to a 2nd-level spell by the feat Extend Spell, and, on round 2, casts shadow arrow. A successful ranged touch attack with that one shadow arrow deals an impressive 5d6 points of Strength damage. There's no saving throw, and Strength 0 means the creature's helpless.)
By the way, monsters don't usually add ability score modifiers to ability damage they deal
Most PCs don't get the opportunity to deal ability damage except as a consequence of other attacks (like the aforementioned chill touch or by employing poison), so, I guess, broadly, this whole issue can be argued, but a quick search of the Monster Manual creatures that deal ability damage or ability drain directly as an attack—like the allip's and the lamia's wisdom drains, the green hag's and the roper's weakness, and the shadow's strength damage—none of whom get any of their ability modifiers added to the ability damage or ability drain they inflict—leads me to believe spells and other effects that grant a creature the ability to deal ability damage or ability drain also shouldn't get an ability modifier to that damage unless they say so explicitly, and the spell shadow arrow doesn't say that.
In Pathfinder this is less clear
As of this writing Pathfinder has made no official pronouncements on the interactions between unusual kinds damage like ability damage, ability drain, and the bestowing of negative levels. Probably the closest a reader can get is this exchange from the Pathfinder Core Rulebook FAQ:
Question
Can you take Weapon Specialization (ray) or Improved Critical (ray) as feats? How about Weapon Specialization (bomb) or Improved Critical (bomb)?
Answer
All four of those are valid choices.
Note that Weapon Specialization (ray) only adds to hit point damage caused by a ray attack that would normally deal hit point damage; it doesn't increase ability score damage or drain (such as the Dexterity drain from polar ray), penalties to ability scores (such as from ray of enfeeblement) or drain, negative levels (such as from enervation), or other damage or penalties from rays.
This could imply that Pathfinder limits damage bonuses to effects dealing lethal damage… except that the D&D 3.5 feat Ranged Spell Specialization (Complete Arcane 82) also limits itself to affecting spells dealing lethal damage and that this FAQ entry uses the same odd phrasing as that feat (hit point damage rather than lethal damage or even just damage), evidence of, perhaps, the FAQ having borrowed rules from Complete Arcane. So before accepting that this FAQ entry Pathfinder limits damage bonuses in such a way, keep in mind that Complete Arcane clarified for 3.5 the idea that, for example, a rogue can deal sneak attack damage with the spell enervation!
Really, since the 3.5 rules are clear enough and the Pathfinder rules aren't, I suggest using the 3.5 rules until Pathfinder decides to rule formally on the subject.
(For further comparison, you can check out the Pathfinder versions of the
allip, green hag, lamia, roper, and shadow. None of them add ability scores to their ability damage or ability drain attacks either.)
1 The inclusion here of favored enemy damage is possibly from the Rules Compendium believing that the favored enemy damage bonus is extra precision damage (as it was prior to the 3.5 revision) rather than a flat damage bonus. (Also see this question.) This is the Rules Compendium's sole mention of favored enemy.