I'm going to go out on a limb and say that there aren't any general rules that govern the visibility of magical effects in the Wizards of the Coast-published Dungeons & Dragons, Third Edition corpus. That is, the game's general silence on this issue is the general rule. A magical effect is only as visible as the specific magical effect's description says it is… or only as visible as the DM rules it is.
To be clear, the Player's Handbook covers casting a spell and that covers a spell's components but each spell's description is a specific, unique, new magical effect. To put general rules on how they all appear defeats the purpose of a magic system that's as confusing, dangerous, powerful, and—above all—vast as the magic system that was developed initially in the late 1970s and from which a direct line can be traced to Dungeons & Dragons 3.5.
In these respects, extraordinary, spell-like, and supernatural abilities are no different from spells: such abilities are diverse and legion, and limiting them by saying all of them follow a general rule for their appearance, whether that's initially when they're activated or for their durations if they're ongoing, would—in this reader's opinion—detract significantly from the game.
In fact, this reader imagines even an extremely conservative general rule about the appearance of magical effects would have an intense effect on actual play. Consider a hypothetical rule like When a creature activates a spell-like or supernatural ability, during that ability's activation time the creature is surrounded by floating, glowing translucent mystical symbols. Once such a rule is enacted, monsters that relied on subtlety can't anymore: every doppelganger that reads a mind is now obviously doing something, as is every druid in squirrel form that activates wild shape again. In short, this reader would view the existence of such a rule as making the game a sadder, less mysterious place.
Addressing the examples
The supernatural ability blinding beauty of the Dungeons & Dragons 3.5 nymph says
This ability affects all humanoids within 30 feet of a nymph. Those who look directly at a nymph must succeed on a DC 17 Fortitude save or be blinded permanently as though by the blindness spell. A nymph can suppress or resume this ability as a free action. The save DC is Charisma-based. (Monster Manual 198)
There's no indication in this description or in the nymph's earlier description that it's obvious when a nymph has her blinding beauty ability activated or deactivated. No mention's made of eye beams or corona around her noggin. The hapless adventurer is left to wonder. An intrepid DM may want to refer to the 2nd-level Sor/Wiz spell blindness/deafness [necro] (PH 206) for more information on how this effect appears even though the blinding beauty ability only says it works like the blindness/deafness spell, but here's entire description of that spell: "You call upon the powers of unlife to render the subject blinded or deafened, as you choose." The DM could rule that calling upon the powers of unlife sends phantom images of skeletons flying from the caster—and, by extension, the nymph—, but that would be an embellishment and that DM's house rule (a term I never use pejoratively, by the way).
The supernatural ability unearthly grace has even less going for it, saying, "A nymph adds her Charisma modifier as a bonus on all her saving throws, and as a deflection bonus to her Armor Class" (MM 198). The unearthly grace ability is on all the time unless the nymph enters an area of antimagic, and there's no indication that a nymph who is in such an area actually looks less graceful or that the subtle haze that softens her sharp features disappears or something.
The same goes for the supernatural ability fear aura of a creature like a lich (MM 166–8) or the tarrasque (240–1) and the supernatural ability tongues that's possessed by creature that possess the subtype angel (11). These creatures don't have radiating from them visible heat-like waves of fear nor do they have floating about them yet in constant motion a loops of glowing alphabets. Instead, you get near and you just fear or hear… the angels talking to you in a language you can understand, respectively.
Addressing other issues
A spell has a visible effect if the spell says it has a visible effect. Spells like the 1st-level Sor/Wiz spell ray of enfeeblement [evoc] (PH 269) and the 3rd-level Sor/Wiz spell fireball [evoc] (PH 231) in their descriptions detail their visible effects. However, only the target is aware of the effect of the 4th-level Sor/Wiz spell phantasmal killer [illus] (PH 260) and only a skilled onlooker could determine that a caster dropped the 1st-level Sor/Wiz charm person [ench] (PH 209) on a city guard, and even then only as the spell is being cast and if that onlooker could see the caster perform the spell's somatic component or could hear the spell's verbal component… and a lot of spells are like that charm person spell—the magic that's created after they're successfully cast has no visible effect.
The Spellcraft skill touches on this briefly by providing these skill uses:
- When casting detect magic, determine the school of magic involved in the aura of a single item or creature you can see. (If the aura is not a spell effect, the DC is 15 + one-half caster level.) No action required.…
- Identify a spell that’s already in place and in effect. You must be able to see or detect the effects of the spell. No action required. No retry. (82)
(The DC for these are 15 + spell level and 20 + spell level, respectively.) Further, the 0-level Sor/Wiz spell detect magic [div] (PH 219) has as the first sentence of its description, "You detect magical auras," strongly implying that normally magical auras are undetectable. That second skill use of the Spellcraft skill though? That's kind of a big deal: by implication, unless the spell says it has an effect that can be seen or otherwise detected, the spell can't be identified, presumably because it can't be perceived at all!
Most magic items don't glow. The Dungeon Master's Guide on Light Generation, in part, says, "Fully 30% of magic weapons shed light equivalent to a light spell…. These glowing weapons are quite obviously magical" (221). However, other magic items don't possess this property. They're case-by-case. A magic item may glow depending upon its description or it may look completely ordinary, either only when not in use or all the time! The mundane way to know if, for example, a bag of holding (DMG 248) (15+ lbs.; 2,500+ gp) is really a bag of holding or just a normal sack? Start filling it up! The typical bag—like most magic items—just doesn't conveniently signal its magical nature by glowing.
(N.b. Sometimes magic items are also high-quality mundane items, but many need not be, so that's not a surefire indicator that an item may be a magic item. Also, the DMG on Random Magic Items explains that about 1/3 of magic items include on them a clue as to how to use them (216). If such a clue is found on a what's thought to be a mundane item, chances are the item's actually a magic item unless the DM's campaign includes extremely clever, cruel, or deceptive craftsmen.)
The existence of the Invisible Spell feat does not mean that all spell effects are visible by default. (N.b. I'm aware that the question doesn't actually say that they are, but bear with me.) First, there aren't more than a handful of feats that are more controversial and ill-conceived than the metamagic feat Invisible Spell (Cityscape 61). Seriously, someone is spoiling for a fight if he claims to know exactly what happens when a wizard casts the 2nd-level Sor/Wiz spell darkness [evoc] (PH 216) or invisibility [illus] (PH 245) when either is modified by the feat Invisible Spell. The feat's effect is unknowable in isolation and mandates DM intervention whenever it's used in a way that's beyond its most basic and obvious uses. That this feat got though editorial in its present form is evidence that Wizards of the Coast still thought of casters as blasters even as late as Cityscape (2006), the company's own vocal messageboards quite to the contrary.
With that out of the way, yes, resources must be devoted to making magical effects imperceptible if the magical effects are perceptible. Unfortunately, the feat Invisible Spell is a terrible game element to use to determine how much value to place on making magical effects imperceptible. No effort was made to make the feat's impact comprehensible outside its extraordinarily narrow vision. Really, the feat offers but one example:
For example, a fireball cast by someone with this feat could be made invisible in the moment of its detonation, but everyone in the area would still feel the full effect (including the heat), and any flammable materials ignited by the explosion would still burn visibly with nonmagical fire.
You, I, or anyone else who'd read the rest of the feat's benefit could've written that example, and it tells the reader nothing about what happens if the feat's applied to the 1st-level Sor/Wiz spell summon monster I [conj] (PH 285–6, 287) et al., the 3rd-level Sor/Wiz spell fly [trans] (PH 232), the 5th-level Sor/Wiz spell wall of stone [conj] (PH 299–300), or the vast majority of other spells.
Because there's no other metamagic feat like it, this reader urges not using the feat Invisible Spell as an example or indicator of, like, anything. If the DM needs to establish the value of making a magical effect imperceptible, the DM should consider the impact of the imperceptibility on the campaign and price it accordingly himself.
Making house rules
A DM can make house rules that mandate magical effects by default have visible effects. That's a fairly significant change, though, and this player—like your player did—would be surprised were such a change made suddenly. It's not that this is a bad change per se, but it is a change that has far-reaching implications, sending ripples throughout the game. It's a change that should be considered thoughtfully and the campaign should change to reflect it. (If anything, I'd suspect such a campaign to have actual monsters be rarer because they'd be easier to detect!)
"What about Pathfinder?"
Almost all of the above applies equally to Pathfinder except that with this 2015 FAQ entry Pathfinder already takes a step in the direction of making magic effects visible by ruling that casting a spell has always been an obviously magical act:
What exactly do I identify when I’m using Spellcraft to identify a spell?…
Although this isn’t directly stated in the Core Rulebook, many elements of the game system work assuming that all spells have their own manifestations, regardless of whether or not they also produce an obvious visual effect, like fireball. You can see some examples to give you ideas of how to describe a spell’s manifestation in various pieces of art from Pathfinder products, but ultimately, the choice is up to your group, or perhaps even to the aesthetics of an individual spellcaster, to decide the exact details. Whatever the case, these manifestations are obviously magic of some kind, even to the uninitiated; this prevents spellcasters that use spell-like abilities, psychic magic, and the like from running completely amok against non-spellcasters in a non-combat situation.…
What this means is that when a piece of Pathfinder art depicts a caster surrounded by a nimbus of energy or juggling glowing arcane symbols or whatever, that isn't a metaphor for casting a spell or what the caster sees when casting a spell but actually what everyone sees when that caster casts that spell! And, according to that FAQ entry's author, the game's designers have always understood that this is how the game works, despite Pathfinder taking its core from Dungeons & Dragons 3.5 that has, as I said, so far as I can tell, no rule like this at all.
