The typical creature doesn't know (and usually can't determine) another creature's Hit Dice
A few rare special abilities can determine a creature's hit points, but determining a creature's Hit Dice is rarer still, usually coming down to just guesswork. (Perhaps a generous GM will allow a creature's typical Hit Dice to be learned as a hint with a successful Knowledge skill check to gain some monster lore about the creature, but that's subject to the GM's whims.)
To summarize, Magic on Aiming a Spell says, "Some spells have a target or targets. You cast these spells on creatures or objects, as defined by the spell itself." However, on Spell Failure says, "If you ever try to cast a spell in conditions where the characteristics of the spell cannot be made to conform, the casting fails and the spell is wasted." Casting a spell on an invalid target is one of these cases that wastes an otherwise perfectly good spell.
Further, according to Saving Throws on Succeeding on a Saving Throw, a caster knows when "a creature’s saving throw succeeds against a targeted spell[, and, because of the target's successful saving throw, the caster knows] that the spell has failed." But when the target is simply—for whatever reason—flat-out immune to the spell, the caster receives no feedback from the spell.
So when a caster that casts a spell on an invalid target—like, for example, a caster that casts the spell cause fear on a 6 Hit Dice foe—, the caster won't know that the spell failed nor will the caster know why the spell doesn't appear to be having the effect it's supposed to have.