Monsters which advance by HD can also take class levels if they're intelligent.
The rules on which monsters can advance appear in the D&D 3.5 Monster Manual p.290, "Chapter 4: Improving Monsters". It's actually somewhat ambiguous and noodly.
Creatures can, in general, advance by HD, class levels, or templates. These are not mutually exclusive, and p.290 gives the example of a creature doing all three (a lammasu with the half-dragon template, increasd hit dice, and character levels).
It then states that intelligent, roughly humanoid creatures "most commonly" advance by class level, and will have "By character class" in their advancement line if they fall into this category. All other creatures (non-intelligent, non-humanoidlike, or both) "can" advance by hit dice.
Right away, we have a seeming contradiction. The lammasu isn't humanoid in form, so how can it advance by class level? How can any creature advance by both?
The answer appears to be that even creatures with without "Advancement: By character class" can advance by character level. Page 291 makes this interesting note:
Generally, only monsters with an Advancement entry of "By character class" receive NPC gear; other creatures adding character levels should be treated as monsters of the appropriate CR and assigned treasure, not equipment.
This implies that characters with only HD advancement can be given character levels. This validates things like the aboleth mage, the hypothetical lammasu, the dragons with class levels in Draconomicon, and a hound archon with a class other than paladin. Draconomicon, p.86 also notes that dragons "rarely" adopt character classses, but describes that they occasionally do (despite advancing by hit dice in their description).
Can any creature advance by hit dice? The rules say non-humanoid or non-intelligent creatures "can" do this, but the Hound Archon, advancing by HD, seems to be an example of a creature who explicitly can despite being humanoidlike and intelligent; the troll, meanwhile, also intelligent and humanoid advances by class.
In terms of the actual Humanoid type, I can't think of any example where a human has advanced by raw Humanoid hit dice. Monster Manual p.310 notes that 1HD humanoids always exchange their first HD for that of a character class, and that only humanoids with more than one HD actually use the humanoid tpye; this suggests that actual humanoid-types can't or don't advance by HD.
More specific rules on this appear in Savage Species, although it may be argued that the Monster Manual is the primary source. However, this serves as a useful rule where the Monster Manual is too ambigious. According to Savage Species chapter 9, "Advancing a Monster", monsters must have a minimum of Intelligence 3 to gain class levels. Some monsters advance only by character class.
Conclusion
If we take the advice in both the Monster Manual and Savage Species for advancing a monster, the following applies:
- Advancing by HD only: Applies to any creature with an Intelligence score below 3, or no Intelligence score. (SS 106) No zombie fighters or cat clerics.
- Advancing by character level only: Applies to some monsters (SS 105). Those who advance "By character class", which means they are intelligent and reasonably humanoid in shape, most commonly advance by this method. (MM 290)
- Advancing by both: Creatures which normally advance by HD, but which are intelligent. (MM 291) Numerous precedents exist, including the aboleth mage, the dragon prestige classes of Draconomicon, and the gargoyle rogues in Elder Evils p.152.
Monsters which normally advance by class rarely advance by hit dice, but at least one example is known: the 6HD advanced ogres in Expedition to Castle Ravenloft p.78. It appears that this is not necessarily forbidden, but it raises the problem that there's no listed hit dice limit or size cutoff, meaning that the rules aren't properly prepared for this eventuality.
