There's no specific rule to this effect. The idea likely comes from the game's rules and assumptions about PCs and how class levels are gained.
As far as I'm aware, there's no specific rule that says anything like "A creature must have an Intelligence score of at least 3 in order to take class levels." Instead, this is a de facto rule that emerges from two pieces of the game's rules/underlying assumptions:
1. A creature with an Intelligence score less than 3 is not a playable character.
From the SRD:
The separate table for Intelligence ensures that no PC ends up with an Intelligence score lower than 3. This is important, because creatures with an Intelligence score lower than 3 are not playable characters. Creatures with any ability score lower than 1 are also not playable.
2. The game mostly assumes that only PCs take class levels by leveling up from experience.
The rules for gaining experience and leveling up (mostly from PHB, p. 58) are all written as something that "you" or "your character" does. For instance, here's what it says about advancing a level:
Advancing a Level: When your character’s XP total reaches at least the minimum XP needed for a new character level...he or she “goes up a level.” For example, when Tordek obtains 1,000 or more XP, he becomes a 2nd-level character. As soon as he accumulates a total of 3,000 XP or higher (2,000 more than he had when he gained 2nd level), he reaches 3rd level. Going up a level provides the character with several immediate benefits (see below).
Similarly, the DMG (p. 260) says:
Experience points (XP) fuel level advancement for player characters...
Even cases where a player controls a creature of animal-level Intelligence, such as a Druid's Animal Companion, don't use the XP + class levels system for advancement; instead, they have individual rules for advancement described by whatever class feature/feat/etc. granted the player control of the animal in question. The closest thing I'm aware of to an exception is cohorts (such as those from the Leadership feat, h/t Carcer), who use part of the XP system for determining your cohort's level (they don't gain XP from encounters normally, but they do use a portion of the PC's XP total to determine their level, and gain class levels like PCs do).
In short, most of the rules in the game for tracking experience points and gaining levels from XP are written to apply to player characters. Though I'm not aware of any definitive statement to this effect, the general assumption of the game rules seems to be that most NPCs and monsters will just be created by the DM with whatever stats they feel are appropriate to the encounter, rather than having their XP tracked and advancing when they acquire enough experience.
Combine 1 and 2, and it means creatures with less than 3 Intelligence never advance by experience-based level progression.
Since creatures with animal-level Intelligence scores can't be PCs, and in general only PCs level up, you end up with a de facto rule that animals never level up. Thus, a player-controlled creature of animal-level Intelligence typically has no way of ever gaining class levels.
That doesn't mean that the DM is never allowed to give class levels to creatures of animal-level Intelligence. If you, as the DM, feel that a vicious bear which has been corrupted by influence from the Shadow Plane is best represented by tacking a couple of levels in Swordsage onto the default brown bear statblock, that's your prerogative. But this isn't some special exception to a rule about class levels and Intelligence score - it's just an instance of the general rule that DMs can build monster statblocks however they want, and aren't subject to the restrictions that the rules place on players when building PCs.