Can a swarm gain class levels, specifically a hellwasp swarm as it has an intelligence score of 6?
Probably not, though the DM can always overrule this
Let's examine some detials of the hellwasp swarm entry:
Advancement: none - This is the most direct way of saying that this creature doesn't progress by gaining class levels (it would have said "By character class" if that was the intention). As a swarm, it doesn't even progress by HD (you just add more swarms for a tougher encounter).
Type: Magical Beast (Extraplanar, Evil, Swarm) - While not set in stone, magical beasts typically advance by HD, not by character class.
Special Attacks: [...] Inhabit and Special Qualities: Hivemind - these, combined, might at first seem like a way to introduce a hellwasp swarm with class levels, but let's take a closer look:
A hellwasp swarm can enter the body of a helpless or dead creature by crawling into its mouth and other orifices. Inhabiting requires 1 minute, and the victim must be Small, Medium, or Large (although four swarms working together can inhabit a Huge creature).
If a hellwasp swarm inhabits a dead body, it can restore animation to the creature and control its movements, effectively transforming it into a zombie of the appropriate size for as long as the swarm remains inside. If a hellwasp swarm inhabits a living victim, it can neutralize the effects of its own poison and control the victim’s movement and actions as if using dominate monster on the victim. The hellwasps quickly consume a living victim, dealing 2d4 points of Constitution damage per hour they inhabit a body. A body reduced to Constitution 0 is dead.
Hive Mind (Ex)
Any hellwasp swarm with at least 1 hit point per Hit Die (or 12 hit points, for a standard hellwasp swarm) forms a hive mind, giving it an Intelligence of 6. When a hellwasp swarm is reduced below this hit point threshold, it becomes mindless.
(All emphasis mine)
So, whether inhabiting a living or a dead creature, the swarm uses it as a basic 'meat puppet' - it can direct its basic actions, but not access its knowledge or skills (remember that without a common language, which is the case here, Dominate Monster allows "only basic commands, such as “Come here,” “Go there,” “Fight,” and “Stand still.”"). The swarm doesn't gain full control of the host creature as would be the case with Magic Jar.
While not inhabiting a creature, the swarm is extremely different from a typical humanoid - it can't really manipulate objects, and won't be able to cast any spells with verbal or somatic components (even material components will be very tricky), it is not even able to speak.
In both cases, these are far from ideal conditions to learn and acquire any PC or NPC class.
Should you as a DM (or your DM) overrule this?
As a PC - no.
Saying this creature is extremely unsuitable to play as a PC is an understatement - unless the campaign is very a-typical and the DM takes special steps for making it compatible to a PC with these strong and uncommon abilities on one hand and crippling limitations on the other, this is likely to be an exercise in frustration to the hellwasp player, the DM and the rest of the group.
As a NPC - maybe, if you really like the concept
This will require some hand-waving and house-ruling (and ignoring the points mentioned above...), and should definitely raise this creature's CR in more than just the class levels it'll gain, but, if you like the idea - a NPC hellwasp swarm can make a cool and memorable minion or even a BBG, but only with some concessions and modifications (allowing it to 'speak' somehow, even if only when possessing a body. Giving it the capability to cast spells, etc.). A sorcerer, cleric, druid or bard seem most suitable, though a rogue, barbarian, ranger or assassin may also work (and, as nijineko noted, the psionics classes complement this concept nicely if you use the rules for them...).
This can provide a nasty 'ace up the sleeve' surprise for a party fighting it, an explanation for a mysteriously unkillable recurring villan or an unsettling resolution to a "the [insert important and previously benevolent role] is acting strange lately" plot-line...
The online SRD mentions that
Intelligent creatures that are reasonably humanoid in shape most commonly advance by adding class levels.
In addition, here it is said that
Creatures who have an Intelligence score of 2 or lower, who have no way to communicate, or who are so different from other PCs that they disrupt the campaign should not be used.
Keeping those points in mind, you might want to avoid making swarms with class levels.
On the other hand, both are merely suggestions and the latter only applies to PCs. There are no actual rules that would preclude a hellwasp swarm from gaining class levels, since it's treated as a single creature. It'd definitely make for an interesting enemy.
Assuming you are the DM or you have the DM fiat.
Surely not a common view but monster can advance in character classes and nothing seams to prevent a sentient swarm to do so.
As @Urist McDorf already pointed out in his answer those are general guide line, and let me tell you why.
Wotc* already did it!
Here and here you can find example of creature clearly not humanoid yet with level in character classes. Dragons take level all the time, they even have dragon's only prestige class.
The point is that classes aren't job, classes are collection of skills, more or less, connected together.
Of my two example before let's take the Skiurid one. They are squirrel like creature which already use hit and run strategy. By adding them level in ninja and thief you make the oldest and more expirienced of them pop-up as an Alpha-Skiurid. That is thematically fitting and build on the existing creature strategy.
Player do this all the time too. Not all rogue are rogue, some are thief, some thug, other expert in dealing with trap and other agile and onorable swashbuckler. And not all samurai have level as samurai.
back on your hellwasp swarm
If immersion and consistency are priority (not always are) in your game not all classes may be good for your swarm.
For example i can easily see a dip in barabarian : "your last blow dropped what seams to be the queen of the swarm, now all the remaining hellwasp are frenzied and aiming for you!".
On the other hand a wizard wasp swarm, for example, may looks silly at first glance, but maybe with the right story behind could work too.
Classes are collection of rules and how this rules are applied in the game world is , often, pure fluff that you can change as need arise.
* I know that some of the article are First party material, I'm unsure if it's true for all of them, but I'm preatty confident that are at least considered Second party.