You’ve pretty much summed it up. As a Rogue, you should max Use Magic Device (it’s the best skill in the game), and thus you could be a kind of mini-caster of your own by using wands, but damage-dealing wands are usually a bad choice so doing that probably wouldn’t have helped you. Ultimately, Swarms are a weird case where you need things that aren’t usually very good. If you expect to be fighting a lot of swarms (as in several per day because the campaign revolves around them), there may be options for you, but if they’re only coming up occasionally then those options are too niche and costly to be worth it.
The long and short of it is, you’ve just learned (at least a small portion of) why spellcasters own 3.x. There are so many different types of monsters, and so many different sorts of defenses, that you need to be incredibly flexible and versatile to handle them all. Spellcasters are; mundanes are not.
It’s not just swarms; they’re just a great example. Swarms, incorporeal or ethereal creatures, creatures with lots of AC, DR, and/or HP, creatures with ways to become undetectable, creatures with ways to prevent you from moving, and so on: every one of them needs a different answer. A spellcaster, thanks to the fact that he can know and prepare many spells of each level, can actually start the day with all different kinds of answers. Got a problem? There’s a spell for that.
Mundanes, on the other hand, typically get class features and feats that cannot be changed once gotten. They also get them in much lower quantity than spellcasters get spells. They’re unlimited use, but as you’ve seen, that doesn’t do you very much good when you cannot use them. By level 12, spellcasters have a lot of spells: it’s a rare day for a smart spellcaster of that level where they go to sleep without quite a few left over. Mundanes can be good for encounters that play to their strengths (typically much higher damage than spellcasters), but in any sort of unusual case, like weird defenses or difficult terrain or whatever, they can be useless.
Unfortunately, this is just a reality of 3.x. It’s reflected in JaronK’s Tier List: the highest-tier classes include Cleric, Druid, and Wizard; the lowest tier classes (aside from NPC classes) include Monk and Paladin. The list is for 3.5, but while Pathfinder helped, for example, the Paladin quite a bit, it did not do nearly enough to eliminate this disparity. Ultimately, Pathfinder is a series of small changes here and there, whose net effect is very little change in the overall balance or feel of the system.
I’d direct you to Why each class is in its tier for more information. What it ultimately comes down to is how flexible you are: how often you have the perfect answer, and how often you have no answer at all.
Given a day’s preparation, a Cleric, Druid, or Wizard can always have the perfect answer, and even caught unawares they almost never have no answer: that’s tier 1.
A Rogue can have a good answer, though by no means a “perfect” answer, to some things, but what those things are cannot be changed once chosen and cannot possibly cover all of the different situations you could get into. That’s Tier 4.
I really wish I could tell you more. But this is just kind of a reality of the 3.x system that is hard to get around. You kind of just have to accept, as a mundane, that you’re not going to keep up very well if the spellcasters are smart.