After this I plan to go druid until I get a few shapeshiftings/day.
Don't.
Multi-classing is a fine thread to walk on. Unless there are specific synergies to exploit (Rogue/Swashbuckler work quite well together for example), by multi-classing you end with a Jack of all Trades, Master of None.
For example, let's suppose that you actually multi-class R3/S3/D5 as proposed. In 5 (long) levels, you will be ECL 11:
you will have Wildshape 1/day, and be able to Wildshape into a Small or Medium animal of at most 5 HD, say a Dire Hawk (MMII, p. 76); a CR 11 Forest Sloth (MMII, p. 106) has 147 HP, by comparison, and its 2 Claws (2d4+7) + Bite (2d8+3) deal an average of 31 points of damage...
you will have 3rd level Druid spells; by comparison a 11th level Druid will have 6th level Druid spells. You get Speak with Plants, the 11th level Druid gets Transport via Plants.
you will make +2d6 (Sneak Attack) and +3 (Insightful Strike) precision damage (average +10) when you catch your opponent unaware, or flank it; by comparison a Rogue 8/Swashbuckler 3 would deal +2d6 more points of damage (average +7), on top of having Improved Uncanny Dodge.
If your only plan to incorporate the Druid class in your build is only to get some weak spells and wildshape, I urge you to reconsider.
TL;DR: Check prestige classes that you find cool or like, check they can really do what you wish for (some are traps), and build your character toward them.
If you need some utility spells (who doesn't?), then invest more thoroughly in Use Magic Device. You have the skill in-class (as a Rogue) and 11 skill points per level given your Intelligence, by next level it could maxed out (at 10 ranks), netting you +12. Activating a wand has a fixed DC of 20, pump up Use Magic Device until you get +19 (rolling 1 is not an automatic failure on skills checks), and you'll be able to use any wand: Wizard's, Cleric's, Druid's, ...
If you worry about escape routes, Anklet of Translocation are indeed a cheap and very useful item for now. You can have several for when you run out of charges. Spider Climb is only 2nd level spell (a full 50 charges wand costs 4,500 gp), and few people think 3D when securing houses; much cheaper than flying.
Which classes to go from there?
You might consider a dip into Swordsage (Tome of Battle), or otherwise use a feat/item to get access to the useful Shadow Jaunt maneuver (Shadow Hand 2, teleport 50 ft. through shadows as a Standard Action). The Swordsage normally relies on Wisdom (bad fit for your character), but only a couple maneuvers really require it so you can just avoid those.
Similarly, a dip into Cloistered Cleric (just one level) would net you 2 Domains (on top of the Knowledge Domain) and 3+Cha Turn/Rebuke Undead attempts per day on top of the ability to use Cleric Wands without a Use Magic Device check. Now the Domains and Turn/Rebuke Undead attempts are not very useful in and out of themselves to you... but:
- Domains sometimes come with passive abilities, such as free feats
- Domains can be converted into Devotion feats (Complete Champion), and notably the Knowledge Domain can be converted into the Knowledge Devotion feat: +1 to +5 Insight on ALL attack rolls and ALL damage rolls, based on a Knowledge check (+5 requires a 36+...) as well as adding one Knowledge skill to your in-class skills (forever)
- Turn/Rebuke Undead attempts can be used to power feats, for example Travel Devotion is 1/day, but you can exchange 2 Turn/Rebuke Undead attempts against another use... as long as have remaining attempts
For example, you could dip Cloistered Cleric with the Knowledge, Travel and Trickery domains, and convert all those into Knowledge/Travel/Trickery Devotion (the latter allow, 1/day, creating a simulacrum of yourself, which gets more and more real as your ECL rises).
Also, check out Complete Scoundrel: the skill tricks inside are "mini-feats" that you can buy using skill points (which you have aplenty). A trick is usable only once per encounter (in general) and you can only have ECL/2 tricks, but picking a couple is quite handy.
But... personally, I tend to build my characters backward: start from a Prestige class I like, and check how to get the requirements to attain it.
For example, in the infiltrator theme, have you heard of the Nightsong Infiltrator class (Complete Adventurer, p. 62)? Not only do YOU get better at infiltrating, but you also grant bonuses to your teammates (Teamwork Infiltration (Ex), study "obstacle" and grant +2 to allies to bypass it).
Or maybe you'd rather be a Spymaster (Complete Adventurer, p. 77) or a Thief-Acrobat (Complete Adventurer, p. 83)?
Unfortunately, none of those are exactly potent in combat situations; I fear. Infiltrators generally are not.
Tome of Battle has two potential Prestige classes that you might be interested in:
Jade Phoenix Mage (dip 1 level in Swordsage and 1 level in Wizard + take the Precocious Apprentice feat to get access) is a fighter/wizard gish; only 1 dead spellcasting level, Full BAB, and a spattering of maneuvers and stances.
Shadow Sun Ninja (requires Good alignment) (dip 1 level in Swordsage to get access; the unarmed variant) is a monk on steroids; it gains extra Shadow Hand and Setting Sun maneuvers as well as abilities thematically based on Light/Darkness.
Initiators are multi-class friendly because your 6 levels of Rogue/Swashbuckler count for 3 Initiator levels (half your levels), so that when you dip Swordsage 1, you count as being Initiator level 4 already (having immediate access to 2nd level maneuvers), and by the end of the JPM or Shadow Sun Ninja (10 levels each), you'll be Initiator 14 (having access to 7th level maneuvers).
I would personally recommend the Jade Phoenix Mage here. Whilst not specifically an infiltrator, at level 18, you'll have access to 5th level spells "for free". There are many spells in the Sorcerer/Wizard list, and you can pick many to help with infiltration (Divination and Abjuration to counter/dispel magical defenses, Transmutation to emulate Wildshape, ...); the Wizard synergizes well with your high Intelligence, and you can have as many spells as you wish in your spellbook.
And yes, you get those "for free" because you will also have a BAB of +15, 11 maneuvers and 3 stances. Plus the capstone is a very cool ability where 1/week you explode (20d6 damage in 20 ft. radius) and 1d6 rounds later reform, dazed but fully healed of all damages :)