Reason #1: Because that's what the feat says. You may not want to go with this if it doesn't make sense, though; read on:
We can try to justify this feat's description. Typically, a sword or gun is positioned in such a way as to be drawn easily. Think scabbards and sheaths. Plus, critically, the handle on a sword and design of a sheath is conducive to quickly drawing; that is immensely important in combat.
However, there are a few problems with drawing other items:
- Scrolls, potions, and alchemical items aren't typically stored in such a way as to be conducive to Quickly Drawing them. You typically don't store scrolls, potions, etc. outside of you. They're often in bags and under cloaks. They're also awkward to carry like that, even if you could.
- They're more fragile; if you're withdrawing a potion or a scroll, you're definitely going to take greater care, and if you aren't, then you risk tearing the scroll, dropping the potion, or splashing it on you.
- They don't typically have grips like swords do, so they're not very conducive to Quick Drawing.
Wands, however, are a different story. Reading through this thread seems to indicate most people think it's silly to not Quick Draw wands. I tend to agree. In the case of potions, I would let players Quick Draw them if they're positioned in such a way as to be conducive to Quick Draw. This post (and almost the entire thread) supports this opinion.
My take has always been that this is due to the holster/sheath, not so much the weapon. When someone bucks up for a custom made bandoleer for their potion vials, for instance, I allow the same effect be emulated with other than weapons. -- Mark (source, post #4)
Ultimately, the rules of the game are at the players' and GM's discretion. If you and your players are in agreement that it makes sense to Quick Draw these things given a current situation, then do so. If your party is more rules-oriented, then just read the rules as-is.
In simpler terms, do what makes sense, and take the standard Pathfinder rules with a grain of salt.
Edit: Since you're looking more for a rules balance answer, here are my thoughts: Most of the time, this isn't going to affect anything. As @KRyan points out, this was originally intended to nerf rogues, however, it's not a terribly powerful feat to begin with. If you let it apply to these items, then you will very likely encounter no problems, and if you do encounter one of the edge cases of abuse (like repeatedly drawing to slow down action), then you as the DM can just say it's being taken to an unreasonable level.
As far as I can tell, there's nothing which justifiably breaks the game, even if there are a couple edge cases which do.