Glazius’s answer defines the bag of rats, and describes its origins, reasonably well in my opinion. In brief, a bag of rats is a tool for abusing “on kill” mechanics by having a large number of easily-killed creatures available.
That answer, however, only covers what the concept is, and where it came from. There is more to discuss here, especially with respect to how big a problem it actually is, and how you deal with it.
In my opinion, the best discussion of the bag of rats, or as it’s often referred to in Pathfinder due to the coinage by third-party product reviewer EndZeitgeist, the kitten test, is The Bag of Kittens and You, by Jade Ripley, aka Lord_Gareth here and elsewhere. Therein, he describes the difficulties in sourcing your rats or kittens or what have you, the difficulty in keeping them alive long enough to kill them at the moment you need them dead, and the difficulty of efficiently killing them all in the middle of a combat.
In short, a lot of times, it’s a non-issue. 3e-era Whirlwind Attack and Great Cleave? Pulling out your bag of rats probably costs a move action—or more, since it’s probably tied shut—which leaves you without a full-round action to actually use Whirlwind Attack. That makes it a minimum of a 2-round maneuver, and in the middle the enemy is free to, ya know, move away from you and your rats. Or just kill you.
Jade brings this to amusing extremes with Countess Felis von Orphanpuncher, who straps kittens to her for ready stabbing. Even this absurd outfit doesn’t necessarily leave the character eminently capable of abusing the kittens strapped to her body.
But other times, it’s much more serious. D&D 3.5e has a notorious spell named consumptive field (and a higher-level greater consumptive field). These gives caster level bonuses for each thing that dies within, and those last for an hour. This is a much, much more serious problem: those bonuses are potent, and you can easily kill your helpless victims from relative safety and comfort, and then go out into the world with absurdly powerful spells.
Unfortunately, as Jade describes in the linked article, a game designer has limited options for fixing this, as any fixed rules are just as liable to be gamed with some loophole or another. The best approach, for a game designer, is to just limit how much benefit you can actually get from a bag of rats, by capping things, requiring the death be near (in time and in space) to where the benefit applies, by having the benefit be based on the power of the dying creature in question (so helpless things offer you little or no benefit), and so on. If you can’t do that, you have to leverage the GM’s judgment, using vague terms like “a real threat,” and let the GM figure out what does or doesn’t count.
As a GM, you should be up-front with your players when you notice that one has an ability that could be abused with a bag of rats or similar, and that might be problematic. Tell them that you will had to adjudicate on an ad hoc basis whether or not a particular creature “counts” for the ability—any enemy you throw at them, sure, but if they start bringing minions or monsters of their own just to fuel the ability, make sure the risk and difficulty of that is commensurate with the reward—and block it if it becomes too abusive. Be open with players about this, and answer their questions about what does or doesn’t work out of game. Don’t force their characters to try things in-game to find out what the rules are—D&D characters are generally assumed to understand their own abilities and that goes into this as well.
But again, trying to state hard-and-fast rules doesn’t work; that will always just leave some loophole, and potentially cut out opportunities that you wanted to work.