According to the Pathfinder bestiary, an adult troll weighs about 1,000 lbs*. They don't give the volume of a troll, but let's assume that the density of troll flesh is similar to that of real-world human flesh: about 61.5 lbs/ft3**. At that density, a typical troll has a volume of about 16.26 ft3.
Bob's pit has a total volume of 10 ft × 10 ft × 50 ft, or 5000 ft3. If Bob's trolls discard their weapons, clothes, and other gear before jumping in, and the chewing action of the pit has the effect of packing them into the hole as tightly as possible, then we can fit at most about 307.5 trolls into the pit. Since trolls are not in fact oozes (and their regeneration will prevent them from being fluidized) and are unlikely to cooperate with requests to drop their gear, a more reasonable limit would be 100 to 150. Further, since the chewing of the pit*** is probably going to reduce its volume by at least half at each chomp, you'd be hard-pressed to keep more than half of that in the pit at any time. So call it a practical limit of 75 trolls in the pit.
Since the pit is chewing on its contents, it's not going to leave a stable surface on the top when it's full. When the pit's "jaws" are open for roughly 3 seconds out of each 6-second round, there will generally be capacity for further trolls to fall in. When its "jaws" are closed for the remainder of the round, any trolls in excess of its capacity will be forced out.
Once the Wall of Fire is in place, the trolls, being intelligent creatures (however marginally), will cease any efforts to run over the pit or climb out as they fear fire over just about everything else. They know that the pit can't really hurt them, so they'll just stay put and hope that either the pit or the fire goes away. On the other hand, if some unusually dim trolls do manage to get themselves killed by some combination of fire, falling, and chewing, then they will eventually get chewed to mush by the pit and consequently fit in more efficiently, potentially allowing more trolls to fall in. However, Bob may have difficulty keeping the pit open long enough for that to make much difference: he's only level 10 or 11, so even Extended, the Hungry Pit will last at most 24 rounds.
[*] That's absurdly scrawny for a 14-foot humanoid, but let's roll with it. Perhaps the 14 feet is from their outstretched toes to their outstretched fingertips. Trolls do have long arms, after all.
[**] That's probably low. Intuitively, large vertebrates need higher proportions of bone to support their bodies than do smaller ones; since bone is the most dense major component of a vertebrate body, the density of a large vertebrate should be higher than a smaller one of similar body plan. However, a quick search didn't turn up any general scaling rules; the closest that I found were body mass estimates as a function of femur and humerus cross-sectional area. But that figure is still a decent starting point for an upper bound.
[***] I'm assuming here that the pit acts like a big mouth, with two sides repeatedly pressing together and retracting. If the pit's "squeeze and crush" action is more like peristalsis, then the pit would keep a constant volume throughout, though less than the volume described above.