Question from the text of the question is:
Would any of the versions of D&D that existed at the time (2002), as written, permit someone to backstab (or that edition's equivalent) with a ballista?
If "as written" means "was this a RAW compliant attack" the answer is NO, due to the damage being far too great for a single ballista shot: 264 points of damage. If that isn't the question, and the question is "can you sneak attack with a ballista" ... that's another matter explored below.
The DM allowed it, so the answer to the title question is obviously "yes," if the game being played was AD&D. The imbedded question (quoted in this answer) asks about versions in print at the time the movie was made. AD&D 1e and 2e were in print at the time.
With the understanding that the game may have been a version of AD&D, versus GURPS or something else, the question is then "Which one?" and the true answer to that is: unknown.
Why this might have been an AD&D 1e or 2e Setting
The DM could have exaggerated for effect.
The DM could have been using a critical hit table. (Common among AD&D 1e DMs even though non within the rules)
The DM could have been amused and chosen to go dramatic for narrative effect. (Rulings not rules approach to DMing, endorsed in writing by the game's author).
Rulings, not rules. Under that basis, this could have been a representation of a 1st or 2d edition AD&D game, since the DM could allow all of the above, and could have chosen to inflate the damage for dramatic effect. These are good enough reasons for a DM to allow a ballista back stab.
Reasons To Not be AD&D First Edition is RAW based, not Rulings based
- Small note: the book the DM consulted didn't look like the 1e DMG, nor the 1e PHB, in terms of what the brief view over his shoulder offered. This leaves 2e or 3d as more likely.
The backstab and triple damage for attack from the rear cited in the scenario (level 5-8 thief) would fit the 1e model(PHB, p. 27), as well as the need to catch Hunk by surprise to achieve that back stab bonus for damage and +4 to attack.
- The original backstab by rules as written wouldn't apply with a ballista. The rule for that sort of attack (p. 27 PHB) cites an attack by "club, dagger or sword."
- That said, Unearthed Arcana expanded the thief's weapons to some bows, but it did specifically list, in RAW, the ballista (which is a siege weapon) as a thief's weapon available for choice.
- The DM's "there isn't anything against it"1 (besides taking almost no time to consult the book, and thus being very unrealistic) looks like a decision that a ballista was a form of crossbow and thus a weapon of proficiency eligible for that backstab (the DM's I played with did not give that damage multiplier for bow shots and ranged weapon, but a DM could apply the logic of "now that bows are thief weapons, all thief attacks can be made by all thief weapons)."
- We don't know what class this thief actually is. If he were a Fighter/Thief, or a player with two classes (some Fighter and most Thief), then weapons restrictions are moot since fighters can use ANY weapon.
The amount of damage finally ruled by the DM, 264 points, argues heavily against an AD&D 1e setting if RAW is operating (likewise for AD&D 2e and D&D 3.0). For that siege weapon, per DMG p. 108, the damage is 2-12 for man sized and 3-18 for larger than man sized for the roughly spear sized projectile. Triple Damage, even on 3-18 base, won't reach 264 points any time soon.
- Insofar as maneuvering a ballista, all the thief needs is a belt of Storm Giant strength, or something similar (RAW available), and he'd be strong enough to handle it with some grace and even stealth. How he got that siege engine into the bar into the first place is a matter left unanswered in the film. The plus to damage from that belt, +12, while it should not apply to this missile shot, would still not, even if tripled, boost the cited damage into the range reported by the DM.
- To mitigate "can the thief use that weapon?" the thief could use that weapon even in a case of non-weapon proficiency (-3 for thieves) being applied to offset the +4 for a surprise attack from the rear. Since the DM rolled a hit, that all comes out in the wash.
Summary
Even though the rules as such don't explicitly seem to support a ballista backstab and that much damage, the GM can allow it in 1e. In 2e, ranged weapons could be ruled as "backstab" inclusive, but the text does not explicitly say so, thus 2e may be in play.
All of these points are irrelevant if "as written" is the key criteria: none of the three editions would produce that much damage (264 points) from a sneak attack/attack by surprise from the rear, even with a ballista.
In that sense, if the question is "can this be done with rules as written" NONE of the D&D versions in print at the time apply since the damage is so far beyond what that weapons does as to beggar belief -- which seems to be the point of what's going on in the scene, a case of exaggeration for dramatic effect.
The DM ruled that it did more damage than it could. In that sense, the AD&D versions that emphasized rulings over rules (1e or 2e) is the more likely candidate for being the version in play.
1 We don't actually know that he consulted a rule when he opened the book. That might have been him stalling for time as he thought through this ludicrous situation. If he DM'd like some of the DM's I played with, the more outrageous the idea, sometimes the more amusing and entertaining it is for the DM. I made a lot of off the cuff rulings when I dm'd since I did not like slowing down play when I could help it.