There are some rules for this in Ghosts of Saltmarsh but they stick to non-gunpowder siege engines. Even in previous D&D versions, cannon were often left to third parties (the presence of cannon being oddly contentious amongst fantasy gamers).
I run a pirate-themed Pathfinder campaign (been going since 2009) where I faced this same issue. There's a lot of earlier ed third party rulesets for ships and naval combat, but many of them either:
- Completely eschew cannon for ballistae and magic stuff, or
- Are super high fantasy so everything's all skyships and gnomish submarines and whatnot, or
- Completely avoid statting and just treat ships as plot devices
- Toss historicity out and have "frigates" and stuff in an otherwise 1600's kind of world
I wanted something vaguely real-world realistic, though playable. If you're running an actual full time naval campaign, some detail in ship sizes, is that a 12 pounder or 9 pounder cannon, etc. are interesting and important. I'd say the Razor Coast: Fire As She Bears supplement is the best balance of these concerns if you're looking for a book (it's Pathfinder statted but should be easily convertible).
The cannon in the DMG, at 8d10 damage, would be a large usually land-based cannon, I'd pin that at a 32-pounder. Ship cannon will usually be smaller. Scale damage up and down the 3/6/9/12/24/32/48/64 pound range.
The main problem with firing cannon on a ship isn't the raw tonnage to carry the gun, it's the force from the gun ripping up your ship or tipping you over. So there's some hard limitations on what you can do.
So you can handwave as much as you want, but the semi-realistic answer for cannon on the 5e ships in the DMG is:
Galley: Due to their design galleys can't side-mount cannon (it'd both destroy the oars and tip them over). They can mount one or more quite heavy ones lengthwise however, which in calm Mediterranean-like seas is a hardcore stand-off weapon. The "list" galley could probably do a single 24-pounder. Large mature RL galleys, probably double the stats for the one in the DMG, could do a 32-pounder with a couple 12-pounders flanking it. They'd also line the sides with swivel guns (small like one-pound guns on a swivel. Think 3d6 damage with a ball or 2d6 in a cone with shot).
Keelboat: Swivel guns only, no proper cannon.
Longship: No guns, just not designed for it.
Rowboat: No guns, unless you want to swim home.
Sailing ship: This is usually code for "you know, a small caravel like you done seen on the TV," Nina/Pinta/Santa Maria style. Probably a 60 foot long one, given that 100 tonnage value. Guns up on the top deck. Realistically, about 4 12-pounders per side max, maybe a 9 pound chase gun.
Warship: Usually code for a carrack or galleon. This is where you get real gun decks going on (which allow for heavier guns, since they're farther down on the center of gravity). "As many as you want," based on the size of the ship. The Mary Rose, a 700 ton carrack (more than 100 feet long), had a dozen 42-pounders and a small complement of 32 and 18 pounders. Just scale up per 100 tons from the caravel.
I find this in general hits a sweet spot of dangerous but not too dangerous (cannon get a lot of penalties to hit generally). Of course it's part of a carefully balanced set of naval combat rules and such.
Now for other siege engines - ballistae don't have the same recoil problems, they're just kinda big. I'd say you could get about 1/3 more ballistae than you could cannon on a given ship. But same thing with the size class - a small boat might have a heavy crossbow but not really a ballista; maybe one could legitimately fit on a keelboat. Catapults can't really be used if you have sails and rigging, they went out with the galleys.