Diversifying isn't enough, the enemy needs coordination too. And even that can be countered by more bards.
As Thomas notes, the wall can only be immune to one form of damage at a time. But that's not a major limitation on this strategy; the Combining Game Effects rules allow you to constantly replace the immunity, as long as your have a sufficient number of Bards with Lyres to provide the reactions:
The most potent effect - such as the highest bonus - is the one that applies, or the most recent effect applies if the effects are equally potent.
So even if the enemy diversifies their damage, say, alternating between bludgeoning from catapults and piercing from ballistas:
- They need to perform this alternation with perfect synchronization during a six second round, and
- It just means you need more reactions (read: more lyre-toting bards) to continually switch the immunity between the various damage types
Frankly, at a certain point it gets kind of absurd to imagine the level of coordination needed here. Trebuchets and mangonels take five actions from beginning loading to firing; ballistas and cannons take three. You're not firing every round, and frankly, while D&D pretends this stuff runs like clockwork, with non-heroes loading and firing, I'd expect them to get out of sync to the tune of a few seconds (enough to ruin carefully timed attack order) quickly. To deal damage to such a structure, you'd need at least as many siege weapons (and/or spellcasters with spells that are equivalent to a siege weapon attack) as the defender has bards and you'd need to make all your attacks, in a single round, perfectly alternating so that two attacks dealing the same damage type don't occur back-to-back.
If you're reliant on magic, your casters will likely be out of siege weapon caliber spells after just a handful of rounds ("five minute workday" is underselling how quickly those higher level slots would run out). If you're reliant on a mix of real-world siege weapons, you can keep it up all day, but you can only pull off the coordinated volley of mixed damage every few minutes, and those bards can be Mending the damage while they wait; normal Mending has a one-minute casting time, so it's not a quick response, but the Lyre grants it as an action, so the bards can be both reactively defending and actively fixing small breaks in the walls live.
In short: Even if the game rules technically allow for it (carefully readying actions to fire dozens or even hundreds of siege weapons or equivalents in a single round to overwhelm the defending bards), practically speaking, the structure is immune to attack if you can keep a dozen or so bards available to defend it.
You need to prevent the reactions entirely
Alternating damage types in sufficient volume to overwhelm the defenders' reactions is a sucker's game. Lyres of Building are almost certainly more common than the high-level casters needed to coordinate an attack of the necessary magnitude in any vaguely realistic way; and if someone is all-in on this defensive tactic and can afford to build them, all they need is to train up a bunch of low-level bards to wield them, which is a lot easier than the enemy hiring/training enough high-level siege casters to overwhelm them.
So don't play it as a numbers game. Find ways to prevent the reactions, e.g.:
Follow Kirt's advice. Don't use your casters to attack the walls. Bring regular old siege weapons, get them aimed on target (it's not like walls can dodge), then have your casters blanket the walls/defenders/ammunition with Darkness, Fog Cloud, etc., so no matter how many bards the enemy has, it doesn't matter, they can't use their reaction to defend. Or no casters, just bring along an Eversmoking Bottle (just uncommon rarity), and you can repeat this all day long (allowing 10 minutes between waves of fire for smoke to dissipate for aiming again), with no limited magical resource expenditure at all. Coordinate enough to fire in waves, focused on one section of the wall, both to minimize the number of obscuring spells needed, and to ensure enough damage has been dealt that Mending-as-an-action is useless (if the damage is greater than a foot in any dimension, Mending doesn't work). That'll leave them with their once-a-day spells for larger repairs, and if you keep up the siege weapon fire all day, with only a couple of lower-level support spells required per barrage, you'll overwhelm that pretty quickly.
This doesn't even require much flexibility from your DM to allow it; the rules for siege weapons like trebuchets and ballistas already require the process of using the weapon to take multiple actions, split into loading, aiming and firing. You only need to see the target when aiming. You only need the target obscured when firing. For emplaced siege weapons, attacking large, immobile targets, it's perfectly reasonable to use the visibility conditions at time of aiming, not at the time of firing, when the two actions are only separated in time by 6-12 seconds.
Use spells that destroy the castle without damaging it. Got Transmute Rock to Mud? That mud is unharmed, but much less useful, defensively. Got Move Earth? Take a few minutes to undermine the walls until they collapse under their own weight (arguably "damage", but not exactly "taking damage"). Animate Object a small (less than 15'x15') tower and have it just walk away (or start bashing the bards). RAW, Disintegrate doesn't do damage to objects, it just turns a 10' cube of it to fine grey dust directly, no damage involved, so use that. etc., etc. You get the idea.
