It applies to any weapon
The trait that applies the extra radiant damage clarifies (emphasis mine):
When the planetar hits with any weapon, the weapon deals an extra 5d8 radiant damage
If you equip the planetar with an adamantine greatsword, attacks with it will also get this extra damage.
You can use Multiattack
In many monsters, their Multiattack action specifies what kinds of attacks they can use with it, but the planetar (among other angels) is special in that it says:
The planetar makes two melee attacks.
This means the planetar can use any weapon given to it with multiattack.
You can probably get the 4d6 slashing damage
The planetar's stat block specifies that the Greatsword action has 4d6 slashing damage. This does not necessarily apply to an adamantine greatsword, which deals only 2d6 slashing damage by default. That being said, a greatsword and adamantine greatsword are basically identical in how you would wield it, so a GM could certainly rule that the 4d6 damage dice are used.
Larger weapon sizes is a rule concerning creating monsters in the Dungeon Master's Guide that could also result in these damage dice:
Big monsters typically wield oversized weapons that deal extra dice of damage on a hit. Double the weapon dice if the creature is Large, triple the weapon dice if it’s Huge, and quadruple the weapon dice if it’s Gargantuan.
Whether or not you could find/craft an oversized weapon is also up to the GM, but this would apply the higher damage dice as well.
Even if it does 4d6 slashing damage, that usually won't destroy a larger ship in two turns
Looking at the larger ships in Ghost of Saltmarsh, destroying them in an average of 2 turns is very unlikely even with an oversized greatsword. Automatically scoring a critical hit helps, but your average damage for each attack is still only: $$8*3.5+10*4.5+7$$
This yields an average of 80 damage per attack, and with four attacks over two turns that gives 320 damage. Then you have to account for the 10% chance that your attacks don't beat the AC of the hull. The larger ship hulls have 15 AC (18 for a warship), so with a +12 to hit you are not dealing damage on a 1 or 2. This reduces your expected damage over two turns to only 288 (240 against a warship).
That being said, gaining advantage does push the expected damage against 15 AC hulls to 317 (300 against warships). The chance that you sink a 500 HP ship (like a galley or warship) in two turns with this strategy is basically 0 regardless.
Without the extra 2d6 slashing damage, you aren't even sinking those 300 HP ships with two turns
If your DM rules that the adamantine greatsword only deals 2d6 slashing damage, you have only:
$$4*3.5+10*4.5+7$$
...or 66 damage per attack. This totals to 261 expected damage over two turns with advantage, and you are only sinking longships and sailing ships in two turns 0.8% of the time. Those 500 HP ships are then completely out of your grasp in two turns.
Magic Items
Sword of Sharpness
If you are looking to deal the most damage to ships with the planetar's Multiattack, you should get yourself an adamantine-coated sword of sharpness. First, to clarify, there is no restriction on coating a magic weapon in adamantine:
Melee weapons and ammunition made of or coated with adamantine are unusually effective when used to break objects. Whenever an adamantine weapon or piece of ammunition hits an object, the hit is a critical hit.
So, if you made an adamantine-coated sword of sharpness, you would also get:
When you attack an object with this magic sword and hit, maximize your weapon damage dice against the target.
This means the expected damage for the 4d6 version jumps up to:
$$8*6+10*4.5+7$$
...or 100 damage per attack. With the 10% chance to fail to deal damage, you are dealing an expected 360 damage over two turns. The galley and warship are still out of reach (dealing 396 and 375 damage respectively with advantage), but are only now in reach with three turns (594 and 562 damage respectively with advantage).
Scimitar of Speed
A slightly weaker alternative is an adamantine-coated scimitar of speed, since it gives you an extra attack (and a +2 to attack rolls). This changes the damage to only 2d6 (assuming its oversized), but you get to make an attack as a bonus action. This gives you three attacks each round for...
$$4*3.5+10*4.5+7$$
...each. This deals an expected 168 damage per round to the warship (193 with advantage), and 188 to the rest (198 with advantage). Slightly less than the *sword of sharpness except against the warship (as the +2 to attack rolls is the most impactful there). In that case, you get enough to 3TK a warship even without advantage.
