Are there any rules around spells nullifying each other? For example, I cast heat metal on a target's armour in round 1. In round 2, I use a bonus action to make the target take the burning damage again, and cast frostbite on the target.
Does the frostbite damage and effect have any adverse effect on the heat metal spell or vice versa?
As mentioned in this question, spells do only what they say they do. If there is any special interaction between spells, the spell description will include it. An example would be the interaction between Wall of Force and Disintegrate. The description of Wall of Force states:
A disintegrate spell destroys the wall instantly
In the example mentioned in the question, where a spellcaster uses their bonus action to deal damage with an active Heat Metal spell, and then uses their action to cast Frostbite on the same target, the target will take fire damage from Heat Metal, followed by cold damage from Frostbite. The two spells will not cancel each other out or interact with each other in any way, because neither mentions the other in its description.
However,
The DM always has the final say and can override RAW. They may decide that the super heating of the armor via Heat Metal immediately followed by the cooling from Frostbite cancels out the damage, or has some other effect like damaging or destroying the affected armor.
Spells do what they say. Neither Heat Metal nor Frostbite say that they are cancelled by, or interact with, each other, so they don't. Their effects are completely independent of each other, and don't depend on each other for anything.
If you need to keep up verisimilitude, imagine what would happen if your poured liquid nitrogen on your left hand, and then 6 seconds later covered the hand in gasoline and lit it on fire. The 'cold' and the 'hot' damage you took don't cancel out at all. You're left with a frostbitten, burned hand.
Your DM is well within their rights to house rule nullification between spells, but there is nothing in the rules that specifically allows for it.
There are no examples of a damage-causing spell nullifying the damage of another. It is, of course, possible for advantage/disadvantage granted by one spell to be negated by the opposite effect caused by another spell, and for non-damage effects such as darkness to be dispelled.
RAW, the target would take both fire and cold damage in your example.
Even in reality, it's possible to suffer from fire burns and ice burns simultaneously - take for example mountain climbers at very high altitude suffering both sunburn and frostbite. One type of burn doesn't negate the other.
What the spells say...
Frostbite says:
You cause numbing frost to form on one creature that you can see within range. The target must make a Constitution saving throw. On a failed save, the target takes 1d6 cold damage, and it has disadvantage on the next weapon attack roll it makes before the end of its next turn.
Heat Metal says:
Choose a manufactured metal object, such as a metal weapon or a suit of heavy or medium metal armor, that you can see within range. You cause the object to glow red-hot. Any creature in physical contact with the object takes 2d8 fire damage when you cast the spell. Until the spell ends, you can use a bonus action on each of your subsequent turns to cause this damage again.
If a creature is holding or wearing the object and takes the damage from it, the creature must succeed on a Constitution saving throw or drop the object if it can. If it doesn’t drop the object, it has disadvantage on attack rolls and ability checks until the start of your next turn.
They say nothing about interference.
