Yes because you are making an attack
Invisibility states:
The spell ends for a target that attacks or casts a spell.
and Spiritual Weapon states:
you can make a melee spell attack against a creature within 5 feet of the weapon.
[...]
As a bonus action on your turn, you can move the weapon up to 20 feet and repeat the attack against a creature within 5 feet of it.
Since using Spiritual Weapon involves making an attack it breaks Invisibility.
Furthermore you can look to the rule in the section "Making an Attack" (PHB, page 194):
If there’s ever any question whether something you’re doing counts as an attack, the rule is simple: if you’re making an attack roll, you’re making an attack.