Spiritual weapon obeys line of sight rules with you as the Caster/Attacker
Jeremy Crawford has explained that spiritual weapon ignores cover, but obeys any visibility concerns. Visibility is determined based strictly on the line of sight between the caster and the target.
Crawford: The spell doesn't require you to see its target. It follows the normal rules for attacking a target you can't see.
Questioner: caster could leave the room and leave a spiritual weapon floating in the doorway for a minute w/ disadv to hit?
Crawford: Yes, assuming there's actually something there to hit.
And the rules for attacking while unseen:
When you attack a target that you can't see, you have disadvantage on the attack roll. This is true whether you're guessing the target's location or you're targeting a creature you can hear but not see. If the target isn't in the location you targeted, you automatically miss, but the GM typically just says that the attack missed, not whether you guessed the target's location correctly.
When a creature can't see you, you have advantage on attack rolls against it. If you are hidden–both unseen and unheard–when you make an attack, you give away your location when the attack hits or misses.
Since it's been explained that attacking while you can't see the target would impose disadvantage, and the circumstances for the advantage/disadvantage both depend on the line of sight between you (the caster) and the target, you would have advantage to hit the target with spiritual weapon if the target cannot see you (and disadvantage if you can't see the target).
This could be explained through the target not recognizing your bonus action to strike, much in the same way that a fire bolt or an arrow shot from the darkness could be seen, while occurring too suddenly to react to.