You make the attack, but the attack originates from the weapon
The rules on advantage and disadvantage are written assuming that the creature making the attack is the one the target sees making the the attack, and is in the place from which the attack 'originates'. That is a reasonable assumption most of the time, so these rules work most of the time. When that assumption does not hold, however, the rules no longer make sense. This is the natural outcome for violating the basic assumptions of the rules.
Knowing that the rules no longer make sense, we have two options - we can apply them anyway - 'This is RAW, even though it is stupid' - or we can adjust them for this edge case situation that they were not designed for. I think the second option is better.
If we modify the rules because we recognize that the original assumptions don't apply, we have some target results: regardless of how we adjust the rules, our special-situation modified rules should (1) draw on the larger body of 5e rules and rulings and make sense within that framework, and (2) add to verisimilitude by seeming like a reasonable adjustment. Any ruling or rules modification that can do both these things is doing well.
You, the caster, are making the attack
The spiritual weapon is a 'spectral weapon'; it is not a creature and perhaps not even an object. It cannot attack without you using a bonus action to make it do so, and cannot reason or decide. You cannot tell it to 'attack when something comes into range' (readied action), you cannot make an opportunity attack with it, you cannot tell it to 'keep attacking that creature on the next round'. You are the author of its intent - it has no capability you do not give it. Most importantly, it cannot select a target - you have to do that. Thus, you have to see the target, or select a square and use the rules for attacking a target unseen.
If you, the caster, cannot see the target, your attack has disadvantage. It does not matter the line of site or proximity of the weapon to the target. No rules give the weapon specifically or objects or forces in general the ability to see. If the sight of the weapon to the target mattered, the weapon would always have disadvantage, because the weapon cannot see.
The target of the attack is trying to avoid the weapon, not avoid you
To be damaged by the spell, the target must be within five feet of the weapon - it is the weapon that is swinging at the target. To avoid being hit, the target needs to avoid the weapon itself; if the target has sufficient distance from the weapon, it cannot be attacked, regardless of the target's distance from you. You, the caster, not being able to see the target may mean that the weapon is swinging wildly and aimlessly - but if it hits the target, it will do just as much damage as if it was well-aimed.
Note that the spiritual weapon spell explicitly says that the weapon is the "spell's effect", while the rules on Cover state that
A target can benefit from cover only when an attack or other effect originates on the opposite side of the cover.
Even though you are making the attack, the attack itself originates from the weapon, and so only cover between the weapon and the target matters.
This situation is analogous to the barbarian's danger sense ability:
At 2nd level, you gain an uncanny sense of when things nearby aren't as they should be, giving you an edge when you dodge away from danger. You have advantage on Dexterity saving throws against effects that you can see, such as traps and spells. To gain this benefit, you can't be blinded, deafened, or incapacitated.
Barbarians gain advantage on Dex saves when they can see the effects of spells, not their casters. If a wizard threw a fireball (the blast of which is not an attack, but is a damaging spell effect) so that it exploded around a corner, the barbarian on the other side would not be penalized for not seeing the wizard - rather, they would have advantage for seeing the bright ball of flame headed their way!
The target is trying to avoid the damage the weapon represents, trying to avoid the attack whose physical point of origin is the weapon. You, the caster, have advantage on attacks with the spiritual weapon when the target cannot see the weapon. It does not matter if they cannot see you.
To summarize:
Your attacks with spiritual weapon are at disadvantage when you cannot see the target. Your attacks with spiritual weapon are at advantage when the target cannot see the weapon.