It's you, both narratively and mechanically
And in fact, that is the conclusion that requires the fewest assumptions.
When a creature is subject to the magic melee attack of spiritual weapon what is considered to be the attacker?
Let's look at the spell text, bit by bit, remembering that spells only do what they say they do:
you can make a melee spell attack
This is fine so far, and not so different to some other spells like Primal Savagery:
[...] Make a melee spell attack against one creature within 5 feet of you. [...]
or Vampiric Touch
[...] Make a melee spell attack against a creature within your reach. [...]
The only difference is that the 'range' of the spell is relative to you. Now compare to Steel Wind Strike:
Choose up to five creatures you can see within range. Make a melee spell attack against each target.
With a range of 30ft, you can attack up to five creatures in that range, but a) it's still a melee (spell) attack, and b) mechanically you haven't moved until this point.
So how does Spiritual Weapon compare?
against a creature within 5 feet of the weapon.
Ok, that's simple. Wherever the weapon is within that range determines what creatures are eligible to be attacked by you. The fact it's you making the attack isn't changed by this last part. It might seem like a strange wording, but it's how regular melee attacks work, except normally the weapon shares your space.
In this way, the plain english reading of 'you can make a (melee spell) attack' makes you the attacker, as there is no defined game term I know of.
So we can either conclude, that for every reference in the rules, that when it says 'attacker' it really means 'the one making the attack, or the weapon if the two aren't sharing the same space' or something similar. Or we can say that the 'attacker', as far as the spell is concerned, keeps it's usual meaning and then work out any interactions without further assumptions.
n.b. if it were the case that only the weapon needed to be seen for things like Uncanny Dodge or Hellish Rebuke to work, there could be circumstances where a ranged attack with a bow could be make where the arrow is visible but the archer is not.
But now we have spooky action at a distance!
Sure, but that's not a problem, mechanically or narratively. We've taken the world view that the attacker is the one casting and controlling the weapon, and thus any references to the attacker must mean them. That's the mechanics sorted.
The implications you seem to be concerned by are valid, but now we've pinned down who the attacker is, we can create a narrative that matches how those mechanics relate to the world the characters see, and the implications therein.
My preferred conceit, is that the caster of Spiritual Weapon (and similar spells) is like a puppeteer. And while you can see their hands are moving you can predict which way the weapon will strike, as soon as they're out of sight, you can't predict how the blade will move or reliably draw line of sight to the caster from the weapon's location.
The Rogue can't use Uncanny Dodge, because they can't see the hands controlling the puppet. The Warlock can't use Hellish rebuke targeting the weapon, and there is no way of connecting back to the attacker from the weapon. Additional narrative conclusions also work: Frostbite imposes disadvantage as you struggle to 'puppet' the weapon due to the 'numbing frost', just as Vicious Mockery does, due to the 'string of insults laced with subtle enchantments' (perhaps mocking your marionette motions); Smite spells that affect you 'narratively' transfer their effects onto the spectral weapon, similar to how Absorb Elements would work Etc.
That should tie up both the mechanical and narrative concerns, meaning there is no reason the caster can't be the attacker in all circumstances.