Spiritual Weapon is not affected by any modifiers to your attack rolls
Per the spell description, it uses only your BAB and Wisdom bonus for its attack rolls, and its damage is strictly set to 1d8 + (caster level/3). Beneficial effects, such as Prayer or Bless are not applied, but neither do detrimental effects impede it, such as being sickened or grappled.
A weapon made of force appears and attacks foes at a distance, as you direct it, dealing 1d8 force damage per hit, + 1 point per three caster levels (maximum +5 at 15th level).
[...]
It uses your base attack bonus (possibly allowing it multiple attacks per round in subsequent rounds) plus your Wisdom modifier1 as its attack bonus.
As you already noted, the spell calls out specifically that feats that you have or combat actions that you take do not impact the weapon in any way. In fact, what is most likely meant by "unaffected by your combat actions" was in reference to things like spells that you may have cast to buff yourself, since casting Prayer is most definitely an action. And before any chuckleheads say "but what about combat actions that my allies/enemies take?" no, those don't (or shouldn't) affect it either.
And that's where it ends, there are no further mentions in the spells description of it gaining bonuses or penalties based on any penalties you may have. The way the spell is worded implies that the weapon is its own entity (essentially being an animated object) and makes the attacks without your intercession. In fact, being knocked unconscious doesn't even end the spell, as it will continue to attack your most recently designated target until that target is dead, or the spell ends.
Each round after the first, you can use a move action to redirect the weapon to a new target. If you do not, the weapon continues to attack the previous round's target.
1It has been stated by one of the devs in a Paizo blog post (though I can't recall which one) that it would be reasonable for a table to houserule that a charisma-based caster, such as an Oracle, Spiritualist, or Medium, who gets this spell on their spell list can use their primary casting stat instead of Wisdom for the spells attack rolls, since at the time that the core book was released, they hadn't really done much with the idea that some divine casters might not use Wisdom as their casting stat. It's noticeable in later books where spells will say to use your spellcasting ability modifier instead of just calling out a specific stat.