The practical answer is that a general -1
−1 to everything is a simpler rule with less bookkeeping that makes the game run faster.
For an in-fiction explanation: Wearing clumsy armour is tiring, uncomfortable and distracting. These all make it harder to think straight and be alert.