1 level of exhaustion affected ability checks, but does it have any implications for passive checks?
Yes, exhaustion would be a -5 to Passive Perception.
You have disadvantage on ability check (which includes Perception).
Here's how to determine a character's total for a passive check:
10 + all modifiers that normally apply to the check
If the character has advantage on the check, add 5. For disadvantage, subtract 5.