Have you spoken to the player about this? And is it actively detrimental to the game? Do the other players feel like he's getting "freebies" because he doesn't "work" as hard for XP?
Because I don't feel like it would overly bother me to have a quiet, follower-type in the party. He's already missing out on a lot of the fun, as I see things. But if that's what he wants, that shouldn't in itself be a reason to penalize him. That's only appropriate if you feel the need to coerce him into behaving differently, and you should only want to do that if he's detracting from your fun or the fun of the other players.
And frankly, a game shouldn’t be about coercion; if he’s not interested in the same game you are, that’s something you all, as a group, should discuss and come to an agreement about. Trying to use in-game rewards to influence out-of-character issues (i.e. what one wants from a game) is almost always a bad idea. If nothing else, I suggest that you at least try to offer the XP as specifically an incentive: “I’m going to offer XP when I think it would be good for your character to become more involved, because I want you to be.” (which you should offer to everyone, of course). Be up front about what you want and be sure to listen to what others want as well.
But in other words, ignore whether or not it's "fair," focus on whether or not there is a problem and whether or not alternative forms of XP distribution are likely to solve problems. I'm kind of dubious about both, to be honest.