Part of the Crossbow Expert feat (PHB, p. 165) says:
Being within 5 feet of a hostile creature doesn't impose disadvantage on your ranged attack rolls.
This is relevant because of the rule on ranged attacks in close combat:
Aiming a ranged attack is more difficult when a foe is next to you. When you make a ranged attack [...] you have disadvantage on the attack roll if you are within 5 feet of a hostile creature who can see you and who isn't incapacitated.
My GM and some players have agreed that the benefit removes your disadvantage on attacks against distant targets (ones more than 5 feet away) while a foe is adjacent to you, but that targeting the adjacent foe still has disadvantage. Is this the correct ruling?