Yes, it's only for a Lucky attacker and a Lucky target.
Lucky can benefit one's attack roll, ability check, or saving throw or an incoming attack roll against oneself. The only way a roll could be affected by Lucky from multiple sources is if it's an attack roll, because that's the only sort of roll there that can be manipulated on both ends (source and target).
If an attacker with the Lucky feat attacks you, they are able to benefit on their own roll yet you're able to benefit against that same roll. This is the only situation that the special canceling rule applies to.
It is very unlikely the canceling rule will see use at the table since generally only PCs have feats and PCs don't usually attack each other, but the canceling rule is there to handle edge cases. Here are some situations the rule would cover:
- Two PCs engage in player-versus-player combat.
- A PC attacks an ally PC who is currently under a mind-control effect that allows a save to end the effect when the target is damaged.
- An NPC enemy designed with character features attacks a PC.
- A PC traitor becomes an NPC enemy in the middle of an encounter.
- A PC applies Lucky to a spell attack against a spectator and misses, and the spectator reflects the spell back at a different PC with the Lucky feat (the spectator's Spell Reflection feature states that the attack is rerolled, so it's the same attack roll that originally benefited from Lucky; the second PC couldn't use Lucky to avoid the reflection).
There are certainly other edge cases, and the rules for the Lucky feat account for the ambiguity and possible shenanigans by including a fix just in case.