Violating the agency of a player as a DM is different from an NPC violating the character's agency
As a player, it is your job to play your character. You know your character's mind, their feelings, and their priorities better than anyone else. And all of this knowledge goes together to allow you to do the role you are assigned in the rules (PHB, p. 6):
The players describe what they want to do...
Naturally, what your character does and doesn't remember is an important element in this decision, so it is unambiguous that agency is being violated here. And your agency as a player may or may not have been violated, depending on some specifics. But before we get into those specifics, it's important to realize that there's a difference between a DM violating your agency as a player and an NPC violating your character's agency.
NPCs can violate character's agency
If a Lich drops a character to 0-hp with a Magic Missile spell, that character's agency has been violated. They may want to stand up and attack the Lich, but they can't because their body is too harmed to allow it. Similarly, if a Vampire uses its abilities to charm a character, that character's agency has been violated. The character normally would want to attack the vampire, but it can't because it is Charmed.
These types of violations are usually not much fun for the player. It means that you don't have much choice in the matter, or at least your choices are severely restricted. If done often enough, these violations can take all the fun out of the game, which is often intended to be a fantasy of empowerment. But they are not a violation of the agreement between player and DM to control their assigned elements of the game.
Messing with your character's memories is an established element of the rules that your DM can inflict upon your character by means of forces in your environment. In the same way your characters can be knocked to 0 hp by damage or Charmed, they can be under the influence of the 5th level spell Modify Memory. And the creature that cast this spell could be so powerful that the DC on the spell was too high for any die roll to overcome (e.g. DC 70). While this could be a frustrating outcome (and one the DM shouldn't use often, lest the game stop being fun for the players), it's something that could happen in the story, and isn't necessarily a violation of your agency as a player.
This could also be a violation of your agency as a player by a DM, depending on how the deal went down
A violation of your agency as a player occurs when your DM decides what your character wants to do for you by virtue of being the DM. While the DM may subject your character to any number of external influences (e.g. magic spells or an ancient dragon's Frightful Presence) that could restrict what your character can do, they cannot assume control of your character without appealing to one of these outside forces. If you say "I stand up and walk away" the DM can't say "No you don't, that would be boring. You sit there and listen." That would essentially ammount to the player being pointless at the table: the DM is controlling everything including you, and you're just here to roll dice.
The removal of your character's memory is not necessarily a violation of your agency as a player: but it sounds like the terms of the agreement were discussed with this outer planes being as a group. So it would be a violation of your agency as a player for the DM to decree that your character agreed to terms you, the player, never heard.
If the "PC in question" had the ability to discuss with the outer planes being privately (as the player discussed with the DM), then the deal might have been altered without your knowledge. This would be frustrating, but not necessarily a violation of your agency. But if the discussion of the deal must have happened in public, then you should get a chance as a player to describe how your character reacts to it. Otherwise, your agency as a player has been violated.