Your player is correct
Making a special mechanical representation of this is very likely to cause problems, and you should not do so. A trait like this simply shouldn’t be in the dice’s hands—it is important to bring it up with care and judgment, in situations where it adds to the game. Otherwise, this sort of thing has a tendency to just derail games and annoy people.
Moreover, permanent disadvantage is not how the game works. If you just aren’t good at something, it is your bonus that should be lower—(dis)advantage is for circumstantial changes from that baseline. I would expect this character to have low Intelligence, and probably not have proficiency in any Intelligence-based skills. But we don’t need to bring special rules into this situation—the usual character creation schemes give s
ample opportunity to choose to assign a low number to Intelligence, except in the rare case where one rolls for scores and then rolls high for all six.
(Or if they have good Intelligence, I would want to talk about what that means. Been there, done that—one of my most successful characters was an absent-minded professor sort, but since he was also a professor of religion, theology, and medicine, he needed Wisdom rather than Intelligence. I just chose not to roll Perception checks when the DM called for one in the middle of him going on about his latest theory. It worked fine. The unfortunate reality is that the ability scores are massively overloaded and sometimes a character is supposed to be good at some parts of what one covers and not others, and the rules don’t handle that. It is rarely worth getting fancy with house rules to “fix” the situation, though—just roleplay it appropriately, and have clear communication about what everyone is doing. I mentioned the problem to my DM, and he agreed that it was more effort than it’s worth to start changing Wisdom classes to Intelligence for this.)
And most importantly, this is not your character. You get the entire world to control—with the exceptions of the handful of player characters. The player absolutely should be pushing back on this, and you need to accept this. They wanted a role-playing quirk. You are trying to push them to accept a potentially-substantial extra disadvantage not seen by any other character in order to have it. That isn’t anything the game calls for (since, per the first paragraph, it causes problems), isn’t what the player tacitly agreed to when they agreed to play D&D (since, per the second paragraph, it isn’t how the game models poor memory), and thus really isn’t an appropriate demand for you to make. It could have been an idea to bring up, if the player was interested you could have tried to balance it out with some other bonus, but they weren’t. That’s their call and that’s when you needed to back off.
All this player is saying is that they have normal kinds of poor memory, the kind that suggests a below-average Intelligence score. What you are trying to do is turn that into some kind of special, severe chronic ailment, the kind that suggests a medical diagnosis and quite possibly an early retirement—certainly from a career as demanding as adventuring. That isn’t what they’re going for and they’ve said as much. That should have been the end of the conversation, and I think it would be appropriate to apologize for pushing on them so hard on this.