You know what you know.
The intent of the sidebar, taken as a whole, is "dunamancy is rare and protected, and it requires DM permission to learn if you aren't a trained dunamancer", but since dunamancer subclasses can use any of these spells as much as they like, this doesn't seem to be for balance reasons. It's just the flavor of the setting.
From a rules perspective, your spellbook is what it is. The sidebar says any wizard can add dunamancy to their spellbook if they're specifically allowed to, and you had permission by dint of your subclass, so you're fine there.
From a flavor standpoint, when you learned those spells, you were in fact a dunamancer, and then later you changed your subclass for whatever reason. You probably won't be able to add new dunamancy spells to your book without the DM providing them to you, but the existing knowledge doesn't evaporate.
But ask your DM.
Changing your subclass is always "with DM approval", and always a conversation to determine why and how it's going to happen. It doesn't actually matter what the sidebar and the "changing subclasses" rules say; it's between the two of you at the table to come up with a story and determine the answer to questions like how it impacts your known dunamancy spells. There is no set of rules that can override your personal discussion with your DM.
If you talk it over with the DM and decide that you're just changing your focus for character reasons, but you'll be allowed to keep your dunamancy spells, then you have the necessary DM permission to keep them. It doesn't matter whether or not that's strictly according-to-sidebar; the DM says that's how it'll work.
If you and the DM mutually decide that you've somehow lost your magical connection to dunamis, then maybe all dunamancy is now out of your reach, and the spells are still there in your book, but useless to you. The DM should probably take into account that you've lost a bunch of spells off your list and make an effort to let you replace them with non-dunamancy spells. Maybe later you'll regain your connection and re-acquire all those lost spells, again with the DM's permission.
If you and your DM decide that your character is intentionally turning their back on dunamancy, believing it to be dangerous or unethical, maybe you rip those pages out of your spellbook and burn them, swearing never again to use that power. In that case, then you have your answer (and the DM should again give you the opportunity to replace them).
Ultimately, it's not about what the book says, because every relevant rule in the book is explicitly prefaced with "if the DM wants to allow it".