Lots of things could happen! But RAW the possession probably doesn't work if the Demonomicon is in an extradimensional space
The description of a Portable Hole clearly states that
You can use an action to unfold a portable hole and place it on or against a solid surface, whereupon the portable hole creates an extradimensional hole 10 feet deep. The cylindrical space within the hole exists on a different plane, so it can't be used to create open passages.
This is pretty decisive that the extradimensional space, whatever other properties it might have, is not on the same plane as the one on which the item itself exists. The description for the Portable Hole is otherwise similar enough to items like the Bag of Holding and the Handy Haversack that I would feel very comfortable applying that reasoning to them as well as the Portable Hole.
There is some discussion of whether or not the extradimensional space in a Bag of Holding and similar items is truly a different plane or not here and here. But regardless, as DM you easily have the authority to tweak that definition if you like.
Why you might want to rule that those extradimensional spaces are their own planes
The major reason to rule that these are distinct planes in this specific case is the Containment effect of the Demonomicon:
When you finish a long rest, if you and the Demonomicon are on the same plane of existence, the trapped creature of the highest challenge rating within the book can attempt to possess you. You must make a DC 20 Charisma saving throw. On a failure, you are possessed by the creature, which controls you like a puppet. The possessing creature can release you as an action, appearing in the closest unoccupied space. On a successful save, the fiend can’t try to possess you again for 7 days.
A save with DC 20 is really, really hard to beat. With a +5 to the saving throw, the creature making the save will succeed only one time in every four. That means that a PC attuned to the book will spend a lot of time possessed. A PC that wants, or ends up with, this book will need some way to deal with this effect, and an extradimensional storage space is one reasonable way to attempt that. Other options, like specifically buffing saving throws or the Charisma attribute are also reasonable but require more work to be effective (more magic items, for example).
Why you might want to rule that those extradimensional spaces are not their own planes
Or at least, not so completely distinct that the possession effect of the Demonomicon is totally suppressed. The Demonomicon is an Artifact-- a unique and incredibly powerful magical item in the setting. Its power is such that the plot of a major part of the campaign might be bent around it, and attuning it should do far more than just give a PC a bunch of bonuses. A party that can obtain and wield the Demonomicon is also going to have easy access to extradimensional spaces similar to a Bag of Holding through a variety of methods. Allowing an easy way to avoid one of the most potent dangers of such a powerful Artifact is a pretty strong decision.
Decide how much the plot is going to revolve around the Demonomicon to find your answer
You will need to think of some way to deal with this item if you allow your players to have it, but if you do not allow the PC to avoid the possession effect then you'll have to deal with that. This is time that the player will not be in control of their PC, which can be tricky to handle unless you relegate it to downtime-- what will you do if the PC fails the saving throw right before a boss fight? It's not even clear how long the possession lasts or how it might be ended outside of the demon's choice to stop. If you don't have a solid plan for how to handle this, it will be easier and less disruptive to your game to allow the possession effect to be more casually avoided.
But I recommend never giving players something like an easy way to avoid a downside without some consequences to balance it out. A clever idea that is cheese-adjacent works for a time, but not forever. It doesn't have to be the possession effect, and could be as simple as being hounded by demons for as long as the book is attuned. If you want more story to involve the book you can play those consequences up, and if you don't you might decide that the book's detrimental effects are enough of a consequence.
In short, Artifacts are amazing story leverage. "What could possibly happen" should be interpreted pretty expansively by the DM to serve that end, rather than just determining how much of a power boost a PC might get-- that's what lesser magic items are for.