Older editions of D&D specifically describe Bags of holding as "always appearing empty", and when you put something into it, it disappears. Based on that, I would say a bag of holding is not an open doorway; it is effectively a permanent teleport spell (or very similar) anchored on the physical bag. So the question becomes whether magically altering the dimensions of the bag messes with the teleport enchantment.
Specific to your question: I would not expect enlarging the bag to alter its holding capacity in any way. The bag is connected to an extradimensional space. At best you're making the opening bigger; but that wouldn't in any way alter that "interior". Think of it like a TARDIS from Doctor Who -- the external "real world interface" can be any size and shape, and at best you're merely altering the doorway itself. The interior doesn't change. (That said, a TARDIS doorway is an open portal, not a teleport; I'm just observing the clear distinction between interior and exterior.)
It's really a DM judgement call as to whether casting a spell that radically alters the real world object to which the enchantment is anchored damages the enchantment itself. Personally, I would decide a percentage chance, and roll to see if they still have a bag of holding, or if the enchantment is broken and they are left with an ordinary bag.
Related: https://rpg.stackexchange.com/a/114723/14729
Also related: There was an early story line in Knights of the Dinner Table relating specifically to the workings of a bag of holding. Worth reading, and pretty entertaining, if this topic interests you.... https://www.goodreads.com/book/show/12216585-knights-of-the-dinner-table