It depends on what you mean by game breaking, exactly.
But it will be seriously imbalanced in that it makes casters that prepare spells much more attractive than casters with known spells.
It turns out that you're not as concerned about Wizards, which were my base example below. This houserule would be especially powerful for them, but my arguments work just as well for other class using prepared versus "known" spells.
Additionally, your players would gain a lot by multi-classing into Wizard with this rule in place.
Advantage: prepared casters
The big issue is that spell flexibility doesn't make the spells themselves any more powerful, but it does make casters that prepare spells effectively more powerful. It will also make a lot of challenges much less difficult for players. And, depending on your players, it might drain some of the interest away from casters that use preparation-- when I play a wizard, choosing which spells to prepare is a major strategic choice and a big element of gameplay.
For example, a Wizard ordinarily has to make a lot of decisions about what spells they want available during a dungeon crawl, because they're stuck with what they pick. Do they take something useful in adventuring, like detect magic, or something useful in combat, like magic missile? It can be a tough call, and the flexibility a magic user has is balanced by the risk of bringing spells that aren't useful at all.
With this houserule those choices and decisions disappear. If a Wizard sees a situation where they might want to cast detect magic, they just need one in-game minute and then they can. After they've tried it, they can just drop it for a combat spell instead. A canny spellcaster might be loaded for bear with combat spells most of the time (plus maybe one or two that can help in an acute crisis as a reaction), and will become perfectly situated to address any dungeon hazard they encounter at any time.
It will be most valuable for Wizards, as they have the largest spell list and the most spell slots. Their spell slots would become incredibly versatile at a (usually) negligible cost, at most nine minutes' wait. The value of this houserule to preparation-requiring casters is a function of spell list size, spell effect variety, and number of spell slots available.
Spellcasters that don't need preparation will become less important. Truly unique effects will be as useful as ever, but outside of those they'll be increasingly relegated to "common" things since the opportunity cost of using a preparer to do them will be higher. It's already often an unusual choice to use a wizard to cast a healing spell, for example, since they can do so many other things with a spell slot that other classes can't. The increased flexibility of changing prepared spells on the fly exacerbates that, leaving non-preparers to fill in the gaps.
Some incentives will change as a result:
- Casters will be well-advised to accumulate as many spells as possible, because they'll all be similarly available at need.
- Some spells, like chromatic orb, will lose some of their cachet, since specific damage types will be more accessible with very little lead time.
- Some class features may become more attractive as well, particularly those which restore spell slots or expand the spell list available to a character.
- Multi-classing to expand spell lists will become extremely attractive, though the synergies that become possible might be interesting enough to add a new dimension to gameplay (just one that happens to be focused more around magic users).
- Scouting and sneaking in dungeons becomes more attractive, as advance knowledge of opponents makes hard-counter preparation much easier. The party will essentially never encounter an enemy with an immunity or resistance that hampers one of their spells, or a weakness they can't exploit, and enemies will have a much more difficult time exploiting weaknesses among the party.
- Casters that don't need preparation will lose some of their value, since they'll gain no benefit while other classes become even more flexible and responsive, and if some of their spells are available (or can be effectively duplicated) by a caster needing preparation those classes will be less useful.
- Finally, I think that you might see a lot more incentive for short rests. Since the casters will need some downtime anyways, it becomes more and more attractive to stretch that downtime a bit farther so the caster gets a perfectly tailored spell list for their situation, and all the other players get the benefits of a short rest as well. Though that's a secondary effect which is under your control as DM, a party that takes short rests more frequently will find challenges easier than if they'd not rested.
Final Thoughts:
I think that this could be a really interesting houserule, but it might work best in a campaign specifically designed to accommodate it. Something high-magic, where the flexibility is expected and built into the challenges. Perhaps with all players taking classes that will benefit right from the start. But in a normal campaign, with players making character choices before knowing about this rule, I think that this would be destabilizing and highly game-able.