A dangerous combination that this could also allow would be to combine Misty Step. which is a bonus action spell, with just about anything else, allowing the spellcaster to pop in from hiding by surprise and immediately cast a devastating AoE spell, or even a touch spell -- or likewise to be able to run up to an opponent, cast a touch spell on him, and immediately Misty Step away to safety. I think that would be an imbalance as well.
As a wizard player, I found myself very frequently wishing I could Misty Step up to the castle wall and immediately blow stuff up in the courtyard without warning, or similar combinations. How often and how badly I wanted to do that is, to me, a very good indication of how unbalancing it would have been to allow that.