You can still cast spells while holding a Two-Handed Weapon
When you look at the statblock for most two-handed weapons (I'll use a Greatsword as an example since it's the typical two-handed weapon), the point for Two-Handed has the following written, emphasis mine:
This weapon requires two hands to use. This property is relevant only when you attack with the weapon, not when you simply hold it.
The consequence of this is that if you're casting a spell this turn (which will usually mean you won't be attacking this turn), you only need to hold onto the sword with one hand while you use your free hand to handle the material + somatic components—which can be done with the same hand, as specified in the Player's Handbook where Spellcasting is described, under the section about Spell Components:
A spellcaster must have a hand free to access a spell's material components—or to hold a spellcasting focus—but it can be the same hand that he or she uses to perform somatic components.
So why does Improved Pact Weapon Invocation give the ability for a weapon to be used as a spellcasting focus?
Given everything I've mentioned so far, it might seem like that feature of the Invocation is redundant. But it does have an important function: for two-weapon fighting.
If a player is holding two weapons, one in each hand, and doesn't have this feature, then attempting to cast a spell is not possible without dropping the second weapon and grabbing the focus, because doing more than that (putting it away + grabbing the focus, for example) requires using the Use an Object Action.
Being able to use the weapon itself as a focus solves that issue, by negating the need to put any weapon away.
Of course, the invocation is still good regardless, because it gives +1/+1 to any weapon that doesn't already have such bonuses. But that is at least one situation where that particular feature of the Invocation is important.
