If a spell has material components, you can cast it without letting go of the crossbow at all.
The rules on "Material Components" state (emphasis mine):
[...] A spellcaster must have a hand free to access a spell's material components -- or to hold a spellcasting focus -- but it can be the same hand that he or she uses to perform somatic components.
Meanwhile the Improved Pact Weapon Eldritch Invocation states:
You can use any weapon you summon with your Pact of the Blade feature as a spellcasting focus for your warlock spells. [...]
Thus, you can be wielding you weapon which is also your spellcasting focus. If you fired your crossbow and then wanted to cast a spell such as fireball you would not have to remove your hands from the crossbow. Material components require that you be holding your focus, which you are already doing and so no hand removing is required.
If a spell has somatic components and not material components; the situation is a little bit iffy
The rules on "Somatic Components" states:
[...] If a spell requires a somatic component, the caster must have free use of at least one hand to perform these gestures.
We can conclude from this that if a spell has only somatic components or only somatic and verbal components, the caster is required to have a free hand, that is, unless they have the War Caster feat which states:
[...] You can perform the somatic components of spells even when you have weapons or a shield in one or both hands. [...]
Thus, if casting such a spell the Warlock would have to remove one hand from their weapon. This is generally considered to be a free action; to require nothing, not even your object interaction:
And then, to fire the crossbow, the caster would have to put their hand back on it (it is a two-handed weapon). Whether or not going from holding a weapon in one hand to holding it in two hands requires some sort of action expenditure such as one free object interaction is unclear by the rules and I go over an incredibly similar situation with a versatile weapon in my own answer to the following question:
I personally would rule that you can go from holding something in one hand to holding it in two hands for free and have not had this imbalance my games by any means.