Use the modifier of the class the spell comes from
Each spell you learn comes from its own class and is labelled as such (a spell learned from Pact Magic is a warlock spell). You must use the spellcasting modifier from that class's feature.
Charisma is your spellcasting ability for your warlock spells
Even if a spell could have been learned from either class, what matters is how the individual learned it. If the character learned a spell from the wizard feature then it is a wizard spell and uses Intelligence even if it appears on other spell lists.
The Multiclassing rules detail that you treat your spells known or prepared separately for each class:
You determine what spells you know and can prepare for each class individually, as if you were a single-classed member of that class