If my pact weapon has finesse and my dex bonus is higher than my Cha bonus can I use my Dex bonus to hit then use my Cha bonus for damage?
When you attack with [the weapon chosen with Hex Warrior], you can use your Charisma modifier, instead of Strength or Dexterity, for the attack and damage rolls. [...] this benefit extends to every pact weapon
(Xanathar’s Guide to Everything pg. 55-56, emphasis mine)
It says “you can use,” not “you must use” or just “you use,” so you have the choice to not do so if you don’t want to. That said, this benefit is pretty close to the reason to make a pact with the Hexblade patron; if your Dexterity is higher anyway, you may be better off with another patron.
Moreover, the choice is to use Charisma for attack and damage, or for neither attack nor damage; it does not offer an opportunity to make different choices. If your Dexterity is higher, why would you want to? In almost all cases, 5e has you use the same ability score for both attack and damage, and this is almost-certainly the best choice here. Well, it’s the only choice here, but it would be the best one if you had another option.