Pact of the Blade does not use Charisma
Pact of the Blade says:
You can use your action to create a pact weapon in your empty hand.
You can choose the form that this melee weapon takes each time you
create it (see the Weapons section for weapon options). You are
proficient with it while you wield it. This weapon counts as magical
for the purpose of overcoming resistance and immunity to nonmagical
attacks and damage.
There is nothing in here that says the warlock wields the weapon using charisma. Nothing in the pact indicates that the weapons function any differently from how they are treated normally for what stat is used to calculate damage from them. The only things that change are being able to summon the weapon and being able to change the form.
Perhaps you are confusing this with the Hex Warrior feature of the Hexblade Warlock:
Whenever you finish a long rest, you can touch one weapon that you are proficient with and that lacks the two-handed property. When you attack with that weapon, you can use your Charisma modifier, instead of Strength or Dexterity, for the attack and damage rolls.
In this case, Hex Warrior does allow you to use Charisma for attack and damage rolls.
It is up to you
The rules don't usually dictate how something must be narratively described. Usually that is the job of the DM possibly in conjunction with the player. It is usually left to the table to decide how a rule/mechanic translates into story and such is the case here.
In reality, fighting with a weapon is never entirely about strength or dexterity or any other stat even though the rules say that is what happens. The game mechanics do not always translate into an easy narrative.
In fact, the ability scores themselves are very nebulous concepts that encompass many aspects of a character. That means that generally there are many many ways for them to be described narratively at a table.
Find a way to describe it that works for your table. Or, don't call special attention to it at all. Just because the mechanic uses charisma doesn't mean you have to narratively describe it differently if you don't want to or if there is nothing that quite fits right.