Charm without speaking is easily possible
For example, the Beholder, one of the game's most iconic monsters, has a charm ray with this effect:
Charm Ray. The targeted creature must succeed on a DC 16 Wisdom saving throw or be charmed by the beholder for 1 hour, or until the beholder harms the creature
It does not need to talk when using it, it is just shooting rays. Likewise, a Vampire can charm just by using its gaze.
In general, it is entirely your decision what a specific magical feature does when you design a monster. Such a feature works by itself. Even though both Beholder and Vampire do have high charisma, if some effect would alter that, the feature would be unaffected. You as DM would need to make a ruling that without Charisma the charm gaze of the Vampire would not work any more, if you thought this is more believable. The feature itself has no text that would indicate so.
Charisma is described on p. 178 PHB:
Charisma measures your ability to interact effectively with others. It includes such factors as confidence and eloquence, and it can represent a charming or commanding personality.
While it can include eloquence, it is not limited to speech. Nonverbal cues make up a large part of communication, and Charisma includes things as personality and confidence. Its mechanical effects are mainly captured in the modifier it provides to skill checks, saves or specific class features. There is no general rule that you would be able to charm anyone with a certain score in the charisma attribute.
P.S. Your idea, by the way, has a famous precedent: in the original Castle Greyhawk D&D campaign, Gary Gygax introduced the mysterious Jeweled Man, a creature made entirely from gold and encrusted in jewels that somehow was impssible to surprise or catch. It turns out, a magical charm was not needed: his players' greed drove them to pursue it. (In that sense, he managed to create a creature that indeed charmed the players, not merely the player characters).