TL;DR: Charisma is often connected to appearance, but has no inherent association with any physical qualities--and definitely does not imply comeliness. The repeated references to Charisma as a physical attractiveness score are incomplete shorthand, referencing the most common use of Charisma for player characters.
Since you're focusing on 3.5-like systems, I'll quote from the best official 3.5 source of Charisma description I've ever found: the 3.5 Monster Manual entry "Ability Scores."
But first, I'd like to point out that this entry describes Str, Dex, and Con as "physical abilities" and the other three as "the remaining abilities" (in other places, Int/Wis/Cha are called "mental ability scores").
Charisma: Reflects the creature's sense of self and ability to influence other creatures for good or for ill (a particularly scary creature often has a high Charisma score). A creature can have a very low Intelligence score and still be Charismatic. A Charisma score of 3 or lower indicates a creature that is barely sentient [aside: I think they mean "sapient"]. A score of 4 to 6 represents Charisma equivalent to that of a strikingly sullen, crass, or retiring human. A score of 16 to 18 reflects an unusually strong presence and force of personality. A score of 20 or higher represents superhuman Charisma. [3.5 MM1, 298]
From this, it is clear that Charisma is not a beauty stat: an exceptionally ugly character might have that appearance represented by a high Charisma too. However, if we read further into this paragraph we find that Charisma is not technically a measure of physical appearance at all.
Strictly speaking, Charisma represents a creature's interpersonal skills. That is, a creature with high Charisma knows how to use its beautiful or hideous appearance to influence others. A terribly ugly or astonishingly pretty creature that doesn't have control over how its features influence others has low Charisma, and a man with an ordinary face who is masterful at influencing people has high Charisma. Remember, Charisma is the base stat for both Intimidation and Diplomacy.
ASIDE: The D&D Fourth Edition description of Charisma says the same thing in many fewer words.
Charisma (Cha) measures your force of personality, persuasiveness, and leadership. [4e PHB1, 17]