This is the wonder of RPGs, they are flexible. But they are not without their trade-offs here.
Probably not an issue
- Spell DCs
- Spell Attack modifiers
- Physical vs Mental
Let's break these out.
Physical vs Mental
Casting is generally tied to one of the three Mental stats (Int, Wis, Cha). If you allowed an Eldritch Knight Fighter to make their casting stat Str this would probably cause power issues because you would be letting them "double-down" on their best stats. Likewise for Arcane Tricksters, Paladins, Rangers.
The game technically has six saves, but Dex, Con & Wis are really the most important saves. If you look at the Save Proficiencies by class, each class only gains proficiency in one of these two saves. They also typically gain proficiency in the save that matches their best stat.
So Sorcerers get proficiency in Cha saves making them really good at Cha saves. Rogues are really good a Dex saves, Clerics are really good at Wis saves, etc. If you make a Wis Sorcerer, they become "pretty good" at Cha saves, but also pretty good at Wis saves.
This definitely changes the nature of the class. You normally don't target a Cleric or Druid with a Wis save because it's probably not going to work. But now it's not so clear. This will affect both PCs and NPC casters.
Available skills tend to be connected to a character's primary stat. So Clerics get more Wis skills and Wizards get more Int skills. If you switch this around, it really reduces the odds of having someone particularly skilled at one of these disciplines. If the Cleric doesn't have a good Medicine score, then who does? Same for Arcana?
Fortunately, some of these can be made up for with specific Backgrounds, but it does hamper specialization usually connected with the given classes.
There is a specific history behind the various casting stats. Int casters had to study, Wis casters are divinely inspired, Cha casters are "naturals". When you switch this stuff around, you're also implying a different world vision than the "classic" D&D.
This is not "bad" per se. But it is significant. Why does a low Int Wizard use a spell book? Are they really going to be book worms if they use Cha to cast? If Sorcerers use Int to cast, but don't need books is there a reason for that? If Clerics use Int to cast, then where does that whole "divine inspiration" thing kick in?
These questions can clearly be reconciled, but they are meaningful. They say something about your game world that is different from say a classic Forgotten Realms world.