5e, as it is normally played, doesn't contain many opportunities for players to permanently weaken their characters in exchange for temporary benefit.  Adding this mechanic opens up possibilities for annoying player behavior.

For example: what if a player uses lots of magic in a short period of time, deliberately driving their character insane in exchange for power -- and then, when their character dies, brings in a new character who does the same thing?  Essentially this player would be treating their characters as disposable, taking advantage of the ability to bring in new characters at will.  This could be hard on your story and hard on morale.

What if a player uses lots of magic in a short period of time, deliberately driving their character insane so that they will have an excuse to attack the other characters?  What if they say: "It's not my fault I opened a portal to the Outer Realms and got you all killed.  The insanity rules made me do it!"  (This has happened to me, though it wasn't specifically tied to an insanity mechanic.  My campaign contained a thing that the characters must never do or it would end the world, and one player had his character commit suicide in order to do it, just to troll us.)

What if a player uses a lot of magic, for a good reason, and then it slowly sinks in that they've permanently weakened their character?  What if they decide they don't want to play a permanently weakened character, and they lose interest in your game as a result?  (This has happened to me: I added an insanity-magic mechanic to a 3.5e campaign I was running, one player started using it, and then just as his character was about to go insane he vanished from my campaign and stopped responding to emails.)

I haven't played Call of Cthulhu, so I don't know how that system normally handles this sort of thing.  My guess is that you develop behavior norms, and when someone violates those norms you remove them from the game.  The worry is that your D&D players might not have the same behavior norms, so they might be more likely than normal to behave annoyingly if you suddenly expose them to insanity-for-power tradeoffs.

Of course, if you have good players, they will roleplay their characters' desire to not go insane, and you'll probably be fine.