There may be problems with a straight import.
I'll do my best to make this answer more applicable than my first. One of the main hurdles I hit writing that one, which I believe may apply to other users, is that I'm not terribly familiar with Call of Cthulhu d20. With that in mind, I'd like to summarize the important bits we'll be dealing with. Once that's done, we can have a context in which to work with the levelless spellcasting system.
To start off with, CoCd20 has many similarities to 3.5 -- the bones of these systems are much the same. All characters have the classic 6 Ability scores, and most scores are between 8 and 20. The Ability Scores boil down to Ability Score Modifiers in the same way, and the modifiers are applied to skills and saving throws in the same way.
There are a few salient differences here. Among these: classes have been replaced with skill templates, and players can choose a defensive or offensive track for levelling. The difference is bonus to saves or full BAB scaling, so it's not too complex.
An important thing to note here is that we can usually expect differences in characters to be limited to about (ASM + 2), which may give a better idea of how things compare between characters.
However, the most important is the introduction of the Sanity mechanic.
From what I understand of d20 CoC, this is an important balancing feature, so I think it's important to understand how it works in order to contextualize the spellcasting mechanic. The other main balancing mechanic is Ability damage -- we'll get to that after Sanity.
Each character has a Sanity score, ranging from -10 (incurably insane) to 99 (paragon of sane). A character has a starting sanity of (5 * Wisdom1).
Maximum Sanity can vary from character to character, and is (99 - Cthulhu Mythos Modifier). In short, the more a character knows about the Mythos, the less stable they are.
When confronted or working with the Mythos, the player characters will occasionally come across disturbing things. In many cases, this results in a Sanity Saving Throw - a roll of d% vs. the character's current Sanity. The Save is successful on a result less than the character's current Sanity.
Individual events chip away at Sanity reasonably slowly -- finding a corpse is 0/1d3, and only damages on a failed saving throw, for example. A friend's death is 0/1d6; torture - 2d10; seeing Cthulhu - 1d10/d%.
Taking an amount of sanity damage that's greater than a portion of your Wisdom1 causes temporary insanity, and greater damage causes greater effects.
To give some perspective, pages 52-57 of the handbook are devoted to a vareity of psychological issues as reference for roleplaying.
This is not a d20-style system3.
Instead, it's a carryover from the earlier d00 CoC systems, and that presents some problems. My inkling is that it doesn't just carry the dice with it -- the Sanity system has a lot of inbuilt assumptions about how the characters interact with the world.
One of the most important assumptions therein is how 'heroic' the characters are. The most effective form of restoring Sanity in the handbook is sustained Psychoanalysis, which restores 1 point/week. This would be a fundamental disconnect with most D&D games I've been a part of, in which downtime is severely reduced. If you intent to import the levelless casting system without modification, you'll have to bear in mind all the assumptions that come with it -- namely , this slower-paced approach.
This definitely feels like a long-term balancing system. You can cast many spells before running up against your limit, but once you get there, it's really hard to recover.
Thankfully, Ability Damage is much more straightforward than the Sanity mechanic. In short, when casting a spell, the caster needs to front a temporary sacrifice. For many spells this is minor -- a point or 3 of Ability, meaning a 1 or 2 point reduction to their Ability Score Modifier.
Some spells (e.g., Call Deity) demand a higher price -- 20 points of damage. However, the spellcasting rules allow for this damage to be mitigated over several casters4.
Additionally, some spells may require permanent reductions to Ability.
Healing Ability Damage
According to the damage information5, temporary Ability Damage heals at the rate of 1/day. This isn't too far off of what I rule-of-thumb for D&D -- about 1 point per long rest seems about right.
t really feels like the Ability Damage is not meant to be a lingering things. In a game of CoC, where recovering the Sanity cost of a single spell takes about a month, recovering 1 point per day of Ability Damage is not too bad at all.
Indeed, it seems to be balanced around short-term tradeoffs -- after most spellcasting, a character should be back in good shape after about a week.
At this point, it seems like it may be fun to do some spell comparisons. Generally speaking, though, it seems like the spells are broadly similar -- Fear out of 5e seems similar to Cause Fear out of CoC. As balancing spells from a 3.5 environment is an entirely different essay, I'm just going to recommend using the equivalent 5e spells whenever possible.
The main differences are in how they're applied -- aside from casting costs, we're really considering the differences between how 3.5 and 5e do spells.
So, what are those costs?
Well, for Blindness/Deafness, 5e considers it to be a second level spell of the Necromancy school. For casting, it obviously takes second level slot.
CoC has the caster pay 3 Int damage and 2d6 Sanity instead of the slots.
Fear is a 3rd level illusion spell in 5e, and takes the requisite slots.
In CoC, Cause Fear takes 2 Wis and 1d4 Sanity.
So what does this tell us?
Well, the way I see it, we have two cases to work with when it comes to bringing in the spell system.
The first case covers spells that have 5e equivalents. In these cases, it's questionable. You'll need to bear in mind how characters interact with the sanity system, and play such that they have to interact with it (whether through spellcasting or not). However, I think it could be a great addition to a campaign.
You'd want to be aware of which classes might want which spells, and adapt spell costs accordingly. A rogue may be tempted to make use of Invisibility6, and the 2 Int damage might not put them off too much. Likewise, Healing Touch's7 low Wisdom and Sanity costs may lead to all non-Cleric characters becoming surprisingly competent healers.
Restricting long rests, and consistently reminding them of the limits of their sanity would be required to keep things from getting out of hand. Regardless, players might really enjoy the added freedom and power that it provides.
The second case is for spells that don't have a 5e equivalent. In these cases, it is of incredible importance that they are reevaluated and rebalanced. As an example, Flesh Ward7 grants damage resistance of 10/+1 for up to 50 points of damage. For the cost of 2 Int damage, and 1d4 Sanity, this is incredibly overpowered for fifth edition.
Bear in mind that many spells in the handbook fall into this category. In almost every case, you'll need to do a rewrite of the spell to ensure that it's even close to balanced. If you're hoping to get some guidance, it'd be useful to find a guide for converting 3.5e spells to 5e.
As other answers have mentioned, you'll want to make sure that your entire group is onboard with the flavor and tone of the game. You'll want to have everyone more-or-less on the same page, and it'd definitely help to have them open to possible changes.
A modification of the spellcasting system like this is huge, and should not be underestimated. It will most definitely change the balance of the game, regardless of how well it's implemented. Bear in mind that this isn't necessarily a bad thing, it's mostly just a thing. So long as you and your group are open to experimentation and willing to find a new level, I think this could be a lot of fun.
If you ever try this out, be sure to let us know how it goes!
1: Note this is the Ability Score, not the modifier
2: AKA "Chapter 4"
4: Multiple Casters, page 128
5: Damage, page 62
6: "Hide From the Eye", page 142 of the CoC handbook
7: Page 142 as well