I'd like to build a magic system
in which every spell is a one-off.
Learn and burn, got it. A way to control this aspect is with the cost of learning, and/or the cost of burning (involving actually destroying the information once it has been used)
"I'm thinking that the time and place of the casting would be important in how the spell was constructed and so a mage couldn't simply trot out the same ritual and get the same effect each time, because this would probably lead to players 'inventing' the fireball spell and then just using it at every opportunity."
The Sun, Moon, and Stars are a great way to alter what effects come into play. You could literally use the night-sky as your proverbial dice and base the outcomes on the time of night and the position of constellations, real or imagined. It would literally take them eons to get the same result, and as such, would thusly require different spells or spell components (physical or otherwise) to acheive the same spell they completed hours, if not minutes ago. If you'd like something a bit more observable, you can use the real world time to affect the different aspects of the spell effect, whether that ice ball becomes a fire ball, or that mind control becomes charm or berserk effect. The balance in this case would come from the position of the stars being favorable or unfavorable.
How do you apply constrains to such a system such that more powerful characters can get more powerful results while not specifically defining what effects are available?
On one end of this equation when you first create spell effects, like in Shadowrun's latest edition, you have to acquire rather expensive spell components.
(NPC Bob tells you "Pure Eye of Newt Extract goes for 500 nuyen these days, up from a measly 20 and change! It's highway robbery (Occasionally)," ).
A more powerful magic user ("Adept", "Magician") can afford those more expensive spell components by virtue of their previous success (power).
On the other end of this equation is the "repair" cost for casting. More powerful magicians could end up paying a smaller cost for the same effect of a less powerful magician, or have the option of paying more of a resource to have a bigger effect.
A friend of mine created a "Nomenclature Casting System" involving "true names" of people, objects, ideas, locations, and the like. Each true word had a value. Bigger constructions of sentences had a bigger value and thusly did a larger effect.
You could try something similar in requiring a "true name" for each spell that is consumed by the caster for that particular target. (The Balance in this case comes from being price prohibitive.)
If that won't work for you, perhaps I can suggest randomly generating each spell from a list each time it is cast. Let the magician choose the element, roll for the different aspects of the spell, let them see what it can/will do and let them pick how to attach it to the world at large. In this instance, most things are available, and are once again dependent on your general ability with magic. As the character's power grows, let them add more aspects to the list and slough off the used ones into the trash bin. The Balance in this case comes from what you are willing to allow the casters to use as a list of effects.
For instance, PC Pryogolath wants to burn NPC Bob to a crisp! He begins summonning.
[Pyrogolath chooses the element 'Fire'. He rolls a d10 and the adjective "Melt" is chosen. He rolls another d10 and the noun "Head" is chosen. He rolls a final d10 and the range "Touch" is chosen. Pyrogolath understands the complex magic that he has just casted and can see its requirements to finish the spell and its effect upon success.] Pyrogolath's hands become redder, and start to glow, bubble, and drip with lava. The next thing he Touches in the Head will Melt in a pool of Lava.
This style of magic changes with each casting and should keep things rather new, pending that you are able to continue adding words to the list and keep track of ones that are used already. Words that are lower on the list could have more power, or that could also be randomized within a rage appropriate for the caster.
And if THAT doesn't work for you (too "crunchy", perhaps?) you could have the players come up with an adjective to describe the action. The better the adjective, the better the bonus. It's incredibly ill-defined, limited to what words you accept, and even nebulous in what sort of benefit it gives the character. (And it makes you break out the Thesaurus!) Once a player has used that word, it can't be used again, or can't be used until some sort of time limit expires. In this idea, everything is available, and power is only limited to the bonuses you give for creative word usage. (It has limited use in a continuing campaign, but breaking out dictionaries and thesauruses will make everyone at the table look like they are reading from tomes. Added bonus!) The balance in this case comes from the completeness of the source they use, as well as a consensus between the other party members and the GM him/herself.
(ex: Magician Crytos the Magificent says "I sublimate his ice armor into steam," [GM >grants +3 for a 3 syllable word and +2 for obscurity!])
If you're looking for something more punishment based, you could borrow from Changeling the Dreaming (White Wolf Publishing) and lower one of the good stats/raise a bad stat ("Banality") of a character for repeating the same action in the same manner over and over again. This allows some repetition, but has the chance to seriously damage the character.
Actually,... I'm going to go ahead and more fully recommend you use Changeling the Dreaming's Magic system and just rename Banality to something more appropriate for your campaign. You get a sphere of influence and the ability to affect a certain type of target. Call "No repeat-sies: if you do, it fizzles," and watch them turn trees into houses and flip people like quarters. The system is already put together. I have had first hand experience with it, and while a few things may be confusing at first, the system gets easier to use over time. "C:tL" uses "Wyrd" as a "mana pool". Out of Wyrd? Out of Mana. You can opt out of spending mana in the first place by following a particular Ban, negating its cost. The Balance in this case comes from difficulty of skipping costs (For instance, Eat naturally formed ice before wreathing yourself in an aura of flame -- Great for Summer/Desert campaigns).
So, in summation, you could:
make it dependent on the motions of the heavens (month/day/year/astrology) or the hands of the clock.
make it a randomized list within a range of possibilites and scratch off the ones used.
make the characters describe it and never use that description again.
Hack away at Changeling: the Dreaming for its magic system and rename things that don't make sense.