Not every attempt in a series of repeated attempts get a roll
Just like the ranger saying, "I look around" doesn't trigger a perception roll if there is nothing to see, not every attempted spell cast doesn't triggers a roll. The dice are rolled when the player attempts to use an ability to move the fiction in a particular direction.
In this case, the player is attempting to "repeatedly try to cast invisibility until it works." That's the action which moves the fiction, so it gets a roll (and only one roll) to resolve it, not one roll per time invisibility is attempted.
That compound action can then be resolved just like any other roll. Here are some examples.
Success, with a Twist
The bottom line for this kind of outcome is that you did eventually make the party invisible, but it cost you something or had an undesirable side-effect.
Expend a resource (1) - You make the party invisible, but you suffer some backlash damage from your failed attempts.
Expend a resource (2) - You make the party invisible, but you accidentally disappeared some of your memories of magic as well. You temporarily lose access to some of your other your Boons/Banes (presumably ones which represent magic spells that you cast).
Overlook an important detail - You make the party invisible, but then get separated because you can no longer see each other.
Waste time - You make the party invisible, but it takes you all day (or several days) to do it.
Failure, but the story progresses
The bottom line for these outcomes is that you tried repeatedly to make the party invisible, and for some reason, cannot continue to do so because you've tried everything you can, or because something interrupted you, or because you think you succeeded, or for whatever other reason the GM comes up with.
The player finds an opportunity for success - You spend all day trying different incantations and different rituals to make the party invisible, but to no avail. That night, while you're ruminating on the problem, you realize that if you had some Unseelie Pixie Dust, you could cast a variant of invisibility that would do what you need. You're pretty sure you saw a trader in the last town you were in selling a vial of it.
The danger snowballs - After the third time you try to cast invisibility, the leftover magic from your spell accidentally creates some Shadows (or other world appropriate illusion-themed monsters, at GM's discretion), which immediately attack you.
The information is false - After a few attempts, your invisibility spell works. Unfortunately, it only works momentarily. You get halfway past the door guard at the front gate before your spell ends with a pop. You're in trouble now, but at least the guards are as surprised as you are!
A quick sidenote on Activation times
The Activation Time for Invisibility is one Major Action, which the Combat rules say can be done in "about six seconds of time". However, applying the rules for Activation Time outside of Combat will make it very hard to make Every Roll Matter (especially since one of the options for Success with a Twist is "It wastes time"), so I recommend treating Activation Time as a Combat rule and not using it for actions which are on a scale larger than a single Combat.