The spell would work & the caster should immediately know something is up & easily figure out what it is!
The spell would work, and the caster shouldn't be confused for more than a second or 2 if at all.
If the spell fails (target succeeds it's saving throw) nothing happens. The caster knows they failed and they can't try again (on this target) for 24hrs. They can, however, cast another spell targeting any known friends/party members known to travel with them if they have such information.
If the caster is able to see an area but simply can't see the target of their scry they should know the spell succeeded and that they just can't see them for some reason. (this is similar to a dark room, the caster knows their target is there they just can't see them)
Scrying is a 4th level spell (minimum CL=7) while
Greater Scrying is a 7th level spell (minimum CL=13) and
Invisibility is a 2nd level spell (minimum CL=3).
Unless the caster had a recent run-in with an illithid he should have an idea that other magics are in play. To be 100% true to the rules you could have the caster roll a spellcraft check to identify that it's an invisibility spell (DC=22 for Invisibility or DC=24 for Greater Invisibility) but I'm also pretty sure a caster of this level has "seen" Invisibility cast before.
Additionally, there is a 5% per CL (minimum 35% chance @level 7) that detect Magic/law/chaos/evil/good & message) function through the scry. If successful, these spells will enable the caster to further identify what magics are at play and where the target is at, if not what they're actively doing.
*Note: corrected caster level to reflect wizard vs sorcerer