Yes, it is obvious they have lied. And they can retry.
The Inveigler published on the Advanced Bestiary says:
Charming Falsehood (Su)
Once per day, an inveigler can tell a lie so convincing that it enchants a single creature that hears it. The inveigler need not be able to see the target of its lie or have line of effect to it, but if the target cannot hear the lie, this use of the ability is wasted. The lie must be one that would cause the target to view the inveigler as a trusted friend or that would make it likely to follow the inveigler’s orders, but it can be as outlandish as the inveigler wishes. If the target hears the lie, it must attempt a Sense Motive check opposed by the inveigler’s Bluff check. A creature that fails this check by less than 5 is affected as though by the charm monster spell (caster level equals inveigler’s character level). Failure by 5 or more means the inveigler has dominated the target as though using the dominate monster spell (caster level equals inveigler’s character level). Charming falsehood is a sonic, mind-affecting charm effect.
Decieving or Lying while bluffing has a retry condition, so it must be followed:
Retry? If you fail to deceive someone, any further checks made to deceive them are made at a –10 penalty and may be impossible (GM discretion).
If the attempt wasn't obvious, there would be no consequences of being caught lying, the character wouldnt take a -10 on future checks for being a considered a liar.
But they have no way to identify the ability
They know they have been lied to, but they have no way of knowing what ability was that. The Spymaster Handbook Player Companion introduced a new knowledge mechanic called Recall Intrigues:
You can attempt a skill check to identify a feat or class feature when you observe it in use, similar to how Spellcraft can be used to identify a spell. The feat or class feature must have some observable effect in order for you to attempt the Knowledge check. For example, you can’t see the internal determination of Iron Will, so this ability can’t identify that feat. In general, if a feat or class feature creates a noticeable effect (such as the extra attack from using Cleave) or has a variable modifier a character must choose to use (such as Arcane Strike, Combat Expertise, or Enlarge Spell), it can be identified. If it creates a static bonus (such as Dodge or Lightning Reflexes), there’s no telltale sign to give it away.
This could be used to identify an extraordinary or supernatural ability used by a class feature, but not those used by a monster. If this monster was known for using that kind of ability (like a siren or nymph), you could try to remember about that by making a knowledge check and use Monster Lore, but this is not the case as this is a template.
Which means they have no way to identify Charming Falsehood as an ability.