Unless the supernatural ability in question has a visible effect, such as causing claws to grow, breathing fire, creating an area of magical darkness or making the target larger. or turning him into a frog...
There is no visible or easy way to tell if your Evil Eye succeded
Supernatural abilities are very similar to spells, in that they are magical in nature, but they are not spells. So mundane methods to identify spells will not work against Su abilities.
For spells we got these rules that are clear:
Succeeding on a Saving Throw: A creature that successfully saves against a spell that has no obvious physical effects feels a hostile force or a tingle, but cannot deduce the exact nature of the attack. Likewise, if a creature's saving throw succeeds against a targeted spell, you sense that the spell has failed. You do not sense when creatures succeed on saves against effect and area spells.
But we got two paragraphs explaining how Supernatural Abilities work and are different from spells:
Supernatural Abilities (Su)
Supernatural abilities are magical attacks, defenses, and qualities. These abilities can be always active or they can require a specific action to utilize. The supernatural ability's description includes information on how it is used and its effects.
Supernatural Abilities (Su)
These can't be disrupted in combat and generally don't provoke attacks of opportunity. They aren't subject to spell resistance, counterspells, or dispel magic, and don't function in antimagic areas.
That meaning, the only visible way to tell if the target is being subject to a supernatural ability, is detecting their magical aura.
But you would have to cast detect magic, use your evil eye, then see if their aura changed. Otherwise, the hex has no visible manifestation and is impossible to identify.
You can, however, identify that a witch has used her Evil Eye on someone (or on yourself), using a Knowledge(Arcana) check. The Spymaster Handbook Player Companion added some guidelines on how to identify class features being used.
This ruling is also supported by this answer.
Supernatural abilities can only be guessed by their effects, like the examples earlier, but what exactly they do should be unclear unless you somehow identified what causes the effect.
Such as when you see a person sucking the blood out of a villager, your first guess would be "a vampire!", but there are quite a few types of creatures that can do that, it could even be somebody with a weird taste of blood, and they all look humanoid and have similar abilities. In this case, you need to identify other abilities or features of those creatures (pointy ears? ugly face? aura of fear?).
Knowing the differences between Zombies and Ghouls can save your group from a total party kill.
For Evil Eye, there is no such visible effect. The description of the ability says:
The witch can cause doubt to creep into the mind of a foe within 30 feet that she can see.
So, there is nothing visible going on, the target simply has doubt on his mind and will suffer the penalties for your hex. The only thing you could do that will clearly tell you that he has doubt is to attempt to read his aura, using the Perception skill unlock from Occult Adventures.
You can also attempt to detect the thoughts of your target and see if he has doubt about the situation.
But do note that there is nothing that says detect doubt, so you will have to check with your GM before investing on it.
On the other hand, being so difficult to identify means you have the upper hand when attempting to remain undercover, the target will have doubts, but will not know exactly why. Meaning that you can apply Evil Eye on pretty much anyone that they will not know what you are doing (sans the knowledge check i mentioned before), as your Hex has no visible effects.