(Disclaimer: the power of the illusions are controlled by the DM unlike most spells, the reason is because illusions might be too powerful and game changing for so little. Therefore, the best course of action is to ask the DM about your question, if you are the DM here are some guidelines but at the end is up to you)
Narrative wise
From the DM to the players, the way they present the illusion is the way it works (mostly, rules of the spell still apply). This is very important, specially in how the magic works in the world, for example, shield: does it make sound when you cast it or while it is active? What about light or shimmers? Some illusions like silent image and minor illusion+ don't make sound. A beast probably won't question why the wall that suddenly appear didn't make sound, but an intelligent creature familiar with a tiny bit of magic, yes, it would.
An interesting spell, narrative wise, is major image. Since major image can create illusion with smell, temperature, and sound, etc, the narrative of the spell should reflect the illusion presented. A wall emerging from the deeps of the earth making crumble sounds and the characteristic smell of earth, dust and cut grass will be way harder to disbelieve than a wall that just appear.
It just odd
This is the most basic way of disbelieving an illusion. Something might be misplaced and should not be there. For example, a cardboard box in the middle of a room undoubtedly will attract attention but the same box in a corner won't be noticed as much. Another example would be that the caster was too greedy and created an illusion so big that there wasn't room for a fake shadow. A giant guppy floating in the middle of the battlefield is a tell too. An example list would be:
- The illusion is in a place where it attracts attention
- The action used by the illusory creature is out of its common behavior
- The spell does not make sense (the wall just pop in from thin air with any "common" warning, like displace air current and so on)
- The creature do nothing at all relevant, e.g. it does not attack.
Failed roll
Some DM will ask for roll checks when you want to create an illusion of a creature that you have not seen or that you usually do not use. These checks vary from illusion to illusion, the most common are arcane, nature and religion. A failed roll just increase the oddity of the illusion, it does not make an automatic failure, the "enemy" still have to pass the corresponding check if the illusion seems odd for him. For example, you know of a giant floating fish, you have seen one once, but you fail (nature roll) to recall the details, but the Orc that you are facing only heard of tales of such creature, he have to pass an history check to remember details of such creature. On a failed, the big fish is safe. This goes for hallucinatory terrain and other spells that require some degree of knowledge.
Phantasmal Force
This is a special case, mainly because of two particular things:
- While a target is affected by the spell, the target treats the phantasm as if it were real.
- The target rationalizes any illogical outcomes from interacting with the phantasm.
If we consider the initial save roll as failed, there is very little that the affected target can do for itself. Number 1 take care of things like thinking it is to odd for it to exist or to be real. Number 2 is particular problematic, since it make any interaction feel real and in order no matter how illogical the reaction or action was. But, there is a solution to this problem, third parties. Since any third party won't see what the affected creature sees, it is very easy for them to connect the dots and establish that something is very wrong with him. They can yell it's in your head or it's an illusion, they can ask what is wrong and latter say there is nothing there or the like. In that case, the affected target have a reason to suspect that it might not be real and act accordingly.