RAW, the ranger learns its actual vulnerabilities.
Let us only consider rules as written. First consider the ability:
At 3rd level, you gain the ability to peer at a creature and magically discern how best to hurt it. As an action, choose one creature you can see within 60 feet of you. You immediately learn whether the creature has any damage immunities, resistances, or vulnerabilities and what they are. If the creature is hidden from divination magic, you sense that it has no damage immunities, resistances, or vulnerabilities.
As we can see, there is nothing in the ability description, that would limit its function in the case of creature masked with an illusion.
Next, let's have a look at some illusion spell. Lvl 5 Seeming seems to fit the bill (it's quite a long description, so just a few excerpts):
This spell allows you to change the appearance of any number of creatures that you can see within range. You give each target you choose a new, illusory appearance
The changes wrought by this spell fail to hold up to physical inspection.
A creature can use its action to inspect a target and make an
Intelligence (Investigation) check against your spell save DC. If it
succeeds, it becomes aware that the target is disguised.
Again, the spell (chosen as an example) says nothing about limiting anyone's abilities to magically discern its vulnerabilities. So when the ranger uses his ability, he gets the information he expects.
...but this is narratively unsatisfying.
To have more fun/fluff/drama/etc., the DM might choose to handle the situation a bit differently. One might for example see the usage of the Hunter's sense as a good substitute for the investigation check against the illusion spell, thus allowing the ranger to see through the illusion — mind that this is not strictly supported by the rules.
On the other hand, the DM might rule that the illusion blocks the ability until the illusion is broken (via the succesful investigation check or physical interaction) — again, not strictly supported by the rules.
Finally, the DM might give just a bit more information to the ranger, e.g. "Your Hunter's sense is telling you, that it is vulnerable to fire damage, but something feels off with this creature." This would hint at the illusion without mechanically changing much. Again, this is also not supported by a strict reading of the rules, but also doesn't really break anything.
But what if there is no creature?
If there is no creature, only illusionary image, there is no target for the ability. So it fails. As can be seen in other questions, using something with the intent of it working (e. g. targeting an invalid target), seems to waste the resources all the same, even if the target is in fact invalid. This is getting into the shady areas that are not very well handled in the actual rulebooks, so the DM might be more or less harsh, but the expected effect is somewhere around: You learn nothing. You lose one use of Hunter's sense.
Please note that I would consider it a different situation from the creature being shielded by divination magic, so you actually learn that you learn nothing, instead of falsely learning it has no vulnerabilities. But that is rather my personal opinion.
