As has been mentioned in other comments, the developers have confirmed that there aren't any concrete rules for this, and whoever is DMing it will have to make a judgment call. However, here are my thoughts. Since it's a special ability (not a spell), the entirety of the rules governing it are covered in its description.
Starting at 6th level, you can vanish in a puff of mist in response to harm. When you take damage, you can use your reaction to turn invisible and teleport up to 60 feet to an unoccupied space you can see. You remain invisible until the start of your next turn or until you attack or cast a spell.
So, when you take damage, you have the choice to use your reaction to turn into mist. If you do, then congratulations! You're an ephemeral, invisible mist. (To answer your recent question: I believe that turning into mist acts as an effect that targets Self.)
Now that you're out of immediate danger, your misty-self must relocate to a new spot. When selecting the spot, three qualifications must be met. First, you have to be able to see it. Second, it has to be within 60 feet. Finally, it has to be unoccupied. Since we know that the rules don't adequately cover the edge case you're asking about (where the space in question is actually occupied, but you're unaware of it), then the DM has to make a ruling. There are basically three options:
- The DM decides to punish you for using your class ability, and declares that the whole thing is negated. You waste your reaction and your class ability, you're still visible, and you stay in the danger zone. (I dislike this option.)
- Same as the above, except the DM also decides to treat the attempt like dimension door, causing you to suffer 4d6 Force damage as you unknowingly try to move into an occupied space. (I hate this option).
- The DM decides to apply some creative reasoning, and refers to the descriptive text of the ability. Now the real job of a DM begins: taking the unique situation at the table and thinking outside the box.
While the rules state that this movement is treated as a "teleport," the descriptive text makes it clear that the narrative intent is that your mist form is moving through the battlefield without having to pay attention to things like difficult terrain, opportunity attacks, or partial obstructions (like prison bars or a wide chasm). However, unlike a "true teleport," you're not actually vanishing-and-reappearing. Instead, your mist form is quickly moving across the battlefield.
With that in mind, if you turn into mist and move toward the intended location, it makes sense that you would immediately be aware of a space that you cannot physically fill. You may not know why (other than the DM telling you that you can't), but that's okay—you're still in mist form!
Now that you know you can't fill that spot (because the DM told you/your mist-self wouldn't fit there), it's time to adjust your landing point accordingly. As long as your final landing spot is within 60 feet of your origin point, and visible from that point, then your mist eventually coalesces and waits until the start of your next turn.
If I'm the DM at the table, that's my ruling. The player gets to use their class ability in the exact kind of emergency situation it's meant for, and I'm spared feeling like a jerk for punishing the player for something that I was hiding from them.