UPDATED ANSWER
That is correct. For perception, your passive score is your floor. The roll is to see if you can get higher.
Jeremy Crawford was interviewed and discussed the Stealth mechanic (about 9:00 in) and he states that for Perception, your passive is your floor.
Whether or not this is considered RAW is possibly up for debate (it was in an interview and not published on released articles/books/twitter and other game designers have said the opposite on twitter - but they do not have Crawford's ability to give "official" rulings.)
In addition - this appears to only be true for Perception. Other skills do not necessarily have a passive score that is a floor. I can see arguments for each skill, but it will be up to the DM as to whether or not there is a floor for an active roll.
PREVIOUS ANSWER
This is currently wrong by the above, but keeping because I think it still applies.
I'm still keeping this here because the rules seem to support it outside of the above quoted interview. Applying a 'floor rule' to one skill but not the others doesn't make a lot of sense to me. It's creating an exception in the rules, or it's creating a new rule where every skill has a passive floor.
Passive Perception is not a Floor to your skill, it represents a different type of check
Passive Perception
A passive perception check is (PHB pp175, emphasis mine)
a special kind of ability check that doesn't involve any die rolls. Such a check can represent the average result for a task done repeatedly, such as searching for secret doors over and over again, or can be used when the DM wants to secretly determine whether the character succeed at something without rolling dice.
Active Perception
If you as the DM are calling for a die roll, it doesn't matter how a passive score is processed, it only matters what that character is actively doing. In this case, the player is (PHB pp 178)
actively trying to detect the presence of something.
And that is determined with their Perception Roll (plus any proficiency). The passive score is completely separated from this particular check.
Is this a narration problem?
It might be! You can add fluff around the passive test vs active test if needed. But just because someone is good at something passively doesn't mean that it becomes their base.
Some PCs can be innately good at things and do have a floor to their score
Rogues get Reliable Talent at level 11 (PHB, pp96) which sets their ability checks they are in proficient to a base roll of 10. If passive scores do represent a floor, it makes this boon not particularly beneficial (especially as you now have a floor generated for all abilities and not just those with which you have proficiency.)
