Your Constitution modifier can decrease your starting hit points
The rules on "Hit Points and Hit Dice" state:
[...] At 1st level, your character has 1 Hit Die, and the die type is determined by your class. You start with hit points equal to the highest roll of that die, as indicated in your class description. (You also add your Constitution modifier, which you’ll determine in step 3.) This is also your hit point maximum. [...]
Nowhere does it make any exception for negative Constitution modifiers. In fact, I don't know if there is any feature where you add a modifier only if it is positive.
Instead of ignoring negative modifiers, feature often impose minimums on the total such as when gaining health at level other than 1:
Each time you gain a level, you gain 1 additional Hit Die. Roll that Hit Die, add your Constitution modifier to the roll, and add the total (minimum of 1) to your hit point maximum.
Notably, a negative Constitution modifier still applies it just also gets effectively ignored if the modifier is sufficiently negative to change the total to a non-positive number.
Applying this sort of minimum to your starting hit points would be awkward to phrase, getting you something like "minimum of the maximum of your hit die", which clearly isn't the best wording (though that doesn't particularly mean much).
Because nothing explicitly states that a negative Constitution modifier doesn't apply and nothing states a minimum on your starting hit points, your Constitution modifier can actually decrease your starting hit points.
A possible example of what the section could have said
There might actually at least one feature that specifies a minimum on the modifier to a roll. The Artillerist Artificer's Protector Eldritch Canon states:
[...] 1d8 + your Intelligence modifier (minimum of +1)
It looks like this minimum applies to the modifier and not the total, though perhaps this feature is just weirdly worded and has a "+" to denote positivity for some reason... Regardless, such a wording does not appear in the section on starting hit points.