From the armor table, the only difference I can see between being unarmored and wearing adventurer's clothing is that the latter imposes a dex cap (+5) while the former does not. Yes, the latter also has the Comfort armor trait, but given that describes armor being sufficiently comfortable that you can rest normally while wearing it, I'd imagine "nothing" is also fairly comfortable.
So what's the purpose of adventurer's clothing? Does it have any impact at all on environmental effects? If a monk has a +6 dex modifier, should they just run around in their birthday suits?
Is it only that adventurer's clothing can have potency runes, while "unarmored" cannot?