Just an out-of-the box suggestion, disregarding magic items :

Disguise self does trick vision, but not the other senses.

To start rumors, you could use :

1. Dogs intrigued by the specific scent, coming to sniff at your character - and running in fear once they touch you.
2. A local blind man, intrigued by either the unusual scent, or the loud steps heard on the pavement - asking around if an ogre just arrived in the town.
3. This testimony could even be approved by a local huntsman, who found a disproportionate footprint in a muddy street.

Once rumors have spread, someone curious with the "observant" feat should be enough to pierce your disguise consistently.