The SRD states clearly (whether you are a Tavern Brawler or not) that:
...you can use an unarmed strike: a punch, kick, head-butt, or similar forceful blow (none of which count as weapons). On a hit, an unarmed strike deals bludgeoning damage equal to 1 + your Strength modifier.
And when you take the Tavern Brawler feat:
Your unarmed strikes use a d4 for damage.
That means the "1" in the above formula (the base damage) changes to 1d4 -- it does not say that it takes away the Strength modifier, so you still keep that modifier.