On the ranged side, there's really only the sling. You could theoretically go whole-hog on ranged Fighter, taking the Archery style to bolster your attack rolls, and just accepting that your damage isn't going to be fantastic.
There are three decent options for melee combat:
Learn martial arts
If you take at least one level of Monk, you'll gain the Martial Arts class feature, which allows you to use your Dexterity modifier when fighting with unarmed strikes and "monk weapons". Monk weapons are all the simple melee weapons that aren't two-handed or heavy, plus short swords. The mace, quarterstaff, light hammer, and club all fit your needs; personally I'd go for the quarterstaff because you can use it in two hands to get a 1d8 damage die. (While your Martial Arts damage die starts at 1d4, it's optional; you can choose to use the weapon's standard damage instead.)
I believe your unarmed strike would also work for your purposes (dealing 1d4 damage due to Martial Arts), should you happen to be caught without your trusty stick.
This will also allow you to throw hammers and make it work with your plan. While light hammers normally use strength when thrown, they are valid monk weapons, so you could use Dexterity when you hurl them at people.
This seems like the best option to me; it has a lot of flexibility and some good side benefits, and it doesn't involve using magic, as you requested. One down-side is that you can't wear armor or use a shield with this method. You'd be focusing on Dex anyway and can add Wis to it via your Unarmored Defense, so with decent stats, you will probably have a respectable AC, but probably not as high as a traditional sword-and-board fighter. You won't be able to use most of the Fighter styles, though. Your best bet is probably Great Weapon Fighting if you're going with a staff, or Dueling if you're using a smaller weapon.
The second easy way to deal bludgeoning damage using wis/cha/dex would be to learn the shillelagh druid spell, which is magic, but it's not too costly. You could take a one-level dip into Druid to pick up the cantrip, or grab the Magic Initiate feat and avoid the hassle of multiclassing. (Although, given you're already pulling in two feats for this, you may not want to spend a third!) That will allow you to use a club or quarterstaff to fight using Wisdom. This also makes your weapon deal 1d8 damage even if you use it one-handed, which is nice.
Note that the spell is cast as a bonus action, so you don't have to give up an Attack action to activate it. As a fighter, you won't have a lot of competition for what to do with your bonus action anyway, unless you pick a couple of very specific Battlemaster maneuvers.
Multiclassing into Druid does put some limits on your armor choices, though, so you should consider the AC loss due to not wearing metal armor.
Strike a bargain
The third way is also technically magical in nature, but does not involve spellcasting as such. If you multiclass into Warlock and choose the Hexblade patron, then you can wield a bludgeoning weapon that uses your Charisma for attack rolls. The Hexblade comes with martial proficiency, so you could use the warhammer or flail in addition to the simple bludgeons. With those weapons, you can go with a shield and hit for d8, or go two-handed and get the d10 from your warhammer.
Unlike the other options, this method doesn't impact your AC, which is nice, but it does come along with some spellcasting ability that may conflict with your design intent.