Don't mention the M word
You say:
I know that they could have magical effects to them (as stated in the Magical Item sections in the DMG) but that's not the situation I'm talking about. I'm talking about the same (or equal, I mean a really well crafted sword could still be better than a badly crafted greataxe) weapon (non-magical), just a way "cooler" one.
I completely understand where you're coming from; not everything needs to be magic. That said, the rules broadly list everything that isn't the standard version of an item, as a magic item, and this gives us a good framework to use for balance.
Take a standard +1 magic weapon for example. This is an Uncommon magic item that doesn't require attunement, and as such the DMG suggests it is suitable/balanced for PCs of 1st level and up.
Now what if your PCs found a masterwork (nonmagical) weapon that was so finely honed that it was easier to swing (+1 to hit) and cut a little deeper (+1 to damage) than normal? In terms of game balance, this is exactly the same* as a magic +1 weapon; but you get to flavour it as high-quality instead of magic.
I would certainly suggest using existing magic item rarities as a guideline for the numerical balance, then designing your own "well-crafted sword" effects that are numerically similar.
Alternatively: embrace fluff
Or, you can go in a completely separate direction, and add features that don't affect rolls at all.
Maybe the sword was historically wielded by the ruler of the nation, and gives the wielder some political clout/claim (if your players like that angle). Maybe the sword makes an unnaturally pleasing sound when drawn. Maybe it makes the wielder feel closer to a particular god/nature spirit. Maybe it just has a good history associated with it, and the player gets to continue that history (e.g. it has always been used in time of Dire Need to defend against attacking giants, in a campaign against giants).
A lot of these things are just "cool", and as a player I'd happily exchange my misc. starting weapon for something that feels cool and more meaningful; even if it doesn't give any numerical bonuses. For many players, D&D is primarily a story-telling device after all.
Or, somewhere between the two
Finally you could pitch your tent between the two options above, and offer concrete effects that don't directly affect attacks. For example:
- The weapons sheds light that can be turned on and off at will
- The weapon makes the hero's cape and hair flow magnificently when they pose
- The weapon plays a theme tune when drawn rapidly (which may or may not give a small bonus to the party such as a bonus to wisdom saves)
- The weapon does double damage to items (OK, it's a damage bonus, but not one that really comes up in combat)
- The weapon slows your falls, as if Feather Fall were always cast on you
- etc.
Most of these would be seen as a direct upgrade to a basic weapon (a greataxe that does 1d12 damage, or a greataxe that does 1d12 damage and makes my eyes glow red?), and you can likely have a great deal of fun coming up with options that would appear to your players' (or their characters') individual quirks.
Now arguably many of these would appear to be somewhat magic due to producing explicit effects. For me, the default high-magic setting of D&D means that a high-quality crafted item would likely have a little sub-cantrip-level enchantment/glamour in it. If that isn't to your tastes however, you can likely flavour these as purely mundane (e.g. the light comes from a glowing core at the base of the shaft, which can be covered or revealed by an opaque sheath; the music comes from violin-like strings inside the scabbard which are plucked by special notches on the blade...)
*as I've written it, there would still be the difference that it wouldn't pierce resistance/immunity to non-magical weapons. Depending on how you want things to play out, either you can say it's made of something like adamantium and so does pierce that immunity anyway; or you can leave it be, and use monsters with this feature when you wish to challenge these players in a particular way.