Background: I have been playing and running Savage Worlds games continuously since 2009, using each current rules edition. I've used several weapon specialization setting rules in various campaigns.
Base Rules
The base core rules apply Athletics (throwing), Fighting, and Shooting equally to all weapons. Situational modifiers can apply, and Edges can make some extreme distinctions, but generally a character with Fighting d10 is a terror regardless of the melee weapon being held.
Note that this applies equally to all three of the weapon combat skills.
Core Rules Options
The Core Rules have a few ways to address weapon proficiency, beyond the relevant skills.
Familiarization: The sidebar on page 36 covers D&D-like proficiency. If the encountered equipment is completely alien (but obvious enough to be usable) then apply a -4 to all Trait rolls to use the equipment. If the equipment is just weird and unfamiliar then the penalty is -2.
Trademark Weapon: These Edges on page 44 represent a bond to a unique individual weapon. Replacing a trademark weapon is a a big deal, and time consuming, so it happens pretty rarely. Being unable to use these Edges can make a character significantly less capable without using multiple die types.
Skill Specialization: An optional Setting Rule on page 141, this follows a similar concept to your proposal. Skill Specialization applies to only selected Traits, so you can apply it to Fighting and Shooting and no other Skill. And if you find your group wants to expand the use, you can extend it to the various Arcane Skills during the course of the campaign.
Your Proposal
Any suggestion on how to appropriately model this? My intention is for players to chose a particular weapon when they pick “fighting” skill and then have it reduced with -2 for a “similar” weapon and by -4 for a “different” weapon.
This is pretty much the Familiarization sidebar. I have found that using it constantly is frustrating due to the frequent math. Also if extreme penalties are used, since they can turn a trivial battle into suicide.
A similar rule is the Skill Specialization setting rule. It works, and depending upon the campaign style and tone it might rarely come up or be a continuous issue. My experiences with it have been mixed, though it did encourage characters to stick with their preferred weapon style.
I’d also plan on letting the[m] advance on “similar” weapons by considering it a skill that is one dice type lower than his weapon of choice. For example, If the player has 1d8 on broadsword, they can bring scimitar to 1d8 and still raise one other skill on one advance.
That's a high price for a minor increase in weapon flexibility. Given the general rarity of Advances and the numerous alternative uses, I would expect no more than one player to use this option more than once. And I would expect that player character to remain less powerful in all situations compared to all other player characters.
My main concern with this approach (as I don’t have a lot of experience with the system) is whether this would unbalance the game compared to casters. As this is not officially in the rules maybe it breaks the balance between this two paths.
Why do you assume the paths are separate? The power of a fighter-mage or fighter-priest is well documented, regardless of the chosen weapon or combat style. (Personally, I have found archer-mage to be the most powerful but I haven't documented the difference.)
Characters generally, and casters in particular, have enough priorities for their Advances that the opportunity cost of gaining additional specializations is not worthwhile. New Edges, Attribute increases, new or improved Skills, and Edges that can only be taken once per Rank are all competing with new skill specializations - and the specializations almost always lose. The only time gaining specializations has been worthwhile is when they are both a) frequently used and b) connected to important tasks.
System Advice
For most tasks, including ranged combat and magic powers, the Target Number (TN) is 4. This makes bonuses very powerful.
A +1 is a large bonus in Savage Worlds. A Wild Card with +1 and a d4 Skill has virtually the same odds of success as a Wild Card with no bonus and a d10 Skill. (83.3% versus 85%).
A +2 to is generally a "guaranteed success" bonus.
This doesn't apply as much to Fighting because the TN is Parry, which can range from 2 to 17 without the use of powers. A Wild Card usually has a 50% chance of hitting a target with equal Fighting skill (Fighting d10 grants Parry 7, and a Wild Card with d10 has a 50% chance of hitting Parry 7).
