I actually agree that it makes sense that you could have a musical instrument that doubled as both a bard focus and an arcane focus, but it wouldn't work the way you seem to think it would.
When you picture a bard using their instrument to cast a charm spell, how do you envision it? More likely than not, you picture them playing the instrument and weaving the magic into their song.
Now picture a wizard casting a charm spell using a staff as their arcane focus. More likely than not, you picture them raising it aloft, or swirling their hand over he top of it as they mutter an incantation.
So, if you were going to use the same item as both your arcane focus, and your bardic instrument, it would be employed differently depending on how you were casting the spell and therefore you would not gain the bard's ability to impose disadvantage on charm resistance rolls for your wizard spells.
I also believe (and this is my personal opinion) that anything employed as more than one sort of focus ought to be ironically appropriate for both spellcasting types independently, which makes using an instrument as an arcane focus a tricky prospect. The first potential example that springs to mind is using a drum as your instrument, and having one of the sticks double as a wand. With that example can you see how each would be employed differently depending on what type of magic you were using?
Similarly, if a cleric/wizard had their holy symbol carved onto their staff to serve as spellcasting focus for both their divine and arcane spells, I would have no problem with it. However, when casting as a cleric, the important part of the focus would be presenting the holy symbol, whereas the important part for wizards would be the movement of the staff.
To clarify, I acknowledge that the rules never specifically say that the bard must play their instrument in order to impose disadvantage vs their charm spells, but when it says they use their instrument, I have a hard time picturing them doing anything else with it to justify the ability (although charming someone by smashing a guitar over their head does have a certain appeal).
Now, because I am a DM who always looks for a way to tell my players "yes", here is my suggestion to fix your problem. It seems to me that the only reason you want to dip bard is to get this ability, but there are a number of options that could grant you more or less the same benefit without a dip into bard. Maybe your DM would be willing to let you quest for a magic item or arcane secret that causes your charm spells to be hard to resist. If you are wanting a high charisma character, maybe your DM would allow you to roll a Charisma(Deception) skill check to disguise the fact that you are trying to cast a charm spell on your target, thereby allowing you to impose disadvantage on your target because they are less likely to realize what you are trying to do. There are lots of interesting ways to gain a similar advantage that could potentially be much more satisfying than dipping a level in bard, and you don't have to be a level behind in your main class in order to do it!