Excalibur is the ultimate sword myth. It's not about the power of the blade but the ability to inspire others. Being worthy of Excalibur is being worthy to lead. To rule. Not being worthy of it is to be a pretender to the throne. The unworthy touching this blade will be the target of ridicule and hatred. Not because the magic in the sword is powerful, it is, but you are defiling something believed in. The sword could cover you in boils and it wouldn't be as bad as the look in a kid's eyes when he see's you reducing the holy sword to a pig sticker.
Rumors have a long tradition in D&D. I see no reason anyone messing with this sword should have any right to anonymity. You're about to be made part of a legend. Anything you do will be magnified out of all reasonable proportion. Didn't pay the beggar you just walked by? Obviously you oppress, abuse, and enslave the poor. It'll be like being on CNN.
Or
Rather than have the downside of wielding Excalibur while unworthy being subtle you could go completely over the top:
