Make the weapon have powers that can't be used (or even known about) by a lower-level player. Frodo wielding the One Ring could only do so much with it -- Gandalf wielding the same ring would be far more powerful.
When Thorgrim the Bold (level 2 fighter) finds a fine steel blade (strangely untouched by the ages) in the barrow of the old king, it seems to just be a longsword. When Thorgrim's party travels to a goblin delving, he discovers that it's +1 against goblins. As he advances to level 3 or 4 or 5, he finds that the sword is now a +1 against giants and ogres as well. At level 10, the sword awakens and begins to whisper to him, whispering the secrets of the old king who once wielded it.
There's no reason the nature of an item has to be fully known by the players from the start. It's old, it's strange, it's otherworldly -- and therefore it's not fully understood.
If you're worried about players discarding a sword that they've decided must be just a plain old longsword, give it more ambience, more interest. It's easy to toss aside a sword you found in a pile of junk in an orc's hut. It's hard to throw out a sword you found clasped in the bony hands of the dead king in his tomb, a sword engraved with runes of a language no one can recognize, a sword made in a long-lost style. The orc's sword and the king's sword might seem to perform the same in battle, but the king's sword has far more interest.