Yes, if the weapon is stored in a container.
If the weapon is not being wielded by the opponent and is in a container they are wearing or carrying (such as a backpack), it counts as any other object and is eligible for retrieving using the second bullet point of Mage Hand Legerdemain.
If you want this to go unnoticed, then you will indeed need to contest your Sleight of Hand skill check their Perception check, but you don't have to if you are okay with them noticing.
No, if it is not stored in a container.
The requirements for the feature are very specific about the object being in a container. If the weapon were strapped to the opponent's back or leg, for example, you would not be able to use Mage Hand Legerdemain to steal it.
This makes sense thematically because something the opponent can reach for easily is far more attended than something the opponent would have to search for in a container.
For more details about what constitutes a container for these purposes, see the following related question with several different interpretations in the answers: Can an Arcane Trickster use Mage Hand Legerdemain to steal the bolt out of a crossbow?