In the Lost Mine of Phandelver adventure module, our party (consisting of three second-level PCs, a fighter, wizard, and cleric, with me DMing) was fighting a singular ogre. We had already taken some damage, due to other monsters, but we were on reasonable amounts of health.

The wizard cast mage hand. Before doing this, he asked me if it would confuse and/or distract the ogre, and I said yes, ruling that even without the wizard moving it (the hand created by mage hand) every turn, it would apply disadvantage to the ogre's attacks. This allowed the party to take out the ogre, taking only 10 damage (or something like that) in the process.

What I want to know is, was my ruling correct by RAW, RAI, and if not (or if nothing is stated) is it reasonable by RAF, and could it be a reasonable house-rule?