In the Lost Mine of Phandelver adventure module, our party (consisting of three second-level PCs, a fighter, wizard, and cleric, with me DMing) was fighting a singular ogre. We had already taken some damage, due to other monsters, but we were on reasonable amounts of health.

The wizard used mage hand to take up the last of his spell slots. Before doing this, he asked me if it would confuse and/or distract the ogre, and I said yes, ruling that even without the wizard moving it every turn, it would apply disadvantage to the ogre's attacks. This allowed the party to take out the ogre, taking only 10 damage (or something like that) in the process.

What I want to know is, was my ruling correct by RAW, RAI, and if not (or if nothing is stated) is it reasonable by RAF, and could it be a reasonable house-rule?