It was an attack.
Taking an action to directly inflict damage, resolved by rolling a d20 compared to AC, is an attack and it should be considered an attack for DM adjudication. Whether that action is firing a bow, swinging a sword, launching a spell, or dropping a rock on a specific person, it is an action to directly inflict damage on a target and is resolved by rolling a d20, applying bonuses, and comparing to AC.
The most common action to take in combat is the Attack action, whether you are swinging a sword, firing an arrow from a bow, or brawling with your fists.
With this action, you make one melee or ranged attack. See the “Making an Attack” section for the rules that govern attacks.
Certain features, such as the Extra Attack feature of the Fighter, allow you to make more than one attack with this action.
Under "Making an Attack" (emphasis added):
If there’s ever any question whether something you’re doing counts as an attack, the rule is simple: if you’re making an attack roll, you’re making an attack.
Familiars can't make attacks, therefore they can't make attack rolls.
The DM is the final authority of the game session, and if the DM allowed the action then it was fine in that instance. But don't count on such favorable cheats in the future.