Combat familiars are a class feature of Pact of the Chain Warlocks.
The basic familiar normally available to users of the spell find familiar is not well suited to battle, as you have observed. There are things you can do, such as using shield of faith, but even then, the familiar is still so squishy, your spell slots are probably best used elsewhere.
Useful combat familiars are basically a class feature of Pact of the Chain Warlocks. The Pact of the Chain warlock feature gives new options for familiars that are significantly more useful than the vanilla familiars, and gives them the ability to attack:
When you cast the spell, you can choose one of the normal forms for your familiar or one of the following special forms: imp, pseudodragon, quasit, or sprite.
Additionally, when you take the Attack action, you can forgo one of your own attacks to allow your familiar to make one attack with its reaction.
This feature can be further improved with the Eldritch Invocation Investment of the Chain Master:
- The familiar gains either a flying speed or a swimming speed (your choice) of 40 feet.
- As a bonus action, you can command the familiar to take the Attack action.
- The familiar’s weapon attacks are considered magical for the purpose of overcoming immunity and resistance to nonmagical attacks.
- If the familiar forces a creature to make a saving throw, it uses your spell save DC.
- When the familiar takes damage, you can use your reaction to grant it resistance against that damage.
I know you didn't ask about Warlocks, but the answer to your question is basically "there isn't much you can do to improve your familiar's combat abilities without being a Pact of the Chain Warlock because combat familiars are the point of Pact of the Chain Warlocks." By using the Gelatinous Ice Cube AL Reward and your DM letting it make an attack, your familiar is already punching well above the weight class of the familiars available to non-Warlocks.