(5 * level) + INT mod
Because no parentheses or other mathematical indicators are given in the Hit Points section of the Homunculus stat block,
Hit Points equal to five times your level in this class + your Intelligence modifier
directly translates to "HP = 5 * [your level in this class] + [your Intelligence modifier]", which in turn means "HP = (5 * [your level in this class]) + [your Intelligence modifier]", because of the order of operations.
Rubiksmoose also notes that Jeremy Crawford has given guidance on similar subjects. One of his tweets is here, addressing the Arcane Ward feature for Abjuration Wizards:
[...] The ward has hit points equal to twice your wizard level + your Intelligence modifier.