Use of a spellcasting focus is a class specific feature.
There may be some confusion from the wording in Chapter 10: Spellcasting. The rules for focuses and components may be construed as permitting any class to use a focus:
Casting some spells requires particular objects, specified in parentheses in the component entry. A character can use a component pouch or a spellcasting focus (found in chapter 5, “Equipment”) in place of the components specified for a spell.
It says "a character", but this should not be understood to mean "any person", but rather, any person with the proper class feature.
To this end, all of the spellcasting classes have a specific class feature that permits use of a focus:
Bard:
You can use a musical instrument (see the Tools section) as a spellcasting focus for your bard spells.
Cleric:
You can use a holy symbol (see the Adventuring Gear section) as a spellcasting focus for your cleric spells.
Druid:
You can use a druidic focus (see the Adventuring Gear section) as a spellcasting focus for your druid spells.
Sorcerer:
You can use an arcane focus (see the Adventuring Gear section) as a spellcasting focus for your sorcerer spells.
Warlock:
You can use an arcane focus (see the Adventuring Gear section) as a spellcasting focus for your warlock spells.
Wizard:
You can use an arcane focus (see the Adventuring Gear section) as a spellcasting focus for your wizard spells.
As you can see, it is clearly intended that use of a focus be a class feature unique to caster classes. This understanding is confirmed in the descriptions of the focuses. For example, the arcane focus description states:
A sorcerer, warlock, or wizard can use such an item as a spellcasting focus, as described in the Spellcasting section.
Without a class feature permitting use of a focus, a character may not make use of a focus.
It is worth noting that anyone can use a component pouch.
The description of a component pouch is a bit different:
A component pouch is a small, watertight leather belt pouch that has compartments to hold all the material components and other special items you need to cast your spells, except for those components that have a specific cost (as indicated in a spell's description).
None of the caster class feature mention component pouches, so anyone should be able to make use of a component pouch for casting spells.
Putting it all together.
In conclusion, the answer to the question is "No, the Magic Initiate feat is not a sufficient condition for making use of a spellcasting focus." But as stated previously, just get a component pouch.