Yes.
You need to hold the arcane focus for it to substitute as a material component.
Spellcasting Focus Substitutes for a Material Component
To clarify: you only need to hold a spellcasting focus to cast spells with a material component, not for every spell. You can tell the difference in the spell description. Two examples, one without (Magic Missile) and one with (Major Image) a material component.
Spell Examples
Magic Missile
1st-level evocation
Casting Time: 1 action
Range: 120 feet
Components: V, S
Duration: Instantaneous
Major Image
3rd-level illusion
Casting Time: 1 action
Range: 120 feet
Components: V, S, M (a bit of fleece)
Duration: Concentration, up to 10 minutes (Basic Rules p. 96)
Spellcasting Rules
From the spellcasting rules in Chapter 10.
Material (M)
Casting some spells requires particular objects, specified in parentheses in the component entry. A character can use a component pouch or a spellcasting focus (found in chapter 5) in place of the components specified for a spell. {snip} A spellcaster must have a hand free to access these components—or to hold a spellcasting focus—but it can be the same hand that he or she uses to perform somatic components. (Basic Rules p. 79)
Spellcasting Focus
A spellcasting focus is:
- A Holy Symbol (Basic Rules p. 48.)
- A Druidic Focus (Basic Rules p. 48.)
- An Arcane Focus (Basic Rules p. 47.)
Note on the Holy Symbol:
A cleric or paladin can use a holy symbol as a spellcasting focus, as
described in chapter 10. To use the symbol in this way, the caster
must hold it in hand, wear it visibly, or bear it on a shield.
The Fine Print
The specific exceptions to this general rule are spells that have a material component that is consumed. Example: Resurrection.
Resurrection
7th-level necromancy
Casting Time: 1 hour
Range: Touch
Components: V, S, M (a diamond worth at least 1,000 gp, which the spell consumes)
Duration: Instantaneous (Basic Rules p. 99)
(Finer print: A spell with a material component that isn't consumed but has a specific cost in GP (example is Identify, requires a pearl worth at least 100 gp which isn't consumed) still needs the material component).
The same rules are in the PHB.
Note: we had a discussion on what constitutes a bardic spellcasting focus here.