Retrieving your component pouch in order to perform the material component of the spell may require object interaction (if in a pocket), or even an action (if in your backpack). However you only need a free hand ready to manipulate your component pouch as part of the spell:
If a spell states that a material component is consumed by the spell, the caster must provide this component for each casting of the spell. A spellcaster must have a hand free to access a spell's material components -- or to hold a spellcasting focus -- but it can be the same hand that he or she uses to perform somatic components.
Instead, keep your component pouch on your belt where you can access it easily. Many spells do not require you to do anything with the materials, you just have to have them.
When a spell describes how material components are handled or used, that handling/use takes place as part of casting the spell, it does not require a separate action.
Note that you do not have to sheath your sword unless totally necessary. You can simply drop it and keep it propped up against yourself or held with your shield arm. However, another enemy could try and grab it or kick it away, so make sure you are careful.