By RAW, the answer is No if not on your turn.
Ready Action states:
First, you decide what perceivable circumstance will trigger your reaction. Then, you choose the action you will take in response to that trigger, or you choose to move up to your speed in response to it. Examples include "If the cultist steps on the trapdoor, I'll pull the lever that opens it," and "If the goblin steps next to me, I move away."
The key here is that unless the trigger occurs on your turn, then your Readied Action does not occur on your turn and you do not have access to on turn aspects of the rules. In addition, they have limited your Readied choices to Action or Movement and not both. The Readied Action is both limited in scope and dependent on a specific trigger.
This includes things like Extra Attack, but extends to Free Object Interactions, which can be found under:
Other Activity On Your Turn
You can also interact with one object or feature of the environment for free, during either your move or your action. For example, you could open a door during your move as you stride toward a foe, or you could draw your weapon as part of the same action you use to attack.
Note that the section is titled On Your Turn. If your Readied Action occurs on your turn, then you can interact with an object or feature for free, but if it does not, then this is not available. And for those wondering, Section Titles are important (emphasis is mine)!
In any piece of writing, context matters. If a rule has multiple sentences, they're meant to be read together. For example, the first sentence of Divine Sense is meant to be read with the rest of the feature's sentences, which explain that first sentence.
But does it really matter?
At first pass, I'd probably say no. Although the designers have gone out of their way to consistently rule what can and can't be done on a turn and the limitation of ready to an action OR movement (and not both.) They clearly have chosen to limit the extent of the Ready Action and give it a high cost (your reaction usage, less impact for an Attack action, etc.) I do not know the reasons, but allowing side cases seems against their intent.
However, should a DM decide to waive it, I don't see this being a cheese-concern situation, but your mileage may vary and be careful about a slippery slope of Ready actions.