Say you attack an object with a normal non-magical weapon to damage an object, and you roll a natural 20. Do you roll double damage against the object's hit points? Or are objects immune to critical hits from normal weapons?
They are subject to critical hits!
The Dungeon Master's Guide (pg. 246) section on Objects never specifies that objects are immune to critical hits. It gives you, as the DM, suggestions of AC and hit points for the object based on its material and size. The "Statistics for Objects" section says:
When time is a factor, you can assign an Armor Class and hit points to a destructible object. You can also give it immunities, resistances, and vulnerabilities to specific types of damage.
In addition, the rules for critical hits on the Player's Handbook never specifies that objects are immune to then. Page 196 says (emphasis mine):
When you score a critical hit, you get to roll extra dice for the attack's damage against the target. Roll all of the attack's damage dice twice and add them together. Then add any relevant modifiers as normal.
The "target" term includes objects, creatures, and other things as well (like illusions). One of the questions answered in the Sage Advice Compendium supports this statement:
Some spells (like eldritch blast) target a creature. Some others (like fire bolt) target objects too. Does this mean that I can’t attack the door with eldritch blast?
The target specifications (creature, object, or something else) in spells are intentional.
Finally, when you hit an object using a melee weapon or ammunition made/coated with adamantine (which doesn't mean they are necessarily magical, see this Q&A), the attack is always a critical hit (Xanathar's Guide to Everything, pg. 78):
Melee weapons and ammunition made of or coated with adamantine are unusually effective when used to break objects. Whenever an adamantine weapon or piece of ammunition hits an object, the hit is a critical hit.