You should tell them
The DM is the players connection with the world, you know how the world works in a manner that the players simply cannot. It is your job to make sure the players have enough information to make sensible judgements about how they interact with the world.
The text in these guides gives you information as a person that you might not have through real life experience.
The players walk into a tavern.
They don't need you to tell them that it is ok to get a table, or order ale, but you might need to tell them if in your world they can order a companion for the night.
The reason is because the players (most likely) don't have any real life experience they can use to understand that is an option, so they might not think about it.
The players have options to cleanse an altar
This is not a situation that occurs in real life so your players have no frame of reference on how to proceed. As a DM this doesn't likely occur in your real life either, so the book gives you information on how it can be done.
It is your job to make sure the players know that these things are possible. I wouldn't say "you can pray here" but you should let your players know that the power of prayer in the game world is more powerful than in real life, and that things like holy water can have this effect. How are they supposed to know otherwise?
How I handle it
I build an environment (out of character) where the players know they can ask me a lot of questions, so I might not outright say 'you found an alter, roll a religion check to cleanse it', but the players know that I give them a chance to improvise, and if they can't think of anything they can ask.
Improvisation involves trust, so generally anything that sounds right I will allow to work, but otherwise they will say "what does my character know about cleansing altars" and I will give them options of prayer or holy water.