No, there is not "a correct way to play NPC/monsters in D&D according to RAW" -- the rules say nothing about it. A DM may play his monsters as smart or as dumb as he pleases.
I play the monsters as smart as they can be, subject to their own limitations. I think it is, though, important to respect those limitations -- some monsters aren't that smart, or that patient, or have other flaws.
I definitely have monsters and NPCs learn from their encounters with the PC party, and alter their behavior as best they can, if they get a second chance.
An example -- I recently had a party that found a small orc tribe that moved into a formerly safe area and were starting to cause trouble. Resolving this involved several encounters. Along about the third encounter, some of the Eyes of Gruumsh came up with plan to surprise and ambush the party, and managed to pull it off, driving the PC party away from what they thought was going to be another easy raid.
However, orcs are not disciplined and patient by nature, so I decided that they were not going to be able to maintain that state of readiness for a long term; the tribe did find a more defensible position and prepared it, and also recruited a few allies, but did not maintain all of the heightened security after the PC party had gone and done something else for a couple of weeks.
Taking them out still wasn't easy, but was possible. If I had had the tribe maintain its alarms and readiness, it would have been very difficult, but I didn't feel that was reasonable for orcs.
Remember the DM is supposed to really be the referee between the PCs and the world, and should be fair about that.