I would like to find a list of monsters that are not OGL - like mind flayers, beholders, etc. How do you identify whether a monster is OGL or not?
In the general case, you can't, because about 200 companies published a near infinite number of monsters for d20.
If you specifically mean "monsters in the small number of WotC monster books that have OGL content" - just the Monster Manual, Epic Level Handbook, Expanded Psionics Handbook (links go to the monster SRDs from each) - if it's not in the SRD, it's not OGL. The MM list is something like:
For third party supplements, the way to know is to check the OGL declaration in the book you are reading, it will say.
The OGL declaration should include a list of "signature" monsters as product identity. Any monster whose name appears in the product identity list is off limits as a name.
Any monster in the SRD, if the SRD is properly built, will be open content, but the name might still be product identity; again, you still have to check the OGL declarations.
If, as is done by Mongoose for several lines, the SRD is OGL but the core isn't, use nothing from the core, use only the SRD.
In a number of 3rd party supplements, the open content declaration includes all creature stats, but puts the names and descriptions in the product identity list; in those cases, rename the monster, write a new description and you can use the stat block. This is a legitimate use of paraphrasing skills you learned in school...