How do I determine how intelligent enemies attack? For beasts and plants and such, I usually roll to randomly determine a player, then attack based off that. But intelligent, even semi-intelligent orcs and such, would they use attack priorities? Would they recognize a caster on sight? Does a devil recognize a character with a bane weapon at a glance? Should I have them roll their own knowledge checks against the players and their gear?
Remember that the NPCs are intelligent. While most of them are not brilliant, they're still going to have combat senses and instincts. Even many unintelligent NPCs have combat instincts (e.g. wolves).
As there is (usually) no physical difference between a caster and non-caster, an orc would have a hard time pointing one out unless they were wearing typical garb or have some other giveaway. Once they cast a spell, though, their cover is blown.
I would recommend doing what makes sense, and this often involves making knowledge checks for NPCs. If it's obvious they'll recognize it, feel free to take 10 and succeed. If your system does not support take 10, then do what makes sense for the NPC. For instance, a devil would have an easy time spotting a bane weapon, but that does not guarantee they will. And, if they do, they're going to behave significantly differently than otherwise.
For the most part, when dealing with NPCs, do what makes sense or just take 10. You don't need to roll nearly as often as the players; remember, the time you spend rolling is time the players spend waiting for you to hurry up and stop rolling. Most of the time I would say: go with what makes sense, and if you're unsure, roll. If your players object and want you to roll, then they will point it out to you, and through this, you can garner a better feel for how your players play.
Just to expand upon this, most animals such as wolves will target a creature in particular as prey focusing upon either the smallest or the most obviously wounded, continuing with the wolf example several of a pack may harass the party as a distraction while the rest of the pack attempts to separate the designated prey away from the party and take it down.
Racial preferences may also affect the initial targeting of attackers for example orcs hate elves and a leader of an assault may demand the satisfaction of killing the elf himself causing the rest of the group to avoid the elf only attacking when attacked themselves.
Kobolds on the other hand hate gnomes and in an ambush scenario a gnome character may find themselves on the receiving end of most of the initial attacks. Alternatively if the fight goes badly for the kobolds and they have no chance of escape or surrender they may throw themselves at the gnome in the hope they can at least take one of their peoples hated foes with them.
This sort of thing really helps gives character to the races and encounters and can lead to bemusing occurrences.