Do spells that have an area effect such as Burning Hands and Sleep apply to friendly creatures or is it okay to point the area towards a mixed group of friendly and hostile creatures and not worry for the friendly ones?
Generally, all in the area of effect are effected by the spell.
Areas of Effect are defined on pp.204-205 of the PHB. Nothing in those general descriptions distinguishes between who might be in those areas.
Specifically, let's look at Burning Hands and Sleep:
Burning Hands: "Each creature in a 15-foot cone.... (PHB p.220)" Since your allies are (presumably) creatures, they'd be affected. They get the same save as your opponents, though.
Sleep: "Creatures within 20 feet of a point you choose are affected.... (PHB p.276)" You can certainly hit your allies with this, or even yourself.
But I really want to hit my enemies and not my friends!
Sculpt Spells, as a Wizard in the School of Evocation (PHB p117), is the easiest way to do so. And, as @KRyan points out (thanks!), the existence of this class feature is pretty solid evidence that spells generally can't work the way you'd hoped: your interpretation would moot* the Evocation Wizard's L2 archetypal feature.
Another option, courtesy of @Khashir and @CrusaderJ:
Sorcerer's Careful Spell Metamagic grants your allies a free save if they happen to be inside the AoE (still not as cool as sculpt for damage spells, but directly addresses the point that allies would normally have to roll saves...). I say 'not as cool' because many spells still deal half damage on a save, so, your allies would still burn... a bit. On the other hand, Careful Spell can work on spells which are not Evocations. This includes many spells which inflict conditions on a failed save, but do nothing on a successful save. This would allow you to use spells like Confusion or Hypnotic Pattern even when your allies are already inside the target area.
I'm sure there are plenty of other ways to selectively-target within areas of effect; hopefully some commenters will be happy to remind me--or even edit them in =)
* I hold it as a doctrine of coreset-construction that we should read all rules, when possible, so as not to invalidate other rules. On this point reasonable rules-lawyers may differ.
The spell description explains the spell
Area spells would describe what they affect. Some examples include:
- creatures - meaning every character in the area
- some subtype of creatures - such as humanoids, or fiends, or beasts, or others
- enemies - presumably the spell has some mystical equivalent to IFF that it knows who does or does not like you
- allies - ditto
- targets of your choice (maybe narrowed down by other criteria)
The spell description can further explain whether all creatures in the area are affected or not. For example, Burning Hands says:
Each creature in a 15-foot cone must make a Dexterity saving throw. A creature takes 3d6 fire damage on a failed save, or half as much damage on a successful one.
Thus unambiguously affects everything in its area. Compare this with Sleep. First limitation:
Roll 5d8; the total is how many hit points of creatures this spell can affect.
Creatures within 20 feet of a point you choose within range are affected in ascending order of their current hit points
(ignoring unconscious creatures).
Then the way to apply the limitations is explained:
Starting with the creature that has the lowest current hit points, each creature affected by this spell falls unconscious until the spell ends, the sleeper takes damage, or someone uses an action to shake or slap the sleeper awake. Subtract each creature’s hit points from the total before moving on to the creature with the next lowest hit points. A creature’s hit points must be equal to or less than the remaining total for that creature to be affected.
Therefore, not necessarily all creatures in the area are affected. An ally which has high enough hit points will be towards the last to be considered, thus the spell might fail on them.
Now, compare both of the above with the wording of Destructive Wave:
You strike the ground, creating a burst of divine energy that ripples outward from you. Each creature you choose within 30 feet of you must succeed on a Constitution saving throw or take 5d6 thunder damage, as well as 5d6 radiant or necrotic damage (your choice), and be knocked prone.
This clearly and unambiguously explains this specific spell allows for freely choosing the targets.
An alternative is that the spell might allow excluding targets. For example Spirit Guardians:
When you cast this spell, you can designate any number of creatures you can see to be unaffected by it.
Ways to alter the way targets are picked for a spell
If there are features that supersede a spell's targeting criteria, then the description of the feature will explain the feature as well. Some of these features include:
Fur the School of Evocation of the Wizard class:
Beginning at 2nd level, you can create pockets of relative safety within the effects of your evocation spells. When you cast an evocation spell that affects other creatures that you can see, you can choose a number of them equal to 1 + the spell’s level. The chosen creatures automatically succeed on their saving throws against the spell, and they take no damage if they would normally take half damage on a successful save.
This this can allow the flames from Burning Hands to be sculpted around 2 targets* but does not work with Sleep as it is not an Evocation spell.
The Metamagic option available primarily to Sorcerers and others who can use Metamagic:
When you cast a spell that forces other creatures to make a saving throw, you can protect some of those creatures from the spell’s full force. To do so, you spend 1 sorcery point and choose a number of those creatures up to your Charisma modifier (minimum of one creature). A chosen creature automatically succeeds on its saving throw against the spell.
Note that this also works for Burning Hands but not Sleep, as the latter does not have a save. It also does not protect from taking half damage, so using Careful Spell with a Fireball would still hurt.
* 1 + 1 (the spell level) = 2 **
** Unless the spell is cast using a higher level spell slot.
If you refer to the RAW, typically it will say something along the lines of "each creature takes XdX damage"
Some spells are less specific which would generally come down to what your DM/Group wants to play it by.
For example the classic Fireball. Some DMs rule that it will hit all units in the AoE. Others will say that it should only hit Hostile units as that is where the harmful magics is directed.