Definitions
We all have our limits and boundaries. Lines and veils are different ways to handle those boundaries in play.
A line is, well, a line — a hard limit, something we do not want to cross. Lines represent places we don't want to go in roleplaying.
"There is no torture in the events in our game. We don't do it, NPCs don't do it to us or to each other. Whether it happens elsewhere in the setting is not an issue in terms of enjoying play."1
A veil is a "pan away" or "fade to black" moment. When we veil something, we're making it a part of the story, but keeping it out of the spotlight. Think of it as a way to still deal with certain themes while avoiding having to describe them in graphic detail.
"Torture does happen in the game world and may happen in our game in some way or another. But if and when it does, we do not role-play it directly or depict it verbally. Everyone is trusted to play their characters as reacting to it appropriately without us having to experience it vicariously."1
Background
This terminology came out of Forge discussion some years ago, plus Sex and Sorcery, a supplement to the game Sorcerer. It's a feature of indie-RPG discussion because the community strives to be inclusive but also features a lot of games that deal with difficult content.
Techniques in Play
You can establish some limits ahead of time, as you can see in the examples above. This is a good approach for stuff you know will come up, because it's central to the game system, the setting, or the genre you have in mind. Oftentimes groups do this using shorthand like "Let's keep it PG-13."
Where an understanding of lines and veils really shines, though, is in helping you to communicate about issues as they arise during play. This is something every gamer should learn since it's part of looking out for each other's fun, and there's no way to perfectly plan your way around every issue before the game starts.
The most important thing is to make it easy to speak up. That means you should cultivate a non-judgmental atmosphere. A basic lines-and-veils discussion kinda looks like this:
"Ouch!"
"Oops. Sorry. Let's fix that."
Don't try to argue someone to give in on their limits. Don't try to assign blame to anyone. The person who's feeling uncomfortable may want to talk about it, or they might not; either way is okay. Identify the problem, fix the problem in a way that addresses their specific needs and desires, make a mental note of it for next time, and continue play.
In many cases, a group can do fine without formal structure for this. However, for tricky subjects or gaming with strangers, formal systems are a big help. Variations include:
Really, lines and veils in play are about communication, attitude, and interpersonal skills:
The Veil or The X-Card doesn't replace our complicated interpersonal toolsets that we've spent our whole lives developing... it signals that now is the time to use them. Maybe the stock solution of "edit out the problem content, keep moving" works, but maybe not. It's case + person + issue + game-dependent.2
However you communicate about limits and comfort, remember that signaling devices aren't a perfect substitute for social awareness. Just because someone isn't explicitly telling you they have a problem doesn't mean everything is fine always. A shy player or someone who's been surprised by a trauma trigger might not be able to assert a line in the moment. So keep an eye on people's actual reactions and adjust your play accordingly.
1 - Examples from Sorcerer author Ron Edwards.
2 - Commentary from Monsterhearts author Avery Alder. Check out the free mini-supplement Safe Hearts (PDF link) for more advice about dealing with emotionally challenging content.