To what extent is it permissible to use aspects of a given system (for commercial use)? Say I wanted to have rules for how a skill check works, do I have to be careful that I am not using the same as an existing system? Are names of things such as skills, feats, classes, and spells copyrighted?
I am a game designer with my own game designs already on the market. My primary system at the moment is currently the number one RPG system on Amazon.com. I'm also a writer with over ten books to my name and dozens of shorter stories. I've read a lot of books with regards to copyright law and checked both the Canadian and U.S. laws concerning copyright, trademark, games, and patents. However, I am Canadian so that might be worth noting for slight changes in regulations.
I am not a lawyer and cannot offer legal advice, etc.
Essentially, I agree with @SevenSidedDie's answer. His research mirrors my own. I do have several points I'd like to bring up regarding my own experience in the field of writing and game design:
Less is More: If you have any doubts about whether something is a copyright infringement, it probably is. I'd steer clear of using exact names, turns of phrase, and anything else which makes it clear you're borrowing heavily from another work. However, game mechanics fall under a different set of regulations so you can probably get away with using a d20 or a system called 'skills'.
Don't Copy Names: I could be wrong about this, but I try to steer clear of any specific name wherever possible. I don't name powers in my book feats. I don't call the powers themselves stuff like Cleave and Great Cleave. In fact, I try to use unique powers wherever possible in the first place. My reasoning is that if everything I've created is 'new' a small slip up like using the abbreviation str once in the entire book will seem less like a copyright violation even though technically it might be.
Don't Copy Phrases: Try not to copy anything in any pattern of 3 words or more from any other work without correctly attributing that. This means no stealing tables, exact skill or power lists, standard D&D classes and so forth.
Use Common Sense: Most copyright laws follow fairly common sense grounds of 'reasonable interpretation'. This basically means that if push comes to shove the 'cumulative' effect of your work will be considered. If your book obviously borrows a lot from another source you could be in trouble. On the other hand, if it's clearly 99% original any slight slip ups will probably be taken as that; slip ups. Consider if someone was to copy your game system. How much copying would you want to allow before you considered someone else was stealing your work?
What you Can Do: The 'game' aspects of a game can be taken to some extent. I'd still recommend against taking a system's mechanics wholesale even if it's technically allowed for the number 4 reason above. I.e. yes, but everyone makes small slip ups and this could be what does you in. If you want to use the basic structure of D&D, you can probably change some descriptions and new names for classes/feats/skills and maybe get away with it. As a game designer myself, I'd personally recommend making your 'own' game anyway because it's usually cooler, more fun, and safer against copyright infringement.
Check out the Competition: Many other game designers have already faced your same questions. Posting here is a great move. Another really smart thing to do is check all the other 'new' RPGs out there across the web. You'll soon get a very good feel for what's allowed (most of it) and what's going too far (a few small games out there). Another good rule of thumb, is to make your game 'more' unique than the other games out there which perhaps steal a few too many mechanics. That way, if there's ever trouble you can point out 90% of the other games out there which already do far worse than you.
Personally, if you're serious about game design, I'd recommend building your own system from scratch. In the end it's more fun, more appealing to players, and won't get you in trouble for copyright infringement. Having a few mechanical similarities usually isn't a big problem, apart from that I'd recommend to be creative.
Thanks for listening. Hope this helps!
I am not a lawyer. This doesn't constitute legal advice. If you require legal advice in a practical matter, retain a lawyer. Yadda, yadda.
However, these are two well-understood parts of game design, so I can comment generally.
Rules cannot be copyrighted, as they are procedures and processes. The correct arena of government-granted production monopoly for procedures and processes is patent law. However, the US Patent Office has affirmed that game rules cannot be patented in the United States. Other countries will differ, but are unlikely to grant patent protection to game rules for similar reasons.
Words ("expression") can be copyrighted. Never reuse words from another game, book, TV show, and so on, without written permission or a license agreement. Copyright only applies to expressions of ideas, so it only covers lengths of text (though the length may be quite short, such as the 140 letters of a tweet). Copyright applies to any work regardless of whether it has a copyright notice – it's automatic in most jurisdictions. The more unique and recognisable a given combination of words, the less safe you are in using them.
Of particular note is that tables are widely considered an "expression" of an idea, so game designers avoid copying them even when there's no other way to represent a set of mechanical details. This hasn't yet been tested in court either way, but nobody wants to be the first in line to find out the hard way. If you look at various D&D simulacra such as OSRIC or Labyrinth Lord, you'll notice the XP tables' numbers don't match the originals. That's deliberate.
Individual names or recognisable terms may also be trademarked, but this really only happens with big game brands since applying for and enforcing trademarks is expensive.
Individual words or compound terms may be permissible in so-called Nominative Use where you merely indicate the name copyrighted by someone else, but this is tricky ground and if you want to attempt nominative use you need to talk to a lawyer. Individual terms that are generically descriptive also don't get copyright or trademark protection, but you should be very sure that the term is generic and not particular to that expression of the rules.
This split is usually summed up in the game design community as (paraphrasing) "game rules can be copied, but their expression – the words used to convey them – can't be copied."
Game designers use rule patterns from other games all the time. Good designers rarely copy exactly though, because good designers know that a rule is almost never going to fit with other, different rules unless it changes to suit them better. However, when straight-up copying is done, it's totally legal and usually not even a point of conflict professionally, because the rest of the game design is going to be different and the overall game will be something new under the sun. We all like to see new games.
Some designers will cite their inspirations right in their book. Readers will likely notice similarities anyways, and why not spread the love of games the designer loves?
However, all of this is very particular to games that are released without a bundled license, such as D&D and other games that are released under the Open Games License or one of the Creative Commons licenses. It's still all true of games that come with a license, but there are other wrinkles to deal with.
I notice that you're talking about "feats", "classes", "spells", and "skills". These four things taken together tell me that you're thinking about using the rules from one of the Wizards of the Coast–produced editions of Dungeons & Dragons. In that case, you have to be extra careful. The 3rd edition of the game is licensed under the OGL. It gives you permission to use various parts of the system and nomenclature that you wouldn't normally be able to use without the worry of legal action, and in exchange it requires some extra restrictions that copyright/trademark/patent law doesn't normally impose, such as avoiding certain terms. But most importantly WotC (Hasbro) is a big fish in a small pond, and even legal uses of D&D stuff will likely get their attention. Not accepting their OGL license for releasing a high-profile game or product is one good way of getting their unhappy legal attention, even if your use is totally above-board. However, using the OGL is a good way to play with WotC's toys and have a reasonable expectation of immunity from legal action. Lots of game designers have gone that route.
(4th edition is a whole other kettle of fish – there is a license, but it's very restrictive. Tread lightly when writing 4e-inspired systems.)
And finally, all this is moot if you're not publishing, designing a game for personal use, releasing the game in an informal way, or otherwise being off or under the intellectual property holder's radar. If you're actually publishing it somewhere (e.g. posting it on a forum or something) there's still the infinitesimal chance you'll be noticed by whoever you've copied from and get a legal nastygram, but the likelihood is small. That's not to say that it's a good idea, but more of a comment on how unlikely the legalities are to be relevant if you're doing it for private use or limited private distribution.