To what extent is it permissible to use aspects of a given system (for commercial use)? Say I wanted to have rules for how a skill check works, do I have to be careful that I am not using the same as an existing system? Are names of things such as skills, feats, classes, and spells copyrighted?
I am a game designer with my own game designs already on the market. My primary system at the moment is currently the number one RPG system on Amazon.com. I'm also a writer with over ten books to my name and dozens of shorter stories. I've read a lot of books with regards to copyright law and checked both the Canadian and U.S. laws concerning copyright, trademark, games, and patents. However, I am Canadian so that might be worth noting for slight changes in regulations.
I am not a lawyer and cannot offer legal advice, etc.
Essentially, I agree with @SevenSidedDie. His research mirrors my own. I do have several points I'd like to bring up regarding my own experience in the field of writing and game design:
Thanks for listening. Hope this helps!
I am not a lawyer. This doesn't constitute legal advice. If you require legal advice in a practical matter, retain a lawyer. Yadda, yadda.
However, these are two well-understood parts of game design, so I can comment generally.
Rules cannot be copyrighted, as they are procedures and processes. The correct arena of government-granted production monopoly for procedures and processes is patent law. However, the US Patent Office has affirmed that game rules cannot be patented in the United States. Other countries will differ, but are unlikely to grant patent protection to game rules for similar reasons.
Words ("expression") can be copyrighted. Never reuse words from another game, book, TV show, and so on, without written permission or a license agreement. Copyright only applies to expressions of ideas, so it only covers lengths of text (though the length may be quite short, such as the 140 letters of a tweet). Copyright applies to any work regardless of whether it has a copyright notice – it's automatic in most jurisdictions. The more unique and recognisable a given combination of words, the less safe you are in using them.
Of particular note is that tables are widely considered an "expression" of an idea, so game designers avoid copying them even when there's no other way to represent a set of mechanical details. This hasn't yet been tested in court either way, but nobody wants to be the first in line to find out the hard way. If you look at various D&D simulacra such as OSRIC or Labyrinth Lord, you'll notice the XP tables' numbers don't match the originals. That's deliberate.
Individual names or recognisable terms may also be trademarked, but this really only happens with big game brands since applying for and enforcing trademarks is expensive.
Individual words or compound terms may be permissible in so-called Nominative Use where you merely indicate the name copyrighted by someone else, but this is tricky ground and if you want to attempt nominative use you need to talk to a lawyer. Individual terms that are generically descriptive also don't get copyright or trademark protection, but you should be very sure that the term is generic and not particular to that expression of the rules.
This split is usually summed up in the game design community as (paraphrasing) "game rules can be copied, but their expression – the words used to convey them – can't be copied."
Game designers use rule patterns from other games all the time. Good designers rarely copy exactly though, because good designers know that a rule is almost never going to fit with other, different rules unless it changes to suit them better. However, when straight-up copying is done, it's totally legal and usually not even a point of conflict professionally, because the rest of the game design is going to be different and the overall game will be something new under the sun. We all like to see new games.
Some designers will cite their inspirations right in their book. Readers will likely notice similarities anyways, and why not spread the love of games the designer loves?
However, all of this is very particular to games that are released without a bundled license, such as D&D and other games that are released under the Open Games License or one of the Creative Commons licenses. It's still all true of games that come with a license, but there are other wrinkles to deal with.
I notice that you're talking about "feats", "classes", "spells", and "skills". These four things taken together tell me that you're thinking about using the rules from one of the Wizards of the Coast–produced editions of Dungeons & Dragons. In that case, you have to be extra careful. The 3rd edition of the game is licensed under the OGL. It gives you permission to use various parts of the system and nomenclature that you wouldn't normally be able to use without the worry of legal action, and in exchange it requires some extra restrictions that copyright/trademark/patent law doesn't normally impose, such as avoiding certain terms. But most importantly WotC (Hasbro) is a big fish in a small pond, and even legal uses of D&D stuff will likely get their attention. Not accepting their OGL license for releasing a high-profile game or product is one good way of getting their unhappy legal attention, even if your use is totally above-board. However, using the OGL is a good way to play with WotC's toys and have a reasonable expectation of immunity from legal action. Lots of game designers have gone that route.
(4th edition is a whole other kettle of fish – there is a license, but it's very restrictive. Tread lightly when writing 4e-inspired systems.)
And finally, all this is moot if you're not publishing, designing a game for personal use, releasing the game in an informal way, or otherwise being off or under the intellectual property holder's radar. If you're actually publishing it somewhere (e.g. posting it on a forum or something) there's still the infinitesimal chance you'll be noticed by whoever you've copied from and get a legal nastygram, but the likelihood is small. That's not to say that it's a good idea, but more of a comment on how unlikely the legalities are to be relevant if you're doing it for private use or limited private distribution.
The Law of the Geek podcast has two episodes about the protectability of game systems under copyright law: Warning: System Failure and OMG, it's the OGL. While it is not legal advice, they are lawyers.
As an aside, Geoff makes the case that Paizo, from a legal standpoint, should have made Pathfinder without involving the OGL or Wizards of the Coast's content. Personally, I think the OGL is useful to Paizo just for its consumer awareness and known compatibility.