First, I'm not a lawyer. I've spent the past 15 years paying attention to the shifting landscape of copyright and trademark and other IP law, so I can point you toward a few ideas that might help, but I'm still not a lawyer and can't give you legal advice that's worth a damn when you actually start publishing.
Infringing others' IP rights
There are a lot of myths about copyright floating around, and it has the predictable effect of freaking out new creators (ironically so, considering it's supposed to create certainty and calm). The important thing to know is that copyright can't be infringed unless you've copied (hence the name) from someone else. It's also important to know that in the United States and some other places, being innocent is no protection from being sued into oblivion by anyone who decides to, and frivolous lawsuits can be used by those with lots of money as weapons against those who can't afford to be sued: even if they would eventually be proven innocent, usually they're bankrupted long before the trial is over.
As a result, the person to talk to about copyright (and trademarks) is a lawyer, not us. Note that in the US, game mechanics can't be patented, so there's at least one bit of law that you don't need to worry about. Aside, you're unlikely to fall afoul of trademark law either, unless your games' titles or major technical terms closely resemble (or resemble when taken in aggregate) an existing game's title or terms.
All that aside, there are thousands upon thousands of roleplaying games published, and hundreds upon hundreds more every year, and very few of those creators have bothered to retain a lawyer. The roleplaying games industry is actually pretty tiny as publishing industries go, and small players simply don't get into legal trouble by publishing a new game. The only people who generally have to worry about legalities are the people producing material designed to work with (or even clone) existing games, and it sounds like that's not your situation.
So really, read around a while about copyright and trademark, get that bit of education that's possible by reading up on Wikipedia and other relatively-reliable sources, and have a good think about whether what you're looking to publish is going to make any existing publisher unhappy. Likely the answer will be no, but by then you'll have a more confident grasp of the legalities and you'll be able to decide better when you need lawyer-shaped legal advice.
Protecting your own work
As for enforcing your own copyright, you're automatically protected if you're in the US, Canada, or the UK. (Possibly elsewhere too, but I'm not as familiar with copyright elsewhere.) Copyrights don't need to be registered anymore in order to be protected. If you do find that others are distributing or copying your work without license on a bothersome scale, that's the time to hire a lawyer to:
- determine who is legally responsible for the distribution (which is not always obvious), and
- send a legal letter asking them to cease and desist
It's worth noting that some money-making RPG publishers have stopped worrying about "piracy" of their work, as there is a growing body of evidence that its net effect on sales is negligible.
As for publishing, the games industry is moving away from up-front print runs and big publishing houses, and moving toward boutique publishing houses (usually at most one or two people), self-publishing, and electronic distribution or print-on-demand (POD) services like Lulu and RPGNow. Even White Wolf, one of the once-great publishers, is now only a few people.
The advantage of print-on-demand is that your up-front financial risk is minimal (and sometimes $0) and you don't have to ship and store a crate of books in your garage; the disadvantage is that the per-unit profit is less due to the larger cut that goes to the printer-publisher (30–40%). Offsetting that per-unit cut, you're usually selling direct to customers so your own percentage is undiminished by wholesale pricing. Print quality used to be a disadvantage too, but for the standard formats the quality of POD books is now very good. And unlike with a novel (and even there not as much as it used to), self-publishing an RPG has no stigma attached to it: the RPG industry has a long history of DIY games being just as good as the "pro" games.
It's also worth knowing that there is no money in RPGs. Unless you become a superstar game designer (there are only two I can think of, and even they have to do non-game work on the side), your games will never be more than a source of pocket change and may in fact be something you pay to keep alive. Wizards of the Coast, the biggest player in the market, barely makes enough money on Dungeons & Dragons to avoid having Hasbro mothball them. Luke Crane, who's one of the few who has achieve publishing stardom independently, is a professional editor and layout designer and has to supplement his earnings from the Burning Wheel with that work.
Keep your day job, do it for love, and ask yourself how much you're willing to lose up-front (scroll down to "Publishing" on that link) before you settle on a publishing route. Some publishing methods are very expensive up-front to the author, and that money isn't guaranteed to be recouped. If your desire is just to have the games published at all and hold the book in your hands, there are many ways to do that now, and some of the cheaper ones may be just as satisfying for you as the more expensive and more traditional 6,000-book print run.
If in doubt, set up your document creation process so that it can generate PDFs if you need it to. There are other formats that traditional publishers accept, but nearly every single one accepts one of the PDF format versions. Besides, if you can generate a PDF of your work, you almost certainly can generate one of the other format they'll accept, whether that's Word, InDesign, or Quark files.