There's three questions you'll ask yourself repeatedly:
- What are the rules supposed to do? (intent, spirit of the rules, etc.)
- How are the rules doing that?
- Is this communicated well enough people other than me?
Proof of Concept Testing
The first set of testing might be playing around with sections of the rules up to full sessions. Playing out a combat scene, screwing around with magic. If you've got friends who are generally into system nerdery, they're good for this.
Players should be testing with good intent - don't try to break or push the limits yet. You want to see if these ideas even work when people are trying to work with them, long before you push edge cases.
Everyone should ask questions while playing! "Hey, does this work like this, or like that?" Write down those questions, because it's going to show you what is important to communicate in your writing. If something is an edge case or not designed yet, just tell them and move on.
Things may not be fun at this point. Let everyone know that. Unconnected fights, playing around with subsystems - it can be very tempting to start fudging and changing everything as you play, but usually what happens is you'll end up pushing your game towards something you already know and not actually learn what these rules actually do.
This stage is pretty early and you may not need to do a lot of this, or you might need to do quite a bit. It depends on how many crunchy subsystems you have and if your core game mechanics push out in weird ways.
Once you think the rules generally work, then you can start playing with people you don't know quite as well, and also have them push on the edges a bit. If that seems solid, you can ask players to start looking to deliberately push the edges of the system.
What happens if you min-max? What happens if you try to spend all of your power points? What's the most broken combo you can do? Is there a way that someone might get NO points? etc.
No matter what you do, players will find breaking points. The issue is not to create a game that cannot be broken (RPGs can always be broken), the issue is to figure out which of these cases you need to pay attention to.
Some of these problems are non-problems. If it requires playing with bad intent, or lots of work to break the rules, you don't have to worry about designing for that. You need to first look for the issues that are pitfalls people MIGHT accidentally fall into, and deal with those. ("Oh, I could see how a sub-genre of superhero stories does this thing, and this game doesn't do that. Is there a rule or advice I can put in about that?"). The second one is pitfalls that are easy to fall into that break the game that maybe you didn't think about. ("Yep, those two powers together are broken. Let me fix that.")
You should also be looking to make sure things CAN BE fun at this point. Be mindful of whether the game even fits what the players want - the guy who doesn't like these kinds of rpgs ("Too much politics, not enough action") aren't going to give you useful feedback other than to be clear on what kind of game it is. It doesn't have to be 100% fun, it just needs to be consistently hitting it more.
Pay attention to what best practices matter, what style of running helps it, what it does differently or how the GM or the players might have to think different about the game. Apocalypse World's Principles work well as an example.
Last part - having people playtest without you leading it, just going from the rules. This is testing your ability to communicate the rules, more than the rules itself. It's really easy to leave important things out - "Of course you play THIS way" isn't common knowledge to everyone.
If you can watch them play, or if they record a podcast or video of the play, it helps. Not everyone is good at self reporting and you can see where things go wrong or off the rails and consider if that's something you can fix in writing or if it's soemthing the group misread or didn't read at all. (This post-mortem video about D&D 5E shows them talking about nearly 50% of the feedback they got was simply people not reading the rules about an hour in.)
Pay attention to where people have questions. Where they say they had one kind of play experience that seems unusual or not to match up with what you expect or have seen in play. If someone's having fun in a way you didn't expect ("Dude! The skill rules change EVERYTHING!") ask some questions, see if they're playing it in a weird way or if they've stumbled upon some best practices you didn't know.
Be really clear on the feedback you're looking for at any stage. Some people's feedback will be "Make your game into this other game I want instead" - that's not helpful.
Don't stress things that aren't finished yet. If you see something works/doesn't work, then you can end play early. Don't grind through endless unfun if you've already seen it doesn't work.
Your rules will go through 3 stages - notes you can understand, clear sparse directions for others, and finally a full text with extra handholding and "encouragment". Don't stress making it a fuller text than what you need at that point.