So, you've built your own system, or a module for an existing system, or even a set of houserules you want to share with people. What do you do next?
Often, but not always, the next step is to playtest it.
Real quick though, three disclaimers:
-Playtesting isn't always possible. If you can't playtest something, you can't playtest it. That's okay.
-Also, rigorous playtesting isn't always possible. If you can playtest your game a little but you can't dump 100+ hours into it, that's also okay. Do what you can.
-Finally, there's no definitive one right way to playtest. What I'm sharing is my approach, but please do what works best for you.
Now, to get started, you need players.
If you've got friends who are willing to help out, great. Recruit 'em.
If you've got friends but you're not sure how they'll feel about the game, ask them.
If you don't have friends who are willing to help out, you can simulate some players. Imagine a person and walk them through the steps of your game, or play yourself as both the players and the GM.
Now, when you playtest, your goal is this: to have fun with the game.
Try to get through a session. Engage the players. Tell an interesting story together.
For any element that breaks but doesn't crash the game, make a note of it and circle back later.
If any of the mechanics break completely in half, rendering the game unrunnable, that's fine. Stop and fix things only to the extent that you can continue play. You don't need to get everything perfect.
When the session's over, ask the players what they thought about the system and if there were any parts of it that they felt strongly about.
Sometimes their answers will tell you as much about the players as they do about your system, but all feedback is good feedback.
If a player is frustrated because they wanted to power-game and your system wouldn't let them, that's useful information! There might be types of players your game clicks with more than others, and maybe you can add some text to let readers know to prepare for this.
Similarly, if a player is constantly getting confused by a dice mechanic, maybe there's another way to explain that mechanic. Try to draft some other versions of that text.
Of course, if the players liked some part of the system, that's worth recording too. If the players light up at your fishing minigame, maybe you can develop that minigame a little bit more.
What you *don't* need to do at this point is worry about whether your game is good, or stress out if the mechanics hit a snag.
The goal of playtesting is to improve a game, so it doesn't matter if the game is good at the start of the process.
I have had systems break in truly spectacular ways, every single mechanic imploding, the moment someone enters a car or tries to assist on a roll. So really don't sweat it if you find some bugs.
If you've got friends helping you, and if they're good friends, they're not going to give you (too much) grief over the system breaking.
And if you're testing by yourself, then you've got even less of nothing to worry about.
After you've gotten your feedback, the final step is making changes to your game. Fiddle with the mechanics, add or remove material, and generally revise the text based on what worked for you as a GM and what the players liked or disliked.
When you're done, you might have an entirely new-looking game, so provided you can keep playtesting (see the earlier note about rigorous playtesting,) schedule another session and see if there's anything more you want to change.