My Approach
#My Approach EvaluatingEvaluating the "goodness" of homebrew is a tricky, subjective thing. My approach will not work for everyone, but it has served me well.
Take the large questions noted here. The more "yes"s you get, the more likely this homebrew thing is to be "good."
Is It Similar To Non-homebrew Things?
#Is It Similar To Non-homebrew Things? StatedStated in a different way: "Is the effectiveness of the homebrew thing (in the game) similar to non-homebrew things?" Make sure to consider quantity and magnitude of a thing's limits and strengths.
For example, a homebrewed weapon doing 100d20 in DnD 5e is wildly bad- look at the other weapons, where you're lucky to be doing 1d12!
If the homebrew thing has similar limits and strengths to non-homebrew things, both in quantity and magnitude, then it's good.
Does it Bring Interesting Choices?
#Does it Bring Interesting Choices? ThisThis question covers the more tricky, often more creative homebrew options. Does this homebrew thing give players (or the DM/GM/Narrator/Voice/Etc.) more interesting play options?
What is an interesting choice? The ideal "interesting choice" has a clear impact on play (how players do things) and has no obvious superior choice. There may be other things that define 'interesting choice' in games, but these two criteria are a good starting point.
For example, sacrificing HP (in a DnD-like game) to cast a spell is interesting. The player needs to weigh their character's ability to stay alive with their ability to cast a spell, and that balancing is interesting.
Does It Fit the Fantasy?
#Does It Fit the Fantasy? ThisThis is super subjective and requires you to know who you play with and what they want to get out of play.
My example here: a group of spear-wielding, shield-bearing, anthropomorphic bunny Hoplites does not fit in your group's Serenity (sci-fi/western) RPG!
A Final Note
#A Final Note SometimesSometimes good homebrew goes bad. If you, as a GM, are tempted to show favoritism to your homebrew thing- like your half-dwarf-half-elf-with-dragon-blood hybrid never dying in spite of the best efforts of your players and no narrative reason, you've taken potentially good homebrew and turned it bad. Don't do this.
Good homebrew stuff, when well implemented, serves to increase the fun of every person who is playing that RPG. It fits in, it is not forced in.
In Summary
#In Summary HomebrewHomebrew is "good" when it is mechanically balanced, interesting, fits the fantasy, and is NOT preferred over non-homebrew.