Homebrew is the creation of one's own rules systems, or material for an existing system. (Note that soliciting ideas for homebrew is off topic for the site.) Not to be confused with the [house-rules] tag.
Homebrew, so called by analogy to home-brewing beer, is the creation of new game material by fans, whether entire game systems or just new rules material for an existing game system.
Homebrew is distinct from house-rules in that house rules are usually small ad hoc modifications of an existing system, while homebrew for an existing system is the creation of entire new subsystems, or of rules-heavy material like creatures, spells, equipment, and similar game material.