There are two completely different and only slightly related topics here. Your house and the game. So, let us split the problem in these two sub-problems.
Nobody can (or should be able to) enter your house, at least not without a warrant, without your permission. Period. If you don't want them in your house, you are completely in the right to deny them from entering. But the distinction that needs to be done is: this shouldn't be related to the game. From your comments, it seems you don't want them to join the game, and you are just using the fact that you own the house to try and enforce your point of view on the game. Specifically, you mention:
We have been playing for awhile and I like our group size (6). He's wanting to add 3 more people. I'm mad because they were just brought over without any of us being asked.
This doesn't seem related to the house. If the game was hosted in someone else's house (e.g., the DM's house), you would still be mad because players joined the game without anyone being asked. Which leads us to...
As discussed in other questions, e.g. Is adding a new player (or players) a DM decision, or a group decision? and How to resolve fundamental differences in perspective between players and DM about the roles each has in decision making? adding new players to a table should be a group decision. If some people want new players and others don't, then you should talk about it, listen to why the other people don't want new players or why they do, and figure out what is best to the table. Who owns the house should not be part of the equation, unless, for some reason, you actually feel unsafe with the new people (e.g. the current players are people you have known for a long time and trust, but the new players are random people, and you don't feel comfortable allowing random people entering your house). But, in that case, that is a very compelling argument to why you don't want them to join, unless someone else is willing to host the games, and when talking about it, the people in favor of bringing new players should listen to it, and I believe any reasonable person would agree that this is a very strong reason to deny the new players.
Nobody can enter your house without your permission, but this doesn't seem to be the issue. The issue seems to be about bringing new players to the table, and in that case, nobody has the final say, it is a group decision and should be talked about.