How to get new players into a convoluted homebrew system?

Question

I included relevant background to illustrate how convoluted the system can become, skip to the bottom if you just want to see the question.

I've gone off the deep end and broke everything by making a homebrew spell system that completely replaced the pathfinder system. Magic was initially destroyed in a cataclysm per my lore, and now everybody can use it as it is gaining more prevalence, though obviously melee classes will implement it differently than ranged.

It is based around a system of magic words that are combined to generate a desired effect: "burn" + an amplifier, ex: "three" results in an above average burning effect. These words can be chained together to create some pretty neat spells, and I've been kept on my toes by some players trying to abuse the literal wording of spells to teleport between planes using a conjuration ritual. The limiting factor is mana, both in terms of a maximum capacity that each player can use each day and in terms of per-round bandwidth. This means a player can only output X mana each round, so they must channel for more rounds if they want to use more modifying words, and have a hard cap on how much they can cast without resting. Additionally, they can make rituals, which effectively store magic over many days to power an exceedingly powerful spell, and can enchant items that automatically absorb mana, allowing X casts of a spell per day for free.

Players are able to learn the words as they are playing, and write combinations of these words down in physical spellbooks. To help facilitate this, I have 'black pages' that contain a 'spell' already written out (they obviously may or may not know what it does depending on what words they know) along with a single magic word and its definition in common. They insert these into their physical spell-books. I find that my players are much more engaged when they have to physically flip through a book, and it helps avoid those 'quickly googling the wiki' moments. Besides this, there is a real world analogue to preparing spells each week, since they must literally construct their spells and write them down for easy access during play, though the smartest players in my games have started memorizing words directly and making free-form spells mid combat, which I find to be a very cool way of showing both that they and their characters have grown.

These players are at a severe advantage because if they are able to overhear enemies casting, they are able to use a particular class of spells to counteract the effects as an interrupting action, assuming they know the words well enough to instantly cast the dispel.

That said, half of my old group dropped. Luckily one of my friends has three buddies who are interested in joining up with experience in RPGs ranging from none to fairly experienced with recent DnD rulesets. The remaining half of my party is used to the new system and highly engaged in it, but I'm concerned that the newbies will be put off or confused by the system, and I unfortunately don't have time to do a multi-hour walk-through at this point.

So, how do I ease the new players into my convoluted homebrew system without neutering the fun of the rest of the party?

Answer 1

Enroll all of your players as play testers

Start them all together at level 1. That way the new players begin with small numbers of options and as they grow, together, their ability to apply these chains and options grows.
The advantage to this is that you don't get overwhelmed and as they discover things that you didn't intend to happen, they get used to applying the spell system in a crawl, walk, trot fun fashion, and you can rule/revise/edit your home brew as the cracks become evident.

"We're in this together!" That's the whole motif of this effort. If your old players are not willing to do this for the new players, you have a different issue to discuss with your entire group: how do we have fun together?

Learning by doing is the best way to get the new player up to speed. You need to encourage your veterans with this new system to act as mentors/helpers to the new guys. That acts as a team building engine, and relieves you of some of the teaching burden that you indicate you'd like to minimize.

and I unfortunately don't have time to do a multi-hour walk-through at this point.

Old lessons

We did a lot of experimenting and home brewing in the olden days (70's) before there was so much material available. We mixed and matched, and cross patched stuff from Chivalry and Sorcery, from the latest article in Dragon Magazine, from some of the Arduin supplements Dave Hargraves published, from EPT, and from new ideas people came up with. Our games in a lot of respects were impromptu play tests. The first spell point system we used crashed and burned, but that doesn't mean point systems are bad. It's just that ours didn't work too well.

We mostly had fun and often tripped over exploits and loopholes as we played.

Since you've re-rigged the magic system, you can expect to do the same.

This play test will be fun (that's your sales pitch)

If nothing else, this play experience will let the players contribute to polishing your home brew. Encourage their active attempts to find loopholes and break the system you have set up. We used to have fun with that too.

Answer 2

Talk to your players.

Let me give you my personal experience having both played in and run games with extremely convoluted (sometimes good, sometimes bad) systems.

Set good expectations.

Let your players know exactly what they're getting in to. Let them know that you think it's intensely convoluted, but also very interesting. You need to get their buy-in on your system before playing, else they will come in with bad expectations and have a bad time. If you let them know that it's homebrew and mildly (or extremely) weird at times, they're much more likely to get into it and love it.

You will also need to talk to your existing players. Let them know newbies are coming in, so that they can assist you with the teaching process. There's no reason you need to shoulder it all, if as you say, you have several players already invested in the system.

Explain your system, in detail.

You will probably need to sit the newbies down for a bit to do this. You will need to do this, especially with those who have no tabletop experience. The act of learning the social flow of tabletop and the rules of Pathfinder ("which one is the d20 again?") can feel monumental, and will slow the game down. Having to learn a difficult system on top of that could grind your system to a halt. As such, you'll probably need to talk to them beforehand. Which leads me to...

You have to give them a tutorial.

I know, you said you don't have time, but you (or one of the players, if you have a natural teacher) will really want to do this, even if only for an hour or two. Most people learn by doing. You can explain rules all day, but until they get the chance to play with it, it's not likely to "click." This will also help answer a ton of the first-time questions in a more relaxed setting. If not, you run the risk of your pacing being broken constantly in the early sessions by questions that should seem obvious, but aren't. Learning takes time, and there's little to do other than give them experience.

My personal recommendation is to take one member of the existing group who's willing to help teach, plus the three newbies, and sit them all down for a silly one-shot (with no consequences) to show them how the rules work without having to tell them. This also avoids potential annoyance to the "experienced" players who sometimes do not want to wait for "newbies" to learn the rules.

As an aside, I would not expect players to take the idea of being handed a set of rules well. There are those of us who love reading rulebooks, but most players do not. If you hand them rules and ask them to read it before game, expect that it will not have been read.

If you truly do not have time to do any of this before running game, you will have to let the rest of your group know, and expect things to go slowly. But please, don't just throw them in; they will get frustrated from not understanding, and are more likely to quit.

Homebrew systems can be a lot of fun, and I wish you luck. Remember, all systems started as "homebrew," and the biggest difference between you and the big guys is they have more playtesters and editors.