Has anyone changed system/setting/genre but kept the characters during a campaign?
Players can become very attached to their characters, even when the campaign/setting/genre or system is getting tired and old. The players could remake the characters in a new system but that feels clunky and jarring. Is there a more progressive/smooth way of doing it?
Assume that the players are keen on the idea; this is really about making the process less painful for all involved.
A real lot depends on the campaign/setting/genre/system… and on the players -- indeed, as @valadil notes in a great comment on the Q. If your setting/campaign allows it, you might want to try an "oscillating" solution.
Introduce (carefully and cleverly) a parallel universe or some alternate plane to which your PCs can eventually travel and have a short adventure strongly related to the campaign and the characters. Have them generate their characters for this alternate world's rules, without introducing any real changes to their personality.
After their (possibly hugely successful) quest that leaves them excited about the other world and their characters' partly new abilities, let them return to the original world. Have an adventure there. Then take them into the new world again, this time for a longer trip.
Repeat this cycle until you have, with your players' approval and support, gradually moved the main plotline over to the new world and system, alongside with their characters.
I've based this on real experience. One of our most successful and longest running campaigns - ran about 10+ years ago - involved Kult, Vampire: The Masquerade, Ravenloft (AD&D), a tiny bit of Shadowrun, and, finally, SLA Industries as well.
I've only shifted characters over to a new system, not a new genre or setting. We went from Shadowrun to Savage Worlds. Shadowrun's rules were just too convoluted for us. We were still looking up rules six-months into the weekly game. Savage Worlds keeps it simple.
To make the conversion, it was clear there was no suitable one-to-one conversion between the two games. The probability distributions were too different. The characters would have become unbalanced in the new system. We jointly decided on an experience level that reflected how much our characters had achieved to date, and simply recreated our characters from scratch, keeping the feel of the old characters.
When playing any new system, we allow respeccing any unused abilities between the first few sessions. This helps the players get comfortable with their characters. Especially when you're shifting systems, this gives players the opportunity to be confortable they're playing the same character.
There was some loss in the translation, mostly in regards to magic, which is a small heartache. After a few sessions, it's generally forgotten. Ultimately we had a better time using the new ruleset.
One of the chapters in John Wick's Big Book of Little Games is dedicated to a mechanic meant to explore this. Called "The Flux," it offers a system that allows characters to change systems while still having access to earlier or different versions of their PCs. You can purchase "The Flux" here.
We have done this in a longer running campaign. We moved from Pathfinder to 5e D&D midway, a relatively easy transition (same genre, same tradition of system, very similar spells, etc.). There are a few aspects to this:
Abilty to translate the character concepts. If the new systems allows you to rebuild your character with similar abilities than it used to have, you're good. Some elements will be weaker in the new version, some stronger, but as long as the character overall does not suck now, and the player likes it this is OK. However, it may not always be possible to get it to work. If you cannot recreate your beloved character, it may be better to retire it, instead of having to play a watered down version. Create a different, new one that works better in the new system. For example, we had a sorcerer that was optimized for fire damage, and after translation lacked heat, so the player decided to retire it, and instead played an elven archer.
Ability to retain story continuity. To keep the story going you want the group to retain their involvement in the game world and their purpose. As long as not all the characters in the group change at the same time, and at least one of them is staying around to provide continuity of purpose, this is not a problem. Any normal group should be able to support some rotation in the cast. If none of the characters makes it across it would be more difficult to keep the stroy going - this has never happend with us, though.
Ability to explain the change in-game. This is not a necessity, and may take some planning to set up, but it is making things a bit more believable. Why do the characters have new skills they never knew about?, or lose abilities? You can give an in-world explanation for the changes. In our campaign, right around the time where we switched systems a magical moon returned to the heavens, heralding cataclysm. The fabric of the world was altered, and the abilities of the characters changed. (This was a coincidence, but it fit perfectly). In the old Greyhawk campaign, when D&D moved to Second Edition, TSR published an entire module in Fate of Istus with adventures to showcase how Istus had changed the world and how it affected the different classes.
The Amazing Engine system was designed so that core parts of characters would be transferable between settings/genres. If you switched from a cyberpunk setting to a fantasy setting, you'd have to leave behind your net-hacking skills, but could learn magic instead.
