I have a cunning plan.
First off, Amber is expressly about this. If you asked for a system to delve into dungeons and slay dragons, D&D makes sense, if you want a system for exploring the angst of the recently undead then Vampire is your game. I haven't played Amber really, and others have recommended it already. Check it out. I am going off the beaten path here.
If you're interested in doing something different- You could hack FATE to do this.
In FATE, characters have a resource called FATE points. These can be spent to invoke aspects of the world around them or of characters around them, to activate certain in game powers, or more importantly for our purposes to declare a minor story detail or to create an aspect of the scene. This is a quite explicit rewriting of the world, though it is a metagame one- This is supposedly the way things always were, instead of the world changing entirely.
While this is entirely a houserule, I tend to let bigger details be declared if more fate points are spent. I have stated/bragged that ten FATE points makes any story detail happen. This works very well- most 'big' declarations are in the two to three fate point range- that's still pretty expensive. The ten pointer has been used exactly twice, and both moments were pretty cool.
What I am proposing is making the declarations explicit in world alterations. While we do not know the exact mechanics of the books in Myst, we do know that they require some kind of expenditure in order to be created. Atrus couldn't make his own way out of the green book without assistance, and couldn't create more worlds from inside. If the spending of FATE points on new scene aspects was an in game 'edit' of a book, and large FATE point expenses could create entire new areas, you could easily regulate the creation of new worlds.
The creation of worlds would be where FATE would shine. The FATE fractal is a construct in the FATE rules- basically, everything has Aspects which describe it, Skills to allow it to defend itself or attack, and Stress and Consequences which mitigate damage. Myst Island for example, might have Aspect "Center of Everything" while the Red, Blue and Green books might have the consequence "Missing Pages." The cool thing is fractals can nest- A building might have the Aspect "On Fire" and contain several rooms, which are subfractals containing their own separate aspects. The Fractal would allow you to quickly write a new world, delving down as needed- one or two aspects to cover the island as a whole when first created, making new sub-fractals for areas on that island, or maybe for interesting puzzles.
From your question, you want creation, exploration and exploitation. The expenditure of FATE points, along with the relevant book allows the creation of these worlds. Entering them, we can explore the world we have created, or we can explore a world left behind by another. Each world presumably has something we want, allowing us to use ('tag' in FATE parlance) the aspects of the world to help us accomplish tasks.
You specify a preferred difference between worlds and ages. I confess, I don't remember enough about Myst to be clear on the distinction, and Wikipedia is of little help to me. However, if I remember right an age is a larger version of a world- and this could be modeled with world being a subfractal of ages, and the creation of an age requiring more FATE points.
You also ask for human or near human characters. That's straightforward- by default, FATE makes characters who are very human in capability and behavior.
Again, Amber Diceless expressly sets out to do what you want to do. However, this would mechanically work just fine, and amuses me greatly by making in character a rule of the system. If it's not to your taste, oh well, but thank you for the question- I intend to try this out in some form or another next chance I get.
Turns out, I'm not the only one who thought FATE and Myst were a great pairing- the awesome folks over at unwritten are actively writing a definitive (not official, but definitive) FATE variant for the D'niverse. They've got the OK from the makers, they've got a successful Kickstarter, and they're actively writing. If you don't feel like creating your own variant, these guys may have exactly what you want in a bit. Kudos to GMJoe for noticing this was going on.