Depending on your campaign's magic level, 5150 GP is a good starting price.
But I'd recommend that you charge more or less based on how generally valuable new spells are to wizards in this edition. (The books may be worthless in the short term, see below). Magic as used by wizards is, in general, rare in D&D 5e:
... practitioners of magic are rare, set apart from the masses of people by their extraordinary talent. (PHB, p. 8)
Since the players are selling it, you as the DM can add modifiers to that basic cost based on how hard to get spells and spell books is in your campaign, or even how easy.
- If magic shops operate freely in settlements all over your setting
(High magic campaign; DMG, p. 38) your players are competing with those
merchants on price. They may have to discount the books to get a buyer. In that case, you may want to reduce the price from that baseline.
- On the other hand, if spells and spell books are extremely rare (Low Magic
Campaign; DMG, p. 38) then what they found borders on priceless. Jack up
the baseline price by multiples, or even by an order of magnitude.
Why use 5150 GP as a baseline?
1st x10, 2nd x5, 3rd x4, 4th x2 + 1st x12, 2nd x6, 3rd x5, 4th x3, 5th x2 = 101 spell levels
Copying a Spell (PHB, Wizard, Your Spellbook sidebar)
For each level of the spell, the process takes 2 hours and costs 50 gp. The cost represents material components you
At a minimum it costs 5050 in GP1 cost to have copied them into the books in the first place. Toss in the basic cost of a spell book, 50 GP each (Basic Rules, Adventuring Gear, p. 48) for 5150.
A point on the specialized knowledge that is required to even create a spellbook in D&D 5e, which has magic as a fairly rare thing(PHB, p. 8). The value of the whole is greater than the sum of the parts. Consider the price of college text books (OK, I'll not begin a rant on that) or consider a piece of meat:
- Case 1: prepared by a gourmet chef
Cast 2: prepared at a fast food chain
The wizard is more like the gourmet chef. The act of creation adds value.
Arcane Traditions
The study of wizardry is ancient, stretching back to the earliest mortal discoveries of magic. It is firmly established in the worlds of D&D, with various traditions dedicated to its complex study. (PHB, p. 115)
Rarity and demand can influence price
With all of the above in mind, this is a golden opportunity for some role play if you and your players like to negotiate and work deals as part of the game.
Who needs spells, who wants spells, and who makes the highest offer?
How discrete do you need to be, in this campaign, when letting the
market (overt or underground) know that a valuable and magical item is
for sale?
Is arcane magic illegal in the kingdom? If yes, now you are trafficking in
contraband so the price goes up, and so does risk.
Are they worthless? Consider this situation: nobody within a hundred miles is interested, since nobody within a hundred miles is a wizard. The party just killed the last two wizards in the region. They are worth zero for the time being. Travel to the next major settlement may change that.
How rare is each spell in your setting?
Uncounted thousands of spells have been created over the course of the multiverse’s history, and many of them are long forgotten. Some might yet lie recorded in crumbling spellbooks hidden in ancient ruins or trapped in the minds of dead gods. Or they might someday be reinvented by a character who has amassed enough power and wisdom to do so. (Chapter 10, Spellcasting, What is a Spell?)
If some of those 4th and 5th level spells are comparatively rare, the price for those spells alone can massively swing the price of the book(s) upwards.
Option: trade for a magic item
These books have significant magic in them; it might be more fitting to try and trade these tomes of magic for a magic item in the rare to very rare category, if such can be had. This is another opening for a mini-quest or adventure to find and negotiate a trade with a special NPC.
1 Using the description in the PHB, it had also cost at least 202 hours of time for someone, but how to measure that in terms of value is an open question.