If someone has a fly speed of 60 feet and moves 60 feet downward and then chooses to stop flying, they would descend another 500 feet totalling 560 feet in a single round. Is this correct interpretation of Xanathar's falling rules (page 77)?
This aligns with the rules for falling in Xanathar's. The only hiccup is that, rules as written, a flying creature only falls in three circumstances:
If a flying creature is knocked prone, has its speed reduced to 0, or is otherwise deprived of the ability to move, the creature falls, unless it has the ability to hover or is being held aloft by magic, such as the fly spell (PHB 191)
You can't reduce your own speed to 0, and are unlikely to want to deprive yourself of the ability to move, so your only option, officially, is to drop prone. This means you need to either keep going until you hit the ground, or "stand up" on a later turn to end the fall, which will cost half your movement:
But if that creature starts any of its later turns still falling and is prone, it can halt the fall on its turn by spending half its flying speed to counter the prone condition (as if it were standing up in midair).
So you can do this, and travel a great distance straight downward, but you're giving up mobility on a future turn (and possibly signing up for 20d6 splatting damage if you fail to arrest the fall for whatever reason).
Your interpretation is correct
If you can choose to stop flying, in addition to your movement speed from flying downwards, you also can drop 500 feet (when using the rules in Xanathar's Guide to Everything, otherwise stopping to fly will make for an immediate, painful encounter with the ground).
The tricky part is what it means to "choose to stop flying". A creature with a natural fly speed cannot simply opt to lose that fly speed. So how would it work to choose to stop flying?
There is a rule in the PHB (p. 191) that discusses things that can force a creature to fall:
Flying creatures enjoy many benefits of mobility, but they must also deal with the danger of falling. If a flying creature is knocked prone, has its speed reduced to 0, or is otherwise deprived of the ability to move, the creature falls, unless it has the ability to hover or it is being held aloft by magic, such as by the fly spell.
While being knocked prone is not the same as falling prone, other answers read it as having the prone condition while flying will cause you to fall, and interpret falling prone as the rules mechanic to stop flying when you have a flying speed, which you can do freely (p. 190 PHB)
You can drop prone without using any of your speed. Standing up takes more effort; doing so costs an amount of movement equal to half your speed.
Stopping to fly
I think this should work, but do you really need to fall prone? This answer assumes that you can just opt to stop flying at any point. To me that makes more sense - unless you have a magical hover speed or other magic that holds you aloft that you cannot get rid of, flying ususally requires an active effort using your wings (or whatever means of propulsion you have). Forgoing this effort should make you fall, without needing to drop prone.
There is also the rule that if you have a speed other than a flying speed, you can switch between your speeds (page 190 PHB):
If you have more than one speed, such as your walking speed and a flying speed, you can switch back and forth between your speeds during your move.
If you switch to your walking speed while in the air, you should drop like any other creature that is in the air and only has a walking speed. If you look for a mechanical rule to just stop flying, you can employ this approach.
One question might be if "during your move" includes after you used up all your movement, that is, when does your move exactly end? The movement rules indicate that "your move" is not really a single event, you can break it up (p. 190 PHB), so I think you should be able to switch speeds even after you used up all your movement. If your DM thinks otherwise, you might opt to move less, switch, fall, switch again and move the rest:
However you’re moving, you deduct the distance of each part of your move from your speed until it is used up or until you are done moving. (...) You can break up your movement on your turn, using some of your speed before and after your action.
Lastly, if you are held aloft by magic like a fly spell, do you need to drop concentration to fall, or can you opt to fall at any time?
Following the logic above, you should be able to fall at will by switching to your walking speed, as the spell adds a fly speed, it does not replace your walking speed. However, that is not how I have seen it in play, possibly due to the exception of such flight from the being forced to fall rule. So you should be prepared that your DM rules if you have such magical sources holding you aloft, you will need to get rid of them to fall.