Rules as Intended, if consistency is to be maintained between Ritual Caster and Book of Ancient Secrets:
Ritual Caster
Choose one of the following classes: bard, cleric, druid, sorcerer, warlock, or wizard. You must choose your spells from that class’s spell list, and the spells you choose must have the ritual tag.
If you come across a spell in written form, such as a magical spell scroll or a wizard’s spellbook, you might be able to add it to your ritual book. The spell must be on the spell list for the class you chose, the spell’s level can be no higher than half your level (rounded up), and it must have the ritual tag.
Book of Ancient Secrets
On your adventures, you can add other ritual spells to your Book of Shadows. When you find such a spell, you can add it to the book if the spell's level is equal to or less than half your warlock level (rounded up) and if you can spare the time to transcribe the spell. For each level of the spell, the transcription process takes 2 hours and costs 50 gp for the rare inks needed to inscribe it.
The constraints are that the spell be physically written down, or inscribed, equal or less than half your level, and you have the time and materials to transcribe it. This follows the rules interpretation provided by Jeremy Crawford and Mike Mearls, where the explained the Pact of the Tome Warlock with the Book of Ancient Secrets must find the ritual spells written down.
You have already identified the prime sources: other spell books and scrolls, if the spell has the ritual tag.
Additional locations could be allowed, or rather included, by the DM for strategic reasons. A story driver ritual spell could be etched into the wall of a dungeon, serving as a key for a future dungeon.