College does not equal College
Or maybe rather, the same word is used for two different concepts. We can most easily start with the seven. They refer to actual (in world historic) institutions as described in Sword Coast Adventurer's Guide:
[...] Sword Coast has some bardic colleges where masters teach students the bardic arts. They hark back to the great bardic colleges of the distant past, particularly the seven elder colleges: Fochlucan, Mac-Fuirmidh, Doss, Canaith, Cli, Anstruth, and Ollamh.
— Chapter 4: Classes > Bard > Bardic Colleges
It then goes on to describe two of the decendants of those colleges (and one other college) and includes for all:
Most of its bards study and practice the methods of the College of Lore
That college or persuasion of bard would appear to the most associated with the actual college institutions, as included in the intro to their subclass is:
The college’s members gather in libraries and sometimes in actual colleges, complete with classrooms and dormitories, to share their lore with one another.
Also, of note that at the time of the publication of all these books (ie. PHB, DMG, SCAG) there were only two published subclasses for the Bard and that some of the newer to make up the now seven were (originally) published in campaign books for settings far removed from the Forgotten Realms.
Also note that any attendance or association with an (actual institution) college is entirely optional for any bard you create, it is given as an idea suggestion among others in the Creating a Bard section of the class description.
Did you serve an apprenticeship, studying under a master, following the more experienced bard until you were ready to strike out on your own? Or did you attend a college where you studied bardic lore and practiced your musical magic? Perhaps you were a young runaway or orphan, befriended by a wandering bard who became your mentor. Or you might have been a spoiled noble child tutored by a master. Perhaps you stumbled into the clutches of a hag, making a bargain for a musical gift in addition to your life and freedom, but at what cost?
For any DMs with their worldbuilding hat on, implementing the bardic subclasses as actual colleges in a given world is discussed on DMG p. 24 albeit briefly among wizard schools and druidic circles.