Optional rules for this can be found in Xanathar's Guide to Everything
On page 85 (or on D&D Beyond) there is a section on "Identifying a spell" that reads:
Sometimes a character wants to identify a spell that someone else is casting or that was already cast. To do so, a character can use their reaction to identify a spell as it's being cast, or they can use an action on their turn to identify a spell by its effect after it is cast.
It goes on to explain in detail what checks are involved (not transcribing everything here for copyright reasons). Do note that the rules presented in Xanathar's are considered optional/variant rules.
Using these rules with Counterspell
Unfortunately these rules do require a reaction to identify the spell being cast which means that your reaction is expended and you can't usually use this along with Counterspell if you are alone, unless the spell is being cast over several rounds.
You might be able to achieve this with two characters: one of them identifying the spell for the other to cast counterspell.1 (A strict reading of the rules may not allow speaking outside your turn, see this Q/A. However, I have never played with a DM that actually enforces this.)
Jeremy Crawford has also addressed this issue (though unofficially) in a series of tweets found on this Sage Advice page2. Notably, these two exchanges are of particular interest:
Harvey Williams @HarvDWilliams 9 Nov 2017
Would it be reasonable that one person could identify a spell and another counterspell it with that knowledge?
Jeremy Crawford @JeremyECrawford 9 Nov 2017
And then again later on:
roscoeivan @roscoeivan 10 Nov 2017
What if you use your reaction to identify it and communicate the spell to some one else who has their reaction left?
Jeremy Crawford @JeremyECrawford 10 Nov 2017
So it seems that though unofficially, Jeremy Crawford also subscribes to the idea of allowing one person to identify a spell and telling another so they can counterspell (at least as of the writing of this answer).
- Suggested by enkryptor
- Found by V2Blast