About 5000 gp big.
The primary in game identifier for diamonds is their value, so this is a question for your DM. Since your DM is the one responsible for world-building, it is up to them how the people of the world relate to valuable gems and place value on their size. There is no such relationship described in general in any campaign setting or rulebook.
To illustrate this point, consider this rule about selling gems found in chapter 5 of the Player's Handbook:
These items retain their full value in the marketplace, and you can either trade them in for coin or use them as currency for other transactions. For exceptionally valuable treasures, the DM might require you to find a buyer in a large town or larger community first.
The important takeaway here is: their full value at the marketplace is the value referenced in the description of costly spell components. If a diamond would fetch less than 5000 gp at the marketplace, it is not suitable for casting a spell requiring a 5000 gp diamond.
So the question a DM must consider when valuing a diamond is:
Since the value of gems is always exchangeable for exactly their value in gold, how much does this diamond fetch at market?
This is the first concern when evaluating diamonds, and any further description is narrative.
Some campaign-specific examples.
Now, in Waterdeep: Dungeon of the Mad Mage on page 2301, a certain diamond is described:
Pedestal. In the middle of the room, two mind flayers grip the top of a 4-foot-tall, three-sided crystal pedestal with a fist-sized diamond floating 1 foot above it.
We are later told:
Treasure. The diamond is worth 5,000 gp.
So this particular diamond in this particular setting and context is worth 5000 gp and about fist sized, but your DM's setting may be different.
Additionally, in Storm King's Thunder, there is a magic item called a Blod Stone (pg. 233). In its description, we do get a single adjective to point us in the right direction:
This item is made from a large diamond worth at least 5,000 gp.
So a 5000 gp diamond may be described as large, at least in relation to other diamonds, whatever that means.
1Level 17, section 16, "Old Dwarven Halls".