If they have experienced it, or if they are familiar with it
Booming Blade has a visible effect (SCAG)
it becomes sheathed in booming energy until the start of your next turn.
If the target is knowledgeable of this spell or has experienced it before, then they should know that moving will make them go boom.
Determining if they know
This is going to get into DM purview here. A DM may also consider the intelligence of the creature and whether or not either of the options below are even available to determine if they are familiar enough with the spell/spellcasting to identify it:
- They have experienced it before
- An Arcana check
Whether or not the bottom option is viable will depend on your DM.
A DM can also reference Xanathar's for guidance in determining the DC under the Chapter 2: Spellcasting:
If the character perceived the casting, the spell’s effect, or both, the character can make an Intelligence (Arcana) check with the reaction or action. The DC equals 15 + the spell’s level. If the spell is cast as a class spell and the character is a member of that class, the check is made with advantage. For example, if the spellcaster casts a spell as a cleric, another cleric has advantage on the check to identify the spell...
This Intelligence (Arcana) check represents the fact that identifying a spell requires a quick mind and familiarity with the theory and practice of casting. This is true even for a character whose spellcasting ability is Wisdom or Charisma. Being able to cast spells doesn’t by itself make you adept at deducing exactly what others are doing when they cast their spells.
As a side note, I personally don't require reactions/actions to be used for an Arcana check and just let the PC roll for it if they ask.
What is Booming Energy anyway?
Whether or not Booming Energy is an perceivable effect also falls under DM purview. However, given that the spell is explicit in the target being sheathed in it, it does seem likely that the target perceives it (either in feeling, sound, or sight.) You could do a Perception (Wisdom) and set a DC you think is reasonable - but I'd likely rule that the effect is obvious to the target given the language of sheathed in booming energy.
We can also look to Jeremy Crawford and his Twitter response on Booming Energy:
Booming blade does make noise. The spell's description says the target is surrounded by booming energy. The DM decides who can hear the booming.