It does not apply, the damage is caused by spending an action - not casting a spell.
As quoted above, Elemental Affinity engages when(emphasis mine):
you cast a spell that deals damage of the type associated with your draconic ancestry...
When you cast dragon's breath, you are not dealing damage. You are granting an ability for whomever it is cast on to(emphasis mine):
use an action to exhale energy of the chosen type in a 15-foot cone.
It is when you use that action, not when you cast the spell, that is dealing the damage.
You are using your action to utilize the effect granted by the spell, but not spell is being cast. The spell is only giving the creature the ability exhale energy.
**You are not meeting the requirement of casting a spell when you are dealing the damage and therefore can not use your Elemental Affinity.
The damage dealing effect is similar to Breath Weapon of a dragonborn.
There is no spell being cast, it is just an ability that uses an action to engage.
When you deal the damage, you are expending an action. Granted, it's an action given to you from a spell, but the damage is dealt from the action - not from casting a spell.
The fact that a spell gave you that ability doesn't change the mechanic of its use. When you're dealing the damage, it is not from casting the spell. It is from using the action granted by the spell (which, again, is not casting a spell.)
Strict vs loose reading
Ultimately, I think there is some ambiguity and it depends entirely on how you read the spell. Given that we are talking about a feat here and one concentration-based spell, I don't think it'd be a huge deal to grant it by stating that the spell is ultimately causing the damage on one breath attack action during the spell duration.
This is somewhat supported by the same initial line. The difference is in granting
cast a spell that deals damage...
to the end-action for the damaging effect and not just the spell itself.
If it becomes overused, then you can always go back to the stricter after discussing it with your player.
But the hard line reading does suggest that this interaction doesn't work given you aren't casting a spell to cause the damage.