Arguably not, but the DM has final say
This seems like a situation where the plain English interpretation conflicts with an implied legal interpretation.
The rule on Targeting in the Spellcasting section states:
A typical spell requires you to pick one or more targets to be affected by the spell's magic [...]
This implies that the spell only "affects" a target during its duration. However, the rules do not explicitly state whether or not the ending/removal of the active effect is to be treated as "affecting in reverse."
Also of relevance, the "Weave of Magic" a little further on describes the lore surrounding D&D's magic system:
All existence is suffused with magical power, and potential energy lies untapped in every rock, stream, and living creature, and even in the air itself. Raw magic is the stuff of creation, the mute and mindless will of existence, permeating every bit of matter and present in every manifestation of energy throughout the multiverse.
Whenever a magic effect is created, the threads of the Weave intertwine, twist, and fold to make the effect possible. [...] A spell such as dispel magic smooths the Weave. Spells such as antimagic field rearrange the Weave so that magic flows around, rather than through, the area affected by the spell.
This block of text seems relevant for two reasons.
First, it states that the Weave is a pervading entity, integrally joined with all creatures and physical matter. Affecting the former is to affect the latter.
Second, it describes the nature of the invisible changes caused by using magic. The text suggests that the Weave is distorted from its natural arrangement by spellcasting and that, after a spell's (altering/positive) effect ends, it might revert back to its original arrangement.
The sum of this is the implication that, in a plain English sense, magic affects creatures both in its active effect and in its termination. This is based entirely in lore, however, and has no corresponding rules support (supporting or disagreeing).
The end result, as so often is the case, is that a DM needs to provide a ruling on this. There is arguably more legal support for "Dropping concentration does NOT count as affecting a creature."
It is also worth considering that, if you take the approach that "affecting the weave in any way is to affect the creatures it flows through (in some lesser way)" then you create a new conflict with spells like Antimagic Field which are intended to prevent spells from affecting creatures.
