Focusing on the specific effect of the Enlarge/Reduce spell:
You cause a creature or an object you can see within range to grow larger or smaller for the duration.
Consider a human who is, for reasons which will become obvious, named Inflatoman.
Inflatoman is the target of an Enlarge spell cast by Engorgo. He grows to twice his size becoming a large creature.
While the Engorgo maintains concentration; Reducto hits Inflatoman with a reduce spell, which clearly causes him to revert to his original size. While these spells have not cancelled each other, their effects have.
So, what happens:
if Engorgo stops concentrating while Reducto keeps concentrating:
- Inflatoman stays his normal size; Reducto's spell has already done all the reducing it can do.
- Inflatoman shrinks to small size; with Engorgo's spell removed the effect of Reducto's reasserts itself.
if Reducto stops concentrating while Engorgo keeps concentrating:
- Inflatoman stays his normal size; Engorgo's spell has already done all the enlarging it can do.
- Inflatoman grows to large size; with Reducto's spell removed the effect of Engorgo's reasserts itself.
An alternative way of asking this is: Is the growth from an Enlarge spell a one shot effect that causes the target to grow at the beginning and then shrink at the end OR a continuous effect that is trying to make the subject be twice the size they would otherwise be? Vice versa for Reduce.