Yes this will work, but...
First off, Enlarge/Reduce can certainly make boulders smaller:
The target's size is halved in all dimensions, and its weight is
reduced to one-eighth of normal.
However, these boulders have to meet the requirements below for Catapult to shoot them:
Choose one object weighing 1 to 5 pounds within range that isn’t being
worn or carried.
So, you're looking at a boulder that is at most 40 pounds in weight. How big is that? I don't make a habit of wandering around the countryside weighing rocks, but if I had to guess I'd say probably a little bigger than your head. So far, we're good to go...
D&D 5e doesn't make rules to handle things like mass, inertia, coefficients of drag, and muzzle velocity...so all we're left with is the actual spell description. And, according to the Catapult spell:
the object and what it strikes each take 3d8 bludgeoning damage.
So, yes you can totally Reduce a boulder to a size that the Catapult spell will work on it, and yes you can even drop concentration after it is shot toward a target, but the spell does 3d8 bludgeoning damage. Period.
The damage comes from the spell, not from what you use as the projectile. You can use a pound of soft butter, 3d8 damage. A pound of feathers, 3d8 damage. You can Reduce a boulder, 3d8 damage.