Ask your DM or, if you're the DM, you need to make a call yourself
You've already quoted the relevant rules text:
For the duration, these plants turn the ground in the area into difficult terrain.[...]
When the spell ends, the conjured plants wilt away.
Putting to one side the instantaneous effect of this spell, when the plants appear they cause an area to become difficult terrain.
Logically, if something did not happen to the plants at the end of the spell's duration then the ground would continue to be difficult terrain. But the difficult terrain is explictly only present for the duration of the spell, the rules text reconciles this narratively by saying 'when the spell ends the conjured plants wilt away'.
What does it actually look like when the plants have wilted away? We don't know exactly, because the rules provide us with no further information.
You've suggested two reasonable outcomes: the floor might be littered with dead vegetation, or the plants might have disappeared completely (because magic). What happens in your game is up to the DM - whether that's you or someone else.
All the rules tell us for definite is that, even if there are a lot of dead plants around now, those dead plants don't inhibit movement through that space.