There are no general rules for ability damage, almost certainly because ability damage in 5e is incredibly rare.
With that said, there are some sources of ability damage, so it's worth taking a look at them. The following effects are the only sources of ability damage I can find:
- The Intellect Devourer's Devour Intellect ability (MM 191), which can reduce a target's Intelligence to 0. If this happens, the target is stunned until it regains a point of Intelligence.
- The Shadow's Strength Drain ability (MM 269), which reduces a target's Strength by 1d4. If their Strength score reaches 0, they die.
It's not much to reach a conclusion from, but it seems clear that the consequences of an ability score reaching 0 depends on what ability score it is. It may even depend on the effect that causes it; perhaps Intelligence reaching 0 from Devour Intellect is different to Intelligence reaching 0 from some dastardly Intelligence-reducing disease.
It's possible, but by no means guaranteed, that a mental ability score reaching 0 causes a creature to be stunned until it regains a point of that ability score, and a physical ability score reaching 0 causes a creature to die. Obviously, this is based on the 2 examples we currently have, so it's not exactly scientific.
With all of that said, if, in your game, someone's ability score reaches 0 for a reason other than these 2 effects, it must be an effect you've added. That being the case, it's up to you to decide what the consequences of this occurring would be.