In my campaign, we have traveled in the wilderness for a while. I made a slight mistake and my DM had me break my leg. However things got a little complicated after that, saying that none of us really knew how a broken leg and its disadvantages actually work. SO we just ended up subtracting 1 from my dex, however I'm not really sure if that's how you do it. For future broken ligaments is there a specific way to do it? Do you just subtract off of your Dex? Or Strength? Or is it a disadvantage on rolls? Or is it something else? What should I do?
While there are no rules for simply broken arms or legs there are rules for losing an arm or leg.
From the DMG on Lingering Injuries
Lose a Foot or Leg. Your speed on foot is halved, and you must use a cane or crutch to move unless you have a peg leg or other prosthesis. You fall prone after using the Dash action. You have disadvantage on Dexterity checks made to balance. Magic such as the regenerate spell can restore the lost appendage.
Lose an Arm or a Hand. You can no longer hold anything with two hands, and you can hold only a single object at a time. Magic such as the regenerate spell can restore the lost appendage.
My suggestion for your DM would be to use these effects but allow a somewhat lower level spell (Regenerate is a 7th level spell!) to reverse the effects. You could also allow it to heal on its own over time though that would likely take your character out of the action for a bit. If you opt for that method an internal injury or a broken rib suggests the time should be "ten days doing nothing but resting."
However things got a little complicated after that....
You're right, this is a little complicated. The tl;dr is "there's no hard-and-fast rule for this, but here are some guidelines from the books and from experience."
The Dungeon Master's Guide has some advice...
The authors knew this would come up, have played lots of D&D, and have a suggestion. You'll find it on p.272 of the DMG: "your speed on foot is halved, and you must use a cane or a crutch to move.... You fall prone after using the Dash action. You have disadvantage on Dexterity checks made to balance."
(That comes from the suggestion after losing a foot or leg; from my experience breaking an ankle it sounds pretty good.)
... and I've got some.
This is the sort of thing--a house-ruled system for lingering/non-HP injuries--that can add a lot of flavor to your table and your campaign. Or it can add a lot of headache.
You and your table-mates should have a conversation about the style of story you're looking to play. Are these near-rubber action heroes who shake off giants' fists? Then disadvantage on dexterity saves until the next long rest might be all you need. Are these gritty Rolands to the Dark Tower coming? Then it may make sense to give your character a limp (see DMG272 again) for all time.
"Ability damage" is the term for what you describe: removing 1 from your dexterity score. It's not a bad way to do things, but it's been largely written away from in 5e in favor of disadvantage. Two reasons for that are the potential to stack with other penalties (and violate Bounded Accuracy) and the headache that comes from the cascade of recalculations needed when an ability score changes: initiative, armor class, skill modifiers, saves, and attack bonus, just to name a potential few.
That second point becomes more important the more frequent such changes are; if you're running a gritty table where recovery from this sort of thing isn't expected, that may not be so problematic. If you're jumping off of buildings and getting hit by boulders and taking and recovering from ability damage every session, I (and the authors of 5e) urge you against it.
Creatures in D&D 5e can't break legs
It's as simple as that. The reason you can't find rules for this is that there aren't any.
For example, consider the Sword of Sharpness, a weapon with the explicit ability to sever limbs. It says:
... the effect of such loss determined by the DM.
You say "we just ended up subtracting 1 from my dex, however I'm not really sure if that's how you do it." Well, it's not necessarily how I would do it but it's how you guys did it and since it's your game ...
The Dungeon Master's Guide has advice on lingering injuries
It's on p.272 or on D&D Beyond.
If you like what they do better than what you did, you can use it. But you need to decide what's right for you.