They will not die
Once they reach exhaustion Level 5 their speed drops to 0 and they can no longer swim and therefore no longer risk more exhaustion.
How far can they go before this happens?
You really need to change your handle if you can't work out a simple markov chain probability calculation like this one :-)
Assume your modifier to your exhaustion check is \$n\$. Your chance of passing a DC\$X\$ check is \${21+n-X}\over{20}\$.
Once you fail your third check you have disadvantage on your checks. This reduces your chance of passing to \$\left({21+n-X}\over{20}\right)^2\$.
You cannot go any further once you fail 5 checks as your speed equals 0.
You can Dash giving you a speed of 60 feet per round, however, water is difficult terrain for a non-swimmer so this means you can only move 30 feet.
If you swim straight down you spend 3 rounds at less than 100 feet, another 3 rounds at less than 200 feet and after that you are deeper than 200 feet.
You need to check based on swimming each hour at DC10, that is every 600 rounds.
You need to check for a forced march at DC10 after 8 hours equivalent. This is after 3 rounds at less than 100 feet, 3 rounds at less than 200 feet and 1198 rounds deeper than 200 feet, that is, after 2 hours 24 seconds.
You then need to make forced march checks every 150 rounds (15 minutes) at DC11, DC12, DC13, ...
This is a Markov chain with 1 starting state and 2 intermediate states (that are all have the same transition probabilities), 2 more intermediate states (that are all have the same but different from the first group transition probabilities) and 1 terminal state. The only complication is the transition probabilities change over time.
Putting this altogether and calculating to 2 significant figures (i.e. a less that 1 in 200 chance of doing better), a person with a +0 Constitution modifier and no proficiency in Constitution saving throws can go 14.5 miles in 4:15:24. A person with +5 Constitution modifier and +6 proficiency bonus can go 24.7 miles in 7:15:24.
Given that the Mariana Trench (the deepest part of the Earth's ocean) is just under 7 miles deep this is not a major impediment. Indeed, even the +0 wimp has a better than a 50% chance of getting to the bottom with 1 or less levels of exhaustion.
Spreadsheet