Resting is not the same as sleeping. The rules make no effort to encourage that distinction, but they're unambiguous on it.
A rest is a period of downtime. When you reach 8 hours, you get the many benefits of a long rest. You're limited to a single one every 24 hours, though.
For humans and most player races, the benefits of 8 hours of sleep are as follow:
- resetting the countdown to exhaustion from lack of sleep
That's all. Elves get that "same benefit" in 4 hours of trance (and Warforged in their 4-hour "sleep-like state"). If they're gonna trance as part of a long rest, they can stay fully alert the other 4 hours.
The source of the (widespread) confusion is clear: for most races, both sleeping and resting take 8 hours of downtime and are once-a-day things. Since 8 hours of sleep also fulfil long rest requirements, you might as well always combine the two.
There's almost no official information about sleep in the books. As stated by the question asker, you can sleep during a long rest and a handful of races can fake a night of sleep in half the normal time. There's also the sleep status effect, irrelevant here. And funnily, a tent has stats even though sleep itself doesn't. So we have to induce the designers' intentions from almost nothing.
Lack of sleep "might" call for a constitution check:
Constitution checks: [...] The DM might call for a Constitution check when you
try to accomplish tasks like the following: hold your breath; march or labor for hours without rest; go without sleep; survive without food or water; [...].
Most of the other problems on that list cause exhaustion:
Forced March: The Travel Pace table assumes that characters travel for 8 hours in day. They can push on beyond that limit, at the risk of exhaustion. For each additional hour of travel beyond 8 hours, [...] each character must make a Constitution saving throw at the end of the hour. [...] On a failed saving throw, a character suffers one level of exhaustion.
Food and Water: Characters who don’t eat or drink suffer the effects of exhaustion. Exhaustion caused by lack of food or water can’t be removed until the character eats and drinks the full required amount.
The designers' intention is probably that the DM call for constitution checks when he feels players have gone without sleep "long enough". Like exhaustion from lack of food or water, exhaustion from sleep could only be removed by actually sleeping.
The fourth edition of D&D had similar sleep and rest rules, but they were slightly more specific, so let's check them out:
Rest and Recovery: [...] At least 6 hours long, an extended rest includes relaxation, sometimes a meal, and usually sleep.
Sleeping and Waking Up: You need at least 6 hours of sleep every day to keep
functioning at your best. If, at the end of an extended rest, you haven’t slept at least 6 hours in the last 24, you gain no benefit from that extended rest.
Interruptions: If anything interrupts your extended rest, such as an attack, add the time spent dealing with the interruption to the total time you need to spend in the extended rest.
Keeping Watch: Adventurers typically take turns keeping watch while their companions sleep. If five characters are in your group, each of you can take a turn on watch duty for
1½ hours and sleep for 6 hours, so that you spend a total of 7½ hours resting.
Clearly, even though they were easy to combine, sleep and extended rests were distinct! Sleep could even be split up, unlike rest, and its lack had no penalty other than preventing extended rests. Still, eladrin (high elves) seem to contradict those rules:
Trance: Rather than sleep, eladrin enter a meditative state known as trance. You need to spend 4 hours in this state to gain the same benefits other races gain from taking a 6-hour extended rest.
It appears the (widespread) confusion is much older than we thought!
As for earlier editions, I think they had sleep itself as the standard long-term recovery mechanic.