A group of PCs without ranged weapons is facing a gargantuan creature (e.g. Tolkien's Oliphaunts). Obviously, if they don't climb the creature, they can only hit a small and non-lethal part of it (such as the paws). However, I suppose that damage is computed normally. What happens if creature's HP are reduced to 0 in this way? Is the creature dead even if no lethal wound has been inflicted?
Yes, the creature is dead when it reaches 0 hit points.
Basic Rules (p. 75)
When you drop to 0 hit points, you either die outright or fall unconscious, as explained in the following sections.
Basic Rules, DM (p. 3)
A monster usually dies or is destroyed when it drops to 0 hit points. For more on hit points, see the player’s D&D basic rules or the Player’s Handbook.
- Exception: a character can choose to make the last blow a knockout blow and just render it unconscious. This fits your "melee attack only" scenario. (No ranged weapons).
Knocking a Creature Out - Basic Rules (p. 76)
Sometimes an attacker wants to incapacitate a foe, rather than deal a killing blow. When an attacker reduces a creature to 0 hit points with a melee attack, the attacker can knock the creature out. The attacker can make this choice the instant the damage is dealt. The creature falls unconscious and is stable.
Hit Points are an abstraction that have been found to be useful for combat resolution in D&D over the years. They represent more than just raw wounds.
Basic Rules (p. 74)
Hit points represent a combination of physical and mental durability, the will to live, and luck. Creatures with more hit points are more difficult to kill. Those with fewer hit points are more fragile. A creature’s current hit points (usually just called hit points) can be any number from the creature’s hit point maximum down to 0. This number changes frequently as a creature takes damage or receives healing. Whenever a creature takes damage, that damage is subtracted from its hit points. The loss of hit points has no effect on a creature’s capabilities until the creature drops to 0 hit points.
There is a term in English called "death by a thousand cuts" which refers to a lot of little wounds eventually killing you. You could apply that analogy to the scenario you described in battling the gargantuan sized beast.
For your Oliphunt, let's think this through: if you hit his feet often enough, or his lower legs, that foot/leg will eventually get weak enough that it won't work as well in supporting that massive weight, so that the beast is more likely to stumble or fall or go to a knee on that foot, possibly making a more vulnerably area more reachable. This can happen for a brief moment within a 6 second combat turn, and not require it to be put into a Prone Condition. (Appendix A, p. 105, Basic Rules, Conditions).
That said, D&D 5e does not get into that level of granular or simulationist detail of vital areas being struck1. It does offer damage bonuses on Critical Hits (a roll of natural 20, Basic Rules p. 75) or emulating hitting vulnerable areas with the Sneak Attack ability of a Rogue that deals bonus damage. (Basic Rules p. 27)
PHB p. 197: Describing The Effects of Damage
Dungeons Masters describe hit point loss in different ways. When your current hit point total is half or more of your hit point maximum, you typically show no signs of injury. When you drop below half your hit point maximum, you show signs of wear, such as cuts and bruises. Any attack that reduces you to 0 hit points strikes you directly, leaving a bleeding injury or other trauma, or it simply knocks you unconscious.
You can narrate the damage being done to this behemoth as follows:
A number of the early hits weaken the large monster to the point that it might stumble or falter and make it easier for a character to reach and hit it in the belly, heart, eye/brain, whatever as it reduced in HP
More and more blood flows out of the various wounds until it can't go on and it falls.
(For a real life illustration: a single wound to the femoral artery can result in you bleeding to death if you don't get first aid/wound repair applied to you. A lot of cuts and attendant loss of blood, even if none hits an artery or major organ, can result in sufficient blood loss that you die.)
1The 5e D&D rules don't have a provision for specifically targeting parts of the body, which sort of makes your basic question moot. Some other RPG's do. An explanation of why this has been avoided in this game can be found in this answer to a related question.