Hypothetically I've managed to shackle up a potentially dangerous opponent. As the risk of them escaping can be large, I want to keep them hurt or hurt them such that they would not be a threat in that situation. As we know, long and short rests allow characters to heal back their HP. However, the manual is vague in determining if my shackled up enemy is able to take these rests. Does it come down again to a GM decision? Otherwise, how can I prevent it from resting and regaining its HP after a period of time?
There are techniques designed to deprive prisoners of rest that are used to this day to break down resistance and weaken their physiology. Bright lights and noises at random intervals, bad food that causes intense gastric distress, imprisonment in awkward positions such as standing or crouching with no way to sit or lie down, "hot boxes" where the prisoner is subjected to extremes of temperature with no water, frequent beatings, throwing dangerous animals such as scorpions into their cell--The list is nearly endless.
In a fantasy world, there are even more options: damaging spells, distressful illusions, creatures with magical damage abilities (such as the death field a bodak, for example, or the stench of a ghoul), or even homebrewed poisons or potions.
Anything that would qualify as making the period of inactivity sufficiently "strenuous" should suffice.
I know you didn't ask about the corollary, but it might be prudent to mention that if you do the same to PCs, players are likely to balk at this and rightly so. The default state of D&D is High Adventure, in which the heroes are... heroes, not victims. It might help to reassure them that this is a temporary state and an opportunity for roleplay. It also helps to point out that their captors could have just killed them instead of imprisoning them, and that you are presenting them with an alternative to a TPK. I recently went through just this scenario, and with that little reassurance at the beginning, they were able to have fun resisting interrogation and trying to get as much info out of their captors as the captors were seeking to gain. All before the inevitable escape and rematch, of course.
Resting is supposed to be, for lack of a better word, restful. From the PHB:
A short rest is a period of downtime, at least 1 hour long, during which a character does nothing more strenuous than eating, drinking, reading, and tending to wounds.
A long rest is a period of extended downtime, at least 8 hours long, during which a character sleeps or performs light activity: reading, talking, eating, or standing watch for no more than 2 hours.
While there is no specific rule that states imprisonment is by definition strenuous, it certainly can be. And if you're looking to keep the imprisonment strenuous enough that the hypothetical opponent can't recover, then the bar you have to cross for it to qualify is fairly low- a long rest is broken if you spend more than two hours watching for trouble, or even one hour walking. You could easily make the argument that, say, hanging from a set of irons would qualify... or invite your DM to try and sleep in such a fashion if they balk.
If your DM has defined short and long rests differently, of course, you'll have to address those if you want to keep your opponent from resting (in my campaigns I tend to treat a short rest as "long enough to catch your breath," for example). And if your DM has previously laid out conditions that keep you from resting- if, for example, they've ruled that you can't benefit from a long rest while wearing heavy armor- you can take advantage of that as well. I'll grant that it's a bit odd to simply keep your enemy in a suit of full plate at all times, but desperate times call for desperate measures, aye?
Keep knocking them unconscious (PHB p. 198):
A stable creature that isn’t healed regains 1 hit point after 1d4 hours.
When they wake up - knock them out again.
As being unconscious is the more specific rule, its effects override the effects of resting.