Can a 'curse' or 'polymorph' transform any creature into a humanoid?
In Volo's Guide to Monsters, a note suggests a 'curse' can turn someone into another humanoid - the 2nd row of the Monstrous Origin table (p. 118) says:
You are the victim of a Curse or Polymorph spell.
Mr. Geddarm's book suggests that one could become a so-called 'monstrous' / let's not judge playable (N)PC via either of these two source-spells. Note: Volothamp is portrayed as an unreliable source (guest writer 'Elminster Aumar' expands on this) / arguably not RAW.
Let us briefly explore each spell type below:
Bestow Curse (abridged):
At the GM’s option, you may choose an alternative curse effect, but it should be no more powerful than those described above. The GM has final say on such a curse’s effect.
This suggests Bestow Curse can do virtually anything, so long as it impacts the recipient less than the listed curse set. This leeway is explicitly subject to table rulings / under DM discretion.
The new form can be any beast whose challenge rating is equal to or less than the target’s.
This should simply NOT work. Though one can argue that the classification of mammal obviously applies to most beasts and most humanoids, 5e D&D is unique in that humanoids are NOT viable / selectable end-forms with this 4th level spell (no other edition had this limitation-severity). Note: this DOES work with the more powerful True Polymorph spell - especially if concentration is held for an hour. At 9th lvl, is not an abuse of RAW, RAI or RAF.
Let us assume that the author(s) of Volo's Guide meant True Polymorph / kept it simple &/or elegant.
Can the Bestow Curse spell alter one's form?
Three possible interpretations:
Rule of Cool:
Why not? This racial modification is far less grievous, arduous &/or impactful than any of the other listed Cursed possibilities in the PhB (q.v. Basic Rules, pg. 218). Cursed race-change = 'modify your character's abilities &/or innate magical powers appropriately'.
A 3rd level spell ('Bestow Curse') CANNOT out-perform a 4th level spell ('Polymorph'). So deeply abusable! A variant human paladin gains long-range Darkvision, more strength - and easily overtakes an entire orc tribe! Not a 'curse' at all, is it? Bestowing this 'curse' on a rabbit and making a pop-up elven ally is also right out of the question ('This curséd rabbit is our archer... and also does our cooking and cleaning! Poor thing.').
Some R.A.W. blending:
Possibly give slight change in mass and appearance - and nothing else. Thus, the aforementioned vuman paladin would look a bit like an orc but be the same size-weight (NPC orc: 'Hm. You thin. Comrade orc, eat this sand-witch!'), yet not gain any magical or racial abilities. Logically this must NOT not work on rabbits... or Purple Worms... or anything that did not start as a living-breathing humanoid in the first place.
The main problem with this third ruling: some racial abilities are amazing curses (arguably the point of said spell &/or Volo's RAW option). Example: Drow sunlight-blindness IS a curse - with bonus universal hatred for any UnderElf that isn't Drizzt. Second example: a PC that looks, smells, sounds and walks as a zombie (no other disadvantages) is a fantastic curse - both story-wise and 5e D&D thematically (especially if the PC goes to town unaware / cannot see his or her appearance).
Basic question / sum up:
If 'Bestow Curse' spell transforms any creature into a humanoid of the caster's choosing (on failed save) - how does any self respecting DM keep power-balance &/or player abuse-ability in RAW-check?