This is fundamentally a question that can only be answered by specific DMs at particular tables
The spell description for bestow curse barely even tries to give guidance here:
You touch a creature, and that creature must succeed on a Wisdom saving throw or become cursed for the duration of the spell. When you cast this spell, choose the nature of the curse from the following options.
- Choose one ability score. While cursed, the target has disadvantage
on ability checks and saving throws made with that ability score.
- While cursed, the target has disadvantage on attack rolls against
you.
- While cursed, the target must make a Wisdom saving throw at the start
of each of its turns. If it fails, it wastes its action that turn
doing nothing.
- While the target is cursed, your attacks and spells deal an extra 1d8
necrotic damage to the target.
[...] At the DM's option, you may choose an alternative curse effect, but it should be no more powerful than those described above. The DM has final say on such a curse's effect.
I mean, it's a little bit overdetermined, since literally everything in a game of D&D is at the DM's option and is subject to the DM's final say. The emphasis on these in the description of the spell itself suggests to me that that the rules as written are not intended to touch on the case of a customized bestow curse effect.
The described guidelines for curse effects also seem like they would be very case-dependent for specific applications. For example, even if we all agreed that a permissible effect of casting the spell could be changing the target's race, changing a bear into a human seems worse than any of the listed effects:
The bear's physical stats would be changed (possibly negatively), it would lose its natural weapons, and it would lose some features of its stat block such as multiattack. Losing a multiattack involving claws and biting and gaining a basic human's unarmed strike is much worse than having disadvantage on ability checks using a single ability score. So even if transformations, generally, would be permissible, this particular one violates what little guidance we have available.
A popular general guideline is to not allow spells to overlap with higher level spells
There are already spells that can definitely change a creature's race, such as polymorph and true polymorph. But the lowest level of those is polymorph, which is a 4th level spell. If bestow curse can transform creatures in the manners described in the question, then polymorph suddenly has a lot less going for it-- you can get mostly the same results off of a 3rd level spell slot. That means that characters will have access to the transformation effect earlier, and less expensively, than the plain text of the rules suggest, and all based on a very vague wording that explicitly charges the DM to evaluate case-by-case.
Thematically, a transformation curse of indeterminate duration seems like it should be relatively uncommon
There are a lot more 4th level spellcasters than 9th, but backgrounds like monstrous origin seem to be fairly rare in-universe. Details of the setting are always up to the DM, but there are implications of having such powerful effects be accessible to so many and they seem absent from settings as described (RAW).
