Dissonant Whispers states:
You whisper a discordant melody that only one creature of your choice within range can hear, wracking it with terrible pain. The target must make a Wisdom saving throw. On a failed save, it takes 3d6 psychic damage and must immediately use its reaction, if available, to move as far as its speed allows away from you. The creature doesn't move into obviously dangerous ground, such as a fire or a pit. On a successful save, the target takes half as much damage and doesn't have to move away. A deafened creature automatically succeeds on the save.
I had an incident where a PC was being grappled and bashed about by an Otyugh. The character was "being flailed about" in the grapple (which also restrains the PC), and used Disonnant Whispers successfully. I had the Otyugh release the grapple by chucking the PC away from it as it's reaction (like an tentacle slam, but away and 2d6 falling w/o stun). It moved the frightening thing (PC) away from the Otyugh, which would have otherwise been dragged in the grapple, and since it used it's reaction, it stayed where it was.
The players, of course, were like "Booo" but I honestly think it would have fit the scenario, since the character was being grappled/restrained by the tentacle and was already being flailed about. Did I make an unfair ruling?