Is the barbarian too far away to melee attack the otyugh?
Rules as written, yes, the barbarian is too far away. A barbarian's melee attack range is 5 feet (Melee Attacks, PHB pg 195), and the Otyugh's tentacle attack range is 10 feet (MM pg 248). If you are grappled, your speed is 0 and you cannot move (PHB pg 290). The Otyugh has a special grapple condition that restrains a target, which also makes your movement speed 0 and gives you disadvantage on attacks (PHB pg 292). The monster's tentacle acts as its weapon, and the monster's "hit box" is in a different area. The same is true of the barbarian, he can attack anything within 5 feet of him, but that doesn't change his position to equal the edge of the range. Thus, the Otyugh is grappling you from 10 feet away, out of your range. Thanks to Airatome for also mentioning that the Otyugh's tentacles are not listed as valid target, as opposed to the Roper (MM pg 261), which has it's tentacles listed as targets with AC and hit points. These targets, and lack of targets, in tentacles are built into the challenge rating.
However, as you have mentioned, it seems rather illogical that you cannot attack the tentacle since it IS wrapped around your body. This is an opportunity for you to role-play with your encounter and be creative. Indicate to your DM what you'd like to do. After that, it's up to him/her if you can do it.