No, the spell does not fail.
The entirety of the rules for this spell are in the spell description. The first paragraph reads:
You touch a length of rope that is up to 60 feet long. One end of the rope then rises into the air until the whole rope hangs perpendicular to the ground. At the upper end of the rope, an Invisible entrance opens to an extradimensional space that lasts until the spell ends.
You touch a length of rope. It rises into the air (and continues to rise until the whole rope is perpendicular to the ground). The upper end of the rope has an invisible entrance to an extradimensional space. As far as the rules are concerned, all of this occurs instantly and simultaneously.
Nowhere does it say that the spell fails under any circumstances.
It's a judgment call by the DM what the rope actually does if the spell is cast in a space with not enough room for the whole rope. But "at the upper end of the rope, an Invisible entrance opens" regardless.
It doesn't say that the extradimensional space opens after the "whole rope" has risen into the air. It just says that it opens at the top end of the rope.
The way I've always seen this played is that the rope simply rises to the ceiling and the door to the extradimensional space is in the ceiling. There's good precedent for that in previous editions of D&D. In AD&D 1st edition, the Rope Trick spell description reads as follows:
The caster causes a rope from 5' to 30' in length to vertically straighten, rising into the air as if afixed by its upper end. The upper end is fastened to an extradimensional space that the caster and up to 5 others can enter, one at a time.
In this version of the spell, it's much more clear that the extradimensional space is already open when the rope begins to rise. This provides a resolution to the ambiguity of the 5th edition version. The DM could simply rule that this is how the spell still works in 5e. There's nothing in the 5e spell description to contradict that.