No, the spell does not fail.
The entirety of the rules for the rope trick spell are in the spell's description. The first paragraph reads:
You touch a length of rope that is up to 60 feet long. One end of the rope then rises into the air until the whole rope hangs perpendicular to the ground. At the upper end of the rope, an invisible entrance opens to an extradimensional space that lasts until the spell ends.
So what happens is:
- You touch a length of rope.
- It rises into the air (and continues to rise until the whole rope is perpendicular to the ground).
- The upper end of the rope has an invisible entrance to an extradimensional space.
As far as the rules are concerned, all of this occurs instantly and simultaneously. Nowhere does it say that the spell fails under any circumstances.
It's a judgment call by the DM what the rope itself actually does if the spell is cast in a space with not enough room for the whole rope. But "at the upper end of the rope, an invisible entrance opens" regardless.
It doesn't say that the extradimensional space opens after the "whole rope" has risen into the air. It just says that it opens at the top end of the rope.
The way I've always seen this played is that the rope simply rises to the ceiling and the door to the extradimensional space is in the ceiling. There's good precedent for that in previous editions of D&D. In AD&D 1st edition, the Rope Trick spell description reads as follows (AD&D 1e Players Handbook, p. 71):
When this spell is cast upon a piece of rope from 5' to 30' in length, one end of the rope rises into the air until the whole is hanging perpendicular, as if affixed at the upper end. The upper end is, in fact, fastened in an extra-dimensional space, and the spell caster and up to five others can climb up the rope and disappear into this place of safety where no creature can find them. [...]
In this version of the spell, it's much more clear that the extradimensional space is already open when the rope begins to rise. This provides a resolution to the ambiguity of the 5th edition version. The DM could simply rule that this is how the spell still works in 5e. There's nothing in the 5e spell description to contradict that.