Unique abilities need to be followed. In this situation, if everyone had the same movement speeds, etc., it would simply be straightforward. But, since you are in a situation that requires the extra effort to ensure that a player's movements need to be followed, to ensure they are not robbed of their special abilities, it simply means you need to spend that effort.
Going "grid-less" does not mean "map-less". When you describe your map to the party, there are several very important things to specify like distances and terrain; because they need this information in order to react to their environment. You can use a reference map - this does not need to be to scale, or particularly detailed, but it can help immensely for the players to understand their environment.
Tracking player "intent" is always going to be blurry. This is the same as declaring a "readied action" like "I am going to cast an eldritch blast at the next thing that walks through that door". What a player intends to do and what a player declares are two different things. In this situation, this is going to require both parties to communicate, in order to understand the layout, and the "intent" of the maneuver.
If you don't want to create a "disadvantage", keep everything further away. For example, if one player has a speed of 40ft, and the rest of the party has a speed of 30, make sure everything (at least starts) 40ft away. This means that the one player with the "extra maneuverability" can close the distance straight away, whereas the rest of the party needs to make extra effort (either by using their action to dash, or taking 2 turns) to close the distance.
But, is it really important? The most important thing to know is whether or not this really is an issue. Talk to the players and the GM and discuss whether or not this is important enough to always ensure this is kept track of 10ft of movement is one thing, ignoring difficult terrain is something much different.