I use my action to toss a Dancing Sword in the air and choose not to attack. It now hovers there. If I stay within 30' of it, and use my bonus action to have it keep up with my speed of 30', but not attack with it, will it stay out, hovering? Is its duration only checked by its making Attacks? This seems to be how it reads. Is this correct?
The sword hovers until it makes a fourth attack, however long that may take
Your reading of the text of the Dancing Sword seems accurate to me:
After the hovering sword attacks for the fourth time, it flies up to 30 feet and tries to return to your hand. If you have no hand free, it falls to the ground at your feet. If the sword has no unobstructed path to you, it moves as close to you as it can and then falls to the ground. It also ceases to hover if you grasp it or move more than 30 feet away from it.
This paragraph stipulates only three circumstances in which the sword stops hovering: it has made a fourth attack, you pick it up again, or you move more than 30 feet away from it. From the general rules about magic and magical items we can further infer that the sword would stop hovering if its magic is suppressed (i.e. it enters an antimagic field) or you die, since dying causes you to lose attunement, and the sword's magical properties don't function if you're not attuned to it.
Otherwise, no limitation is given, and it should hover indefinitely if none of those conditions are met; you could conceivably throw out the sword and maintain it hovering nearby indefinitely. It doesn't even require the use of your bonus action to stay in the air, so you could have it serenely floating nearby while you sleep, or if you're knocked unconscious but not yet dead (or suffering any other condition which precludes the use of actions).