You go with the slowest member's traveling speed
And this makes complete sense once you consider if they are carrying equipment with a low Carrying Capacity, or if they are sick, or if they are fatigued/exhausted from lack of proper alimentation or dehydrated. All those things should be considered when traveling with a large group of people.
Not only that, but you will need people Foraging for food for 20 or more people, possibly more than one hunter doing so, after all, the Survival DC is 10 + 2 per extra person fed. A group of 20+ people means a DC 48 for a single hunter, or two hunters foraging with a DC 30, or a group or three with a DC 22 and so on. The more foragers you got, the easier this will be.
The Foragers also will have to move at half their overland speed, which for a humanoid character, that usually means a 15-foot base land speed, which is already slower than the base land speed of dwarves, halflings and gnomes, or those carrying a medium load, or people wearing medium and heavy armor (like guards and soldiers).
If you look at the Pursuit rules, you will notice that the base speed of any party will always be the base speed of the slowest member. That is why when pursuing a quarry, it makes sense to send those with a higher speed ahead to scout.
Another aspect to be considered is the Terrain. Depending on what type of terrain these people are traveling, this could slow their progress per day considerably, as seen on the rules of Exploration and Movement. Traveling through roads and plains is quick and easy, but trying to hide from enemy troops by avoiding roads and going into woods will slow down the group even more.