For humanoid monsters in the Basic and Expert sets there is a clear pattern of creatures with the same number of Hit Dice appearing in smaller numbers in random encounters if they have better armor and damage, and in larger numbers if they have lower armor and damage. This pattern continues to simple predatory animals like wolves, boars, and bears.
Among higher HD creatures there seems to be a trend towards smaller groups as these tend to have additional special abilities beyond just direct damage attacks and only armor for protection.
Is this simply something that was eyeballed by the writers when having comparable creatures with the same number of HD, or is there any kind of measure by which a fitting group size for a creature of a given HD value can be judged when creating new custom monsters?