Why isn't an Animal Companion autonomous when a Familiar is?

Question

I was very surprised and a bit confused about how a wizard's familiar seems to have more autonomy than a ranger's animal companion. According to the PHB page 93:

The beast obeys your commands as best as it can. It takes its turn on your initiative, though it doesn't take an action unless you command it to.

Conversely the Find Familiar spell, PHB page 240:

Your familiar acts independently of you, but it always obeys your commands. In combat, it rolls its own initiative and acts on its own turn. A familiar can't attack, but it can take other actions as normal.

I have two issues with how this works. First is just conceptual: Why would an independent animal have less independence than a familiar? This seems backwards to me. The second is more practical: If the character get knocked unconscious, the PHB seems to imply that the animal companion would sit around waiting, while the familiar would still go around doing stuff.

Why is this the way it is, and how do I practically work with it?

Answer 1

First, as Derek Stucki mentioned in the comments, the errata for the Ranger's companion animal clear up some of this:

Ranger’s Companion (p. 93). Like any creature, the beast can spend Hit Dice during a short rest. If you are incapacitated or absent, the beast acts on its own, focusing on protecting you and itself. It never requires your command to use its reaction, such as when making an opportunity attack.

Out of game, the rest of the discrepancy is about balance. The Wizard's familiar cannot attack, but the animal companion can and often does. Allowing the Ranger's companion animal to attack independently essentially gives the Ranger an extra Attack action each turn, which would break the action economy.

This Q/A addresses the issue of balance if a Ranger could give their companion a single command to Attack and then have the beast continue attacking without giving up any further actions of their own. It concludes that it would indeed be overpowered.

In game, we can understand the discrepancy when we understand that a Familiar is not merely a beast, it is a spirit (celestial, fey or fiend) in the form of a beast.

Answer 2

The Animal Companion is just that; an animal. It has animal intelligence, acts like an animal, thinks like an animal, etc. It's actions are either the result of you commanding him or his instinct.
Meanwhile, the familiar is a magical creature. It has a higher intelligence score than animals, which means they have thoughts of themselves and can act independently.

So, quite simply because animals are less intelligent than magical creatures.