There are two basic approaches to this sort of question. One is to find rules for goats, and then look for a way to make them ride-able, and aloisdg has given three options along those lines.
The other way is to find something that's already ride-able, with roughly the right stats, and make it a goat. For instance:
Take one Riding Dog.
Change "dog" to "goat". Change "bite" to "gore". This is a minor downgrade since the damage changes from B/S/P to P, but since you're not planning on using this for combat, it doesn't matter. No need to change the Trip special attack. Optionally, move one skill point and the Skill Focus feat from Perception to Climb.
Voila! You now have a Medium-sized riding goat.
In the end, the stats are pretty similar to the ram that aloisdg suggested. However, reskinning is sometimes easier to sell to a GM. There's less need to worry about balance issues, especially in this case where the reskin is based on a "standard issue" mount for small PCs, and it reduces the number of rulebooks required (something that, IME, GMs often appreciate!)