Yes, they would still benefit from Expertise in Wild Shape
As stated in the part of the description of the druid's Wild Shape feature you quote:
You also retain all of your skill and saving throw proficiencies, in addition to gaining those of the creature. If the creature has the same proficiency as you and the bonus in its stat block is higher than yours, use the creature’s bonus instead of yours.
In addition, another bullet point in the Wild Shape feature description says:
You retain the benefit of any features from your class, race, or other source and can use them if the new form is physically capable of doing so. However, you can’t use any of your special senses, such as darkvision, unless your new form also has that sense.
So, to see whether your Expertise class feature is retained, we need to look at the description of the rogue's Expertise feature:
At 1st level, choose two of your skill proficiencies, or one of your skill proficiencies and your proficiency with thieves’ tools. Your proficiency bonus is doubled for any ability check you make that uses either of the chosen proficiencies.
Nothing in the feature description specifies a particular anatomy needed to benefit from Expertise (...it'd be weird if it did, honestly). Thus, you do retain the benefit of the Expertise feature.
Putting it all together: You still benefit from the Expertise feature in Wild Shape. You retain all your skill proficiencies in Wild Shape, in addition to those of your new form. And if the creature is also proficient in the same skill, you use whichever total bonus to the skill is higher between yours and the beast's (for a skill you have Expertise in, your own bonus will likely be higher).
Note: Your own modifier for a skill may change depending on the ability score that the skill is associated with. Per the Wild Shape description:
Your game statistics are replaced by the statistics of the beast, but you retain your alignment, personality, and Intelligence, Wisdom, and Charisma scores.
Thus, your own Strength, Dexterity, and Constitution scores are replaced by those of your new form while in Wild Shape. For those skills that are tied to these three scores (or rather two, since Constitution isn't associated with any skills), your modifiers are also changed accordingly.
For instance, if your own Strength modifier is +1 but your new form's Strength modifier is +5, your Athletics skill modifier will increase by 4 - assuming the beast is not also proficient in Athletics with a higher modifier (in which case you would use the beast's Athletics modifier instead of yours).
Correspondingly, your modifier for a skill may actually decrease if the new form is not proficient in the skill and its corresponding ability score is lower. For instance, if you have a Dexterity modifier of +3, and the beast's Dexterity modifier is -2, your modifier for the Stealth skill would actually decrease by 5.
You can determine your new modifiers for the skills associated with Strength and Dexterity by recalculating them based on your new ability score, and then comparing them to the beast's statblock. If the beast is not proficient, you use your own recalculated modifier. If it is proficient, then you compare your recalculated modifier to the modifier given in the statblock; use whichever of the two is higher.