Ignore the Loading Property
As you stated, the text for Crossbow Expert says
You ignore the loading property of crossbows with which you are proficient.
It does not say to ignore it in specific cases, but to generally ignore it if you are proficient.
If you are ignoring the property, then the property is not relevant for Speed Factor.
Return on Investment
Note also that the character has chosen to invest in a very limited resource (ASIs) in order to improve their initiative and crossbow speed use. It makes sense logically that ignoring the Loading property lets them fire faster, and letting them fire faster improves their Speed Factor initiative. In addition, they have significantly invested in this as opposed to taking another feat or an ASI.
If the player is looking to purely offset the -5 for the loading property (they don't have extra attack and therefore bypassing loading isn't an issue), then the Alert Feat may be an equivalent choice.
You gain a +5 bonus to initiative.
You can't be surprised while you are conscious.
Other creatures don’t gain advantage on attack rolls against you as a result of being unseen by you.
Alert vs Crossbow Expert
So which is better for your player? That's going to be up to them and will matter if they have more than on attack during a turn and if they plan on using the crossbow in melee range.