That's correct. The spell (like most spells) consistently uses "you" to refer to the person who cast it. In fact, the entire chapter on spellcasting uses "you" to refer to the caster. I could go through and provide a bunch of quotes with the word "you" bolded, but it seems unnecessary.
It seems to me that you see the need for it to be the person who cast the spell to use the gem as a bad thing. However, the whole point of this spell is to safeguard something valuable to you so that it can't be stolen or lost. If you're a Wizard, this is probably your spellbook, which is absolutely vital to your functioning as a character and probably useless to the rest of your party. There's generally no one else who you would want to be able to find or retrieve it.