My group has a bag of holding. Inside the bag is an item that is enchanted, but they don't know this. If one of them casts detect magic, will they sense this enchanted item inside the bag of holding, or will it just sense the bag itself?
Detect Magic will only detect the Bag of Holding.
Detect Magic says:
you sense the presence of magic within 30 feet of you.
so whatever is magic must be within that distance.
The Bag of Holding, alongside the Handy Haversack and Portable Hole, contains an extradimensional space, meaning its contents are not considered to be on the same plane as the bag/haversack/hole holding them.
Since an extradimensional space is more than 30 feet away, anything on it cannot be detected by Detect Magic.
Though admittedly, if the bag is within 30 feet of you, DM fiat may be that its contents are as well, and thus could be detected. However, even if so, you would not be able to know, as long as the contents stay within the bag, due to the following text from Detect Magic
If you sense magic in this way, you can use your action to see a faint aura around any visible creature or object
because arguably, as the bag itself would have an aura, you would not be able to see the aura of anything within it, regardless of whether its contents are on the same plane (it would be extremely difficult to see a glow or aura, within a glow or aura).
Your group does have a viable option however:
If the bag is turned inside out, its contents spill forth, unharmed, but the bag must be put right before it can be used again.
So, if you want them to use Detect Magic on the items within, you could hint at them to upend the bag.