I would rule yes, they can
with a note below.
The requirement for exerting your will is that you can see the target, that is the blindfold. Unless you are in darkness, you would be able to see the blindfold. The amount of light needed to see the blindfold is open to the DM's ruling. As a general rule, you would be able to see the blindfold if at least the blindfold is within dim light area.
The blindfold being worn is actually not a concern, because you won't resist.
Note: Stricter (and ridiculous) reading would require you to still perform the contest. The DM may grant advantage to spellcasting ability roll and disadvantage for strength roll for not resisting, though.
If the blindfold is secured with a knot, more realistic DM might consider whether you can't see the knot. Most of the times you can't see the knot, thus won't be able to untie the blindfold. You might still be able to remove the blindfold, though, if it's not tightly secured.
Note: Stricter reading would disregard the position of knot. You can see the blindfold, so you can manipulate the knot wherever it is.
So, I would ask myself these questions:
- Is the blindfold in darkness? If yes, the answer is no.
- Is the blindfold secured with a knot? If yes, most likely the answer is no.
- Else, yes, you can remove the blindfold.