When trying to lockpick a padlock with a rusty nail I found in the guano excrement of a cell in a Bugbear cave, my DM had me roll for lockpicking. I rolled a 7 and he ruled that I rolled badly and that my character stabbed himself in the hand with the nail and that in the future his lockpicking skill would be reduced by 1.
The text on the Player's Handbook says the following about ability checks:
To make an ability check, roll a d20 and add the relevant ability modifier. As with other d20 rolls, apply bonuses and penalties, and compare the total to the DC. If the total equals or exceeds the DC, the ability check is a success—the creature overcomes the Challenge at hand. Otherwise, it’s a failure, which means the character or monster makes no progress toward the objective or makes progress combined with a setback determined by the GM.
The only thing I can see in this definition that would correspond to losing skill scores is the word "setback". When looking at it on the online Merriam Webster dictionary, I see a few definitions:
setback: a checking of progress setback: defeat, reverse
When looking at checking, this is what I find:
a sudden stoppage of a forward course or progress
By these definitions, I do not see how lowering my ability score follows the rules.
The Cambridge online dictionary defines setback as:
something that happens that delays or prevents a process from developing:
Well sure, me stabbing myself would certainly delay or prevent the process of opening the padlock from developing. So by this definition the DM is following the rules.
I am therefore curious as to whether skills being reduced on failed ability checks is valid according to the rules.