Don't overthink this
Breathe easy - you cannot do this wrong. Some DMs will favor ability checks over contests and some will lean more heavily on passive over active checks: each of these DMs is playing the game by the rules because "the DM decides what happens".
How to Play
First of all, focus on the fundamental rule of D&D 5e:
1. The DM describes the environment.
2. The players describe what they want to do. ...the DM decides what happens, often relying on the roll of a die to determine the results of an action.
3. The DM narrates the results of the adventurers’ actions.
(PHB p.6 "How to Play", paraphrased)
Note that rolling an ability check is never a requirement. In general, ability checks are only called for where:
- the result is uncertain. Convincing the nothic to do what it wants to do doesn't require a roll. And
- failure is an option. That is to say, failure presents new and interesting options to the players that are significantly different to what success would provide. Failing to convince the nothic not to eat you is going to lead to a combat which is different from not having a combat. And
- there are no re-runs. If the characters' can just keep trying until they succeed then just let them succeed. Failing to convince the nothic not to eat you means that its going to try and eat you right now.
Ability Checks versus Contests
Page 174 of the Player's Handbook calls out the distinction between an ability check which is made against a DC set by the DM and contest where each creature makes an opposed ability check.
For every ability check, the DM decides which o f the six
abilities is relevant to the task at hand and the difficulty
of the task, represented by a Difficulty Class.
Sometimes one character’s or monster’s efforts are
directly opposed to another’s.
Both participants in a contest make ability checks
appropriate to their efforts. They apply all appropriate
bonuses and penalties, but instead of comparing the
total to a DC, they compare the totals of their two
checks. The participant with the higher check total wins
the contest. That character or monster either succeeds
at the action or prevents the other one from succeeding.
It is worth noting that the outcomes of contests are more random since they involve at least 2 rolls (more if advantage/disadvantage).
In addition to that, p. 175 introduces the idea of a passive check:
A passive check is a special kind of ability check that
doesn’t involve any die rolls. Such a check can represent
the average result for a task done repeatedly, such as
searching for secret doors over and over again, or can
be used when the DM wants to secretly determine
whether the characters succeed at something without
rolling dice, such as noticing a hidden monster.
The author is suggesting:
A contest: the characters' Wisdom (Perception) is being opposed by the goblins' Dexterity (Stealth).
A passive check: presumably because this is specifically the example given as to when these are appropriate, "noticing a hidden monster".
Equivalently, the DM could call for the players to make an active Wisdom (Perception) check and use the Goblins' passive Dexterity (Stealth): this is mechanically equivalent. Note that an active vs active check is not mechanically equivalent - an active check improves the characters' chances by 2.5%.
The advantages of the author's suggestion are:
- the player's remain in ignorance of why the DM is rolling dice - this can ratchet up tension if you get in the habit of occasionally rolling the dice and making "hmmmm" noises even when there is nothing to roll for. In contrast, asking the players' to make a Wisdom (Perception) check clue them into the fact that there may be something they are missing - unless you also occasionally call for these for no reason as well.
- Active checks are appropriate when the characters are being, well ... active. Be alert for when this happens "I look into the bushes" should trigger an active check. Active checks should never give worse results than a passive check: actively looking for you missing sock is not going to decrease the chance that your eye will light on it by chance.
Is a roll called for? If the characters offer the nothic what it wants at trivial cost why would you roll? Other than the fact that the damn thing is insane and you may want to model this with randomness.
What you should determine is if this is a contest or not. It becomes a contest when "one character’s or monster’s efforts are directly opposed to another’s". In a negotiation this is not, or not necessarily, the case. If you decide that it is a contest then a Charisma (Persuasion) by the character is opposed by a Wisdom (Insight) by the nothic with advantages/disadvantages applied depending on the inclination of the nothic to do what they want anyway. If you decide that it isn't then just set a difficulty representing how hard it is to persuade and insane creature to do whatever is being asked.
Passive checks are inappropriate here.
Don't roll unless you have to.
A contest is appropriate when there is active opposition and the skill of the opponent is relevant to the outcome. For example a Charisma (Deception) check is more likely to be a contest then a Charisma (Persuasion) check.
Passive checks are appropriate where a creature is not actively engaged.
If there is a passive check and an active check it is irrelevant which is which. An active Dexterity (Stealth) check vs a passive Wisdom (Perception) is exactly the same as a passive Dexterity (Stealth) check vs an active Wisdom (Perception). This allows you to let the most appropriate person roll the dice: the DM if you are keeping secrets or the players if you want them more engaged in the outcome.