In one fight I want 2 duergar and one kobold to fight my party. I want the party to see them all and then let the duergar become invisible. Once they are, do they need to roll stealth to sneak up on the party and does the party than roll perception in order to search for them? Or can I just go with the duergar up to one ranger and attack him (with surprise??)
Invisible is not the same as Hidden
An invisible creature is unseen; a hidden creature is unseen, and unheard, and unsmelt etc. being invisible fulfils the precondition for hiding - that you cannot be seen clearly - but you still need to take an action to Hide: that's why Hide is an action.
If the duergar merely turn invisible then everyone knows exactly where they are and can follow their progress as they move. They would attack with advantage and be attacked with disadvantage but could be targeted without a problem.
If they turn invisible and take an action to Hide, then when they move, only people whose passive Wisdom (Perception) with disadvantage (assuming vision is their primary sense) beats their Dexterity (Stealth) can track them. Otherwise the PC must spend an action to Search, engaging active Wisdom (Perception) also with disadvantage, or attack where they think they are.
Read the PHB, page 291, or look at the section on your DM screen if you have it;
For the situation in question, the duergar is impossible to see without magic. This means that any ability check which requires sight (e.g. a spell such as Diine Word, targeting creatures you "can see within range" will not work)
Also check PHB pages 194/195, in the section entitled "Unseen attackers and Targets".
In your example of the duergar sneaking up on the ranger, it specifies that it would have advantage on attack rolls, and that whether the attack hits or misses, it has given away its position.
Duergar invisibility drops when they make an attack, as well as some other situations - see MM page 122.
According to rules as written, if the ranger misses (e.g with an arrow) while the duergar is invisible, they won't know why - was the duergar somewhere else, or did it sidestep the arrow? This doesn't work for me - if it breaks against a duergar's armour, rather than flying past and hitting a wall, they will know that they had the right location.
You have more senses than sight
Even if someone is invisible, you are aware of its presence. You can see its footprints, hear its movement, feel the breeze it stirs as it moves. This means you can still attack it, but not as efficiently, and it can not suprise you.
In game terms, your attack is with Disadvantage, and it attacks you with Advantage. You can not make Opportunity Attacks against them.
You can make an opportunity attack when a hostile creature that you can see moves out of your reach. (PHB p195)
Adding to @Dale_M and @András answers above.
There is a rule of attacking invisible enemies in the DMG, in wich the players calls where he think the enemy is, and then they roll the attack (At disadvantage). Only if they call the right position, the attack has a chance to hit.
(Q): What if they say a broad position, say, kinda in the front?
(A): well, it is board. ¿What is front? Is it right in front? kinda to the left? if the player does not specify, you can say "Ok, 1-2 is kinda to the left, 3-4 is right in front, 5-6 is kinda to the right" And if the Duergar is in that position, they miight get hit.
(Q): what does it matter where they aim if they cant see him, shouldnt they auto-miss and carry on?
Yes and no. Imagine the ranger says: I'll shoot to the left!, and rolls a 18. Now the ranger, after doing a clean shoot and it going straight ahead can call out "¡He's not here"!. Or maybe aim for the right, or the center next.
(Q): how can the players pin down where they are?
(A): Now...here comes the rulings. I think the best idea is to divide the status on 3:
1)- I have no idea where he is: The characters have no idea where the invisible character is. All they can do is guess where he is, attacking at random directions.
2) I hear noises to the Right: The characters sense a position where the enemy is (Right, Front, Behind). but they cant pinpoint where exactly he is
3) He is over there!: The characters, thanks to noise or smell can pinpoin the exact position where the enemy is. But since the enemy still is invisible, it still is hard to hit him
What i usually do is either make the enemy roll for stealth, or trust in the invisibility do the work. And use the passive perception (and passive stealth) to see if the PC can sense them.
-If they take the hide action and they overcome the passive perception of the party, they are on the status one. The party can take the search action to pinpoint where they are. If they totally flop the roll (-5 of the Party Passive perception) then the party can pinpoint where the enemy is
-If they relly on the invisibility they are on the status two. unless he has a really high (+5 of the party passive perception) Passive Stealth. if they have a passive stealth lower than the passive perception of the party, then the party can pinpoint where they are.
You can also track the perception of the party separately IE: The 15 Pass.Perception Rogue knows where the Duergar is exactly, but the 11 Pass.Perception Barbarian only has a sense where he is. The poor 8 Pass.perception Mage has no idea where the god damn duergar is, and relies on the indications of their friends.
They can move around freely.
It's been my experience that once something is invisible in the heat of battle that's all there is to it. They can move around freely. However, if when they pass close to one of your party members you want to roll against their passive perception to see if they hear them move past that would be interesting.
I may use that myself now.