No, you cannot draw your dagger and attack with it and retain your Duelist feature
--------------------------------------------------

The core argument behind this answer is that you can only attack as a bonus action with your offhand weapon when you use your action to attack.

> **When** you take the Attack action and attack with a light melee weapon
> that you’re holding in one hand, you can use a bonus action to attack
> with a different light melee weapon **that you’re holding in the other
> hand.** (Basic Rules, p. 74 / PHB. p.195)

You don't have this bonus action otherwise.

>You can take a bonus action only when a special ability, spell, or other feature of the game states that you can do something as a bonus action. You otherwise don’t have a bonus action to take. (Basic Rules p. 69)

You can draw your weapon as part of an attack...

>You can also interact with one object or feature of the environment for free, during either your move or your action. For example, you could open a door during your move as you stride toward a foe, or you could draw your weapon **as part of the same action you use to attack.** (PHB p. 190)

...but that would remove your +2 bonus
>When you are wielding a melee weapon in one hand **and no other weapons**, you gain a +2 bonus to damage rolls with that weapon. (Basic Rules, p. 25)

So, here is what you are trying to do.

1. You declare an attack. 

2. You can use Two-Weapon Fighting to attack with a weapon *that is already in your hand*. You can subvert this by drawing your weapon as part of your aforementioned attack declaration. 

3. Since you now have another weapon in your hand, you lose the Duelist feature of +2 damage.

If you forgo drawing your dagger, you are not allowed a second attack since the dagger is not in your hand during your attack action. It's either one or the other, you cannot combine two-weapon fighting and duelist in the same attack action.