When you use Portent to replace a roll with advantage or disadvantage, you replace one of the two d20s.
This has been clearly addressed by the 2018 PHB errata:
Advantage and Disadvantage (p. 173). In the first sentence of the fourth paragraph, both instances of “reroll” have been changed to “reroll or replace.”
The relevant portion of the rule on Advantage and Disadvantage in the PHB and Basic Rules now reads (emphasis mine):
When you have advantage or disadvantage and something in the game, such as the halfling's Lucky trait, lets you reroll or replace the d20, you can reroll or replace only one of the dice. You choose which one. For example, if a halfling has advantage or disadvantage on an ability check and rolls a 1 and a 13, the halfling could use the Lucky trait to reroll the 1.
Previously, that portion of the rule only applied to features that let you reroll a d20 after the initial roll, but now that has been expanded to features that let you replace the die instead of rerolling it - like the School of Divination wizard's Portent feature (PHB, p. 116):
You can replace any attack roll, saving throw, or ability check made by you or a creature that you can see with one of these foretelling rolls. You must choose to do so before the roll, and you can replace a roll in this way only once per turn.
As of the 2018 PHB errata, it is clear that when you use Portent to replace an attack roll, ability check, or saving throw that you have advantage or disadvantage on, you replace one of the two die rolls (before you make it), and then roll the other die. You then apply the advantage/disadvantage rules as normal.
For reference, the Portent feature replaces only the d20, not the entire roll; modifiers are still added to it. This is supported by this unofficial tweet from March 2015 by rules designer Jeremy Crawford:
When using a Diviner's portent die, are modifiers added to the portent die, or is the portent die used as is?
The portent die is intended to replace a d20 roll only, not any modifiers applied to it.
As well as this unofficial tweet from April 2016:
Sorry, follow up: and does diviner's portent replace roll only or roll plus modifiers? Thanks again.
A Portent roll replaces a d20 roll, not modifiers.
As such, it is clear that, post-errata, Portent replaces only one of the dice when rolling at advantage or disadvantage. After advantage or disadvantage is applied, modifiers are added to the die as usual to determine the total for the roll.