I am aware that you did not ask for a math-based answer, and Ilmari already did a wonderful job, but I decided to provide one anyway, since in my honest opinion the math approach is one of the best strategies to generalize and it allows one to use the results in other contexts.
Using the Order Statistics is the way.
When one has to deal with advantage/disadvantage mechanics, one has to consider the beloved order statistic. The formulae are given below, while at the bottom of this answer I provide the mathematical details.
Given two rolls of a die with \$d\$ faces, \$X_{(2)}\$ denotes the outcome of rolling with advantage and \$X_{(1)}\$ the outcome of rolling with disadvantage. The expected values are, respectively,
$$
{\rm E}[X_{(2)}] = \frac23d+\frac12-\frac{1}{6d}, \quad {\rm E}[X_{(1)}] = \frac13d+\frac12+\frac{1}{6d}
$$
which provide the same results as Ilmari's answer. Below is a plot which depicts the behaviour of these functions for dice of an arbitrary number of sides; the case for \$d=20\$ is highlighted. Note that \${\rm E}[X_{(2)}]\$ and \${\rm E}[X_{(1)}]\$ are not linear functions of the number of faces, but, on the other hand, for large \$d\$ the last term (\$1/(6d)\$) is close to 0.
Denoting with \$X\$ the result of a straight roll of a die with \$d\$ faces, the expected values of the differences between rolling with adv/disadv can be calculated as
$$
{\rm E}[X_{(2)}-X]=\frac{d^2-1}{6d}, \quad
{\rm E}[X_{(1)}-X]=\frac{1-d^2}{6d}
$$
These differences are strictly greater than 1 (in absolute value) for \$d>6.16\$, as shown in Ilmari's table. Roughly speaking, when \$d=6k\$ (i.e. when the number of faces is a multiple of 6), then the difference between the average straight roll behaves as \$\pm k\$, depending on whether you are considering adv or disadv. When the die has a number of faces which is not a multiple of 6, then depending on how you approximate results (flooring, ceiling or rounding) you may consider an higher or lower difference. For example, for \$d=20\$, the difference is 3.325, which you may consider as \$\pm\$3 (floor or round operation) or \$\pm\$4 (ceiling operation).
An interesting fact is that the function describing the expected value of a straight roll is the average of the two functions describing the expected value of rolling with adv/disadv:
$$
\begin{eqnarray}
{\rm E}[X] &=& \frac12d +\frac12\\
&=& \frac12\left( \frac23d+\frac12-\frac{1}{6d} + \frac13d+\frac12+\frac{1}{6d}\right)\\
&=&\frac12\left({\rm E}[X_{(2)}] + {\rm E}[X_{(1)}]\right)&
\end{eqnarray}
$$
Mathematical details
Consider a fair die with \$d\$ faces: since the outcome of such a die follows a discrete uniform distribution, the chance of getting the result \$x\$ is given by the probability density function (pdf) \$f(x)=\displaystyle\frac{1}{d}\$ and the cumulative distribution function (cdf) of \$x\$ is \$F(x)=\displaystyle\frac{x}{d}\$. Recall that the cdf \$F(x)\$ provides the probability of getting a result of less than or equal to \$x\$.
Advantage
Denote with \$X_{(2)}\$ the maximum outcome of two rolls of a die: using the formula for discrete distribution the probability of being \$x\$ (the highest outcome) is given by\$^1\$
$$
\begin{eqnarray}
P(X_{(2)}=x)&=& F(X)^2-(F(x)-f(x))^2\\
&=& 2f(x)F(X)-f(x)^2
\end{eqnarray}
$$
Substituting the formulae for the cdf and the pdf and after some algebra one has that
$$
P(X_{(2)}=x) = \frac{2x-1}{d^2}
$$
It is now possible to compute the expected value:
$$
\begin{eqnarray}
{\rm E}[X_{(2)}] &=& \sum_{x=1}^dxP(X_{(2)}=x)\\
&=&\frac{1}{d^2}\left( 2\sum_{x=1}^dx^2 -\sum_{x=1}^dx\right)\\
&=& \frac23d +\frac12 -\frac{1}{6d}
\end{eqnarray}
$$
where we used the fact that
$$\displaystyle\sum_{x=1}^dx^2=\frac{d^3}{3}+\frac{d^2}{2}+\frac{d}{6}, \quad \displaystyle\sum_{x=1}^dx=\frac{d^2+d}{2}
$$
This formula gives the same results as stated in Ilmari's answer. Now, compute the expected value of the difference between rolling with advantage and normal rolling, denoted with \$X\$:
$$
\begin{eqnarray}
{\rm E}[X_{(2)}-X] &=& {\rm E}[X_{(2)}] -{\rm E}[X]\\
&=& \frac23d +\frac12 -\frac{1}{6d}- \frac{d}{2}-\frac12\\
&=& \frac{d}{6} -\frac{1}{6d}\\
&=& \frac{d^2-1}{6d}
\end{eqnarray}
$$
Disadvantage
Using the same formula as the previous case, one can compute the expected value\$^1\$ of \$X_{(1)}\$, which is the outcome of rolling with disadvantage:
$$
P(X_{(1)}=x) = f(x)^2+2f(x)-2f(x)F(x)
$$
Then
$$
P(X_{(1)}=x) = \frac{1-2x}{d^2} +\frac{2}{d}
$$
which leads to
$$
\begin{eqnarray}
{\rm E}[X_{(1)}] &=& \displaystyle\sum_{x=1}^dxP(X_{(1)}=x)\\
&=& -\frac{2}{d^2}\displaystyle\sum_{x=1}^dx^2 +\left(\frac{1}{d^2}+\frac{2}{d}\right)\sum_{x=1}^dx\\
&=& \frac13d +\frac12 +\frac{1}{6d}
\end{eqnarray}
$$
The expected value of the difference between a straight roll and a roll with disadvantage is
$$
\begin{eqnarray}
{\rm E}[X_{(1)}-X] &=& {\rm E}[X_{(1)}] -{\rm E}[X]\\
&=& \frac13d +\frac12 +\frac{1}{6d}- \frac{d}{2}-\frac12\\
&=& -\frac{d}{6} +\frac{1}{6d}\\
&=& \frac{1-d^2}{6d}
\end{eqnarray}
$$
\$^1\$ The computation is left as an exercise for the reader.
