The official rules—or lack thereof
To establish what the rules say—which isn’t much for the 3D case—here’s what we’ve got:
When determining whether a given creature is within the area of a spell, count out the distance from the point of origin in squares just as you do when moving a character or when determining the range for a ranged attack. The only difference is that instead of counting from the center of one square to the center of the next, you count from intersection to intersection.
(Magic Overview → Spell Descriptions → Area)
This establishes that simply measuring things, with string or rulers or anything else, is not the official rule—though it is honestly the most sensible one. However, should you handle things that way, note that large areas—as in, even in the 2D case—will have deviations from where a character can reach, because of the approximations used in the official method. For what it’s worth, you will only start to see such deviations once you start dealing with radii of 50 feet or more, though.
Diagonals
When measuring distance, the first diagonal counts as 1 square, the second counts as 2 squares, the third counts as 1, the fourth as 2, and so on.
(Combat → Movement and Positioning → Measuring Distance)
This establishes the “5-ft., 10-ft.” rule that we’re familiar with. This is in the combat section on movement, which doesn’t really touch on 3D movement—that has its own section elsewhere.
Buuuuut,
[nothing about diagonals at all]
(Carrying, Movement, & Exploration → Movement → Moving in Three Dimensions)
The rules we have for 3D movement do not mention anything about diagonals. There just isn’t any rule about them here.
Extrapolating 3D rules from 2D rules
Without an explicit “this is what you should do for this case” rule, we have to come up with something. There are a few options here:
Just substitute “cube” for “square”? Concern about diagonal speed boost.
We could imagine that the rules just “fall back on” the existing rule for diagonals, for all it talks about “squares.” After all, it is true that the rules very, very often use “square” (or “circle” or “rectangle” or other 2D shape) when they should use “cube” (or “sphere” or “rectangular prism” or other 3D shape). But this isn’t done consistently, nor do I particularly get the impression that it was done in a kind of jargony “this is what the words mean in our context” kind of way. Rather it seems, often, to have been done more in a “we were only thinking about the 2D case and never considered the 3D case at all.” That is, very often, when we read “square” and understand “cube,” what we are doing is extrapolating from the rules. Usually, this is very straightforward—mentally replace “square” with “cube” and you’re done.
So we can’t really “just fall back on” the “default” rule about diagonals, treating “square” as “cube.” Or, we can, but that isn’t “just” what we’re doing. Really, what we are doing, is extrapolating. That is, we cannot really say there is a rule for this at all—we’re making up a rule, in the hopes that it is consistent and useful.
But are 3D “double diagonals” well served by this rule? Is our extrapolation consistent and useful? Is this what we are “officially” what we are supposed to do? It is not—at all—clear to me that any of these are true. After all, it lets you move vertically for free, if you were already moving on a diagonal. It makes moving on a “double diagonal” better than moving on a straight line, which seems very wrong. Intuitively, to me, this rule seems very strongly like a case wherein the devs simply forgot about 3D movement entirely—which is not terribly surprising.
Apply the same logic to double diagonals? Very complex.
So what would we use here, to be consistent with this rule but to account for “double diagonals”?
The “5-ft., 10-ft.” is an approximation of having diagonals cost \$1.5\times\$ the distance, which itself is an approximation of having them cost \$\sqrt{2}\times \approx 1.414\times\$ (Pythagorean theorem says a right angle with \$a\$ for the legs will have a hypotenuse of \$\sqrt{2}\times a\$).
A “double diagonal” will be the hypotenuse of a right angle with legs of \$a\$ and \$\sqrt{2}\times a\$, so the hypotenuse will be \$\sqrt{3}\times a\$, so we need an approximation of \$\sqrt{3}\times \approx 1.732\times\$. If we round that to \$1.75\times\$, we need “5-ft., 10-ft., 10-ft., 10-ft.” (so moving four squares costs 35 ft. of movement—\$1.75\times\$ the 20 ft. it would usually take.
Obviously, “5-ft., 10-ft., 10-ft., 10-ft.” is a pain, and also I’m not sure from the face of it that it’s fair to start with 5 ft. on the first square as we do with the “5-ft., 10-ft.” scheme. It’s also less clear how to combine it with “single diagonal” movement in the same turn—you probably shouldn’t be able to go 5 ft. for a double diagonal square and then move on a single diagonal for another 5 ft.
The most accurate way to resolve this is to imagine the “5-ft., 10-ft.” rule as actually being “7.5-ft.” every time—then it’s really “7.5 feet (rounded to 5 feet), 15 feet (rounded to 15 feet, so 10 feet beyond the first).” For the double-diagonals, we’re looking at 8.75-ft, which is still rounded down to 5 feet the first time, and then 17.5 feet (rounded to 15 feet total distance), 26.25 feet (25 feet), 35 feet (35 feet).
Maybe easier to see in tabular form. Here, \$d\$ is the actual, unrounded distance, \$\lfloor d \rfloor\$ for the rounded distance, and \$\Delta \lfloor d \rfloor\$ for the cost of the latest step. Each step should cost what’s listed as \$\Delta \lfloor d \rfloor\$.
\begin{array}{c|c|c}
\textbf{Straight Line} & \textbf{Single Diagonal} & \textbf{Double Diagonal} \\
{
\begin{array}{c c c}
d & \lfloor d \rfloor & \Delta \lfloor d \rfloor \\ \hline
\phantom{0}5 & \phantom{0}5 & 5 \\
10 & 10 & 5 \\
15 & 15 & 5 \\
20 & 20 & 5 \\
\end{array}
}
&
{
\begin{array}{c c c}
d & \lfloor d \rfloor & \Delta \lfloor d \rfloor \\ \hline
\phantom{0}7.5 & \phantom{0}5 & \phantom{0}5 \\
15\phantom{.0} & 15 & 10 \\
22.5 & 20 & \phantom{0}5 \\
30\phantom{.0} & 30 & 10 \\
\end{array}
}
&
{
\begin{array}{c c c}
d & \lfloor d \rfloor & \Delta \lfloor d \rfloor \\ \hline
\phantom{0}8.75 & \phantom{0}5 & \phantom{0}5 \\
17.5\phantom{0} & 15 & 10 \\
26.25 & 25 & 10 \\
35\phantom{.00} & 35 & 10 \\
\end{array}
}
\end{array}
Combining single and double diagonals then becomes possible by leveraging those fractions—7.5-ft. + 8.75-ft. is 16.25 feet, so the second step when moving single-diagonal then double-diagonal is going to cost 10-ft, but the 1.25 feet “extra” is less than the 2.5 feet “extra” from two double-diagonal moves. By tracking that extra you can keep track of how far a character has actually moved.
Which to use? Simple substitution far superior
So how does that end up comparing against the “5-ft., 10-ft.” rule?
This is a 30-ft. hemisphere. The cubes marked in red are included if we use the “5-ft., 10-ft.” rule but not if we use the “5-ft., 10-ft., 10-ft., 10-ft.” rule. I spent a ridiculously long time making this (and got it wrong the first time, if you check the edit history), which is itself a damn good reason to never use the “5-ft., 10-ft., 10-ft., 10-ft.” rule. Another is, this difference isn’t very large. It will get larger at larger radii—basically every 20 feet in the radius, you’ll get a larger discrepancy—but massive radii are rare and it’s pretty unlikely that you’ll have a target right on the edge in a corner anyway.
And the other nice thing about this is that any two points that are within a given 2D distance of one another, and also are no more vertically separated than they are horizontally, can be sure to be within the 3D model as well. So you can just flatly ignore 3D effects for any two points that are closer vertically than they are horizontally. You can also just mentally flip the horizontal and vertical axes, so anything more vertically distant than horizontally can be considered solely on the vertical axis (though you’ll probably have to work harder at that because of the presumed lack of grid). That is a huge advantage.
Ultimately, I still don’t know if the “5-ft., 10-ft.” rule was supposed to cover 3D and “double diagonals” too, or if they just never thought about 3D and it’s just what we’re left with, but it’s pretty clear that you should use that rule. I just desperately wish the rules had explained all of this—or even just asserted it—because until I did the work myself, I didn’t believe it would work out well at all. Hopefully this answer will convince you and save you that effort, as well.
Conclusion
Use the “5-ft., 10-ft.” rule; it’s vastly simpler in practice, and the discrepancies are extremely minor.
Unfortunately, even with this simpler approximation, 3D movement and distances are very difficult to work with. I don’t recommend them—if you actually bother with this mess, I salute you, because I honestly think it’s insane. But unfortunately, unclear rules and complicated calculations—even if not as complicated as they could be—are the reality of 3D movement in D&D 3.5e. I strongly recommend a gentleman’s agreement to just keep things grounded, or houserule in some form of abstract flight—here’s mine.
