The rule for Marvelous pigments states:
When you complete the painting, the object or terrain feature depicted becomes a real, nonmagical object.
The area of the Portable Hole is a magical item and therefore can't be created or expanded by Marvelous Pigments. This much is very clear.
However, as David Coffron points out, the rule doesn't say the extra-dimensional space created by the hole is magical only that it exists on another plane:
You can [...] place [the hole] on or against a solid surface, whereupon the portable hole creates an extradimensional hole 10 feet deep. The cylindrical space within the hole exists on a different plane.
The space created by the object that is the Portable Hole is talked about as a distinct and separate entity by the rule and its existence on another plane does not make that space inherently magical.
In other words, the Portable Hole seems to be a magical item that creates a portal to an extradimensional space that is connected to the item when the item is unfolded but treated separately otherwise. The object is magical, the space is not (and therefore the pigments could be used on it).
It is possible to imagine a person climbing into this extradimensional space and painting a cavity onto the wall of it to expand the space's capacity. One could argue that the rules do not declare that walls exist in the hole but I believe these must exist: the rule on Portable Hole says
Any creature inside an open portable hole can exit the hole by climbing out of it.
How could a creature climb out of a 10 foot deep hole if there are no interior surfaces? Can you even have a space if there is nothing to delineate it? We've departed from supportable arguments either way at this point.
Therefore, it is difficult to decisively say one way or the other if it is possible to use Marvelous Pigments to achieve something like what you are asking but a DM would not be out of line to rule one way or the other.