Untyped penalties stack, even from the same source
From the Bonus Types section under Magic:
Usually, a bonus has a type that indicates how the spell grants the bonus. The important aspect of bonus types is that two bonuses of the same type don’t generally stack (...), only the better bonus of a given type works. The same principle applies to penalties - a character taking two or more penalties of the same type applies only the worst one, although most penalties have no type and thus always stack. Bonuses without a type always stack, unless they are from the same source.
[End of paragraph no further text on Bonus Types follows]
The sentence that excludes stacking for untyped effects from the same source only mentions bonuses, so penalties are not excluded from stacking when from the same source.
You should be able to stack the penalties from Hypnotic Stare and a spell that creates an untyped penalty.
There is one exception
Note that under Combining Magical Effects, Stacking Effects there is an exemption from bonuses from multiple castings of the same spell:
Spells that provide bonuses or penalties on attack rolls, damage rolls, saving throws, and other attributes usually do not stack with themselves.
Since you are combining a non-spell with a spell, this should not affect you in this case.