I do not know of any published oWoD rules to develop new Disciplines. I heard that there are some for nWoD (Chronicles of Darkness, to be precise), but not sure.
I can give you some guidelines from experience of some attempts to create a new Discipline, plus some experience of my friends doing it.
When designing a new power for an existing Discipline or a brand new Discipline...
- Think when and where will it be used. WoD is full of powers that are win buttons with the right help from your Storyteller, but also useless without one (Animalism 1 being the most iconic example). If you are a player, consult your ST if the power you came up with will be a good fit for a campaign. If you are the ST, think if you are ready to make it a good fit.
- If it can be used in combat for dealing direct damage, compare it to existing options. Use AnyDice to measure effectiveness.
- Most powers in oWoD are more or less comparable in usefulness to other powers of such level. So, Presence 1 and Dominate 1 give you a relatively similar levels of power, so do Protean 5 and Obtenebration 5. It is hard to compare the effectiveness of Potence 5 and Dominate 5, and oWoD Disciplines really do different things as opposed to D&D spells, but try to make your power in line with existing ones. There are some overpowered options, like Obtenebration 3, and Combination Disciplines are very often OP, but be careful when designing something like this.
- Most of Disciplines that require you to roll dice depend on one single Attribute and/or Ability, so characters can be built around it.
- Be ready to change your power if you find out that you messed up. It happens. Just tell your players that you messed up and provide a fix. Preferably tell in advance that the power is a subject to changes. If you are a player, always be ready for fixes, even if not notified in advance.
- Design your power for your group, knowing what you can expect from it.
- If creating some power as a plot device rather than something that your PCs will be able to put their hands on, just make an estimate of what is achievable by its roll, and then adjust it on the flight for the current situation.
- Remember that learning Disciplines normally takes time. It doesn't happen in a minute. If you plan someone to learn it, consider this time.
- Use common sense.