I'm planning on making a campaign that uses elements from monster hunter where my players are the hunters and they must kill the monster. I want to give the enemies unique attacks I've made myself, but I don't know how I should calculate the to hit modifier for those attacks. How is one supposed to handle calculating that to-hit modifier? Is there advice or guidance in the rules?
DMG at "Dungeon Master's Workshop: Creating a Monster."
On pp.273-282 of the Dungeon Master's Guide you'll find guidance on building a monster of your own, which will help you create one that "fits" pretty well within the 5e bestiary.
Note: this is enough to get you what you titularly asked for: the to-hit modifier for a monster created from scratch. But from the body of your post I think more'll be in order.
Then you playtest. Because there are so many things about terrain, synergies, conditions, monster tactics, and party composition that influence the difficulty of a combat encounter beyond just the "tale of the tape."
Then you go to grad school.
Or at least, you take an online course from a prestigious, non-accredited institution. The Angry institution.
The Angry GM has a number of good posts on the theme of building monsters. As usual I'll warn that his posts are liberally salted with vulgar and rude language, but there's good stuff in there. Once you get past the rambling introductions....
Now you've got some seriously interesting monsters. And you playtest again. Because it's D&D, and anything can happen once a group of characters decides that lethal force is the only way forward.
Custom Monsters, or Custom Attacks?
If you're just creating custom attacks to add to existing monsters, then use the same attack bonus they already have for their other attacks, potentially modified by the stat you want their new special attack to be based on. For example: if you're adding a steel-feather-dart attack to a griffon, you can look at their basic attacks at +6 to hit, and say 'Those are based on their +4 strength bonus, but this will be a ranged attack, so I'll replace that with a +2 dexterity bonus for a total of a +4 to-hit.)
If you're adding a save-based attack to a creature that doesn't otherwise have any, the basic save would be their attack bonus for the proper stat, plus 8. (Generally, physical special attacks should be based on strength or dexterity, breath weapons and similar biology-based attacks should be con-based, and overtly magical stuff should be int, wis, or cha, but that's just general guidelines.)
If you're making a custom monster, that's much more complicated, so my advice is:
Steal, Steal, Steal
The quickest way to create a new monster is just re-skin an existing one and tack on some minor mechanical changes as needed. Nobody needs to know that your Agate Defender is secretly an ogre wearing a hat that says "Construct". If you describe its stone fists crashing down, they won't realize that it's actually using a greatclub weapon. And if you stole the Dragonborn's breath weapon to make that 'crystal shard spray' attack, they won't be the wiser.
Really, the only thing to watch out for in this case is any special abilities that might make the creature significantly tougher than it otherwise would be. For example, adding flight to a basically melee-focused monster isn't a big deal since they have to come in close to fight anyway, but adding flight to a ranged combatant may make them significantly tougher, since some of the party can't use their favored combat style. Similarly, adding defensive abilities to a high-damage, low-health monster (or conversely adding a big attack to a Sack of Hit Points monster) may make them significantly more effective.
It's more art than science in a lot of ways, but grabbing an existing monster that's close to your target CR and fills kind of the same role can very quickly get you 70% of the way to being done, and no worries about how many HP or what the attack bonus ought to be.
A lot of times it's more like 98% -- a reskinned monster may work fine just as it is. Describing a squad of hobgoblins as a troop of terra-cotta warriors with no real mechanical changes (beyond treating them as Constructs, and of course any flavor-related effects, such as being vulnerable to Shatter) requires virtually no effort on your part. Just remember that constructs and undead tend to have special resistances and immunities, so if you're using either of those types, you should add or remove the appropriate qualities.