In my experience, playtesting isn't about having an awesome game. But having a game that gathers data to make future games awesome. So your players will need to buy into the goal, and be willing to have after-encounter discussions about what they liked or didn't like about the creature, how they think it could be improved, how hard or easy they thought the encounter was, and so forth.
Start of with as vanilla, plain, ordinary a set of by-the-book characters as you can. These should have standard stats and abilities straight from the PHB, no special powers gained or overpowered items. No custom or add-on rules, nothing that makes your character more powerful than the typical, generic, characters that come with published modules.
Level them up to what you think the creature's CR ought to be, but again, make sure they are text-book advancements. Use only things affordable off the starting equipment at higher levels rules from DMG, for example.
Then have your test party go up against the monster(s) in fairly generic ways, more than once. With each encounter, rotate who plays what class, so your data will reflect the different playing styles each player brings to that PC class.
Track how hard it is to overcome over a number of fresh encounters with a published monster of the same CR you believe your creature should be -- full health and full spells at the start of round 1, etc. Use hard data, not just subjective questionnaires, so maybe track things like:
- Percentage of hits scored by PCs
- Percentage of hits scored by Creatures
- Damage dealt by PCs
- Damage dealt by Creatures
- Rounds to completion of encounter
- Did the PCs win or lose?
- PCs reduced to zero or lower HP during encounter (How many? How fast?)
(Or whatever metrics will help you align the creature with your expectations.)
This is your baseline, so you may want to find a creature as similar mechanically speaking to yours as possible. Find similar attack styles or abilities, etc. to remove as many variables as you can. Fight this encounter with your party at least once or twice, but maybe 3 or 4 times, gathering data to use as your baseline.
Then do similar encounters with the same PCs against your monster. Again, repetition may help improve your dataset here.
If your creature went down too fast or too slow relative to your baseline, then adjust the CR up or down. Or change the monster stats. Then reset and try again.
Ask for subjective feedback, too. But your primary focus for setting the CR should be baselining your beast against published beasts.
(Not that published creatures are perfect. Every gaming group approaches encounters with different tactics...)