Working on a theory after a couple of Vol. 3 rewatches:
Essentially, the flaw in Rocket's design was the same as Batch 90's flaw. The High Evolutionary's plan was always doomed because his idea of a perfect race is one which is incapable of violence but still creates beauty through art and music, and that's inherently impossible.
To begin with, Rocket is a favorite of the High E, who explains to him that work on the prototype Humanimals is stalled because of their rage, noting that they're supposed to be "peaceful - like you" (emphasis mine).
Rocket, to all appearances, had the correct temperament. He was gentle, attentive, and most importantly, his obedience was absolute. He wouldn't even admit to enjoying music until he got permission. Paired with his cleverness, this made him exactly the kind of resident that the High E wanted for his perfect world, but he still was never meant to live there because he didn't have the right kind of body. The High E wouldn't care about Rocket's suffering but we can assume that he considered all that surgery inefficient.
So, the goal was to create a race that had a body like the Humanimals (which presumably reproduce on their own), but a mind like Rocket. But then Rocket solves the filtration problem and suddenly his genius has gone too far. I thought in my first watch that the High E decided at that point to execute Rocket because it damaged his pride to see his creation surpass his own intelligence, but there may be more to it. High E could have rationalized his way into taking credit for the idea, but an unpredictable element had just come up in the mind that he thought he had perfectly designed.
Rocket had taken a step from "useful" intelligence into creative genius, and all of a sudden, the meek little 89P13 was a threat. If his mind was allowed to continue its development, it might come up with any number of dangerous ideas: freedom, self-reliance, devotion to something other than its maker. Those are no big deal if they're limited to one caged raccoon, but Rocket's mind was supposed to be the blueprint for the perfect race.
Ordering Rocket's execution the very next morning was the logical next step, since the High E needed to know how he had made such a leap and couldn't get any answers by asking questions. He may have also been thinking he needed to get rid of the one subject who could potentially cause trouble in his lab - he even anticipated that Rocket, having discovered his own capability, was likely to set up an escape plan before the morning.
But with all this focus on Rocket's unique mind, the High E still never even considered the possibility that his creation could attack him. Quite an oversight, but then, until that very moment, Rocket had never considered that possibility either. His rage at Lylla's murder brought something completely new out of him, something which might otherwise have been dormant his entire life no matter how hard and lonely that life was.
I think it's significant that the block in making the Humanimals is rage. High E saw the problem as something being added to a mind whose natural state was peace (it looks like he even chose animal species that he perceived as friendly and harmless), but all along, that wrath was a part of Rocket too, as innate as his brilliance and inseparable from it.
When we see the "finished" Counter-Earth, populated by docile Humanimals who are smart enough to live their lives but don't question the order the High E imposed on them, I assume he's been working on it ever since Rocket's attack and this is the result that he originally thought he wanted. I mean, a 1980s cookie cutter suburb isn't my paradise, but I can buy that this is how he visualized his perfect world. By this time, though, he's aware that he fumbled in their making - note the new contempt for "rote memorization" - and he must know it had to do with the One That Got Away.
Certainly he's aware of Counter-Earth's decline, or he wouldn't have been so quick to give up on it when Quill shared his observations. The Humanimals mostly seem like good people (I'm still grieving for Vampire Bat Mom), but they'll never match the achievements of the original Earth. Something is missing.
So when Rocket resurfaces, it naturally follows that nothing else matters to the High E. The obsession isn't (just) about revenge; Rocket is the key to everything. He has the secret of what went wrong to make him disobey and attack, and also what went right that let him innovate an escape and then survive for years on his own.
High E is right about that, but he's dead wrong if he thinks he can learn the secret by dissecting Rocket's brain. It's been staring him in the face from the beginning but he's always refused to accept it: he didn't even have a hand in what makes Rocket special. Creative thought arises from the emotions of sentient life as God wills. We can nurture it or suppress it, but we can't engineer it.
Maybe Batch 90 was angry because they had been ripped away from their natural state and forced into a madman's game. Maybe they had a right to their rage.
Rocket became himself the day he learned to hate his maker, but he never stopped learning and growing, never stopped proving that he was more than what he was made for. In the end, he knows and we know exactly how violent he can be, and it's a story he shares with the entire human race of the real world.
He's offered revenge and he chooses peace. So can we.