(manga spoilers for chapters 90+)
Reading Beldaruit's recount of Qifrey's past while as a reader you have more context than he does is brutal.
For one, Beldaruit was convinced that what Qifrey sought every time he went to the surface was clues to feed his "hatred" and "fixation". Then you read it from Qifrey's point of view and you learn that he was being tormented by the silverwood already and simply wanted to know what the fuck had happened to him that made him shiver at every kind gesture he received. Like, for Qifrey to simply want answers as to why his own body reacted badly at every opportunity he had to feel safe and protected, only for Beldaruit to mistake that as an obsession with revenge is heartbreaking.
He also says that Qifrey's friendship with Olruggio did not alleviate that desperation... when it was, in fact, that very friendship that made Qifrey all the more focused on finding out. It was because Qifrey wanted that friendship that he wanted answers. Not to mention that, according to Qifrey's retelling of those "sneak-outs"... they were probably the happiest memories of this part of his life.
Beldaruit saw him as a boy so blinded by hate that he could not appreciate his friendship. What truly was was a boy leaving a miserable place to have fun adventures with his friend, yearning to be able to enjoy those moments to the fullest.
To be clear, I do not condemn Beldaruit for not knowing nor Qifrey for not sharing the truth. It's part of the silverwood vicious cycle. Once he found out, Qifrey had no choice but to leave Beldaruit in the dark. But it is a little sad that through Qifrey's reluctant secrecy, Beldaruit has assumed the worst.
On the other side, there's also this.
Beldaruit claims that the only reason Qifrey didn't resort to using forbidden magic to continue his quest for revenge was because that would mean he would risk getting caught, which would lead to the Knights Moralis erasing the memories he'd built up so far. He's practically saying that Qifrey appreciated this new life too much to risk it - which is, admittedly, a very wholesome view.
And then you read it from Qifrey's point of view and you learn that Qifrey was perfectly willing to go through that again.
In a moment of desperation, he said that it didn't matter if he lost his memories again since they'd been wiped over and over before, what difference would one more time make? And at this point, Olruggio didn't call his bluff - because I do believe it was not a bluff. From a detached point of view, Olruggio erasing his own memories of only that day was much less of a drastic measure and a more long-term solution too than Qifrey forgetting everything all over again, but given the situation, logic was the last thing on Qifrey's mind. He could only think and act on his feelings, and what his feelings told him in that moment was "Your memories matter less than Olruggio's. You matter less than Olruggio." I'm not even touching on how terrified he must have been feeling at the prospect of carrying that burden alone.
So, yeah. In this second case, Beldaruit actually assumes something positive behind Qifrey's reason to preserve his memories. Instead, we learn that Qifrey continuing on that life while avoiding dealing with forbidden magic was purely in order to honour Olruggio's sacrifice. Not because he wanted it for himself, but because Olruggio wanted it for him.
I do believe that in the years that followed Qifrey did build an honest appreciation for his life (that's the conclusion of that later arc, after all), but while they were kids? He was probably holding on by his fingernails to not give everything up.
And again, I do not begrudge Beldaruit for not knowing; Qifrey had to put a wall between them, so even though it's clear Beldaruit cares for him, he could only work with what little he had about him. On one thing he assumed something worse than the truth, on the other something better. But even the first was out of worry. He was not judging Qifrey, he was merely worrying. When he tells Coco to become his apprentice, it's because he fears she will be the cause for Qifrey to "relapse" to his old habits, endangering both her and himself. He aspires to separate them because he wants to protect them both.
In any case, it's quite heartbreaking to know just how wrong he is about him, especially knowing how crucial he's been to Qifrey's life. Just another person Qifrey had to leave in the dark.