Look at the reaction he gives when Virgil is named as the one responsible for Remus' impact:
From the way he talks he sounds smug and self-certain, like he already knew what was going on and he's glad to be proven right.
vs. here, when Logan throws in Patton's name as another source of the problem at hand:
He's confused, thrown off course, he literally says "record scratch" out loud to verbalize how this has derailed his prior bragging.
(With extra ?s and !s in the captions for emphasis, no less)
The others are all similarly surprised, understandably so, because they know Patton as a primarily good force and have difficulty believing at first that he could be the problem.
But Remus, arguably the one most shaped by Patton's rigid beliefs, the one who exists because Patton wanted to separate him from c!thomas, is just as shocked at the notion that Patton is at fault.
Unless... he didn't blame Patton from the start.
Remember the line used to reference Janus in his song earlier?
"No longer will you deceive yourself about the ugliness within you"
I'm starting to think that Janus was really the one most directly involved in keeping Remus away from the others, presumably hiding him under layers of omission and denial because he knew c!thomas didn't want to see him.
And Remus, only really able to interact with/exist around Janus (and, presumably, not Patton much at all) only saw him as the source of his confinement, and as such, as the one to blame.
Until, of course, the courtroom fiasco, at which point he decided to challenge the moral compass that c!thomas has tied himself so tightly to by revealing just how much he's been repressing.
Tldr: Remus didn't know that Patton was the one who decided he should stay away because Janus is the one who actually kept him locked up, so revealing that Janus was just doing what Patton insisted on was a surprise for him.
