To be honest, that line of being not able to love a woman because he looks down on everyone always makes me think that Ohba is quoting/lifted it from Dostoevsky. This place in “Crime and Punishment”, a dialogue between Raskolnikov’s buddy (think Matsuda, about the same personality) and Raskolnikov’s younger sister (think older Sayu) about him:
“You have said many curious things about my brother's character, and...have spoken impartially. That's good; I thought you were in awe of him,” Avdotya Romanovna observed with a smile. “It also seems true that he ought to have a woman around him,” she added pensively.
“I didn't say so, but perhaps you're right about that, too, only . . .”
“He doesn't love anyone, and maybe he never will,” Razumikhin said bluntly.
“You mean he's unable to love?”
“And you know, Avdotya Romanovna, you resemble your brother terribly much, in everything even!” he suddenly blurted out, unexpectedly for himself, but, recalling what he had just told her about her brother, he immediately blushed like a lobster and became terribly embarrassed. Looking at him, Avdotya Romanovna could not help laughing.
I mean, some more about Raskolnikov:
A feeling of the deepest revulsion flashed for a moment in the young man's fine features. Incidentally, he was remarkably good-looking, taller than average, slender and trim, with beautiful dark eyes and dark blond hair. But soon he lapsed as if into deep thought, or even, more precisely, into some sort of oblivion, and walked on no longer noticing what was around him, and not wishing to notice. He only muttered something to himself from time to time, out of that habit of monologues he had just confessed to himself.
An anxiety with no object or purpose in the present, and in the future nothing but endless sacrifice, by means of which he would attain nothing - that was what his days on earth held in store for him... What good was life to him? What prospects did he have? What did he have to strive for? Was he to live merely in order to exist? But a thousand times before he had been ready to give up his existence for an idea, for a hope, even for an imagining. Existence on its own had never been enough for him; he had always wanted more than that. Perhaps it was merely the strength of his own desires that made him believe he was a person to whom more was allowed than others.
“Crime? What crime? ... My killing a loathsome, harmful louse, a filthy old moneylender woman who brought no good to anyone, to murder whom would pardon forty sins, who sucked the lifeblood of the poor, and you call that a crime ?”
Yeah. But he loves his family and falls in love later actually. When he drops his “believing he was a person to whom more was allowed than others” act.