The Other World's Books Depend on the Bean Counter is a story that focuses on different kinds of workaholics and what you give up for what you believe is right and why that is, in many cases, fucked up, and can also compound your issues down the line. And why, in many cases, it's important to call that out and demand better for yourself and other people. And that the systems we are born into are not inherently good just because they've always been there.
Seiichirou is a workaholic because who lives in a capitalist society that trains people to become office drones: working all day, all the time, even in your free time, even when you're sick and/or tired, for the sake of (usually someone else's) capital. He deprioritizes his personal relationships not because he doesn't like having them - canonically he cares deeply about his parents and had multiple girlfriends, and even reached out to aid a stranger and cares about common people, this is not an unfeeling person who finds people and relationships unlikable - but because he has been trained to prioritize his work, and that makes him forgetful in personal matters. As he notes on multiple occasions, it's his workaholism that led to the downfall of his relationships, not because he found being in the relationships annoying or difficult. And because his relationships kept failing because he deprioritized them, he has little experience making them work long-term:
“To make things perfectly clear, it wasn’t as if Seiichirou was inexperienced in love.
It had not been an unusual thing for him to have a girlfriend, nor was he a virgin.
His ex-girlfriends, however, had almost always broken up with him after asking the cliché question, “What’s more important, your work or me?"
Aresh’s initial actions had been to save his life, and afterward, his excessive concern and management of Seiichirou’s health might have been some sort of atonement for the man who had been accidentally abducted from another world. Or maybe Aresh was just incredibly good-natured—maybe his personality wouldn’t let him leave something alone once he had decided to take care of it.
However, that didn’t explain their kiss the day before. Seiichirou had tried to purchase some medicine (nutritional tonics), and Aresh could have just left Seiichirou with his slight discomfort. But instead, Aresh had gone out of his way to heal him with magic and suppressed the incoming magic-sickness with a touch that might be considered the most profound act when choosing a partner—a kiss.” - Light Novel Volume 1, [Chapter Four] Aged
There's another entire paragraph where Seiichirou dithers over how Aresh might be interpreting their relationship.
This is how the anime portrays the same scene in Episode 3:
An even briefer explanation of his past relationships with no reference to his workaholism (and that he is, on some level, aware of it), a brief thought on how "strange" his relationship with Aresh is, and a very open-ended thought where he's kind of starry-eyed about Aresh's actions, before he realizes he's being silly/distracted while trying to work, and goes back to work. I think a central issue at this point with folks who've read the light novels or even at least the manga versus folks who are anime-only is that you're nearly getting two different Seiichirous. Light novel Seiichirou is awkward and has experience issues, but he's thoughtful, and way more aware of what things with him and Aresh mean (like he's very obviously the older, more experienced person in his relationship with Aresh in the novel). Anime Seiichirou is far more detached and remote, and seemingly clueless. He has feelings, but they're left more open-ended. And this isn't entirely why folks are having odd interpretations of his behavior, but I think it's part of it, on top of the societal issues that Seiichirou in particular and Aresh are navigating.
I think a lot of people mistake Seiichirou's frustration with other people in Romany for disgust or aggravation that he has to talk/interact in general, when... no. Seiichirou was kidnapped and is also being bullied around by a class system that doesn't exist in modern Japan, and he is confronted by a bureaucratic system that benefits the most wealthy at the cost of the least, and is likely to endanger him, and he gets frustrated when other people (mostly the privileged) think this is an ignorable problem (his anger is a lot more evident in the light novel and manga, which is especially evident in his first meeting with Yurius). I think people forget Seiichirou volunteered to work, which would generally involve interacting with other people. He also seemed quite happy to join an office with coworkers, and frequently interacts with his coworkers, whom he canonically likes, even if he occasionally gets exasperated with them. It's not people he hates (in fact, it's the people for whom he is doing much of what he does - to better things for them now and in the future, as he believes it is everyone's duty to do), it's the way he's been forced into this messy place and can't get home and may be in for worse if someone doesn't step in and fix things.
Take Seiichirou's confrontations with Yurius. Age and class are central to the issues here. Seiichirou is aggravated that a man younger than him - so you've got elder getting annoyed with a younger person - is not only wasting his time with nonsense (seeing Seiichirou as a rival for Yua's affections, when it is extremely obvious that's not happening to anyone not blinded by religious dogma, propaganda, and a crush), but also abusing his authority to threaten Seiichirou into dealing with this. So not only is Seiichirou's time being wasted with utter bullshit by someone who has better things to do and could be doing worse to other people who don't see a crown prince abusing his authority to be an issue, but also it is extremely dangerous for Seiichirou to refuse to participate/be around Yurius' tantrums and accusations. Because Yurius is the crown prince. Of course Seiichirou is going to be aggravated with the whole mess of dealing with any part of this. A commoner from another world given sanctuary and lodging on the crown's charity is not going to say, "Fuck you" to the guy next in line to be king (much as Seiichirou basically repeats it in his head). That's how you end up homeless, starving, in prison, or executed. That Seiichirou is angry at this entire situation is extremely understandable. I don't think he hates Yurius, he just finds Yurius annoying and expects better of him.
Now look at Seiichirou's confrontations with Yua, and yes, I do mean confrontations:
“K-Kondo! This is an invitation from the Holy Maiden! Your work can wait, so please hurry!!”
At his boss’s insistence, Seiichirou left with the man who had come to retrieve him.
. . . Seiichirou didn’t remember Yua being worried about him, and if he was to reveal his true thoughts, he would be directly criticizing the kingdom…” - Light Novel Volume 1, [Chapter One] Employed
“If Seiichirou was being honest, things had gotten to the point where he considered Yua someone to be avoided at all costs. He knew that she had no ill will or ill intentions, but he came to understand that, for their respective positions and in all other aspects, interacting with her would not lead to anything good.
But Seiichirou also knew that Yua did not understand this.” - Light Novel Volume 1, [Chapter Seven] Set Out
Yua is essentially now of a higher class than him: as the Holy Maiden she's sort of nobility, after a fashion. She has a lot more say than a tag-along most people don't know what to do with. She also has the ear of the crown prince and who knows what other nobility, and can use magic. He cannot refuse to meet with her, and she could do a lot with her power, if she chose, to screw with him. Unfortunately she's also a teenage girl from modern Japan who is deeply unaware of the power she wields and that begging Seiichirou to switch spots in the caravan with her isn't the same as swapping seats during lunch or something. Seiichirou is at risk whenever he interacts with her: whether from her own retribution or from the retribution of those around her, like Yurius or her caretakers (as has happened with the storeroom assault), no matter what he does. That Seiichirou is angry about this entire situation is also extremely understandable, especially because his life is in danger because a 16-year-old has more power than she understands. That he knows Yua has to hear things multiple times before the lesson sinks in is also quite frustrating. But he's also not entirely angry at her: he understands she is a teenager and uneducated in some things, and still gaining experience, and she is also a victim of kidnapping and not being told all she needs to know, because those around her want to manipulate her to their own ends. This means that he can be angry at her for some things, but shouldn't be for other things, and that too is frustrating. He is, nonetheless, trying to help her regain her autonomy, because he cares about people and feels it's the duty of older generations to teach the younger ones. That interaction itself is not the part he hates. He hates that they're being forced into it, as a matter of survival, and that they're both in danger.
You can see the comparison quite well with how Seiichirou treats Sigma, which is as a happy teacher/mentor nurturing a younger person to their full potential:
The difference is, Seiichirou wasn't forced to interact with Sigma. Sigma holds no power over Seiichirou, and Seiichirou technically does hold some over Sigma as an adult with money and connections while Sigma is a child from a poor background with no connections, but he's not a noble, or a teacher, or a police officer, or anything else. Without abandoning his post, this is about as close to a "two strangers who happen to meet and befriend each other" can get. And Sigma is not treating life like high school. Seiichirou is fond of Sigma. He wants to nurture Sigma for the accounting department. And he's aggravated when Ist nabs Sigma first. That's not the behavior of someone who dislikes dealing with people and uses their work to avoid such things. Seiichirou goes out of his way to create opportunities for Sigma to better himself.
Most of his other interactions are frustrations anyone who's done any customer service work or worked with other departments would understand. Someone who keeps sending in their forms wrong? Who keeps coming to harass you with the same questions/demands over and over again? Add in that Seiichirou is trying to stop the kingdom from running out of money and some rich guys who never thought about this before don't give a shit, in part because they're lazy.
Seiichirou likes Norbert. They're friends. He's at times confused by Norbert's behavior and why Norbert isn't abusing his authority more - as he could, and as his peers do - and why he isn't ambitious. And Seiichirou is frustrated at Norbert's incompetence, but I think most people know at least one person like this. Someone you like, who's a really nice person... who's bad at their job. And the thing is, Norbert isn't even bad at his job! He was just born really privileged and placed in a department that didn't actually really do any work until Seiichirou came along and pointed out they were supposed to. And once Norbert gets to work and trains, he's actually pretty decent! But it is understandable that Seiichirou is exasperated with the whole situation. Not because he hates having friends or coworkers, but because Norbert's shortcomings often lead to more work for Seiichirou and every moment is precious so wasting it is bad.
Norbert also just enjoys life a lot more than Seiichirou does, in large part because he was raised in privilege and can do that; he never had to work for food and housing, or for that for his relatives. This is also an issue that comes into play when you look at Aresh and Seiichirou's relationship. More on that later. It's not that Norbert likes people more than Seiichirou (in fact, Norbert seems more scared of people than Seiichirou is). It's that they have different priorities, in large part due to how they were raised.
Camile, besotted as he is with Seiichirou, is the terrifying boss who will middle manager you if he can. Many people have met someone like this (ignoring the flirty stuff): someone who sees you as an easy target for a middle manager position.
And I'm sure there are middle managers who love their jobs. But a lot of us do not want that at all, largely because it's grueling and a lot of responsibility and often involves managing other people in particular. Seiichirou has experience with this. He doesn't want it. He doesn't hate Camile, but he understands the power Camile wields, and that it can be used against Seiichirou at any time. Seiichirou is very aware that Camile is the most powerful man in Romany after the king.
In fact, one facet of Seiichirou that aggravates Aresh in particular is Seiichirou's ability to get around to different people. Seemingly at all times he's talking and interacting with someone new in or around the government in some fashion. He knows the prime minister, he knows the prince's half-brother, he's taking over the accounting department, he's getting close with the Holy Maiden, he's getting close with Ist and the mages, he's buddies with random children and one of the head priests of Abran, he's bullying the other knights, he's getting close with diplomats... Seiichirou is many things, but avoidant of other people (so long as he doesn't have to go to the cafeteria because he considers it a waste of time that could be better spent working), he is not.
Aresh is a workaholic, too. He just largely ignored the paperwork side of things until Seiichirou came along. As is explained in the light novel:
“As an extraordinary prodigy of both magic and swordplay, Aresh had made a name for himself at an incredibly young age. With his marquess pedigree and his good looks, Aresh had become a symbol of the Third Royal Order, which, as an order of magic, stood apart from the other Royal Orders. He had been appointed as commander at a young age.
Despite his meteoric rise to the top, perhaps because he had already had everything in life, he was totally indifferent toward status and prestige—he had no serious interests and was perpetually bored. Due to this, he was dispassionate about his responsibilities as commander and wouldn’t have minded at all if he was removed from his position that very moment." - Light Novel Volume 1, [Chapter Two] Promoted
What's also interesting about Aresh is that he doesn't ostensibly care as much about his work until Seiichirou shows up, but also he does sacrifice a lot for his work, including risking his safety, frequently. That's dedication. His troops also seem to like him, not just respect him. And after Seiichirou, he starts getting even more dedicated to his work so Seiichirou and the public at large have fewer dangers to face.
His aggravation with Yua isn't that he hates interacting with women or girls or whatever. It's because he disliked the workload he faced with the summoning for her, is used to certain behavior as regards his position at court and as a noble and just culturally (which Yua does not understand) and he disliked being forced to be her teacher:
“For the [Holy Maiden Summoning], Aresh had been forced to do many unreasonable tasks to carry out the lost secret art. He wondered if there had been any other method they could have used. . . .
Ever since childhood, Aresh had been the object of affection for many girls because of his good looks and his family’s status, but he was used to this, and ladies of marriageable age would distance themselves before trying to catch his attention. The Holy Maiden, however, perhaps because of her otherworldly nature, interacted with Aresh in a very friendly manner, with no regard for his social status at all. Or maybe she believed that her rank was the highest of them all.
. . . [Yua] had appointed Aresh, commander of the Third Royal Order, of all people, to be her magic tutor. In the first place, why would a student be appointing their own instructors? Aresh was neither a teacher nor a magician—he was a knight.
He had initially objected, claiming that he was too busy, but a superior had twisted his arm into becoming her tutor." - Light Novel Volume 1, [Chapter Two] Promoted
He goes on to explain that he's annoyed that she ignores her studies during their lessons, that Yurius, who is present a lot, should be teaching her the things she's having Aresh teach her, and that she's deeply clueless about the timeliness of the miasma situation and the local culture.
If you look at his initial considerations of Seiichirou, it's confusion about Seiichirou's own priorities, especially given how Seiichirou could choose never to work, and a simple matter of just minding his own business. He has enough to deal with without getting more involved with clueless otherworlders he finds more annoying than anything else.
There's nothing here about him disliking dealing with people. It's just that no one really came to his interest before Seiichirou showed up and he just focused on his job. There's nothing that I've seen indicating Aresh dislikes people in general or having relationships. But not everyone is interested in being with other people all the time, especially in a physical and/or romantic relationship. Heck, Aresh might even be demisexual. Seiichirou's presence introduces him to the benefits of being in a relationship: that he enjoys being around someone he's close with and finding things they enjoy, and that he likes taking care of people (which he did anyway as he's in charge of protecting Romany). His issue, primarily, is that he is inexperienced in a lot of social niceties in regard to interpersonal relationships and particularly in being in a romantic and physical relationship. So what he falls back on is being a noble, where he can order people around and do as he pleases.
So what you have are two people who work for different but similar reasons: neither of them have to work, as Aresh is a noble and Seiichirou has a guaranteed stipend and housing. Both of them have issues with romantic and physical relationships: Seiichirou's relationships all ended because of his workaholism, and Aresh hasn't been in a relationship prior to being with Seiichirou. And when they are pushed together due to chance and their odd choices, that lack leads to issues, especially when they both have strong beliefs that make them unwilling to change their habits. Aresh comes from a class where you can choose to work, but don't have to. Seiichirou comes from a class where you have to work. Both problem solve in their own way. And Seiichirou, like most of his time in Romany, constantly comes up against that class barrier: that Aresh is noble and Seiichirou is not. It is obnoxious that Aresh orders him around all the time and Seiichirou basically has no say, even if it's generally for Seiichirou's own good. And Aresh actually does try to learn about Seiichirou's past in Japan and what he's like as a person (in the light novel and manga), because understanding the idea of capitalism (let alone questioning the class system he was born and raised into) is extremely hard for him to do, and the only person who could properly talk to him about it, Seiichirou, is often busy doing other things and largely too polite to have a very frank privilege talk more than the one or two times it kind of went badly.
Seiichirou is also far more aware of his own limitations, and has taken to seeing them as a personal failing (to a degree). He knows he has feelings for Aresh, and Aresh has feelings for him (in the light novel and manga, mostly; the anime cuts out most of it) but he's not sure how to avoid the usual disaster, and he also believes Aresh deserves better/will find better and move on. Aresh, meanwhile, refuses to back down, and faces all his problems about the same way: headfirst and fighting. So when this combines, when Seiichirou wants boundaries and Aresh ignores them, that creates issues. And you can argue that Seiichirou should be bullied because his lifestyle is killing him and nothing else will work, but that doesn't mean Seiichirou is wrong for being frustrated about the whole thing. Or that Aresh is wrong for seeing injustice and trying to fix it, or correct in all the ways he goes about doing that, even if many of his choices do lead to benefit.
To a degree, yes, they are two people who kind of closed themselves off to different things and bring out different sides of each other when forced together. But it's not because either of them dislikes people in general or dealing with messy people problems. A central theme of the story is about autonomy and when it is respected/disrespected, and when to put up boundaries, so of course their central conflicts are going to be about that. It's kind of unfortunate that I also think a large part of misunderstanding their characters at this point is due to whether you've read the source material or only engaged with one of the adaptations, particularly the anime, which cuts so much out.
Read the novel in English. Listen to it as an audiobook.
Read the manga adaptation in English