Some spicy Kallus headcanons >:3 (or maybe they're more sad, decide for yourself)
Kallus has slept with multiple people in his life, but before Zeb the only person with whom he had ever really gotten into it was Jovan, with whom he was in a never-made-official-will-they-won't-they situation in their time at the Royal Imperial Academy.
They started out switching, but as Kallus built his walls higher and higher he become more and more of a top, his need to stay in control making him become the dominant one between them, until that need eventually became so overpowering that he didn't feel free enough to lose himself into it at all.
Already from his Academy days it frightened him how distracting youthful hormones could be, and he trained to completely desensitise himself to attractors and gain full control of his body that way. He succeeded; even when he was still with Jovan, he eventually could only get aroused when he would actively allow himself to. At first he would still masturbate in private to take the pressure off, but found that to only increase it in the long run, so he ended up even suppressing that desire until it as good as disappeared.
For this reason he was spectacularly good at assignments which included the seedier parts of a planet; no matter how many attractive hookers threw themselves at him, he would remain unfazeable. Many Imperials, even within ISB, would see such assignments beneath them, and by not only being one of the few to actually do them but also fulfilling them brilliantly, Kallus managed to quickly rise to prominence within the ranks.
He has slept with other people after Jovan, but always faked his orgasms, always wanting to fully stay in control of himself. He has gotten good at this; during undercover work he more than once used his body to get the information or whatever else he was after, even pretending to date people for a prolonged period of time if needed, and they fully believed him. None of it was real, of course, not even the sex.
By the time he first encounters the Spectres, he has mostly ranked out of having to go to such lengths to fulfil his task, and it has been a long time since he has known the touch of another -- or even himself -- in any other context than combat. Zeb and he huddling together is the first time he is in such close contact with anyone in years.
Once he finally joins the Rebellion on Yavin-4 and he and Zeb start figuring out what they mean to each other, this becomes an issue, especially when Zeb wants to get physical. Kallus knows he can fake it, but he doesn't want to do that, not with Zeb; he wants things between them to be real and meaningful. But in order to get there, he has to knock down all those walls he has contructed around himself for the past twenty years, and that is not as easily done as it is said.