I'm gonna project a little and say that Kaeya was confused for a girl a lot when he was a kid because of his long hair and soft face. His father didn't like that he wasn't super masculine and would get angry that he "got a daughter instead of a son." This caused Kaeya to have self-esteem issues.
When he's taken into the Ragnvindr Clan, Diluc and Crepus refer to him as she and he begins to think they don't like him even though they seemed to treat him so well. It isn't until the beautiful blonde maid who had bathed him the night he arrived (Adelinde?) called him a lad that he realizes they were just mistaken as they trip over themselves apologizing.
The way they treat him doesn't change and Crepus and Diluc still loved spending time with him. Crepus never gets angry at him for wanting to read the fantasy and adventure novels in the library. Diluc doesn't seem annoyed at his plants and even helps him take care of them. Outside of teaching him basic self-defense, they don't force him to learn how to fight. They don't force him to learn politics and let him run and play like a kid.
He decides it's not so bad here.
Bonus:
Years later, he's sitting at the bar taller, with broader shoulders, and a more masculine build that makes it harder (but not impossible) to be mistaken for a girl. He teases Diluc about how he thought he was going to have a sister instead of a brother and how relieved he must have been to play with swords and pretend to be knights and pirates instead of playing dress-up and doing each other's hair.
Diluc looks him in the eye and tells him not to pretend like they hadn't done each other's hair often as kids and that they hadn't tried on their father's suits.
He leaves his flustered brother at the bar to be teased by Rosaria and Venti.