Kasumi's penchant for acting tough and blaming herself that she couldn't stop a wanderer or a situation probably stems somewhat from when she was younger mainly because of Caleb telling she couldn't be tough or handle situations. His constant reminders of people capable of overwhelming her in a fight thanks to the bond memory that was presented also probably attributed towards this mindset. Kasumi wants to be part of a situation, not to be told she couldn't, even if the other person had some good intentions underneath the cruel actions ( locking her in an attic for a day because Caleb wanted to hang out with friends but was told it was to protect her from 'bullying', which I digress isn't great at all ).
Adding to the fact in some 4 star cards, even Rafayel points out she could get overpowered in certain situations even if she does fight wanderers all day for her work but backtracks his opinion when Kasumi tells him that she could had handle the situation with the clerk that was somewhat obsessed with her if they only didn't have a taser. And his response is, 'no one is saying that you couldn't' after she tries to justify what went wrong. Haven't really explored the other's 4 star cards but Kasumi really didn't care for the things Sylus also said to her during their initial three day meeting, adding to the fire of wanting to act tough and get better at bluffing because of it. Kasumi is hard on herself because of many factors, even her job is because she never wants to feel powerless again.
In all, there's a wall she puts up sometimes but it's understandable to say the least. And it's one reason why her relationship with one of her childhood friends is strained. But then again, sometimes it seems like Caleb likes shooting himself in the foot, Kasumi thinks.