It's so funny and iconic that the most plausible theory as to how the Lannisters ended up in Casterly Rock is that Lann the Clever pretty much just gaslit the Casterlys out of the way by making them paranoid and getting them to turn on each other while he was just wandering around being A Problem until they fled because they thought (rightfully) that they were being haunted by a malevolent presence... Typical Lannister behaviour.
Casterly #1: I found my treasures in your room! You're stealing from me?!
Casterly #2: Noooo I was framed and also the voice that howls in the courtyard at night whispered to me that you want to sleep with my wife!!!
Lann the Clever, soon to be King of the Rock: