Why Alviarin Wasn't so Smart and Brilliant
Alviarin had a brilliant Darkfriend carreer by her own standards and by applying continuously a few traits that she had learned and mastered. She killed a young man in her teen years which showed her ability to keep composure during stressful events/situations and this is likely what led her or was close to her beginning to channel the One Power.
Due to her ability to remain calm she had managed to survive both as a Darkfriend novice and Accepted, and as a Black Ajah Aes Sedai.
But let us compare her with a few other Black sisters.
When Liandhrin met Moghedien, she almost immediately tried to take advantage or look for a way to twist her Power against her. Liandrin was brave to use her own hatred of Nynaeve to come close to Moghedien and earn her trust, however little that means to people who serve the Shadow. Liandrin is not tempted to try against Moghedien when she was in her full wake but just and only when she saw a weakness in her composure. That is, Liandrin used her now-or-never chance. She failed because she underestimated (doh) the Forsaken's ability to channel under extreme pressure - after all, the Forsaken were Aes Sedai, for which we have some proof from Graendal's ability when she is cut during the Last Battle in her fight with Aviendha. And of course, she underestimated by a large margin how old and experienced Moghedien in fact is compared to herself .
When Alviarin met Messaana she did the opposite. She never underestimated the Forsaken and she never looked at the possibility that she could fail in her own way in terms of serving the Dark One. Alviarin was quick to discover the Forsaken was hiding among Aes Sedai and that there was no way that she herself might have matched her in a fight. But Alviarin also assumed that Messaana had full control over her environement. Messaana was a good leader and a strong administrator, but she did in fact little more than hide herself and use Alviarin to run most of her errands in the Tower. Messaana knew Alviarin was intelligent and besides teaching her a few weaves and occasionaly checking up on her, she did little but use the already-set network of the Blacks. When Alviarin ended up between her and Elaida, both irrational personalities and with a blindspot each, she was bound to fail. When Messaana was punished for not attending the cleansing of saidin, Alviarin peed her pants, not because she saw something which set her off, but mainly because she had thought of Messaana as invincible and all-knowing. The fact that Messaana felt she had enough power to ignore an order from Moridin speaks of her blind spot for the rest of the world besides the White Tower at this point. Alviarin never saw her as vulnerable or faulty. In her own thoughts, she refers to Messaana as some long-awaited teacher or leader who had come to help her. She almost relied on the Forsaken. The same way that Ishamael stretched his arm and chose her to be the Black Ajah leader, all the while, there was no way that she could have been sure there is anything more than chance behind his choice, as the Black, naturally, don't keep a record of their deeds.
Alviarin wasn't particularly smart either, when she decided to press Elaida too harshly. This did result in a near-collapse inside the Tower, but her rigid methods led Elaida to be suspicious of her enough so that Alviarin at one point later couldn't be sure anymore if Elaida had not uncovered the Black Ajah itself. Alviarin believed, knowing Elaida, that she had managed to find a way to twist her arm good, but then again underestimated the woman in this Tower who managed to usurp an entire Amyrlin Seat.
Alviarin was calm, thoughtful and emotionless, but at some point these qualities weren't enough for where she was anymore. One mistake of Alviarin's was mainly that she never gathered an acomplice during her time as a Keeper. She survived the Tower reuinion mainly because of the chance which had led Messaana to teach her to Travel. Which by itself was only due to the fact that Messaana probably only needed no more than a few errands run outside the Tower, for which she herself never thought of using another but her main and most important servant.
Had Alviarin not Travelled to Tremalking for a long period of time, would she have kept her position as Keeper? Yes, if she had managed to manipulate Elaida more carefully and not enough for Elaida to hope that she is a traitor just for the sake of losing her.
We can almost deduce for Alviarin that she was a thought-out careful planner, but her rigid actions and the fact that an unexpected change of her plans sent her to her rooms to think it through meant that she wasn't well-prepared for quick action, even if it were necessary. Compare her to some of the other Blacks, like Liandhrin or Sheriam, two other leaders among the Black, compare her even to Moghedien, she was much more set in her ways and much less flexible.