I am not a database administrator, but you have probably been in those situations where you are forced to do it. The best situations are when you have total control of the database such as creating the tables, setting up the correct indices, and writing correct queries in away that you get those thousands of records in 1 second.
BUT this isn't an ideal world and we live in a world where we use frameworks and have to use their built in querying systems. I make use of SHOW PROCESSLIST; and SHOW FULL PROCESSLIST; in mysql. I also turn on the mysql-slow log and restart my configuration. My issue with these methods is that it only shows me when the problem STARTS -- and the queries it shows me at those times might be unrelated to the problem. I find the issues are always writing these huge temporary tables to disk, and my servers I/O is constantly waiting and waiting and waiting. What the hell is going on?
I have heard of Percona Toolkit, and I am going through those binaries to help me find out what is going on... but is there anything else out there? Yes, I've already run the mysql-tuner.pl script, mysql-tuner-primer, blah blah blah. These scripts gave me hope and then the hope was taken away with the server crashing once again...
So my question to you in the tumblr 'verse is: what tools do you use to figure out why MySQL is misbehaving?
02/24/2013 Edit: I found the same issues with malloc and some excellent articles on fixes for the issue are: https://github.com/blog/1422-tcmalloc-and-mysql and http://jamesgolick.com/2012/7/18/innodb-kernel-mutex-contention-and-memory-allocators.html