Everyone Should Know How To Do Taguchi Experiments
Empiricism is an important virtue; it's great to try things and find what works for you, and there's a noble tradition of people doing that and blogging about it. We should be doing this more.
The problem is that experimenting on yourself can be pretty slow and tricky and inconvenient.
This post is about Taguchi arrays, which fix that by allowing you to get much more useful information from far fewer experiments. The method is commonly used in industry for process optimisation, but it isn’t complicated and is very easy to apply to DIY experiments. (Thanks to NightHawkInLight for making me aware of this!)
The default way kind of sucks
Suppose you're wondering what will work best to improve your sleep. You've got one of those sleep tracker devices that gives you an integrated Sleep Quality Score, so you've got a clear metric to optimise. Let's do some empiricism!
Now, what interventions to test? A lot of people say melatonin is good. Apparently 0.3mg is the ideal, but maybe 1mg works better, and it's more commonly available. Other people swear by magnesium. You can get magnesium glycinate easily, but some say magnesium l-threonate is better, although it's more expensive. You've heard exercising helps, but is cardio or strength better? And you've got some of those blue light blocking glasses, but they look kind of goofy. There’s a ton of things to test.
The big piece of accepted wisdom here is to only change one thing at a time. If you change more than one thing and your sleep gets better, you won't know which thing did it. Or, one thing you change might help and the other might make things worse, so they cancel each other out and you learn nothing.
Changing one thing at a time, we need at least 8 experiments:
In principle we could do this in 8 days, just trying each thing once.
But that's going to be really noisy. Just one measurement per item is hardly going to tell us anything; we should try each thing on more than one night to get more signal. Maybe do each one for two or three days? That's 16 or 24 days, which is kind of a lot. Give each thing a week and that’s 2 months of experimentation.
And also, we care about finding a good overall setup, not just which individual things work. Probably there are some interactions here, like, what if magnesium works really well, but only if you also exercise? We'd never find out using this experiment.
To really know, we'd want to try every combination. So, 3 levels of melatonin (none, 0.3mg, 1mg), times 3 options for magnesium (none, glycinate, l-threonate), times 3 exercise options (none, cardio, strength), times 2 for red glasses (yes or no), means... 54 days, to try each thing once. That's not great.
There's got to be a better way!
This is where Taguchi Arrays, aka Orthogonal Arrays come in!
You go to an online Taguchi array generator like this one, add in your factors and their levels, and it spits out something like this:
Before you start, it’s a good idea to randomly shuffle the order of the rows, so secular changes or cumulative effects don’t bias things.
Then you go through the rows one night at a time. Each night you follow the protocol listed, and each morning you record your Sleep Score for that protocol.
Now to analyse the results. The genius of orthogonal arrays is that for any pair of factors, every possible combination of their levels appears the same number of times. This means that you can safely compare averages.
So if for example we want to know how magnesium glycinate compares with magnesium l-threonate, we can take the average of the sleep scores of nights we took glycinate, and the average of nights where we took l-threonate, and just see which is better. This is safe to do, because all the other variables are equally represented in those sets, so they won't bias things. You can go through and check - the 3 glycinate nights have one of each of the melatonin levels, and one of each of the exercise levels, and two have glasses and one doesn't, and the exact same is true of the L-threonate nights, and of the "no magnesium" nights. And this nice property is also true for all of the possible values of all other pairs of factors.
So using just 9 nights of experiments (just one more than the original 8 night experiment which told us very little), we get at least three nights of data to average together for each level of each factor, rather than just one! And we're testing all possible combinations of levels for each pair of factors at least once. With just these 9 experiments, we can find out a huge amount of information, almost as much as we'd get from the full 54-day factorial experiment!
Now, this works best for situations where you expect the result to be approximately a linear combination of the factors, with only simple interactions between pairs of factors. For example, if there were some major interaction between three factors, like if melatonin worked wonderfully but only if you do cardio and also don't wear light-blocking glasses, this wouldn't catch that. But real life experiments tend not to have much of that kind of thing.
And of course if you have a situation like, magnesium only starts to help with sleep on the third night in a row you take it, this won’t detect that. But that’s a problem with using ‘sleep score on the same night’ as the metric, not an issue with Taguchi methods.
I hope that this easy to use experimental design method gets widespread adoption. It really lowers the bar for getting useful information from DIY experiments. Try it for yourself!