Ralph's Personal Experiments

I've decided to run a series of personal experiments and document them in an effort to understand more about the experimental subject matter specifically and about personal experimentation more generally.

[ralphjpassarella]

Personal Experiment #2: Comparing precision and accuracy between multiple accelerometers – Protocol

My second personal experiment aims assess the (relative) utility of digital health activity trackers. This experiment isn’t an N-of-1 trial like our first installment. This is more of a classic measurement error / calibration experiment. So without further adieu, the design…

Aim: This experiment aims to assess the precision and accuracy of three commercially available activity trackers at measuring steps taken by a user of the device.

Experiment: I will wear three activity tracker devices, the Fitbit One, the Fitbug Orb, and the Jawbone UP, as directed by each manufacturer. For fourteen consecutive days I will wear the three devices, take 500 steps, and manually count each step using a commercially available step counter.

Prior to beginning the 500 steps, I will record the number of steps registered on each device for the day for a device’s baseline reading. After taking my 500th step, I will record the number of steps registered on each device. I will subtract the baseline step count from the final step count to arrive at the number of steps measured by each device during my 500 steps.

I will perform two trial runs of 100 steps each before the first experiment to familiarize myself with the process of uploading data from each device.

I will record measurements from the three devices being tested in the following order: 1) Jawbone UP, 2) Fitbit One, 3) Fitbug Orb

Primary outcome: Accuracy of each device compared to the “gold standard” measurement of self-recorded steps via step counter.

Secondary outcomes: Precision of each device over the course of the 14 trials. Relative accuracy of each device compared to the other two devices.

Analysis plan: [I’m going to write an updated post with the analysis plan before looking at the data. I figure I’ll make myself do it if I just get started with the experiment.]

Here is a link to the google spreadsheet where I’ll be collecting the data.

Potential sources of error:

- Additional steps counted by Fitbit, Fitbug, and/or UP either 1) before experimenter begins counting steps or 2) after 500 steps have been recorded but before readings are taken from a given device.

- Miscounting of steps by experimenter when operating the manual step counter – this could include over- or under-estimating total number of steps taken while measurements are being recorded.

- Error introduced by changing the terrain over which the 500 steps are taken

Materials:

- Step counter

- Fitbit One

- Fitbug Orb

- Jawbone UP

- Laptop with internet access

- iPhone to collect data from Fitbug device

[ralphjpassarella]

Personal experiment #1: Lunch purchasing intervention – Results

I’m not terribly certain how to write up the results of my N-of-1 personal experiment. I think the raw data is most compelling to me and is probably most intriguing to anyone reading this, so here they are in a google spreadsheet.

By way of a tally, I purchased lunch 9/20 days during the 20-day run-in period when I did not send myself a text message reminder, and I purchased lunch 10/20 days when I did send myself a text message reminder during the 20-day treatment period. The intervention did not appear to have a “significant” impact (I put “significant” in scare quotes to humor myself).

Following my two grocery purchases during the 8-week period, I went six days and four days without purchasing lunch respectively. This might seem obvious, but I found that having chips, particularly Sun Chips or Baked Lays, correlated with my not purchasing lunch (data not recorded). As I ran out of my favorite chips, I began purchasing lunch again.

I did not track the following variables as I had intended per my methods section: perceived hunger level, bank account status, cash in wallet, friends’ desires to eat lunch outside the office, weather

Other “supplementary results” include:

- My average wake time was 4:50

- My median wake time was 4:45

- My average self-reported stress level at lunchtime on days when I purchased lunch was 5.21. It was 5.23 on days when I did not purchase lunch. I’m paying no mind to significant digits if you’re wondering.

- My average self-reported workload on days when I purchased lunch was 6.68. It was 6.71 on days when I did not purchase lunch.

[ralphjpassarella]

N-of-1 emotions

In the medical research world, the N-of-1 trial is one of several sexy new alternatives being championed by a pocket of forward thinkers in the field as an alternative to the old, crusty, oft-economically-unappealing randomized controlled trial (RCT). And deservedly so! (1) The potential for N-of-1 study designs to revolutionize medical research tickles the same emotional and intellectual senses as the potential for personalized medicine to alter the process the healthcare system uses to manage its patients.

But, much like my feelings toward personalized medicine, I've found my unbridled excitement for N-of-1s replaced by a feeling of intellectual resistance. It's a resistance akin to what I felt when I heard a scientist throw out "GWAS" as the solution that would allow us to achieve personalized medicine, or when I hear a startup say they're going to use "machine learning" to change my behaviors with nary an explanation of how or why. The potential impact is there, but I need to look around a bit more before I leap. I need to take a step back from the excitement and begin to understand the complexities, the potential upside, and the likely underappreciated shortcomings of N-of-1 trials. (2) Maybe this series of experiments has yet another secondary outcome -- my understanding of N-of-1 trials.

Notes:

(1) I've considered starting a blog from time-to-time precisely to use sentence fragments for emphasis in my writing. After years of being told fragments are bad, using them liberally is quite...well, liberating.

(2) This analogy is a bit immaturely developed in my mind at the moment, but I included it here among other reasons because I would love to get a casual reader's thoughts on complexities N-of-1 trials introduce into the process of data analysis and of drawing conclusions from the data.

[ralphjpassarella]

Personal Experiment #1: Lunch purchasing intervention - Protocol

Treatment: (1)

I'm going to send myself a text message every day at 11:30am encouraging me to make a sandwich in the office instead of purchasing lunch from a market. 

Outcome:

Number of days out of 28 that I made lunch for myself during treatment (SMS reminder) compared to number of days that I made lunch for myself during a 28 day run-in period.

Other measures:

wake time, perceived hunger level, bank account status, cash in wallet, friends' desires to eat lunch outside the office, weather, meeting frequency, meeting locations for meetings that border lunch hours, evening plans, purchase (or lack of purchase) of breakfast, perceived stress level, perceived workload, grocery purchase

Experiment schedule:

Four week run-in period (n-of-1-esque):

I will conduct a two week run-in period to assess my lunch purchasing habits at baseline.

Four week treatment period:

I will send myself a text message for the next four weeks at 11:30am encouraging myself (is that proper grammar?) to make a sandwich for...well, myself, rather than purchasing lunch.

Predictors:

- Debit card statement

- Cash in wallet

- Presence/absence of purchased lunch

(2)

Notes:

(1) Major h/t to Ian Eslick's http://personalexperiments.org. I used his template entirely for the study design.

(2) I designed this experiment 2.5 months ago and only recently decided to blog about the design and execution of my personal experiments. I feel like it's appropriate to include the experimental epochs given the topic of this blog.

[ralphjpassarella]

O hai personal experimentation

Having spent the last twelve months of my life running a startup company that aims to help its customers conduct higher quality research, I've recently engaged in more than a couple philosophical conversations about the utility of the scientific method with my colleagues, my friends, and, with increasing frequency, myself. 

So I've decided to start a blog and run a series of personal experiments to test two overarching hypotheses that are summed up nicely by two of my favorite quotes (that also happen to adorn a blackboard in our office).

1. "Science is the battle against the bewitchment of our intelligence by data." - Emanuel Derman

I believe it is important that we think scientifically about our world. That we view uncertain decisions as hypothesis tests. Data can tell you a lot about what happened and can sometimes be used to predict future events with a high degree of accuracy. But I believe very strongly that blindly relying on data to make decisions places us, as individuals, at risk of numbing our intellectual capacity and curiosity and us, as a society, at risk for experiencing more societal black swans. I hypothesize that conscious personal experimentation will serve to heighten my reliance on scientific decision making other areas of my life. 

2. "The first principle is that you must not fool yourself, and you are the easiest person to fool." - Richard Feynman

It turns out human beings have a less than perfectly objective ability to interpret patterns and data from our world. We see patterns where none exist. We weight data that confirms our beliefs too heavily. We unfairly discount data that doesn't align with our current beliefs. (1) It's a bit of a grim situation if you look at it from a certain angle. That said, humans have used the scientific method to make amazing discoveries that conflicted strongly with an existing paradigm and required them to overcome these cognitive biases. That strikes me as a useful skill. I hypothesize that regularly setting up and executing personal experiments will heighten in me a sense of healthy skepticism when I view the results of my professional work and my other personal work, which will ultimately lead to more objective interpretations of the fruits of my labors.

With that, I set off on this personal experimentation journey. To anyone who reads this or any other post of the blog, I welcome your opposed views, aligned view, analysis of my data, or just about any rumination my writing spawns that you feel like sharing. (2)

Time to go science.

Notes:

(1) This is pulled straight from the book Bad Science by Ben Goldacre. If you'd like a quick read that gives you an overview of people who do bad science and a bit of insight into why they do it, it's worth your time.

(2) You can email me at [email protected] and follow me on Twitter @ralphpassarella.