Reproducibility

file: command not found

[staff]

Once more for old time’s sake

🔥 With your help, we passed Title II net neutrality protections. Now we need to defend it.🔥

On December 14 the FCC will vote on Commissioner Pai’s plan to repeal Title II rules. This week he tried to justify that decision with a “myth busting” explainer where he makes a lot of sweeping claims he doesn’t think you’ll fact check. 

So let’s go through his big points:

❌ Mr. Pai claims ISPs won’t block access or throttle content

These are the real facts. Before Title II, the internet was so “free and open” that… 

Comcast blocked P2P file sharing services (EFF).

AT&T blocked Skype from iPhones (Fortune) and, later, wanted FaceTime users to pay for a more expensive plan (Freepress).

MetroPCS blocked all streaming video except YouTube (Wired).

In today’s media market where the same huge companies make and deliver content, Commissioner Pai wants us to trust that corporations won’t use their dominance to bury competitive content or services. 

❌ Mr. Pai claims Title II keeps ISPs from building new networks

Here’s another claim Commissioner Pai doesn’t want you to fact check, but:

AT&T’s own CEO told investors that the company would deploy more fiber optic networks in 2016 than 2015 when the FCC passed Title II protections (Investor call transcript). 

Charter’s CEO said “Title II, it didn’t really hurt us; it hasn’t hurt us” (Ars Technica).  

And Comcast actually increased investment in their network by 10% in Q1 of this year (Ars). 

❌ Mr. Pai claims repealing Title II won’t hurt competition

As we mentioned above, ISPs tried to interfere with the services their customers could access and courts had to step in to stop them.

The FCC tried to craft net neutrality rules in 2010 called the Open Internet Order but the ISPs sued and won. The courts told the FCC that the only way to guarantee a free and open internet was using their Title II authority. Without those protections, any of these things would be legal:

Your ISP launches a streaming video service and starts throttling other streaming services until they’re unusable.

Your phone company cuts a deal with a popular music streaming service so it doesn’t count towards your data cap but lowers your overall data limit. If a better service comes along (or your favorite artist releases new tracks somewhere else) you can’t use it without incurring huge data fees.

A billionaire buys your ISP and blocks access to news sites that challenge their ideology. 

Repealing Title II would be like letting a car company own the roads and banning a competitor from the highways.

❌ Mr. Pai claims there won’t be fast lanes and slow lanes

Let’s break this down: We won’t have fast lanes and slow lanes, we’ll have “priority access” and…non-priority access? Well gosh.

🚨 Please help us protect Title II one more time! 🚨

This week we co-signed a letter with more than 300 other companies—businesses Mr. Pai gleefully ignores—urging the FCC to retain the Title II internet protections. Now we need you.

Go to 👉 Battle For The Net 👈  to start a call with your representatives in Congress. Tell them to publicly support Title II protections. 

The FCC votes on December 14.

We’re only powerful when we work together.

Oh, also: that post about automatically unfollowing the #net neutrality tag—it’s not true. It’s really not. That’s not who we are. Whatever happened, we haven’t been able to reproduce it. We tried. A lot.

But if it were true—which it’s not, we feel compelled to say again—THAT’S EXACTLY WHY YOU SHOULD CALL YOUR REPRESENTATIVES and demand a free, open, and neutral internet.

We can do this one more time, guys! ❤️

[travalicious]

[mrs-jamie-wellerstein]

ONE OF MY FAVORITE COLD OPENS EVER

[lickmetilllicecreammmm]

I laughed out loud in Starbucks because of this

[iamabagfullofcats]

This is a masterwork of storytelling which is also a rollercoaster from beginning to end.

[totallyamelia]

To my Tumblr family

@channel @david

To my Tumblr family,

I think I’m finally done crying enough to actually write something. I didn’t cry when Murphy told me I was being let go because I was upset to lose a job. I cried because I felt as though I was being ripped away from my family.

Crap, I’m crying again. That really didn’t take long! Gosh, I really hate crying in airports. It must be something about trips to Duluth because this now makes it two for two! I guess I’m not as dead inside as I pretend to be.

I’m crying because y’all are so great and I cherished every day I got to spend with you. The last (nearly) three years have been some of the best of my life. I got to wake up each morning and go to a place I loved and be around people who genuinely brought me joy. Even on the worst days, there were always smiles to be had. And on top of that, I got to work on an amazing product that I truly believe in and has done so much for its users. I got to make something people wanted to use. I got to make something special.

I’ve always referred to Tumblr as home and the people as my family. What that meant to me is something that’s hard to put into words, but I want to try.

HQ was a place that was safe for me in a world that still really does not want people like me to exist. Even New York City isn’t a safe place for trans people, but Tumblr was. Tumblr was a place where I got to truly be myself, unapologetically. I got to just exist. I got to let my guard down, something I very rarely do.

Tumblr was a place where I found my voice, grew, and discovered who I am. When I joined Tumblr in 2014, I was only nine months into living as an openly trans woman. I had been struggling to really find comfort and confidence while existing in the world. I didn’t know my voice. Mostly, I was just angry at the world. But Tumblr gave me a place to work that out and learn what my voice was and how to use it. Tumblr helped me grow as both a human and an engineer and it gave me the confidence to be me.

That’s the Tumblr I will miss because it wasn’t a job to me, it was a home and family that also happened to give me a paycheck. Over the last three years, Tumblr introduced me to some of the best humans I’ve ever met in my life. People who are kind and caring. Supportive and encouraging. Smart yet not full of themselves.

Thinking about my time at Tumblr, I don’t know what else I could ask for. On top of all of these things, I got to work at a place where I felt like I made an impact for the better. I got to work somewhere I could speak up and fight for what I believe in. I could fight for equality, diversity, and my coworkers. I could fight for our users. Sometimes, that even meant yelling at David in front of the whole company at All Team or speaking up during Prod Reviews with a personal but relevant anecdote about the importance of our decisions. I wasn’t always right and it didn’t always change things, but I got to be in a place where I could speak up and fight those fights.

I got to, somehow, lead Tumblr’s femgineers and organize our monthly lunches. It was never lost on me that I was a trans woman leading a group of women and not once did anyone ever make me feel like I didn’t belong in the group or as its organizer. When I get bummed or jaded about this increasingly bonkers world we live in, it’s all of you that give me hope for the future. Seeing the fucking amazing work that some of you are doing while still in your early 20s and only relatively recently out of college gives me so much optimism for what you’ll achieve and the mark you’ll leave on this world. I’m literally crying again just thinking about it. My voice could never do you all justice, but I hope it helped.

I know this is already long, but I have so many memories rushing into my mind right now. I’m hesitant to start naming them because I know for every one of them, there will be dozens I’m neglecting to mention. It’s hard to list everything when literally every day of the last three years was memorable.

But to just name a few…

There was my first Tumblr Halloween party where Steck drunkenly told me about the discussions they had about me after my interview. He kept saying he wasn’t supposed to tell me, but over the years, I think he told me a dozen times. Almost every time he was drunk. I, of course, knew better than to trust him. Mohonk! I mean, what else do I need to say? Pellcar on the way to Monhonk! Dying tonight on the boat cruise. All the femgineer lunches. Flying to WWDC in Mint chocolate lass. Two Grace Hoppers, the first of which was one of the most affirming weeks of my life. Shitposting and coalescing in #new_york. Caragh’s completely insane stories and my DMing her with the weirdest fucking shit a coworker has ever said to another. Gatorade Gala and Chainschella. The Kenny themed party! grund. Sleeping in conference rooms after mind blowing hack days. Holiday parties followed by Sward bringing everyone McCoulton’s the next morning. The goddamn lizard election! The 9west temperature wars and 9east clapping. Constant sandwich arguments in #holy_war. Brian Michel dressing up as a reblog on Halloween. Friday coffee with the old Ads Team crew. Coming in the day after the election and having Ken hug and comfort me as I cried in the freight elevator. All the comforting we did for each other after the election. Ethan DMing me “food” right before lunch every day. Movies We Love. The aesthetic channels. Sam constantly asking me if I was happy and giving me heartfelt hugs on a near-daily basis. Finding sisterhood for the first time in #heyladies. Robbie’s hack day projects. Taking selfies of myself to approve every single PR. All the GIFs. The bad puns with Keith, Cherokee, and Josh. The security capture the flags. Daily cat photos in #KittyTime. Turning around at my desk to see Lucy Lawless in the elevator. For better or for worse, Prima Punisher. Flying with my Falcons. The dozens of you who helped my former startup’s Kickstarter be a success. Snowboarding trips to Killington and Montana. Trolling Joey with Slack bots. Bonusly points from Graham for every lol chain. 🍆🕳.

I could honestly go on and on with memories all day, there are enough to fill an entire blog and they’re coming faster than I can type them. I wish I could mention everyone here, but if I didn’t mention you by name, please know that you had an impact on me and were a part of this family. If our time at Tumblr overlapped, you were (are!) important to me and made my life better.

Looking forward, I will be fine. Don’t worry about me. Instead of worrying, use that energy to keep Tumblr great. That is my ask to my family, keep Tumblr fucking weird af. And SPEAK UP. You work in a place where you can use your voice. Do it! Fight for Tumblr users. Fight for each other.

And worst case scenario, Chris already has the git branch to delete Tumblr.

I want to thank everyone who came to Madison Square Park yesterday while I sat crying on bench. I really don’t cry very often and I have a thing about letting people see me cry, but just seeing all of you and how much you cared helped me so much. I also want to thank everyone who reached out to me via text, Twitter, Facebook, and other Slack orgs. And especially to Michelle (happy birthday!!!) and Shah whose messages about the impact I’ve had made me start crying all over again. Okay, Amelia, get it together.

I hope to keep in contact with all of you. I hope you’ll reach out to me on Tumblr, Facebook, Twitter, Snapchat, text, other Slack orgs, and wherever else. Let’s hang, get lunch, eat dinner, grab drinks, play volleyball, and just plain be weird. I’m terrible at keeping in touch with people, but I want that to change. Please don’t hesitate to contact me anyway you know how because, and I know I keep saying this, I love you all.

Finally, thank you to Amanda Brennan for bringing me to Tumblr and making the last three years possible. Thank you to Brian and Sherry for allowing me to have a heads up the the night before so I could process and react with dignity without being blindsided or reactionary. And thank you to David for building something so wonderful.

I love you all so so so much, Amelia

[sha384]

[engineering]

Jetpants Integration Testing

Tumblr is a big user of MySQL, and MySQL automation at Tumblr is centered around a tool we built called Jetpants. Jetpants does an incredible job making risky operations safe and reliable, even fairly complex tasks like replacing failed master servers, or splitting a shard.

While Jetpants is an incredibly effective and valuable tool for Tumblr’s day-to-day operation, it has remained very difficult to implement a meaningful testing framework. Integration testing at this level is very challenging. In this article I’ll go through these challenges and how we’ve tackled them at Tumblr.

Requirements

Jetpants operates under the assumption you’re managing MySQL daemons on a fully functional host, and that it can:

ssh to the target system

manage processes via service or systemctl commands

copy data around between systems

allocate spare servers from the asset management system, Collins

Right away this means we have some challenges with respect to infrastructure testing:

We need a Collins deployment

We need an environment with spare servers running MySQL

We need these spare servers to actually be servers, not light-weight Docker containers

Problems

For most of the life of Jetpants, these requirements were fulfilled using actual hardware in a testing pool in our datacenter. This wasn’t ideal, however. Running a test which allocated more replicas, or tested shard splitting means using an extensive amount of real hardware that takes hours to reprovision. Testing changes to the Collins code meant talking to a real Collins deployment. What if we messed up?

This test strategy has all the hallmarks of manual testing. It doesn’t prevent regressions. Test coverage of our featureset is spotty based on what was interesting at the time. Public contributors can’t run the tests.

For a new user to pick up Jetpants and Collins, it can be very difficult to get started. Jetpants requires Collins to be configured it certain ways that aren’t publicly documented. When I first built the testing environment, I had to regularly compare what I had to our actual deployment to figure out why Jetpants wasn’t working correctly.

Solution

During a Tumblr hackathon earlier this year, I devoted my time to developing an isolated, automatic testing system. We have since integrated this system directly into Jetpants and are using it in our day-to-day development and testing.

Our test framework is based on the NixOS test framework, the same framework NixOS uses to verify it is safe to release a new version. These tests use QEMU to start an isolated environment of at least one VM, and NixOS configuration to build the VMs.

Our testing framework adds lots of tooling on top to let us create robust tests. By default, a test has a running Collins instance, a master database server, and one replica. Simple options allow provisioning additional spares or additional replicas on that initial master.

Below is an example test we’ve written for performing a dead master promotion. This is where the current master database is dead, and we replace it with one of the existing replicas.

Here you can see what a test looks like, and how easily we can express the components and phases of our tests:

import ../make-test.nix ({ helpers, ... }: { name = "shard-dead-master-promotion"; starting-slave-dbs = 2; test-phases = with helpers; [ (jetpants-phase "shutdown-master" '' Jetpants.pool("POSTS-1-INFINITY").master.stop_mysql '') (phase "jetpants-promotion" '' ( echo "YES" # Approve for promotion echo "YES" # Approve after summary output. Confirmation. ) | jetpants promotion --demote=10.50.2.10 --promote=10.50.2.11 '') (assert-shard-master "POSTS-1-INFINITY" "10.50.2.11") (assert-shard-slave "POSTS-1-INFINITY" "10.50.2.12") ]; })

Running this test first provisions the base environment, by

starting Collins

starting 3 Linux systems running MySQL

creating a master-replica relationship between one MySQL server as a master, and two MySQL servers as replicas, then loading in a default schema, and naming it the POSTS-1-INFINITY shard

Once all this preparation is done, our test phases begin.

First we shut down the current master, to simulate a dead master situation. We then run the jetpants promotion command which will replace the old master (10.50.2.10) with a new master we have selected, 10.50.2.11. jetpants promotion will prompt for confirmations, so we echo approvals to its stdin.

We continue by validating that the jetpants command did what we expected, and verifying the master and slaves.

Initial Results

Through this testing, we have already identified and fixed several race conditions and very old interface bugs. Nix’s functional nature allows us to create and tear down test VMs in minutes, as it isn’t a convergence-based configuration management tool. The stability of the test framework, and consistency of its results have allowed us to more aggressively change the underlying code in Jetpants while remaining confident our tools will work correctly during our day-to-day production maintenance.

Jetpants has been under continuous and vigorous development at Tumblr for many years now, and I’m excited about where the future will be taking MySQL automation at Tumblr.

- @beta

[sha384]

[engineering]

Jetpants Integration Testing

Tumblr is a big user of MySQL, and MySQL automation at Tumblr is centered around a tool we built called Jetpants. Jetpants does an incredible job making risky operations safe and reliable, even fairly complex tasks like replacing failed master servers, or splitting a shard.

While Jetpants is an incredibly effective and valuable tool for Tumblr’s day-to-day operation, it has remained very difficult to implement a meaningful testing framework. Integration testing at this level is very challenging. In this article I’ll go through these challenges and how we’ve tackled them at Tumblr.

Requirements

Jetpants operates under the assumption you’re managing MySQL daemons on a fully functional host, and that it can:

ssh to the target system

manage processes via service or systemctl commands

copy data around between systems

allocate spare servers from the asset management system, Collins

Right away this means we have some challenges with respect to infrastructure testing:

We need a Collins deployment

We need an environment with spare servers running MySQL

We need these spare servers to actually be servers, not light-weight Docker containers

Problems

For most of the life of Jetpants, these requirements were fulfilled using actual hardware in a testing pool in our datacenter. This wasn’t ideal, however. Running a test which allocated more replicas, or tested shard splitting means using an extensive amount of real hardware that takes hours to reprovision. Testing changes to the Collins code meant talking to a real Collins deployment. What if we messed up?

This test strategy has all the hallmarks of manual testing. It doesn’t prevent regressions. Test coverage of our featureset is spotty based on what was interesting at the time. Public contributors can’t run the tests.

For a new user to pick up Jetpants and Collins, it can be very difficult to get started. Jetpants requires Collins to be configured it certain ways that aren’t publicly documented. When I first built the testing environment, I had to regularly compare what I had to our actual deployment to figure out why Jetpants wasn’t working correctly.

Solution

During a Tumblr hackathon earlier this year, I devoted my time to developing an isolated, automatic testing system. We have since integrated this system directly into Jetpants and are using it in our day-to-day development and testing.

Our test framework is based on the NixOS test framework, the same framework NixOS uses to verify it is safe to release a new version. These tests use QEMU to start an isolated environment of at least one VM, and NixOS configuration to build the VMs.

Our testing framework adds lots of tooling on top to let us create robust tests. By default, a test has a running Collins instance, a master database server, and one replica. Simple options allow provisioning additional spares or additional replicas on that initial master.

Below is an example test we’ve written for performing a dead master promotion. This is where the current master database is dead, and we replace it with one of the existing replicas.

Here you can see what a test looks like, and how easily we can express the components and phases of our tests:

import ../make-test.nix ({ helpers, ... }: { name = "shard-dead-master-promotion"; starting-slave-dbs = 2; test-phases = with helpers; [ (jetpants-phase "shutdown-master" '' Jetpants.pool("POSTS-1-INFINITY").master.stop_mysql '') (phase "jetpants-promotion" '' ( echo "YES" # Approve for promotion echo "YES" # Approve after summary output. Confirmation. ) | jetpants promotion --demote=10.50.2.10 --promote=10.50.2.11 '') (assert-shard-master "POSTS-1-INFINITY" "10.50.2.11") (assert-shard-slave "POSTS-1-INFINITY" "10.50.2.12") ]; })

Running this test first provisions the base environment, by

starting Collins

starting 3 Linux systems running MySQL

creating a master-replica relationship between one MySQL server as a master, and two MySQL servers as replicas, then loading in a default schema, and naming it the POSTS-1-INFINITY shard

Once all this preparation is done, our test phases begin.

First we shut down the current master, to simulate a dead master situation. We then run the jetpants promotion command which will replace the old master (10.50.2.10) with a new master we have selected, 10.50.2.11. jetpants promotion will prompt for confirmations, so we echo approvals to its stdin.

We continue by validating that the jetpants command did what we expected, and verifying the master and slaves.

Initial Results

Through this testing, we have already identified and fixed several race conditions and very old interface bugs. Nix’s functional nature allows us to create and tear down test VMs in minutes, as it isn’t a convergence-based configuration management tool. The stability of the test framework, and consistency of its results have allowed us to more aggressively change the underlying code in Jetpants while remaining confident our tools will work correctly during our day-to-day production maintenance.

Jetpants has been under continuous and vigorous development at Tumblr for many years now, and I’m excited about where the future will be taking MySQL automation at Tumblr.

- @beta

[sha384]

CAP

me we came up with a patch for jetpants, by using the following, sneaky test case:

db = pools.first.slaves.first loop do db.collins_secondary_role = "foo" db.collins_secondary_role = "bar" end

me I’m sure the Jepsen guy would be impressed by our efforts to defeat the guarantees of our DB

them who needs any of the letters of CAP, really?

[instagramchild-blog]

Y'all mind if I praise God real quick ?

[drugfriendly]

I’m proud of them

[sha384]

hello

[sha384]

when projects `git clone` libraries at compile time

[classicprogrammerpaintings]

Abandoned GitHub repository

Caspar David Friedrich

1810

Oil on canvas

[throwmylifeawayforjustice]

Take breaks to shower. We will not presume to know the details of your friendships and love life, but we can only guess that following this advice will pay off in the end.

(via throwmylifeawayforjustice)

[sha384]

$ nix-shell --pure --run "file"

/run/user/1000/nix-shell.myqhmq/rc: line 1: file: command not found