For obvious reasons, I recently got interested in how to build websites that are widely accessible but also resistant to censorship. Naturally, my first instinct was to run off and come up with my own blue-sky designs of the most resilient, censorship-resistant website in the world. But censorship is not new and I realized it would be smart to learn from the past: in particular, The Pirate Bay and WikiLeaks, which both continue to operate even under immense pressure to shutdown.
Domain
The first problem to sort out for any website is the domain name, and both these websites have had recurring issues with their domain names being seized. TPB tried a ‘hydra’ approach, where they registered thepiratebay under many different regional TLDs with different small registrars, and they would direct users to any that would work. They hoped this would help protect them from domain seizures, as well as circumvent ISPs that blocked their main .org domain. But unfortunately, their regional TLDs were consistently seized, leaving them with only their .org domain. TPB is able to keep their .org domain because they do technically follow US copyright law: they respond to DMCA takedowns [1]. WikiLeaks is also a .org domain, but has had legal help from the EFF with keeping their domain operating [2].
The Pirate Bay
The Pirate Bay uses Cloudflare as its edge network and splits traffic between a number of cloud providers. Any one cloud provider suspending them won't cause them to go down because they can easily move the traffic to other clouds or add new ones [3]. It's also unlikely to cause data loss, as they can replicate data between clouds. In addition, Cloudflare's edge network functions like an anonymizing layer, preventing the general public from identifying which cloud providers they use and pressuring them, which almost certainly makes it a lot easier for them to retain services.
WikiLeaks
WikiLeaks on the other hand has chosen to build out what's essentially their own edge network [4]. They rent space in datacenters to colocate their own machines, and these machines serve their website's traffic. New documents are submitted through a Tor hidden service, where they likely get stored in the cloud for review. Since document submission is anonymous and relatively low traffic, it would be possible to also anonymize payment for these services by getting someone sufficiently far removed from the project to pay and reimbursing them in cash. Note that I don't have any evidence they do this, but it makes sense to me that they would because keeping documents in the cloud prevents them from being physically seized, and keeping payment sufficiently far removed from key project members prevents their systems from being identified through financial records.
Conclusions
Operating as a non-profit under a .org domain seems most likely to minimize domain seizures.
Anonymity is centrally important to censorship-resistance. It prevents retribution against the clients of a service, and allows the service itself to blend in among its vendors' customers.
Censorship-resistance is one of many network effects of edge CDNs, and this is because they have diversity in their points of presence. Even if content is taken down in one region, it will still likely be available in others, and therefore accessible even where it’s blocked through a VPN.
In my last post, I discussed Income Share Agreements (ISAs), including their many downsides and how they’re prone to being more expensive than traditional loans. In this post, I wanted to discuss something that I think is a much safer solution to the same problem: Income-Based Repayments (IBR) for a loan.
IBR loans are similar to ISAs in that the minimum payment is a percent of your income, but there are a few key differences:
The agreement is still structured as a loan, which means interest accumulates over time rather than the borrower having to hit an arbitrary repayment cap. The faster the loan is repaid, the less interest expense there is.
The expiry period of a loan is much longer than that of an ISA (25 years vs 5 years).
This solves the same problem that ISAs sought to solve by minimizing the likelihood that the loan becomes a financial hardship, in addition to not having edge cases that are unfair to the borrower. Loans also have many desirable features that an ISA doesn't such as allowing pre-payment, consolidation, and refinancing.
Finally, IBR loans consitute better investments than ISAs because having a fixed interest rate and a long expiry period makes the long-term returns more predictable, even if the short-term payouts are not.
Income Share Agreements, or ISAs, are contracts where a borrower receives something of value, and in exchange they give the lender a percentage of their income every year for a fixed number of years. A lot of people have been proposing them as an alternative to traditional student loans.
Unlike loans, which are two dimensional (term and rate), ISAs typically have about five parameters:
Income Threshold. The yearly income under which you make no payments, and at/over which you must make monthly payments. Usually $50k.
Income Percent. The percent of income the payment must be equal to. Usually 10-20%.
Repayment Cap. The maximum amount of money that may be paid pack. Usually 1.5x prinicpal.
Payment Count. The number of monthly payments required. Usually 24 or 48.
Expiry Period. The amount of time after which the ISA expires regardless of repayment status. Usually 5 or 8 years.
As such, the contract ends when one of three events happens: the repayment cap is hit, the required number of payments has been made, or the contract expires. Lenders usually seem to want to maximize the likelihood of hitting the repayment cap, and they do this by setting fairly low income thresholds relative to the profession they’re training people to enter.
The benefit of an ISA over a traditional loan is that there’s less risk to the borrower because the structure of ISA payments minimizes the chance that they become a financial hardship. When a borrower makes less income, their ISA payment is less or nothing at all.
What’s worth considering though, is that professionally successful individuals would expect to pay much much more over the life of an ISA than they would with an equivalent loan. That’s because ISAs have all the same failure scenarios as a typical loan (the borrower dies, the borrower is unemployed), plus anything that could cause the borrower to make less money than expected, compounded by the fact that ISAs create no minimum obligation for the borrower. So if someone chooses a lower-paying career path after school, that’s already a failed investment, and in fact you’d expect more ISA borrowers to do this than those that received a traditional loan.
As a method of financing, ISAs disconnect the decision to receive debt from having any intention or ability to repay the debt, which encourages riskier behavior, which drives their price up, which drives away safer borrowers, and creates a negative feedback loop. As an investment vehicle, ISAs have completely unpredictable returns that can only be measured historically, and they more tightly couple the labor market with the financial market which are already prone to their own negative feedback loops.
Recently I was asked about the possibility of using MLS in groups with “hidden members”. That is, groups where the creator is known to all participants but the participants don’t know each other. This is the use-case of broadcast TV, private Twitter accounts, Instagram stories. The answer is no, MLS doesn’t work here.
The main issue is that MLS isn’t secure against malicious insiders, and therefore isn't suitable for most broadcast use-cases. A lesser issue is that MLS is designed for homogeneous groups, and would be wasteful to use in a scenario where one member has special authority.
Which solution is more appropriate depends on how many receivers there are, and if it’s possible to interact with them all regularly.
If there are too many receivers to regularly process all their messages, then it's not possible to achieve MLS’s strong post-compromise security properties and the best solution is a Broadcast Encryption. With broadcast encryption, receivers are assigned a decryption key by the sender that allows them to decrypt messages, while also allowing the sender to efficiently revoke their access later. Delerablee's broadcast encryption has the special ability to handle an arbitrary number of new members and revocations with constant-size ciphertexts.
On the other hand, if it's possible to interact with all receivers regularly, the sender could establish one-on-one MLS groups with each receiver. The sender could use these groups to share a common symmetric key with all receivers, to which it would encrypt broadcasts. This approach is about as expensive as maintaining an MLS group between all receivers, but mitigates the risk of group sabotage.
The conclusion of my previous post was that whether to rent or buy in my area likely comes down more to individual factors rather than financial ones. But what was interesting to me while doing that analysis, is that I realized I honestly had no interest in buying a condo that’s roughly equivalent to my current apartment.
While I happily currently live in a 1 bedroom apartment, any condos I’d genuinely be willing to buy would either be 2 bedroom, or the same size in a higher-demand area. That’s because my current apartment is over-fit to my job: it has an abnormally short commute to the office for how far away it is, and a dining area that works as a small home office. However if I was to buy a condo, I wouldn’t be able to over-fit it to my current work situation. I’d want to choose one which is reasonably close to a large number of job options, or has a real office in anticipation of working from home more often. Any condo would have to be better than any apartment, to make up for the loss of flexibility in where I live.
Also if I’m honest with myself and if it were possible, it would be severely tempting to buy with cash than with a mortgage. I think this is essentially because the monthly cash outflows are the smallest once the initial deal is done, and you have the deed to a property. It feels the most secure, and the most like being genuinely free. And in fact, I’ve met people that value this sense of freedom enough that they’ve bought houses that are much worse than what they could’ve afforded with a mortgage.
Looking purely at the psychological aspects, buying with a mortgage is really the least attractive option because it’s the most invasive, you’re functionally still renting, but you have the least flexibility. That explains why it’s priced the lowest when you compare objectively equivalent apartments/condos, but why renting or buying with cash is often still more attractive when you compare subjectively equivalent properties (ones you’d actually be willing to rent vs buy).
A while ago, I tweeted that “it’s essentially always cheaper to rent in urban areas than it is to buy property.” I believe this is conventional wisdom called the Rule of 16, where you divide the price of a house by the yearly rent of a similar apartment. If the ratio is over 16, it’s “better” to rent in that area than it is to buy and vice versa. I’m not sure where that rule came from and I’ll try to derive it later, but I decided to check this assumption by modeling it.
The approach I took was to calculate the price of a perpetuity that would pay my rent, bills, food, and shopping expenses assuming a modest degree of inflation. I compared this to the strategy where I buy a house with a mortgage with 20% down, purchase a 30-year annuity that pays the mortgage, and a separate perpetuity that pays for bills, food, shopping, property tax, HOA, and repairs. I also checked a third approach, which is to buy a house with cash directly instead of using a mortgage.
So the output of each model is the number of dollars that I need today to cover all my expenses until my untimely demise, assuming that I’m able to get some given return on my savings. Here are the results, as a graph:
For reference, my rent is $3k/mo and the price for a similar condo is about $650k. Obviously buying with cash is always slightly more expensive than a mortgage because mortgages are heavily subsidized. However I was surprised that despite living in an area where the Rule of 16 says I should rent, it’s essentially always long-term cheaper to buy.
In general, the difference between buying and renting is so small in the 7-10% range that whichever is “right” for an individual likely comes down to individual factors that are harder to quantify.
The only financial skill I have that I haven’t seen other people do better than me is DuPont analysis. DuPont analysis tries to find the drivers of a company’s Return on Equity by factoring it into three parts:
Profitability as measured by Net Profit Margin, or how much profit a company is able to keep from its revenue.
Efficiency as measured by Asset Turnover, or how much revenue a company is able to produce from its assets.
For three CDN companies I’m familiar with, the DuPont breakdown looks like this:
The profit margin for Akamai stands out as the only positive one, since they’re the only profitable company of the bunch. Cloudflare and Akamai both have a higher degree of leverage because they’ve both raised money by issuing convertible debt, while Fastly only has minor obligations like unpaid invoices and wages.
It makes sense that Akamai, as an older profitable company, would prefer to raise funds through debt instead of diluting their equity. But it doesn’t make sense that Cloudflare would be so highly levered, or that they would be the least efficient and also somehow more profitable than Fastly. I suspected this was due to the convertible debt they issued, and since this was done opportunistically instead of out of need, I removed it from the company’s total assets and liabilities. This adjustment makes Cloudflare the most efficient, least levered of the three:
It’s also worth pointing out that between the two unprofitable companies, Cloudflare is more profitable than Fastly, and you can see Cloudflare’s net loss decreasing as revenue increases while Fastly’s net loss is increasing with revenue:
Fastly’s cost of revenue (servers / bandwidth / essential employees) is consistently about 40% of revenue and their operating expenses (engineers / marketing / etc) are consistently about 80% of revenue (expenses = 120% of revenue). Cloudflare’s cost of revenue is consistently about 20% of revenue, however their operating expenses grew 10% Y/Y despite revenue growing 55% Y/Y. Fastly had a similar degree of year-over-year revenue growth, but their R&D expenses grew just as much and their G&A expenses doubled. This largely indicates that the company isn’t making more efficient use of its employees as it scales, while Cloudflare is.
In summary, DuPont analysis helps us see:
Akamai’s relative strength comes from its financial position of already being profitable and able to take on huge amounts of debt.
Cloudflare’s relative strength comes from being highly resource efficient.
The natural question that comes up when thinking about disruptive innovation is: How can incumbent companies successfully navigate the transition to a disruptive technology?
The answer I’m familiar with is basically that managers invest in the new technology, and let the old and new compete. This acknowledges the risk that the new technology might fail, and also captures the upside if it succeeds. As the previous company/department begins to decline, the other starts growing just as quickly and you already have an ownership stake in it.
However, what if the new technology is lower margin? Or if you’ve made investments in the old technology that haven’t fully paid off yet? Then encouraging the transition would be against the manager’s interests. They would rather slow or halt the development of the new technology as much as possible.
The answer here is still to develop the new technology, but just enough to make creating a startup in the area unattractive while not substantially competing with existing products.
The first real example of this I recognized was with serverless. All major VPS providers like AWS / GCP / Azure offer serverless platforms, but they’re all artificially handicapped such that people don’t see them as suitable replacements for a VPS. As long as there’s no competition, they can leave things undeveloped. But as soon as there is competition, they have the resources and market lead to stop it.
It’s often cheaper than running the same application on a VPS. Right now, this may be mostly because it’s under-priced. But it does actually require fewer physical resources, since applications scale on-demand and many applications can share the same physical host more efficiently.
Possibly also falling under “cheaper,” it has a lower operational burden. Developers simply upload their code and the cloud provider handles process management and scaling.
Applications that run on serverless platforms are faster because they’re always hosted near the end-user.
It’s such a good business because all of the benefits above come from strong network effects which entrench incumbents. Being able to charge a low price is the benefit of having a lot of customers and high hardware utilization. Being near end-users is the benefit of having built a large, distributed network of data centers. Reliability and scalability are the benefit of significant and prolonged technical investment. Additionally, serverless platforms have proprietary APIs that create lock-in and having a brand that's trustworthy enough to build a business on top of propretiary APIs is a significant accomplishment.
The flip side of all the aspects that make it a good business also make it a good product because application developers get access to the upside of all these network effects, without having to build them themselves. It ends up being a democratizing force more than a centralizing one.
E2E encryption possibly fits into the model for disruptive technology:
It’s disproportionately valued by a small set of people.
Established companies are unable to effectively deploy it because they either consider plaintext data valuable, or they’ve built a product which is technically unable to be offered in an end-to-end encrypted fashion.
However, it’s not clear that sufficiently developed E2E encryption is able to provide the same service better than an unencrypted alternative would be able to.
Disruptive innovation requires the creation of a new disruptive technology. So if Wikipedia is an example of disruptive innovation, what technology did they create? Like most people, I hear “new technology” and my mind naturally starts looking for machinery and gears combining to accomplish something that nobody thought was possible before. But that’s not here: Wikipedia is widely considered technically unremarkable. Instead, what stands out to me about Wikipedia and makes me think “that shouldn’t work” is anonymous contribution.
Most people’s knee-jerk reaction to allowing anonymous contribution (in any context) is that vandalism will prevent a project from being widely useful or trustworthy. At first this is true, it does. But people have natural desires compatible with Wikipedia’s goals: they want to share their knowledge, they want to fix mistakes they find, they want to revel in open-source ideals. In most other scenarios in life, these desires are stifled by processes that were put in place to prevent vandalism outright. For example, this level of openness is never found in OSS because vandalized code could result in malware being distributed to users.
Restricting the ability to contribute to those with “power,” maintainers, encourages territorial (exclusive / hurtful) behavior and reviews create a burdensome workload for the maintainers. It also discourages outside contributors that don’t want to wait for a review of their change, don’t want to deal with follow-up arguments with the maintainer, don’t want a sense of wasted effort if their change is rejected.
Anonymous contribution on the other hand has a very low transaction cost: if you want to make a change, there’s almost no reason not to. And for maintainers, there’s no social obligation to continue doing work they don’t want to do. Changes are accepted and published immediately, as are reversions.
Where people’s knee-jerk reaction fell short, is in understanding that vandalism can become better controlled over time as more people get involved and the right low-touch technical investments can be made.
Alternate Title: Cloudflare’s Plan for World Domination (Not Really (Maybe))
Cloudflare’s browser product will mark the beginning of a two-sided network. That is, we’ll have answers to the two following questions:
Why would more users of Cloudflare’s browser make a website want to join Cloudflare? Because this would give them access to very accurate information about the types of users on their website. For example, whether or not someone is a bot (security), or if the device being used is compatible with company policy (compliance). Highly-accurate and detailed analytics of almost any type could be collected.
Why would more websites on Cloudflare make a user want to use Cloudflare’s browser? Cloudflare claims that a remote browser is already slightly faster and more bandwidth-efficient than a normal browser, but if a website is also built on Cloudflare’s platform (using Railgun, Argo, cache, Workers), that would make it extraordinarily fast to render because everything could be computed locally, and would possibly make new online experiences possible.
The important point is that, while one-sided network effects make it difficult for newcomers to directly compete against incumbents (hence the importance of disruptive innovation), they don’t necessarily help incumbents compete against each other. Two-sided network effects on the other hand, favor winner-take-all outcomes, especially in something like the browser market where people have a strong preference to only use one browser.
(NOTE: This is speculation based on public information.)
What do people like about their jobs (besides money)?
Social interaction: A sense of community
Autonomy: An ability to decide how they work
Variety: A defense against boredom
Feedback: Knowing if they’ve done a good job
Sense of contribution: An understanding of how their effort helps the company achieve its mission
In my experience, 2 through 4 have almost always been provided.
The first item, social interaction, is touch-and-go for me. When I was an intern, I was explicitly excluded from team meetings and events. Sometimes the team I’m on has bonded surprisingly well. The months after COVID-related lockdowns started were about as isolating as unemployment.
The last item, a sense of contribution, is offered only when I demand it as a condition for contributing anything at all. These days, most of what I hear is about whether or not some chunk of work will make money. And guess what? Nobody cares about money that’s not theirs. Instead, repeatedly and compellingly remind people why their work--their individual effort--makes the lives of others better.
Wikipedia is a classic, underrated example of disruptive innovation. They started out by serving the low-end market of people that want information immediately, for free, and are fine with that information not necessarily being high quality. This was previously a market that print encyclopedias like Britannica served with door-to-door salespeople, but while trying to grow their revenue they naturally pursued higher-margin opportunities with universities and academic researchers.
These high-value customers needed accuracy above all else and Britannica implicitly started investing more in their product’s accuracy than its distribution channels, abandoning their mission to be the “household reference of choice” and letting their low-end customers languish. As Wikipedia developed, it accumulated a broad array of rapidly-updated and increasingly trustworthy information. This slowly made it a better choice for more and more use-cases, than any print or highly-curated online encyclopedia.
In the end, Britannica eventually did have to adopt a lot of Wikipedia’s practices to satisfy their chosen customers, but only after they permanently lost the vast majority of their addressable market. If they had continued to defend their low-end customer base they might be a colossal business today, but a need for short-term growth meant they happily surrendered any avenue for long-term growth.
Surprisingly often, well-managed companies fail to compete against startups that pursue their customer base. By not competing, these large companies lose market share and often go out of business, even though they have substantially more resources than their startup competition.
The concept in business theory that explains this pattern of David beating Goliath is called disruptive innovation and it comes from the creation and commercialization of a new, disruptive technology. Disruptive technology is characterized by the following:
It starts out by serving a niche or low-end market. Startups are able to gain initial traction here because their market is infertile enough that they have no competition from more powerful, established companies.
As initially formulated, the new technology doesn’t offer any value to a mainstream customer. But as it becomes more refined, it’s able to satisfy the needs of more and more of the established companies’ least desirable customers, better than the established company can.
It’s unintuitive, but this often creates a lot of growth and value in the established companies because shedding their least valuable customers decreases cost and improves margin, which allows the company to better pursue and develop their high-value customers.
