Thoughts Heap @romanandreg - Tumblr Blog

Using Nix and nixpkgs-mozilla for Rust projects

TL;DR: Go to the Implement a shell.nix with nixpkgs-mozilla section

Introduction -- why use nix if rustup exists?

Rust dependency management is top class, rustup facilitates downloading and managing different versions of Rust without much effort, and also specify what version of Rust you want in your project via the rust-toolchain file. As an author, I can ensure collaborators of a project I'm working on will download the exact version of Rust that I intended to use to develop my project.

That being said, since I've become more familiar with the nix package manager, I've been striving to use it for all my dependencies; this package manager is like rustup, but for every dependency in your system, not just the Rust toolchain. If you need a C library, it is likely nixpkgs has it.

The cool thing about nix as a package manager, is that you can use it the same way you would use apt or brew, but you can also specify dependencies on an specific sub-directory, like you would do with virtualenv or rbenv files. When you execute nix-shell on a directory that has a shell.nix file, boom! You are on an environment that uses only the specified dependencies, no problems with conflicting libraries installed on your system or anything like that.

Another great feature of nix, is that it allows us to pin down a package repository in a way that will make it's packages use the same version -- always. This is similar to having your project running on a docker image or virtual machine pinned to an specific version of ubuntu, and having the apt install commands always resolving to the same libraries and binaries every time.

How does it do it? Well, this is not a tutorial on nix per-se, just a pitch, and a rationale about why, you, as a Rust developer, would like to try this approach to install the Rust toolchain. There are some great resources out there on how to get started with nix.

What is nixpkgs-mozilla?

It seems some folks at Mozilla also see the value of nix, and use it internally. Given this, they maintain a repository of nix packages that install the latest version of their tools and software. The mechanics about how to integrate it on a project are not trivial if you are not familiar with nix, that's why I'm writing this blogpost.

Just to clarify, you can install Rust without Mozilla's repository, but there is a big chance that the version of the Rust compiler you download is not going to be the latest stable version; also I think the Mozilla packages allow you to cutomize the Rust toolchain more easily (target platform, stable vs nightly, etc.) and it understands the rust-toolchain file convention.

Implement a shell.nix with nixpkgs-mozilla

1) Install the niv tool to manage project deps

niv is a tool that sits on top of nix, it allows us to pin a package repository ("attribute set" in nix lingo) and custom packages ("derivations" in nix lingo) using a UX friendly CLI.

Note, niv might become obsolete in the not so distant future, the nix community is working hard on a feature called flakes, which tries to replicate what niv does in a native way.

Is also worth mentioning, niv is not the only way to pin dependencies, you may be able to use nix alone for this, however, I like niv just because it enforces a standard and a way to add packages (derivations) easily.

2) Initialize niv in your directory, and add nixpkgs-mozilla

To add the nixpkgs-mozilla package repository ("overlay" in nix lingo), we are going to use niv add.

$ niv init $ niv add mozilla/nixpkgs-mozilla

Make sure to also update the nixpkgs repository to nixpkgs-unstable, the nixpkgs-mozilla overlay relies (as of Jul 2020) on code that is not in the stable branch.

$ niv update nixpkgs -b nixpkgs-unstable

3) Add nixpkgs-mozilla overlay

We are going to modify the default nixpkgs "attribute set" with the Mozilla "overlay" overrides. Now, every package that gets installed that has a Rust dependency, will use the project's Rust version by default. This behavior happens because we are overriding the special rustPlatform derivation, which is used to build Rust programs in nix.

$ cat <<\EOF > nix/pkgs.nix let # import URLs for all dependencies managed by niv sources = import ./sources.nix; # import the package repository (overlay) specified on nixpkgs-mozilla mozilla-overlay = import sources.nixpkgs-mozilla; # define project's package repository project-overlay = # self and super are convoluted stuff, check out docs for nix overlays self: super: let # use the utilities from the nixpkgs-mozilla to build the "rustup toolchain" mozRustToolchain = self.rustChannelOf { rustToolchain = ../rust-toolchain; }; # We want to get the rust package with all these extensions mozilla-rust = mozRustToolchain.rust.override { extensions = [ "rust-src" "rust-std" "rustfmt-preview" "rls-preview" "clippy-preview" ]; }; # rust-src derivation is a tree of deriviations, we need to get the "src" attribute # from one of it's paths mozRustSrc = (builtins.elemAt mozRustToolchain.rust-src.paths 0); # We need to modify the structure of the rust source package that comes # from the nixpkgs-mozilla to work with an structure that works on upstream nixpkgs. rustSrc = super.runCommandLocal "${mozRustSrc.name}-compat.tar.gz" {} '' # get contents on directory in place tar -xf ${mozRustSrc.src} --strip-components 1 mkdir out # modify the directory structure to work with development/compilers/rust/rust-src.nix mv rust-src/lib/rustlib/src/rust/* out tar -czf rust-src.tar.gz out # vaya con dios mv rust-src.tar.gz $out ''; in { rustPlatform = super.makeRustPlatform { cargo = mozilla-rust; rustc = (mozilla-rust // { src = rustSrc; }); }; mozilla-rust = mozilla-rust; }; pinnedPkgs = import sources.nixpkgs { overlays = [ mozilla-overlay project-overlay ]; }; in pinnedPkgs EOF

4) Now build a shell.nix file that contains all the dependencies you need for development

$ cat <<\EOF > shell.nix # pkgs contains all the packages from nixpkgs + the mozilla overlay { pkgs ? import ./nix/pkgs.nix }: pkgs.mkShell { buildInputs = with pkgs; [ # the package/derivation we built on nix/pkgs.nix mozilla-rust # other rust utilities. Because we override the rustPlatform # package on nix/pkgs.nix, these dependencies will use the same # version as our mozilla-rust dep. rust-analyzer ]; } EOF

5) Add a rust-toolchain file

There are various valid formats for this file, to be frank, I didn't find a place with examples, so there has been a lot of experimenting. Let us use the desired rust version we want for our project

echo '1.45.0' > rust-toolchain

6) Execute nix-shell, and see all your dependencies get downloaded and installed

$ nix-shell $ which rustc /nix/store/vqqfzjjnk9zd3ps28pjxcxwzckbwlfvj-rust-1.45.0-2020-07-13-5c1f21c3b/bin/rustc

Note, this process may take longer than usual given you are compiling these tools with the Mozilla's Rust toolchain, and that is not going to be on the cache server of nixpkgs. If you want to build these packages once and share with the rest of your team, I recommend using cachix or roll out your own binary cache.

7) Bonus: use direnv to call nix-shell.

This will automagically integrate a dynamic dependency environment to your editor for freeeee (if your editor supports direnv, which is likely).

Conclusion

So there you have it, it took me a while to integrate nixpkgs-mozilla's Rust with the rest of the nixpkgs ecosystem. Hopefully this will empower you to upgrade and keep track of Rust upgrades (as well as your other deps) sanely.

#nix #nixpkgs #rust #rustPlatform #overlay #direnv

On passing parameters by value in Golang

TL;DR: Choosing to use pointer arguments for functions for the sake of better performance is premature optimization, and you should consider the drawbacks of pointers in regards to accidental complexity.

Introduction

When working in Golang, how many times have you considered the choice of receiving a pointer instead of a value as the argument of a function? We usually don't spend much time thinking about this given it is known that pointer arguments perform faster than values.

However, in other languages (cough, Haskell, cough), doing this, is one of the most pervasive anti-patterns. Why? The repercussions of mutating a variable are many, especially if you are using an awesome language with green threads like Golang. In this blog post, we'll explore some of the drawbacks of using pointers.

The scope of change

Let's start with explicitly defining what the pointer contract means in your business logic: whenever we pass a pointer as an argument to a function, we are giving this other function a rubber stamp to make changes on the behavior of their caller.

This approach, by extension, adds a mental overhead to the algorithms you run. As soon as you pass a pointer argument to a function, you cannot guarantee that this function is not going to modify the behavior of an algorithm in unexpected ways (unless the documentation is evident on what the function does, and is also accurate about it).

For sure, this situation is unlikely to happen when you use a general purpose library function, but is not impossible -- and I argue, it happens more often than not -- to have an unexpected behavior after calling a function from an API that changes for the sake of solving business needs. The priority is not in the UX of the code or the repercussions of its many callers, but the business value that a particular change offers when using a specific set of inputs.

However, now you argue, we have unit tests to solve this kind of issues right? Well yeah, up to a point, the combination of values and mid-states makes it hard to test every single combination that could exacerbate an issue from mutation.

I prefer not to give a function the power to change state like that, and instead, make this issue impossible to happen in the first place by passing parameters by value.

Pointers and concurrency

With the advent of multi-core CPUs, concurrency is becoming more prevalent as a way to squeeze as much performance as we can from our computers.

Golang is one of the few languages that make concurrency easy, via the usage of goroutines; however, it makes it also very easy to shoot yourself in the foot. Values are mutable by default, and if by any chance you use the same reference to multiple goroutines unexpected things tend to happen.

If you add that to the fact that you are passing values as pointers, things become accidentally complex very quickly.

How expensive is it to receive parameters by value?

The points above may sound reasonable, but you still think to yourself, won't somebody, please think of the performance?, that's fair, let's figure out how expensive is it to pass parameters by value to a function.

Let us build a benchmark that allows us to assess what are the performance implications of passing small and big sized values as parameters:

package vals_test import ( "testing" ) type F2 struct { f1 int64 f2 int64 } type F4 struct { f1 int64 f2 int64 f3 int64 f4 int64 } type F8 struct { f1 int64 f2 int64 f3 int64 f4 int64 f5 int64 f6 int64 f7 int64 f8 int64 } type F16 struct { f1 int64 f2 int64 f3 int64 f4 int64 f5 int64 f6 int64 f7 int64 f8 int64 f9 int64 f10 int64 f11 int64 f12 int64 f13 int64 f14 int64 f15 int64 f16 int64 } type D2 struct { d1 F2 d2 F2 d3 F2 d4 F2 } type D4 struct { d1 F4 d2 F4 d3 F4 d4 F4 } type D8 struct { d1 F8 d2 F8 d3 F8 d4 F8 } type D16 struct { d1 F16 d2 F16 d3 F16 d4 F16 } func updateValD2(val D2) D2 { val.d1.f2 += int64(10) return val } func updateRefD2(val *D2) { val.d1.f2 += int64(10) } func updateValD4(val D4) D4 { val.d1.f3 += int64(10) return val } func updateRefD4(val *D4) { val.d1.f3 += int64(10) } func updateValD8(val D8) D8 { val.d1.f3 += int64(10) return val } func updateRefD8(val *D8) { val.d1.f3 += int64(10) } func updateValD16(val D16) D16 { val.d1.f3 += int64(10) return val } func updateRefD16(val *D16) { val.d1.f3 += int64(10) } func BenchmarkPassRefD2(b *testing.B) { var d D2 for i := 0; i < b.N; i++ { updateRefD2(&d) } } func BenchmarkPassValD2(b *testing.B) { var d D2 for i := 0; i < b.N; i++ { d = updateValD2(d) } } func BenchmarkPassRefD4(b *testing.B) { var d D4 for i := 0; i < b.N; i++ { updateRefD4(&d) } } func BenchmarkPassValD4(b *testing.B) { var d D4 for i := 0; i < b.N; i++ { d = updateValD4(d) } } func BenchmarkPassRefD8(b *testing.B) { var d D8 for i := 0; i < b.N; i++ { updateRefD8(&d) } } func BenchmarkPassValD8(b *testing.B) { var d D8 for i := 0; i < b.N; i++ { d = updateValD8(d) } } func BenchmarkPassRefD16(b *testing.B) { var d D16 for i := 0; i < b.N; i++ { updateRefD16(&d) } } func BenchmarkPassValD16(b *testing.B) { var d D16 for i := 0; i < b.N; i++ {s d = updateValD16(d) } }

We have two versions of the updateVal function, one receiving a reference, and another receiving a value. We benchmark different sizes to assess how different is the slowness when using small and big values. Executing the above code gives us the following result:

$ go test -bench=. goos: linux goarch: amd64 BenchmarkPassRefD2-8 2000000000 1.37 ns/op BenchmarkPassValD2-8 200000000 6.71 ns/op BenchmarkPassRefD4-8 2000000000 1.37 ns/op BenchmarkPassValD4-8 200000000 9.77 ns/op BenchmarkPassRefD8-8 2000000000 1.35 ns/op BenchmarkPassValD8-8 100000000 14.7 ns/op BenchmarkPassRefD16-8 2000000000 1.37 ns/op BenchmarkPassValD16-8 50000000 24.9 ns/op PASS ok _/home/roman/tmp/playground/vals 19.197s

What do you know? Passing parameters is slow!

The first thing we can notice is that even with small values, passing parameters by value is around 6x slower than passing them by reference. The bigger the value, the slower it becomes, having it 24x slower when having a struct of 16 attributes (not your average use case). Now, is this slowness relevant in the bigger scheme of things? I argue that it may not be.

There is this table about latency numbers every programmer should now that may give us some clarity; for any access to RAM we do in our code, we get a 100ns tax, meaning, if we do have any IO related code around our functions, the latency of those calls is going to eclipse any performance degradation you may get from receiving values as parameters.

Conclusion

Does this mean you should not worry about parameters by value ever? Well, no. If you are executing a pure function that receives big values as parameters in a tight loop, it may make sense to pass parameters by reference to speed up the performance of your algorithm.

Our key takeaway is, before passing pointers as arguments for the sake of performance, make sure that this piece of code is the real bottleneck of your program, this way you reduce accidental complexity from your application.

Golang's Sprintf luxury tax

I wanted to get a sense of how expensive the different mechanisms to build strings are; I’m used to use printf like functions in the languages I work with, and I wanted to check how performant (or not) was in Go. I came up with this benchmark code:

https://gist.github.com/roman/3c116382d5a245b0752ff8336cc39572

Which gave me the following results when running the code

$ go test -bench=. -benchmem 2>&1 goos: linux goarch: amd64 BenchmarkSprintfGreeter-8 20000000 100 ns/op 32 B/op 2 allocs/op BenchmarkBuilderGreeter-8 30000000 48.9 ns/op 24 B/op 2 allocs/op BenchmarkConcatGreeter-8 100000000 16.9 ns/op 0 B/op 0 allocs/op BenchmarkBuilderReplicate2-8 20000000 80.8 ns/op 56 B/op 3 allocs/op BenchmarkConcatReplicate2-8 20000000 114 ns/op 64 B/op 3 allocs/op BenchmarkBuilderReplicate4-8 10000000 128 ns/op 120 B/op 4 allocs/op BenchmarkConcatReplicate4-8 5000000 268 ns/op 176 B/op 7 allocs/op BenchmarkBuilderReplicate8-8 10000000 191 ns/op 248 B/op 5 allocs/op BenchmarkConcatReplicate8-8 3000000 607 ns/op 528 B/op 15 allocs/op BenchmarkBuilderReplicate16-8 5000000 287 ns/op 504 B/op 6 allocs/op BenchmarkConcatReplicate16-8 1000000 1595 ns/op 1712 B/op 31 allocs/op BenchmarkBuilderReplicate32-8 3000000 553 ns/op 1016 B/op 7 allocs/op BenchmarkConcatReplicate32-8 300000 3770 ns/op 6064 B/op 63 allocs/op PASS ok _/home/roman/Projects/playground 23.791s

From what we can tell from the Greeter benchmarks, in the simplest use case, a Sprintf is twice slower than using strings.Builder, with just concatantion using + being almost 10x faster. This is in the case of a single string build (no iterations).

In the scenario where we are accumulating a string in a loop, we can see the benefits of using strings.Builder vs. + at four iterations, where strings.Builder is around 38% faster.

So, the lessons learned are:

Avoid Sprintf if you can make do with string concatenation (or if performance is not that important compared to convenience);

Always use strings.Builder when dealing with string accumulation in loops.

#golang #performance #strings

I keep this with me all the time now, because he (Nicolas Winding) says he asked himself, whenever it says in the script the guy comes through the door, I wonder, why doesn't he come through the window, or up through the floor, break through a wall, and, he kind of goes through all the difference choices of what _it's not_, to get back to why it has to be this one; and that is something that I always kept with me, that idea "what else could it possibly be?".

Oscar Isaac GQ interview

We have tried to demonstrate by these examples that it is almost always incorrect to begin the decomposition of a system into modules on the basis of a flowchart. We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others. Since, in most cases, design decisions transcend time of execution, modules will not correspond to steps in the processing…

David L Parnas - On the criteria to be used in decomposing systems into modules (1971)

#software #decomposition #system design

The incremental nature of complexity makes it hard to control. It's easy to convince yourself that a little bit of complexity introduced to your current change is no big deal. However, if every developer takes this approach for every change, complexity accumulates rapidly. Once complexity has accumulated, it is hard to eliminate, since fixing a single dependency or obscurity will not, by itself, make a big difference. In order to slow the growth of complexity, you must adopt a "zero tolerance" philosophy. John Ousterhout - A Philosophy of Software Design

#software #complexity

Fast Haskell coding with cushions

Previously this week, the awesome "Matt Of Lambda" published a blog post about the benefits of using ghcid to develop Haskell applications; if you haven't checked it out, it is an excellent read on how to get started with something like REPL driven development

Personally, I've always have used GHCi for development, from trivial tasks like checking my types and experiment with code to running my test suites and the main executable of my projects. This approach was usually way faster than the regular change/compile/run cycle that Haskellers developers typically do, but it came with some challenges:

Lack of resource de-allocation

Given that most applications are designed to run only once, there are no real incentives to develop our software in a way that can be cleaned up and started again without killing a process.

Code gets unusually slow after a few runs

Because of the previous point, every time you reload code in the REPL your application leaves resources hanging in memory (e.g., threads), and the REPL starts to get slower (and confusing) if dangling threads are spitting strings to stdout.

Port already allocated errors

Sometimes, you will get errors like a TCP Socket port is already bound, this happens when you reload your code, and your application leaves socket resources open after a reload.

Solving some of these problems

Given my tenure in Clojure land for a big chunk of the past four years, I had the opportunity to see how "REPL driven development" should be. Their tooling is oriented to have a REPL open at all times, and they run everything there, from unit-tests to executables. Following their example, I started to implement a few libraries that help me find this REPL nirvana.

teardown

Inspired by Reactive Extensions Disposable API, this library is a smart wrapper on top of IO sub-routines whose whole purpose is to perform a cleanup. This library will:

When a resource cleanup fails with an error, this API makes sure that this error does not affect the cleanup of other unrelated resources

It keeps track of the resource name and how much time it took to de-allocate; if the de-allocation fails, it also shows the error that made it fail

It ensures that de-allocation sub-routines are executed exactly once, avoiding errors of the "already de-allocated" nature.

You can take a look at the documentation site of this project, bear in mind though, this API is not a good fit for most applications.

componentm

Inspired by Clojure's component, this library provides approaches for you to build all the resources of your applications transparently and safely. Some of the features this API offers:

It keeps track of initialization time for each declared component of your application

Builds a dependency graph for all your allocated components and then it guarantees they will be cleaned up in the right order (e.g., if component B depends on component A, the library makes sure that B is cleaned up before A)

If any of the resources fail on initialization (e.g., a Database is not running), this API will rollback all the previously allocated components in the right order and then throw an exception with detailed information

Makes use of teardown for resource cleanup, getting all the benefits of that API. This dependency is an implementation detail, so users of componentm don't need to understand teardown.

It will execute the initialization of your components concurrently if components are built using an Applicative interface.

To create ComponentM value, you can use buildComponent:

buildDatabasePool :: LogFunc -> IORef Etc.Config -> ComponentM (Pool SqlBackend) buildDatabasePool logFunc configRef = do config <- readIORef configRef connString <- getConnString config poolSize <- getPoolSize config pool <- buildComponent "databasePool" (runRIO logFunc (createPostgresqlPool connString poolSize)) (destroyAllResources) runMigrations logFunc pool return pool

Although not all functions are defined in the previous example, we can see that buildComponent, which creates a Pool of PostgreSQL Database Connection and also, a call to destroyAllResources which de-allocates that pool resource.

Following is how the output of your program would look when it cannot connect to the database:

2018-05-22 07:44:02.317225: [error] # Application Failed Application failed on initialization, following are the exceptions that made it failed: * ComponentAllocationFailed "db migrations" - the following error was reported: | `- libpq: failed (could not connect to server: Connection refused Is the server running on host "localhost" (127.0.0.1) and accepting TCP/IP connections on port 5432? ) Following, we have the information of application resources cleanup: ✓ servant-playground (0.000036s) ✓ http-server (0.000033s) ✓ db migrations (empty) ✓ databasePool (0.000003s) ✓ logReloader (empty) ✓ config (empty)

You'll notice there is an indication of the errors that our initialization script throws, as well as the cleanup of all resources that we have allocated so far. You have total visibility on the lifecycle of your program.

componentm-devel

To tie the knot, I created a library that provides an easy to use API that automatically reloads your application everytime you call it with the ComponentM builder of your application. It provides a single function runComponentDevel that receives the same parameters as runComponentM1

Conclusion

I like to say ComponentM implements a Monad to avoid the usual pyramid of withResourceX functions at the beginning of your application. It composes them all together once, so you don't need to worry about it. If you like these APIs, make sure to check out a sibling library called etc, it provides an API to manage the configuration values of your app as a map with multiple sources in a declarative way.

You can check out a full-fledged example where we setup an app using servant and persistent-postgresql using the libraries mentioned in this blog post.

Shameless Plug

If you would like to learn more about how to build robust Haskell applications, be sure to join my workshop at LambdaConf 2018

When you repeat a mistake it is not a mistake anymore but a decision

Annonymous

Programming is suddenly not about reading documentation anymore, nor about navigating executable code, but about learning how to quickly recognize the shapes of the pieces of a puzzle, knowing that as long as two pieces fit together, it will be alright. Haskell is beautiful, it makes programming fun.

Renzo Carbonara

Lightning fast CI for Haskell projects

I've been working in a few projects at a time in Haskell for the past year, and one point that has been dragging is how much time it takes for a CI job to finish, given this, I started to experiment with other solutions to improve my build feedback loop.

As of a few weeks, my main goto for CI was TravisCI. There is a lot of Work done by others around how to build and test Haskell projects for MacOS, Linux, multiple stackage snapshots, etc. I was however never satisfied with how TravisCI wasted so much time doing the same things over and over again. Some of the builds would take a 20 minute time just to finish, stopping my feedback loop towards quickly closing PRs I was working at the time.

Looking for alternatives

After going through some alternatives, I landed at CircleCI via a tweet from them:

After I started experimenting with CircleCI, there were many things that I liked:

It allows me to specify my docker images as builders - This is huge, I know most of the time spent in TravisCI is compiling and installing tools and a minimal amount of time was testing my code. By giving me the opportunity to build images with those tools baked in, I could start a PR build from an environment that didn't spend time compiling the stuff I used to test my libraries.

It allows you to run several jobs in parallel, and not just that, but also enables to build pipelines. This feature is excellent, I can keep my compilation matrix going, and not just that, I can add other jobs to release my software. I haven't built an open-source app just yet, but I will keep this in mind for that.

I got builds to become 95% faster when the build uses a previous cached output (yes, that's ~2 minutes instead of ~40 minutes). With this flow, I can do quick tweaks to my PR builds and merge them quickly.

Execution of TravisCI

Execution of CircleCI

What I use to build my software

I rely on two docker images:

romanandreg/haskell-tools:latest - This image contains hlint, stylish-haskell, brittany and refactor binaries to run all those tools as part of my CI pipeline.

romanandreg/stack:latest - It contains a stack binary and a cabal-1.24 binary (to run the resolver), and I used this image to build my project in various snapshots.

My CircleCI config file makes use of this docker images that have all the tools I need to build my project.

How I build my project

I heavily rely on Makefiles to build my projects. Why? I do this because I want to be able to run my CI scripts on my local machine. The CI scripts are no different from what I would use on local development.

But seriously, why Makefile and no other tool? Makefiles are the best-supported tool on Unix systems, you have it available everywhere, and it does its job very well; granted the syntax is awful, but I have learned to live with that.

Organization of my Makefiles

After much working with Makefiles, I've learned that is a bad idea to keep all your tasks in a single file, this way I cannot re-use well-defined tasks in many projects; re-usability suffers when I need to modify the files per project details. If some common tasks are self-contained in a file, I have to copy/paste that into a new project, and voila, things work as I need them to work. The files I'm currently using:

make/tools.make - Contains all tasks related to linting and formatting the code of a library

make/sdist.make - Contains tasks to build a release of the library, and test it with the stack nightly version

make/solver.make - Contains tasks to execute the resolver so that I can modify my stack.yaml file to support other snapshots dynamically.

I'm planning to add other Makefiles as I go, to script out the cabal-install setup for testing my projects with cabal-install. This functionality is something I need to port from my previous TravisCI build.

Where can you find an example

I believe the best two projects that are implementing this approach are teardown and etc, just go through their .circleci/config.yml file and their Makefiles.

Should you try CircleCI?

I highly recommend spending the time to try CircleCI, if you are typically waiting for a PR build to finish to push a new release up, this tool will provide a fantastic feedback loop for your workflow.

#haskell #ci #circleci

Paranoia is good engineering

Mike Nygard

With every abstraction, the time comes when you must peel the onion, shed some tears, and see what's really going on

Mike Nygard

#releaseit

Capataz-0.1 released 🎉

I'm happy to announce a new version of the OTP style supervision tree library.

Some of the new features:

Creation of multi-level supervision trees

Now the Capataz system can be composed of multiple supervisors; there is always going to be a root supervisor created everytime we call the forkCapataz function, but now we can attach to it other supervisors that monitor sub-systems of an application. This feature is pretty outstanding given it allows us composing robust sub-systems together.

Changes in the API to build only valid configurations

Now, the signatures of some functions changed only to allow the creation of Worker processes that make sense; we no longer can create workers that may do nothing accidentally or that have generic names.

More examples

We added a new more useful example that showcases how to create a static supervision tree.

Feel free to check the project on Github, Gitbook and Hackage.

My Last Day at Unbounce

Today was a bit of a bittersweet day; I officially stopped working at Unbounce.

It wasn't an easy decision to make, but I believe it is the best one I can go with if I want to move forward in my personal BHAG, remodeling Vancouver, BC as a Functional Programming hub where strong typed functional programming jobs are not an uncommon ordeal.

My time at Unbounce taught me so much:

Efficient usage of tooling for monitoring, logging and track business data analytics for production systems

How to communicate backend services using AWS tooling

How to do Product Discovery, Product Planning and Customer Success in a data-informed fashion

How to interview people that can couple with values one cares about

Sensitive ways to deal with unplanned and urgent work

And many other things...

It also got me the privilege to work with amazing people like Raymond Huang, Ho Yan Leung, Macarena Poo, Meagan Sobol, Teresa Duke, James Brennan, Kieran Cormack, Gabe Scholz, Ainara Sainz, Steve Mynett, Cheryl Lee, Arthur Gonigberg, David Dossot, and many other names (I can't possibly fit on a single paragraph). You taught me standards of excellence and professionalism by example.

Unbounce has been the company I've worked for the longest time, and, in my opinion, that is the best compliment I can give. Thank you Unbounce, I will remember times there as some of the bests of my career so far.

Now, I am preparing for my next adventure at Calabrio, where I'm going to be working full time in Haskell and hopefully make a positive impact on their business and my BHAG.

#life

Introducing Haskell to your Company

I've heard time and time again about how Haskell may be a risky technology choice for business. Some companies might be in a situation where this statement may be true, however, in the end, is always a matter of "it depends."

I would like to explore some of the points that companies may go through before picking Haskell (or any other language for that matter) and distill different strategies that may mitigate the risks of using a new language/stack.

Haskell developers are hard to find

There is no denying that finding Haskell developers can be tricky; however is not impossible, here are some strategies that your company recruiters can try out:

Hire remote developers

There is a big pool of Haskellers around the globe that are looking for an excellent shot to work on Haskell, but haven't found one in their city.

Get acquainted with your local Haskell Meetup

The same people that are looking to work remotely on a Haskell gig are likely to be participating in the local Haskell Meetup, and even though they are not looking for a new job, they might feel tainted to work at your company if you bring Haskell into the equation.

Advertise Job Positions in well-known functional programming channels

For Haskell, some of the channels I would recommend are:

Functional Jobs This page is a great spot to advertise jobs in any FP language (Clojure, Scala, Haskell, OCaml, etc.)

Haskell Weekly An excellent source of news and articles for Haskellers, it provides a space for advertising Haskell jobs (The postings don't last very long which may be a good sign).

Sponsoring an FP Conference Some functional programming conferences provide a great space for advertising your desire to hire Haskellers, some of the most popular ones are:

Curry on

Compose Conference

LambdaConf

Host Public Haskell Study Groups

Now is more approachable than ever to start a study group to learn Haskell, there are proven guidelines on how to run one and also well-established study materials. Making people learn Haskell together is also an excellent opportunity to grasp participants skills to teach others as they learn the language.

As it was the norm with Python a few years ago, it is likely that the Haskeller you hire will be a very competent developer. Whoever learns Haskell is plausible she would be a very passionate and motivated individual and understands the repercussions of side-effects pretty well.

You might be saying that these strategies are not so different from the ones you would use when trying to hire developers in other languages (and yes, that's the point I'm seeking to convey).

Is important to bear in mind that if you are looking for a skilled senior developer in any technology, the search will still be challenging. For example, some companies where I've worked before spent months (some cases a year) to fill a Ruby or Java position.

Haskell concepts are hard to understand

Many friends that have attempted to learn Haskell complained about how hard is to understand the core concepts, and I completely understand their pain. Haskell is a challenging language given how different it is to other mainstream languages; nevertheless, I argue that once you get into an advanced beginner level, the leverage you get in comparison to other languages is quite substantial. How so? Haskell in nature is a very constrained language, not giving you "enough rope to hang yourself"; many of the common pitfalls other languages have by providing "easy" (not simple) APIs will just not happen in Haskell; bye-bye null pointer exceptions.

How can we mitigate the onboarding pain? I think first and foremost we need to acknowledge the Haskell pyramid effect. Paraphrasing what Steven Syrek says in one of his posts:

Haskell is capable of representing extremely abstract mathematical entities, but so is math itself, and you hardly need to understand linear algebra to balance your checkbook.

On the same lines, you don't need to know Category Theory to do a Web APIs with Haskell.

Secondly, we need to acknowledge that most of the interfaces in Haskell are very abstract, and humans (sadly) suck at fully grasping those on the first attempt. You'll need to allow mentees to digest an abstract idea a few times (no, your "Monads are like burritos" blog post won't help), and see many different concrete examples of an abstraction before they can understand it fully. Luckily there are resources like The Scala Red Book and The Haskell Book that you can use as resources to do study groups and go one step at a time into the fascinating world of FP.

My company cannot afford to invest in new tech

The first thing to keep in mind is: If your company is working on a Monolith architecture, it makes no sense to invest in any other technology, is simply not viable for any business to "reimplement all the things" in a new language.

If on the other hand, a Microservice architecture is in place, there may be wiggle room to implement some parts of your system using a new stack as long as you don't have the "too many technologies" problem already[*].

[*] I like to imagine adding every new language to a company as tearing the company's soul apart, like Horcruxes

The trickiness comes when developing applications on an entirely new tech, is not just the challenge of a new language, but all the operational costs that go with it:

How can we debug Haskell code when things go awry?

How can we track garbage collection, memory allocations, performance, etc.?

What options do we have to deploy Haskell applications?

Is the library ecosystem mature enough to cover my business requirements?

All of the above are valid concerns that you need to address and sadly you cannot give reliable answers unless you have done production Haskell somewhere else and had the scars to prove it. If you are not experienced in production and got enough people interested in working with Haskell, try to tackle non-risky projects and assess how you would do the operations around it, make failures be less expensive for the business and learn as much as you can in the process.

Also, it might be a good initiative to provide example projects that showcase how you would tackle Real World™ challenges (HTTP APIs, Usage of AWS/GCE, etc.). These codebases will give guidance to less experienced developers on how to get started and let them feel empowered and productive.

Does this sound like a lot of work? You bet. When advocating a new technology inside your company, you unofficially start having two jobs.

Haskell is more of an academic hobby language right?

I've heard this one before despite the fact that Haskell has been around for the past 20 years, and big companies trust it for fundamental concerns of their business. There is also many stories of companies with some levels of success in Haskell.

Haskell is a language that started as an investigation project trying to answer the question: can laziness be used in production settings for profit? (spoiler alert, not so much). However, in the past ten years, it has had lots of production quality libraries developed on top of the Glasgow Haskell Compiler (GHC), conduit, ekg, wai/warp, async, stm to name a few.

I can argue Haskell is great if you are trying to do performant HTTP servers, reliable CLI applications, and systems which need to excel in parallel/concurrent environments.

People are afraid of change

I've had experiences where some people are very afraid of trying Haskell given how different it is, sometimes they go as far as to reflect there is something wrong with me for not going with standard mainstream languages like C/C++/Java.

Dealing with developers in this mindset is a bit tricky, most of the times this would be the well cherished cynical developer that doesn't believe in silver bullet solutions (and that is great). Starting with an honest conversation about the weaknesses of Haskell and then go on its virtues is a good start to get them going. Move the conversation to more of facts vs. passion argumentation, and you and this person will learn way more about the language.

Conclusion

Here are some non-extensive bullet points on how to get Haskell started at your company, do you think I missed something obvious? Please feel free to share comments and impressions; I'm always in the quest of spreading Haskell knowledge into people so that the language can gain more traction in production settings and your feedback may help in this quest, so highly appreciated.

#haskell

Perfect is the enemy of good

Voltaire

PolyglotConf 2014: Notes on Introduction to Distributed Systems

The session on Introduction to Distributed Systems was definitely one of my favorites this year. I can argue it was on the same lines of "how to become a better programmer" on Polyglot 2013's conference, because I could compile a lot of call-to-actions.

The session was led by Fred Hebert (author of Learn You Some Erlang) and Jeremy Pierre (Askuity's Distributed System guru) explaining the core problems you may find in distributed systems in layman words. The first concept to get tackled was the CAP Theorem, in which Fred used a metaphor of two groups of people separated on an stranded island trying to communicate and coordinate. The second one was mentioned by Jeremy, commonly known as the 2 General Problem or The Byzantine Generals Problem which was really enlightening.

After those two first intros, a lot of recommendations came by from the session leaders and Saem Ghani. Here is a summary of the different resources a person should read in order to get a better understanding on Distributed Systems:

1) End-To-End Arguments in System Design[pdf] by J.H. Saltzer and others.

2) Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services[pdf] by Seth Gilbert and Nancy Lynch.

3) Virtual Time and Global States of Distributed Systems[pdf] by Friedemann Mattern

4) Two Phase Commit Protocol, I also read this blog post that tackles this concept with some illustrations.

5) In Search of an Understandable Consensus Algorithm[pdf] a.k.a the Raft algorithm by Diego Ongaro and John Ousterhout

6) Idempotence is Not a Medical Condition by Pat Helland.

Bonus blog-posts and papers:

8) The 8 Fallacies of Distributed Computing[pdf] this is an explained version done by Arnon Rotem-Gal-Oz

9) Dynamo: Amazon’s Highly Available Key-value Store[pdf] by a bunch of people. This paper was emphasized as a suggested reading by Jeremy and Saem.

10) Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications[pdf] by a bunch of people. This paper was emphasized as a suggested reading by Saem.

11) Notes on Distributed Systems for Young Bloods[pdf] by Jeff Hodges. This blog post was praised by Jeremy, stating that he tries to read it at least 2 times a year.

The session was an amazing experience, and I would recommend for people missing the PolyglotConf this year to try for the next one.

#polyglotconf #distributed systems #links #review

Trending Blogs

Recently Viewed Blogs

Thoughts Heap