Pretty cool, we have to say.
We were putting together something similar and then this came out on Twitter first! So, we honored the first rule of the Interwebs - hyperlinking...
art blog(derogatory)
YOU ARE THE REASON
"I'm Dorothy Gale from Kansas"
taylor price
we're not kids anymore.
Sade Olutola
Keni

Product Placement

shark vs the universe
hello vonnie
almost home
Misplaced Lens Cap

JVL
Claire Keane
🪼
tumblr dot com
2025 on Tumblr: Trends That Defined the Year
$LAYYYTER
Not today Justin
No title available

seen from United States

seen from United States
seen from United States

seen from Japan
seen from United States
seen from United States

seen from United States

seen from United States

seen from United States
seen from United States
seen from Lithuania
seen from Türkiye

seen from United Kingdom
seen from United States

seen from Türkiye

seen from United States

seen from Malaysia
seen from China
seen from United States

seen from United States
@kivralabs
Pretty cool, we have to say.
We were putting together something similar and then this came out on Twitter first! So, we honored the first rule of the Interwebs - hyperlinking...
Implementing Single Sign-On (SSO) at Kivra
Hello, Internet! It's Kivra Labs.
We work with two types of customers at Kivra. The first is our partners who send in their content to communicate with their customers (end users) (like, Skatteverket) and the other are the end users themselves. Our success has been built on being able to understand both types, be able to solve problems for both, and know how to get them to like talking to each other. :)
The Use Case
A prevalent example is our partners wanting to redirect end-users from a content in their Kivra inbox, straight into the partners' system. An example is an insurance company wants to communicate something important to the end user and prompt them to check their current policy on their system.
The prompt is achieved by our Active Content in the Kivra product. The Active Content is our take on an embedded iframe that is included with the partner's content. So, when the end user logs into Kivra to read the communication from the partner, they will see the Active Content above the content item.
So, to summarize, our partner wants the user to log into Kivra, read the content item, click on the link in the Active Content, and then be able to look at their current policy on the insurance company's website.
In order for this to work, the authenticated user needs to be able to transferred to the sender's system, authenticated, and then logged into the system, without the user having to re-login or be asked to re-authenticate.
The authenticated user on Kivra clicks the button in the Active Content.
The user is sent automatically to the sender's system.
Their credentials are verified.
They are logged into the sender's system.
The process is seamless for the user. The real fun is under the hood, tinkering in the engine of the technical solution.
The Solution
We used SAML2 to implement the use case.
Final Thoughts
Apart from the usual teething problems and tweaks, this solution proved really successful for us. Our partners have seen its value and we're now getting more requests to implement this on a wider scale.
Have you implemented SSO into an external system? How did you do it? Let us know on Twitter!
the inner pain of developers, in words.
Hello, Internet! It’s Kivra Labs.
This week, we bring you the macabre and the funny rolled into one. It will hurt because it’s so funny. Or, is it the other way around?!
Anyway, a great read and find this week on HN.
Our Battle-Ready Technical Architecture
Hello, Internet! It’s Kivra Labs.
Oh, thanks for all your love on Twitter! It means a lot to us.
This week, we thought we'd share our experiences with our technical architecture, after we teamed up with Skatteverket (Swedish Tax Authority, SkV for short) on their Pengar till Påsk campaign ("Your tax return money back by Easter").
Skatteverket launched a large and successful campaign to encourage Swedish citizens to get a digital mailbox. The carrot was that if the users sign up to a digital mailbox, set up Skatteverket as a sender, and send back their pre-filled declaration to them by a certain date in March, those users could get their tax return money before Easter!
We saw some astronomical traffic across all our channels. Some stats for you:
The Challenge
In the lead up to the campaign, our infrastructure team was aware of a peak load value for our systems. We use a variety of load balancers across our architecture and with our hosting provider, so we knew that we weren't going to survive the load!
What We Did
We had a baseline of total concurrent sessions that we could withstand, so we increased this significantly. This involved, of course, tweaking our configuration so that our product wouldn't buckle from the pressure.
We also had executed a suite of load tests. We were expecting more than 100,000 hits from SkV at peak load.
Finally, we also needed to make sure that our integration platform with SkV could handle the increased load during the campaign.
Done.
That's all for this week. You can read a summary of the entire SkV campaign, in Swedish.
Our Backend Project and Build Architecture
Hello, Internet! It’s Kivra Labs.
Following on from our post last week about frontend, the backend guys started grumbling about systemic and structural injustice. So, it's their turn to shine this week!
A 7-person strong team, they build and maintain the very core of our product and systems. Most days, they mostly discuss Erlang, flame the eternally-burning embers of emacs vs. vim, and where to go for lunch.
There's really a diversity of technologies within the backend ecosystem, so strap in!
Programming languages
We're an Erlang shop. 'Nuff said.
Java is used to integrate with the Swedish Tax Authority's (Skatteverket) API and there is Haskell that powers the API for our business intelligence system. You'll find Python in various integration scripts and powering our in-house sentient Slack bot (SSB). Otherwise, they use a plethora of Bash scripts to handle integration and operations tasks.
Our interface for monitoring infrastructure and handling service alerts is written in Clojure. (This is tied into PagerDuty and Sentry.)
IDE/Developer tools
While the team is caught in the honorable war between emacs and vim, a single lone ranger uses Sublime.
Package management
The guys use rebar3 for all the Erlang projects and Maven for the Java projects.
Databases
We use Riak KV for most of our services.
You will find however PostGreSQL and SQLite, too, though!
Testing
The backend team uses EUnit, a whole lot of Common tests, and JUnit to keep Java happy. (Always keep Java happy!)
Source Control, Hooks, Deployment et al
GitHub is our source control system. Can there be any other?!
We have post-commit hooks wired into Slack so that our channels receive notifications of commits, new pull requests, and when comments are posted on GitHub.
A custom bash script lies in our repos that triggers an automated build on BuildKite upon a commit to an active branch and of course, the hallowed master branch.
We are heavy users of Docker images and containers. When changes are to deployed live, we run an Ansible playbook that runs a git pull on latest master and shoots it up to a Docker image on our production environment.
Finally, our projects are happily married to Sentry and PagerDuty to help us track, in real-time, errors and raise alarms for the person assigned to operations duty. (More about that in a future post!)
Our Frontend Project and Build Architecture
Hello, Internet! It’s Kivra Labs.
We're going to start to dig into some technology this week.
The frontend development team at Kivra is 8-person strong and has a really great mix of skills. Our work is driven by great, informal discussions (on Slack and IRL), commitment to understanding the needs that come from the product team, and then translating those needs into great features that we build. We believe in first make it work, then make it pretty, then make it fast.
We treat all our applications as a microcosm of microservices, designed to communicate with each other and remain as autonomous as possible. This makes for sleeker and easier management and maintenance.
Here's a quick run-through of our project and build architecture!
Most of our recent projects are built on Angular 1.4 or 1.5, some of our older ones were built on 1.2. We're porting slowly our architecture to a component-based one and trying to slurp up every last drop of the ES6 juice.
Like most JavaScript projects online today, modular dependencies are defined and managed with npm's package.json file.
We've used Grunt in older projects, newer ones Gulp, and most recently, we're going all-out 2017 by leveraging webpack.
Jasmine is our unit testing tool of choice.
GitHub is our source control system. Can there be any other?!
We have post-commit hooks wired into Slack so that our channels receive notifications of commits, new pull requests, or when comments are posted on GitHub.
A custom bash script lies in our repos that triggers an automated build on BuildKite upon a commit to an active branch and of course, the hallowed master branch.
We are heavy users of Docker images and containers. When changes are to deployed live, we run an Ansible playbook that runs a git pull on latest master and shoots it up to a Docker image on our production environment.
Finally, our projects are happily married to Sentry and PagerDuty to help us track, in real-time, errors and raise alarms for the person assigned to operations duty. (More about that in a future post!)
How do your frontend architectures look? Let us know on Twitter or here!
See ya next week, Internet.
Our Wacky Wednesdays... Are You Wacky Too?
Hello, Internet! It's Kivra Labs.
Next week Wednesday is our monthly Wacky Wednesday, where we developers (front- and back-end) get the opportunity to work on side projects, burning ideas, or just something interesting for a day. It's a time-honored tradition here at Kivra and we thought that we could share with you some highlights from our recent Wacky's.
1. Egil from backend wrote a plugin for Slackbot to set up multiplayer poker games in the office! Written in Python, it's nifty and the cause of much entertainment/distraction in the office.
2. Our stealth frontend developer Tor took a deep-dive into machine learning and prophecy to see whether we could predict if users would use our service to pay their services or not. If it sounds super-brainy, it is because it is and includes the following words - binary decision tree, machine learning, and logistic regression.
3. and, finally our CTO Bip entered the world of Web-Meta and explored common GNU makefiles for building and testing web applications. James Franco has nothing on Kivra when it comes to meta, you see.
We kept it nice and short this week. But let us know on Twitter or here - do you do something like Wacky Wednesdays? We'd love to hear from you!
... and we’re back!
It's been a little while that we haven't posted on here. You know, life, work, and...
- CRAZY amounts of new users on our app
- #1 spot for our iOS app on Apple Store Sweden
- stuffing our faces silly with diet-destroying AMAZING cakes by @kivracookies
So much exciting stuff is happening here at Kivra. We’re going to start out nice ‘n’ light this week.
All tech departments at startups engage in *productive* banter all day long. Right?
Right?
Anyway, we have our fair share of banter here at Kivra and so, we present our top 5 inside jokes lately.
Lock your rebar dependencies or die
When developing with erlang and using libraries outside of OTP you've most probably come across rebar. Rebar is a build tool that aids in handling dependencies, doing releases, eunit, etc. In short, rebar is a pretty awsome tool.
What's not awsome is when rebar deps, either your own or your dependencies deps break their API which in turn render your app broken. Luckily there's a simple fix for this. Let me explain.
This is a pretty common sight in a typical rebar.config
{deps, [{oauth2, ".*", {git, "git://github.com/kivra/oauth2.git", "master"}}]}.
Well, maybe not using oauth2 as a dep, but specifying "master" or "HEAD" as the branch to track. This means that every time someone commits anything to oauth2 that breaks the API it will render your app broken.
When using rebar for dependency handling it accepts three types of forms:
{deps, [app]}. {deps, [app, "1.0.*"]}. {deps, [app, {VCS, "URI", REV}]}
where VCS is one of: git, hg, bzr, svn, and rsync. URI tells VCS where to find the dep, i.e. "git://github.com/kivra/oauth2.git" and REV is the dependecies revision. When using git you have a plethora of ways to specify revisions (This only applies to git). Here I'll list the possible ways:
{git, "git://github.com/kivra/oauth2.git"} {git, "git://github.com/kivra/oauth2.git", ""} {git, "git://github.com/kivra/oauth2.git", "HEAD"}
These will all get translated to "git checkout -q origin/HEAD" the equivalent of doing:
{git, "git://github.com/kivra/oauth2.git", {branch, "HEAD"}}
The last two methods are:
{git, "git://github.com/kivra/oauth2.git", {tag, "TAG"}} {git, "git://github.com/kivra/oauth2.git", {branch, "BRANCH"}}
Which get translated to "git checkout -q TAG" and "git checkout -q origin/BRANCH" respectively.
This means that it's pretty simple to lock the versions your app depends on. A good strategy is to go on a descriptive "tag" if it exists, such as "v1.0". What we end up doing a lot when using a lib that isn't tagged is to use the git hash such as:
{git, "git://github.com/kivra/oauth2.git", {tag, "4cf6d7e686"}}
The git hash can be a prefix, git supports shorter hashes if they are found to be unique. I.e. in the above case we could've just written {tag, "4cf6"}.
But even if you're a good citizen and correctly lock your deps you might still get hit by your deps deps not being locked, or your deps deps deps. The rabbit hole goes deep.
A comparison of web servers for running Chicago Boss REDUX
Some time ago we did a number of benchmarks measuring the current state of Chicago Boss and it's web servers. Since then we have made some small changes to improve performance and reduce potential bottlenecks. To begin with, Chicago Boss is a very nicely designed system with clear separation of concerns, interface, etc.. Chicago Boss adheres strictly to OTP which is both good and bad. One of the first things we noticed after running fprof and eprof on a system under load was that to much time was spent waiting for messages sent and queued between gen_server:call/2. Then we saw that many gen_server instances didn't updated its state after it started, but still needed to keep some information about the adapter, connection, etc.. so we decided to convert these to regular modules and use a common ets-table with {read_concurrency, true}. Through these changes, we got down the wait time in gen_server modules in the system under high load. On an average we see a ~40% better throughput with no errors (using the session cache backend).
We also made changes to how Misultin gets initiated with the result that we got rid of the many error 503 we saw in the previous tests. More specifically, we increased the number of max_connections from 4096 to 8192.
A lot of time is spent during the creation and sessions gets for each request, reducing the number of session gets saw drastic improvements in performance. Using the session mock backend the number of sessions will grow and ultimately hamper session lookup times as is seen from the graphs below. Using the session cache backend (which uses memcached) we saw a bit higher latency due to memcached but over time we saw better overal performance due to the O(1) nature of memcached.
Using the cache backend for sessions had another positive side effect in that there was no errors generated. So the session cache backend can be a good idea even for a single server setup as it provides a very predictable experience.
All the changes have been pushed to Github.
Below are the new tests (# 2) with the old test as reference:
A comparison of web servers for running Chicago Boss
Chicago Boss is a web development framework much like Ruby on Rails but written in Erlang. At Kivra we use Erlang extensibly and continuously evaluate and test new technology. We contribute to a lot of Open Source projects and our main technology interest is massive data at massive scale.
Yaws is currently not supported with Chicago Boss but since we like Yaws and would like to see how well it works with Chicago Boss and in comparison with the other web servers in this test, we've done some patching to Chicago Boss to support this. Apparently there's more to be done for this integration to be perfect as is evidenced by the graphs below.
This comparison is of purely academic purpose too better understand where in the chain certain things break and where we should focus our efforts in achieving even better performance, up-time and scalability. When we choose a web server for our projects we tend to look at a number of factors where speed is but one of them.
Also, bear in mind that the numbers in this test is for the web server being run through simple_bridge and fed into Chicago Boss. Depending on the quality of the simple_bridge implementation for that web server it might yield significant different results, which is evidenced by the raw misultin vs. Chicago Boss+misultin graph below.
Tech: To be able to run a http load test of somewhat massive scale there's a bunch of things to tweak to achieve maximum performance. For this test we've decided on the bare minimum so standard kernel, scheduling, etc. just changes to open file descriptors and minor changes to the ip stack.
Each connection we make from the client machine requires a file descriptor, and by default this is limited to 1024. To avoid the Too many open files problem you’ll need to modify the ulimit for your shell. We also ended up tweaking the client machine "ipv4/ip_local_port_range" to be able to open as many ports as needed. This can be changed in the current shell on the client machine:
# echo "32768 65535" >/proc/sys/net/ipv4/ip_local_port_range # cat /proc/sys/net/ipv4/ip_local_port_range 32768 65535 # ulimit -n 65535 # ulimit -n 65535
We're running httperf which is a tried and proved http load testing tool and we tweak some things before compilation to handle the load.
The server is a 16GB 8CPU machine running Erlang R15B with kernel-poll enabled:
Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:true]
We also tweak some parameters on the server machine:
# ulimit -n 65535 # ulimit -n 65535 # sysctl -p net.core.rmem_max = 33554432 net.core.wmem_max = 33554432 net.ipv4.tcp_rmem = 4096 16384 33554432 net.ipv4.tcp_wmem = 4096 16384 33554432 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_mem = 786432 1048576 26777216 net.ipv4.tcp_max_tw_buckets = 360000 net.core.netdev_max_backlog = 2500 vm.min_free_kbytes = 65536 vm.swappiness = 0 net.ipv4.ip_local_port_range = 1024 65535 net.core.somaxconn = 65535
We've used Roberto Ostinelli's excellent "A comparison between Misultin, Mochiweb, Cowboy, NodeJS and Tornadoweb" as a model to be able to correlate somewhat. There is of course allot of difference between raw web servers like the one in Ostinelli's test and running ChicagoBoss+ErlyDTL+Simple_Bridge+<web server>. So don't try to draw too many conclusions if comparing since there's also other factors that come to play.
We've used a simple controller which generates a list of random length(1-20) and passes it to a view for rendering. This way we get to test a normal MVC type pattern with ErlyDTL parsing.
Controller:
-module(loadtest_test_controller, [Req]). -compile(export_all). index('GET', []) -> {A,B,C}=erlang:now(), random:seed(A,B,C), R = random:uniform(20), List = getList(R), {ok, [{list, List}]}. getList(R) -> lists:foldl(fun(N,A) -> [[{val, N}]|A] end, [], lists:seq(1,R)).
View:
<html> <head> <title>{% block title %}Loadtest{% endblock %}</title> </head> <body> {% block body %} {% if list %} {% for l in list %} -{{ l.val }}<br/> {% endfor %} {% else %} Nope {% endif %} {% endblock %} </body> </html>
For the standalone misultin test we used this:
-module(load). -export([start/1, stop/0]). % start misultin http server start(Port) -> misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]). % stop misultin stop() -> misultin:stop(). % callback on request received handle_http(Req) -> {A,B,C}=erlang:now(), random:seed(A,B,C), R = random:uniform(20), List = getList(R), Req:ok(io_lib:format("<html>~n<head>~n<title>Loadtest</title>~n</head>~n<body>~s</body>~n</htm l>", [printList(List, [])])). getList(R) -> lists:foldl(fun(N,A) -> [N|A] end, [], lists:seq(1,R)). printList([], A) -> A; printList([H|T], A) -> V = lists:flatten(io_lib:format("~p", [H])), NewAcc = V++A, printList(T, NewAcc).
Here's the actual results from the tests:
The ideal graph would be linear from 1000 response/sec (start) to 12000 response/sec (end). Which is basically what we get when running misultin standalone.
Here we see that all web servers top out around 2000 res/sec. There's clearly something strange with Yaws since it dies and start generating error which is seen in the graphs below. Probably something in the simple_bridge API bridging Chicago Boss <> Yaws since we know from experience that Yaws standalone is a rock solid web server.
This is an aggregated view of all the errors that occurred. In the case of misultin we started getting "connection reset" errors after a while. Yaws gave us "connection timeout" which means it doesn't respond within the given five seconds. Mochiweb is pushing along nicely with two spikes that account for any errors given.
This is the average response time in ms for a request. The Y scale is O(log n) which means that there's a certain overhead (such as setting up sessions, etc) when serving a request but that overhead doesn't grow. This is good, you want a O(log n) or O(1) here.
This graph is basically the reverse of the error graph showing at how much req/sec each web server starts to crumble under the load and start generating errors.
It's been great fun performing these tests and now comes the really fun part of using this insights and try to create something that performs even better. Misultin standalone is really the school book example of how a web server should perform. Since we don't get those numbers when running with Chicago Boss there's clearly a great deal of potential here.
We used misultin standalone as a reference and could just as easily have used Yaws or Mochiweb and they would probably have performed equally well. But this test wasn't about which web server performs best but how good they perform under Chicago Boss which as we see from the tests are a whole other thing.
Hope you enjoyed this!