#!/sys/hero

Cassiano Aquino Security R&D at Ubiquiti Networks Ex-Zendesk

[syshero]

DNS Round Robin, Consul and Java

Hello everyone! Today, for a change, I will be talking about Consul from Hashicorp instead of NGINX.

I will start with the assumption that you're aware of what a DNS round robin is and how it's supposed to work and evolve from there.

I would like to start saying that the "problem" we faced, it's not related to Consul, but the solution that we looked for was for Consul.

Today Rafael while troubleshooting some uneven load distribution across his services, noticed an unexpected behavior.

He had his service correctly configured in Consul, the DNS response from Consul was returning the expected information, but still, only one node was receiving all the requests.

First of all, we started to take a look at java DNS lookup functionality to understand what could cause this issue, especially that both of us saw Greg Banks presentation about Java's internal DNS caching, which threw us out a little bit.

After doing some tests disabling Java's internal DNS caching and still having the same issue, we started to break the problem into pieces and test piece by piece, which we should've done since the beginning.

We started doing direct DNS queries to Consul to verify that the DNS response was what we expected to be, and it was, so we know that the problem is not Consul.

The next step on our DNS path is the local dnsmasq, which also proved to be working as expected after some simple DNS queries.

Now we were to a point where the DNS path leads us back to Java-land, which we tested before and looked OK.

But just to be sure that Java was the issue and not something else, we tried to use curl, after all the DNS path for both will be the same, and to our surprise, curl had the exact same behavior that Java had.

After this test, we can safely assume, that what's causing this issue, it's not only on Java-land but we also know that's NOT on DNS land, so what is the problem? Where to go now?

While trying to search for more information, regarding DNS round robin, we found this post written in 2012 by Daniel Stenberg curl lead developer, which gave us more clues about our problem.

The short version is that most applications used gethostbyname to do DNS requests, which will just give you a randomly ordered list of hosts, which makes DNS round robin work as expected.

But gethostbyname was built to support IPv4 and with IPv6 a new function was required, this is when getaddrinfo comes to play.

The "problem" is that getaddrinfo implements RFC3484 which returns the address always in a specific order, thus breaking DNS round robin, which relied on the random order from gethostbyname.

The idea is that when using getaddrinfo, the application should control the desired behavior, for example, grabbing a random IP from the list, or trying in order until you find one that's working.

In one hand, this is good as it gives applications more control over the desired behavior, but in the other hand, it makes the behavior inconsistent as each application do whatever they want.

Now that we knew what we were facing, we had some options, change our internal application to control this, or figure out a way to make our application work as it worked on the good old gethostbyname times.

And for our surprise, Consul had a clever workaround for this issue, which basically limit the size of the answer, and returns a random IP from the list instead of returning the full list and assume that the client application handle it as expected.

Obviously, this has some trade-offs, for example, DNS TTL needs to be shorter as the DNS RR will rely on returning a different address for each query, which may increase load on your DNS infrastructure and Consul.

But this will solve our issue until we have a more consistent behavior from applications using getaddrinfo.

For some extra information, you can also take a look at this post at Consul's mailing list.

That's all for today!

Thanks

[syshero]

NGINX+ - Replicated Key-Value Store

Hello everyone,

And I'm back with more NGINX for you!

If you have any question, not only about NGINX, that you think that I can answer for you, feel free to leave a comment!

I've been monitoring NGINX questions on Twitter, but let's say that people have a really bad time asking questions, especially when they have a limited amount of characters to do so.

Today I would like to show how two NGINX+ modules can work together to create a replicated and load balanced RESTfull key-value storage.

The current implementation of NGINX+ API has support for a key-value store, but not to replicate it, but nothing that can't be solved if we use the mirror module.

The concept that I will show below has two NGINX+ nodes, registered in Consul.

The servers have a service called nginx-kv, and tags that identify which node is the master or slave of the cluster.

This is optional and can be done using plain DNS but I've used Consul because it's another powerfull technology that can augment NGINX functionality.

On the master node we have the following configuration:

https://gist.github.com/caquino/10d4c33e66da6520a34abc03fb268fd6

The API is accessible via the URI /replicated, and every request is mirrored to /mirror.

On the /mirror location, a header X-Replication-Source with the value master is set, and I will get back to it as it's really important.

And any request that is not GET or HEAD is mirrored via proxy_pass to the slave node.

We have no interest in mirroring GETs and HEADs to the mirror as this would waste resources.

On the slave we have the following configuration:

https://gist.github.com/caquino/5a57bddc1dba8cb0ab8411dd680b61af

The first thing to note on the slave, we have a variable that we concatenate the header X-Replicated-Source with the request_method, this way if a request comes from the master nginx, it will not be affected by the if clause.

The if clause then checks from any write operations, that is not coming from the master node, and redirect them to the master server.

This is a similar approach to LDAP updateref; the slave server is aware that it's a slave, and redirect writes to the master.

With this in place, a simple load balancer can be added in front of the two nodes, and reads will be load balanced and writes will be sent to the master server.

I quickly presented this concept at nginx.conf 2017 in Portland, where I said that I would share more information in a blog post, took some time, but here it is!

[syshero]

NGINX+ Automatic PURGE when content is changed

Hello everyone, I’m on fire! After such a long time without posting finally I got some content and time to update my blog.

One of the biggest challenges for content caching is expiration of this content, too short lived and your cache is not effective, too long and you will serve stale data.

The sweet spot for caching is exactly to cache the content while it’s valid, which is hard as you may not know for how long your content will be valid in the moment that’s being cached.

To help with that NGINX+ has a cache purge functionality, that allow applications to send a request to NGINX+ and ask that an specific content to be removed from the cache.

This requires extra application logic, which is fine, but sometimes you need to be able to purge content for applications that are not aware of the caching in front of it.

For applications that uses the same URIs to read and write data based on methods, for example, lets imagine that we have an URI /profile/syshero.

If a GET is sent to /profile/syshero, the application will render and serve the output to the client

If a PUT/POST/DELETE is sent to /profile/syshero, the application will do an write operation and change this content

This concept is similar to how REST API works, which makes this approach compatible with a large amount of REST applications.

The desired solution would:

Cache or serve cached content for GET/HEAD requests

Purge cached content for any other HTTP method e.g: POST, PUT, DELETE, PATCH, MKCOL, etc

The following configuration achieves that by using the NGINX+ mirror module together with the purge option.

https://gist.github.com/caquino/ab3aabe5b0ba966a581ad649d7bbffe0

This configuration receives any request to /, forward to the backends servers and also mirrors this request to the location /mirror.

The location /mirror will return a 204 for GET and HEAD methods as we want to avoid duplicating all our requests to the backend, and also to acoid purging the cache on every request, and for any other method that are not GET or HEAD purge the cache for the specific URI.

As can be seen on the following test:

https://gist.github.com/caquino/7a82a740a2cdbc5a67607dcacd4cb126

Simple as that!

One important thing to note is that as you can see, different variables are required on both location blocks as the proxy_cache_key, and why is that? This is necessary because of how the mirror module works. To be able to have the same value on both blocks it was necessary to use $uri and $request_uri.

I would like to thank all of NGINX+ Support team for the constant help and support but specially Dmitry Pryadko that helped me figure our how to filter out methods as limit_except was not working as expected.

Also I would like to thank Bruno Paiuca for asking this question on twitter which got me thinking about on how to solve this problem.

@nginx you need an option to purge cache after post/put/delete to works with our Rest APIs. proxy_cache_purge_after POST, PUT, DELETE

— Bruno Paiuca (@BPaiuca)

February 10, 2018

Thanks everyone! See you next time!

[syshero]

NGINX - Unique Request Identifier

Hello everyone! I’m back and this time faster than expected, right?

One useful information to have while debugging applications is an end-to-end request id.

To be able to achieve that you need not only NGINX but also application changes, to forward the ids but also to log them.

On this post, I would like to share how to implement the NGINX side.

One way to achieve that is setting a header for both the client side and to the upstream called X-Request-ID for example.

The behavior expect is that, if X-Request-ID is set, NGINX will forward the same value to the next upstream, if the header is not set, NGINX will generate a random request identifier and add it to the request.

To achieve that we use a map configuration as can be seen in the following configuration.

https://gist.github.com/caquino/2b6b92d39740f967a8e2ad634f03cb25

On my previous implementation, we used a Lua piece of code to generate the unique request identifier, and this had an specific formatting similar to:

12345678-1234-1234-1234-1234

To guarantee backward compatibility, I used an NGINX map to format the $request_id variable to a format that matches my old setup.

https://gist.github.com/caquino/cebdee88499e123c6529a20690131b89

Using map you can apply regular expressions to variables and transform the variables based on the regular expression.

This is one of the many reasons why I think map is one of the most versatile and powerful configuration options for NGINX.

I hope this helps you!

[syshero]

NGINX+ Chained PURGE

After a really long winter, I'm here again to post about my favorite subject, wanna guess? NGINX! (Like it was hard to guess)

I will start with a story, which may or may not be based in real life, you get to choose.

Once upon the time, Mr. J worked in a big company that used NGINX as their caching layer.

As the company grew, so their infrastructure, and now they needed to run multiple NGINXs (What's the plural of NGINX BTW?), and seemed to be an easy task, turned into a complex problem.

Their applications used NGINX as a cache, and now doing cache PURGE was a lot more complex than it was before, as they needed to do it on every NGINX server as the caches were isolated from each other.

Mr. J was also worried about how this would work in the Cloud™, what about Autoscaling Groups? All that fancy functionalities that I can leverage BUT now it's much harder to do PURGEs.

How do I keep a list of the servers? Do I need to call the Cloud™ API?

So after doing some research, he had some ideas, one was to use a shared storage between the caches, but he felt that this would increase the complexity even more and steered away from this option.

What about using Lua? after some research, he had some issues to overcome.

How to maintain a list of the NGINX nodes dynamically

We have a way to solve this, as we use NGINX+ and Consul, we could use an upstream to have a list of the NGINX servers.

Lua has no access to the list of servers in an upstream without extra modules

This one is a little bit more complex, and Mr. J didn't want to add unnecessary modules to his NGINX+, which made him remember that NGINX+ has an API that can be used to query the upstream servers list.

Lua has no access to external HTTP requests without extra modules

Lua has a function to do internal HTTP requests, ngx.location.capture(), which will not work to call other external endpoints, but if we join forces between ngx.location.capture and proxy_pass we can make it work.

But Mr. J continued to move forward with the mission in his mind of doing it using only what he had available to avoid increasing the complexity of his system.

After a lot of research and hard work (not really, it was done in a couple of hours, but don't tell anyone.) Mr. J managed to come up with a solution for his problems.

Let's imagine a really simple NGINX+, which works well with a single cache node:

https://gist.github.com/caquino/a4692d5abefba4141ad6f5e9fa49ee05

On this example, NGINX+ uses Consul to populate the backends upstream server list.

Nothing special here, no PURGE or anything just yet

Now, to allow PURGE, Mr. J decided to go with an internal invalid domain, and this was the configuration used.

https://gist.github.com/caquino/7cd202a7d15462b48c649ac829daf63b

Basically, to purge any content, he can even use cli tools like curl:

https://gist.github.com/caquino/e5bce397e4106e041d01fccf7a7755f5

When executing a PURGE, NGINX+ will return a 204 No Content.

But still, Mr. J needed to turn this simple configuration in something that would be aware of other NGINX+ caches, and forward the PURGE requests to the other instances.

And finally this was the solution that Mr. J found to his problem:

https://gist.github.com/caquino/52b9612167325f43361aea80856fed56

Let's break these into pieces and explain the important parts, first NGINX+ has an upstream called caches configuration, that's not used by any proxy_pass:

https://gist.github.com/caquino/14ba8849cb2610f9ffa8202b9d72d560

This is used as a catalog to allow instances to find each other.

Another important part is to declare where the NGINX+ Lua module will find the cjson module, as Mr. J used Ubuntu, after simply doing an apt-get install lua-cjson, he added the following line to the configuration:

https://gist.github.com/caquino/fa8df1de60ee61489c3acf97f4786327

And now it comes the important part, the most complex one:

https://gist.github.com/caquino/e5f319f2df589c3fa7eb95e290da329c

When any content needs to be purged on all nodes, a PURGE request to /purger/index.html should be issued on using the host purger.local.

This will trigger the Lua code running on this location, that will call NGINX+ API and fetch a list of servers on the upstream caches.

For each server on the upstream list, Lua will execute an internal ngx.location.capture() to a proxying location, that will then send the request to the destination server.

And here it's the solution in action:

Caches register themselves to Consul as the service named cache, and they are immediately available to NGINX+ to be used as servers on the upstream configuration.

With this solution, the PURGE request can be sent to any NGINX+ server and it will forward the PURGE request to all other NGINX+ nodes in the pool.

And Mr. J lived happily ever after.

I hope that Mr. J adventure helps you solve your caching issues! See you next time!

[syshero]

Dublin DevOps Meetup - Lazy DevOps

Yesterday, 10th August 2016, I’ve presented in our Dublin Devops Meetup, that happened on Dublin Zendesk’s Office.

My talk was about my experience building a PaaS, what’ve learned, my mistakes and how I solved some of my problems.

I would like to say thanks to everyone that came to Zendesk’s office to see our talks.

The presentation is available at my github

Thanks!

[syshero]

nginx.conf 2016

Hello, friends!

I hope you're doing all right!

As you know, between September 7 and September 9 in Austin, Texas, the nginx.conf 2016 will happen!

It will be an awesome opportunity to learn from the NGINX experts and to join the community.

The good news about it is that NGINX kindly gave a gift to my blog readers, a discount code that will give you $400 off 2-Day and All Access pass!

Just go to the nginx.conf website and buy your ticket and use the code SYSHERO on the checkout, simpler than configuring NGINX right?

It's almost 50% discount for the All Access Pass and more than 50% for 2-Day Conference Pass! Don't miss this opportunity to improve your NGINX skills!

[syshero]

Thinking outside of the box with NGINX series - proxy_cache_bypass command

Hello, everyone! Time to get back to our series of posts about NGINX!

This time, let's talk about an old friend, the proxy_cache_bypass command, which we used multiple times in my examples but I never dedicated a post to it, so it's time to fix this error!

At first glimpse proxy_cache_bypass feels like a simple command, one where you can bypass caching when necessary, but after reading the documentation with more attention, there was a piece of it that caught my attention.

Note that the response from the back-end is still eligible for caching.

So basically proxy_cache_bypass not only allows your to bypass the cache but also will update our cache with the new file working similarly to a cache purge/refresh functionality.

If you want to not update cached files, you should use proxy_no_cache instead, which will have the desired behavior.

Let's imagine that you want to bypass your cache when the requests are from your office, to do this you can use the geo module together with proxy_cache_bypass, and as a bonus, the cache will be updated as your office browse through your website.

https://gist.github.com/caquino/20a2f3ba36835b5fe344f0429df49a3f

Another really useful feature of proxy_cache_bypass is that you can pass multiple variables as argument to it, and if any is set it will enable the bypass action.

Some other posts I did that I used proxy_cache_bypass to update our cache:

Thinking outside of the box with NGINX series - split_clients command

NGINX passive cache invalidation

NGINX “cache purge” emulation

See you in the next post.

Thanks!

[syshero]

imapfilter - cleaning up your mailbox and Gmail tricks

Hello friends,

For the ones of you that do not know imapfilter, you should check it out, but basically, imapfilter is a IMAP client where you can write a set of rules and apply this rules to your mailbox.

imapfilter rules are written in Lua, a really simple and powerful programming language.

To start you should create a folder in your home directory called imapfilter, to do so you use the following command:

mkdir ~/.imapfilter

The configuration file should be saved at ~/.imapfilter/config.lua, after having this configuration file in place, just by running imapfilter will be enough to have your configuration executed.

To make things a little bit more interesting, let's dissect my imapfilter configuration, where I will explain some basic features and some Gmail particularities that you should be aware if you use it.

Let's start with some basic options that I use and I will explain why.

https://gist.github.com/caquino/cf523a99ebfe29a2d8ebb374bc6df58f

options.timeout: will specify a timeout for imapfilter to wait for the IMAP server response, if you have huge mailboxes OR if you're planning to use the special "All mail" folder in Gmail, it's a good idea to have a big timeout as operations may take a while.

options.create: this will create folders that you reference in the configuration if they don't exist.

options.subscribe: this will subscribe to the created folders, basically when you create a folder does not mean that it needs to appear on the list, this will make it appear.

options.expunge: deleting messages in folders just mark this messages for deletion, moving the messages to trash, this will remove deleted messages, this behavior can be different based on IMAP server configuration, this is the most common configuration thou.

Now let's define our account login options.

https://gist.github.com/caquino/a31480066518213f4a364a559e26ab35

This block will create an object called account1, which represents our mail account.

It's a self-explanatory block, but I do have a remark if you use 2-factor authentication on google you need to generate an application password and put it here as your password will not work.

Now, let's start to see some basic rules I use, you will need to change the examples to match your needs, as this one was made based on my personal needs.

https://gist.github.com/caquino/27d8b48b22dda1a544e2e19aa4fd86ad

On this block, I've created a code that, in the case of Gmail, archives read and unflagged messages older than a specified period, for example in this example 30 days.

For Gmail the function delete_messages() archive instead of delete, keep this in mind while doing your own configuration.

https://gist.github.com/caquino/989eb350ba975fabc05522895c6c0070

Now we will remove meeting invites older than 90 days, don't worry if you accepted, it will be on your calendar and will not be deleted.

Note as we really want to delete this messages, instead of using delete_messages() we are moving the messages to the Trash folder on Gmail.

Now let's say we want to delete messages from some senders and based on some subjects, instead of having individual code for each, it's simpler to use arrays to do it.

https://gist.github.com/caquino/1fe8b1115865b6c1e07dfc776158deaa

Again we are moving to Trash, as we really want to delete this messages.

Now let's clean up some folders where I store automated emails.

https://gist.github.com/caquino/74fe45880aabf3e3269d1344672095e1

And to finish things up we need to delete messages on our Trash by using the following code.

https://gist.github.com/caquino/6ccac22055e1a2ea435e0ccfd80adaac

One thing that I like to do is to move messages older than a specific period out of my INBOX to a "Cleanup" folder.

https://gist.github.com/caquino/cadee6720fc566b0f5787e010a1140ab

All this examples should be put on the same file, called ~/.imapfilter/config.lua, or you can use includes if you want, but now you have a good baseline on how to clean up your mailbox using imapfilter.

Thanks!

See you on the next post!

[syshero]

Thinking outside of the box with NGINX series - map command

I don't think this is really thinking outside the box, but I could not do a series of posts about useful commands without talking about map, arguably one of the most useful commands on NGINX, especially that I'm sure map will be used in most of our posts.

You should think about map as a "switch" statement like many programming languages have, it's how you will be able to take advantage of it.

Basically, the syntax for map is the following:

map $variable_to_evaluate $return_variable {

 value_to_match value_to_return;

 default value_to_return_by_default;

}

Where the variable_to_evaluate can be ANY variable accessible by NGINX, which includes HTTP headers, Cookies and etc.

You can think of map as a good substitute for most "ifs" that you plan to use, especially considering that if is evil, basically map is a safe replacement for if in most of the situations.

Let's review our rate limiting example that we used geo to create a whitelist, now we will use map to do the same thing, if the client sends a cookie called "whitelistrl", he will bypass the rate limiting, but remember, this is not a good idea for rate limiting control as cookies can be easily injected to any http request.

https://gist.github.com/caquino/ad3e6bc545258880e437a8aec4470159

Another functionality is that maps can be nested, one map can call another map or even combine geo/map/split_clients.

Let's try another example, what about using map and geo to allow us to only whitelist clients from our customer network that contains the whitelistrl cookie? sounds like fun!

https://gist.github.com/caquino/9c724bad73421b495e2ad6225b3c1e48

One thing to note thou, the order of the nesting matters, map accepts nesting other commands but this is not true for all commands, for example if we had tried to call map from geo, it was not going to work, so always test before deploying to production friends!

Also, you can see examples of map usage in posts right here on my blog! Some examples are:

Persistent backend selection using cookies with NGINX

Using map for upstream configuration on NGINX

Disable NGINX cache based on cookies

Avoid caching 0-byte files on NGINX

See you on the next post! 

If you have any question or suggestion use the Ask-O-matic link on the top of the page.

Thanks!

[syshero]

Thinking outside of the box with NGINX series - geo command

For the second part of this series let's talk about another command, this time, let's talk about the geo command.

The geo command, as the name says, was created to allow geolocation information, but actually you can think of geo as a "switch case" where you can match network blocks and get a result from it.

So let's say for example that you want your QA team to always go to your preproduction environment instead of production, with geo this can be done easily, provide your team have a fixed IP address that you match.

https://gist.github.com/caquino/514c4b62c78101441bf1efef4aa371e8

Another possibility is to route a single server or a group of servers to a different upstream, for example, to do regression tests in a debug server, this is possible if you use NGINX to route the API calls through your application stack.

The geo can also be used together with the rate limit module, allowing you to specify rules for specific netblocks, let's say you have a global rate limit and you want to allow your special customer to be whitelisted.

https://gist.github.com/caquino/535e2e16976ea065a1526fc3a02c15dc

The idea here is to use the variable $binary_remote_addr for every request except for the customer network, for the customer network we will return an empty value.

On the next post, we will talk about the map command

If you have any suggestion for a post or if you even have NGINX questions fell free to contact me through the contact link here on the blog.

As the idea of this series of posts is to show different ways of using NGINX configurations, I would like to ask for your collaboration sending questions or even guest posts.

See you in the next post, thanks!

[syshero]

Thinking outside of the box with NGINX series - split_clients command

I think it's time for me to start a series of posts about useful NGINX commands, and different ways of solving common problems using NGINX and to start this series, let's talk about split_clients.

split_clients was built with A/B testing in mind, but it's not the only possible way of using it, it's way more powerful than it's original design.

let's start with some examples, the first one I like to call it limited caching and renewal, the idea is that you have a cache in front of your application, where X% of the requests will hit the upstream servers and update the cached version.

For this example let's consider that we have an upstream group called limitedcache and that 10% of my requests will be sent to the backend and renew my cached file and the other 90% of the requests will receive our cached page instead.

The key used for split_clients on this example is a "pseudo-random" variable that tries to split 10% of the requests in a different bucket to trigger the bypass action on NGINX, that's why date_gmt is used on it.

The key commands in this example are split_clients, proxy_cache_bypass, proxy_cache_use_stale and proxy_cache_lock.

Again we are abusing proxy_cache_bypass to create a poor mans "PURGE" functionality.

https://gist.github.com/caquino/e21e378530a648dd7e4fb282b152cc7a

Another interesting usage of split_clients, that's closer to the A/B testing idea is that you can use it to do load testing/regression tests on new deployments.

The idea here is that you can send X% of your requests to a set of backends running the new version of the code that you want to test.

On the following example, we will send 10% of our requests to a group of backends called testing that is running our new code that we want to test.

https://gist.github.com/caquino/b08c8034941157d32161dccc2412155a

But remember, this example will only work well for stateless applications if yours need persistence check it out this example

You can also find some really nice usage of spit_clients at this post under Splitting the Cache Across Multiple Hard Drives and also how to overcome ephemeral port limit using split clients

On the next post, we will talk about the geo command

If you have any suggestion for a post or if you even have NGINX questions fell free to contact me through the contact link here on the blog.

As the idea of this series of posts is to show different ways of using NGINX configurations, I would like to ask for your collaboration sending questions or even guest posts.

See you in the next post, thanks!

[syshero]

SSL acceleration using NGINX and backend SSL connection identification

One common trick to optimize SSL termination performance is to have a proxy terminating all the SSL connections and proxying the connection to a plain HTTP backend.

This is theory works very well and without any code changes, but in practice, you discover that after doing this your application can't tell the difference between HTTP and HTTPS clients.

The most common issue is that the application creates a redirect loop trying to redirect the user to HTTPS, because the application is not able to know that the user is already using HTTPS. 

But how to solve this issue? 

The most common used method is to use an HTTP Header to identify SSL/non-SSL connections, this way the application is able to know which protocol was used on the request.

The header used for this purpose is X-Forwarded-Proto, which can be added to the upstream request using proxy_set_header, as we want this header to be sent to the backend servers and not to the clients.

With this header in place, the application can take actions by evaluating the value set on it.

The following example sets the X-Forwarded-Proto header and forwards the connection to our backend using proxy_pass.

https://gist.github.com/caquino/6f477ad445d1e508141263d055d136dd

[syshero]

Zero-downtime deployments using NGINX

Hello, everyone!  missed posting here, I should say that I need to start posting again!

As many of you know, some application servers do not support zero downtime deployments or even rolling restarts, and no, I'm not talking about unicorn, unicorn supports this out of the box.

One possible solution to be able to do zero-downtime deployments on such application servers is to use NGINX and some really basic shell scripting.

The concept is that you run a local NGINX in front of such application server and during a deployment you spin up a second instance of the application server in a different port, having temporarily two copies of the application running.

For the sake of this example let's say we have a configuration file called zerodowntime.conf and the application runs on the ports 8001 and 8002.

To be able to execute the zero-downtime deployment you just need to switch the "down" flag on the upstream configuration to the old backend, which will not receive any new connections, but will finish processing the existing ones and remove the down flag from the new instance, which will start to receive all the new traffic.

I've added a configuration example and a simple shell script to do the backend switch, but this can be easily improved, like for example adding some curl requests to the backends to check if they are up.

After the configuration changes, the last step necessary is just to send a reload signal to the NGINX process and voilá! your new application server is processing all the new requests!

The following configuration is a simple example of how to achieve such functionality with minimal configuration. 

https://gist.github.com/caquino/32e50922059aadcc75e787c62ae956c8

I hope this will be useful for some poor soul suffering trying to do zero-downtime deployments.

Thanks!

[syshero]

Dublin DevOps Meetup - Nginx

Today, 17th April 2016, I've presented in our first Dublin Devops Meetup, that happened on Dublin Zendesk's Office.

My talk was about my experience using Nginx, what've learned during the years, my mistakes and how I solved some of my problems.

I would like to say thanks to everyone that came to Zendesk's office to see our talks.

The presentation is available at my github

Thanks!

[syshero]

Persistent backend selection using cookies with NGINX

Hello folks, sorry for not posting more often here.

Today an old good friend, Alex,  sent me a "Bat signal" asking for help with an unusual NGINX configuration, I wonder why my friends only remembers me during "weird" incidents :P

Basically the requirement was to run a multiple copies of a legacy application on the same URL as it does not support folder prefix or subdomains.

The desired functionality was that /app1 configures nginx to use backend1 and return to /, and all the subsequent requests on / should be sent to backend1, the same with /app2 to backend2 and /app3 to backend3.

So the solution we designed is:

A request to http://server/app1, sets a cookie to be used for backend selection, and redirect to /.

subsequent requests to /, will use the cookie value and select the backend based on it.

So on this example it was used a combination of map, regexp named matches and cookies using NGINX.

The configuration below achieves this functionality, mapping /app[0-9] to specific backend listed on the map { }

https://gist.github.com/caquino/27dd864ab756eb71e99f

Thanks!

[syshero]

Using map for upstream configuration on NGINX

Hi guys, sorry for not posting any content here for such a long time.

Some of you should have noticed already that I like using map for a lot of stuff.

Another neat usage of map is to simplify your configuration, let me give you a quick example.

Imagine that you have the following configuration:

https://gist.github.com/caquino/d133219ec08a50285394

To simplify your configuration you can use map with the following configuration:

https://gist.github.com/caquino/41477d0fca3b78075a68

This helps reducing the mess on the configuration made by a lot of location { ... } blocks with similar configurations.