Tech Talk @code-snippets-tech-talk - Tumblr Blog

Benchmarking NoSQL solutions

SQL world is easier to understand as compared to NoSQL world. NoSQL landscape is very confusing with multitude of solutions loaded with buzzwords. NoSQL solutions are often divided into categories based on solution type like key value, document or columnar. Sometimes they are also divided based on CAP theorem.

However, it's quite hard to classify an application as key-value or document or columnar. It is very much possible to architect an application to use any kind of nosql database. For example, one can store document as json in a key value store database. Or one can decompose the document into columns and store them in a columnar database. Similarly, a document or columnar database can easily fit in for a key-value solution.

This blog is focused on benchmark performance results between 3 popular NoSQL solutions - Redis, Couchbase, Cassandra. The test was done using the following scenario

DB server was running on AWS m4.large instance

DB Client was running on AWS c4.xlarge instance

Only a single DB server was used to compare performance in single instance solution

Both server and client were in same availability zone

All DB solutions were used as key-value store.

Data was written into database first, compacted, optimized and then test was 100% read workload

DB client was larger than DB server to ensure that server can be pushed to limits

First test was done with just 1,000 entries into the database

Redis was able to give 50k ops/sec, couchbase maxed out at 29k ops/sec, cassandra maxed out at 18k ops/sec. However, the biggest difference was in response times, where redis performed around 150 microseconds per response, cassandra starting reaching 1000 microseconds (1ms) for simple key value read.

Next, the test was done using 500k entries in the database

Here, Redis and Couchbase maxed out at 16k ops/sec. Redis response time was 250 microseconds compared to 375 microseconds for Couchbase. Cassandra started going beyond 1ms when pushed beyond 9000 ops/sec.

So far, all the data could be stored in RAM, since the data size (+indexes) was less than RAM size. So, all the databases were reading from RAM. Going further ahead on number of entries required configuring Redis to use swap along with RAM which is not recommended by Redis. The next test was done using 3M objects, each with 4k size. The total data size was approx 12GB and indexing (on primary key) led to approx 16 GB database size on a 8GB RAM machine. The reads were spread randomly across the 3M objects. Swap used was coming from EBS-SSD. Note that only Redis used Swap, Couchbase and Cassandra were not configured with Swap.

Cassandra was too slow to even compare. So, I dropped it out of the test after first read. Redis performed faster than Couchbase.

However, an interesting thing happened while I was prepping up for 3M test. I ran out of disk space while testing Couchbase. Couchbase became unresponsive, so I rebooted the machine. Once rebooted, couchbase failed to start and read the data from disk. I spent hours trying to get it to read the data, but ultimately, it led to data loss with Couchbase.

Next, I did the same 3M entries test, but read was confined to only 5% of the entries. So, the DB was storing 3M objects, but only 125,000 objects were read from DB.

Here, performance looked similar to 500K test. Redis with the help of Linux OS swap management was able to load keys into RAM. Couchbase already has logic to put cache in memory. However, Redis did outperform couchbase on both ops/sec and response times.

Summary:

Cassandra is quite slow as compared to Redis & Couchbase

Couchbase process list is huge and many components working together. It looks like a product built by acquisition. It's stability for production use is questionable.

Redis is blazing fast. It provides much more ops/sec on similar hardware. So, ops/$ spent is very good for Redis.

Redis Labs already has product for storing cold values on Flash. Hopefully, AWS will provide Flash as cheaper memory, expensive disk option sometime in future.

Elasticache now supports Redis Cluster. If your application can afford (from $ perspective) to put data into RAM, consider that option strongly.

#nosql #redis #cassandra #couchbase #elasticache

Load Balancer 101

This blog is part of Cloud School series of blogs

Load balancers are quite common in architectural designs. Load balancers are primarily used for spreading requests across a pool of servers. They are also called reverse proxies. Note that they are different from forward proxy (like squid). Forward proxy sits close to the client and reverse proxy sits close to the server.

Load balancers have the following features

Persistence - Load balancers can maintain session persistence which allows the requests from same session to go to the same backend server.

Health Checks - Load balancers can periodically check health (and adjust weights) of the backend servers. This ensures that service remains up even when a few backend servers are down.

Metrics - Since all the requests and responses pass through load balancers, they are often used for collecting the service metrics

SSL offloading - Load balancers often terminate HTTPS/SSL and store client facing certificates. Connection from load balancer to backend can be encrypted again often using a low CPU intensive cipher like AES-128.

Request Queuing - An often overlooked but great feature of load balancer is to queue the requests instead of overload the backend servers. This improves the service response time and can handle small traffic spikes without affecting service health.

DOS protection - Load balancers can maintain metrics on requests per ip per min etc and can throttle/deny abusers.

Maintenance page - Load balancers can direct traffic to a backup pool or show maintenance page when backend pool is not available.

#cloud school #load balancing #health check

Rolling deployment leveraging cloud capabilities

I recently took part in a panel on Application Deployment hosted by Electric Cloud. Check out the recap video for interesting discussion and pitfalls around the topic of Deployment. During the panel, I talked about deploying software in the cloud leveraging the features of rapid provisioning and pay by the hour pricing. The deployment model I talked about in the panel discussion and am mentioning here builds on concepts like immutable servers/phoenix servers, blue green deployment, cloud layers.

The idea originated from AWS implementation of Elastic IP. AWS allows one to re-map the Public IP address to an internal server. AWS is essentially doing a 1:1 NAT where the public IP traffic is forwarded to a private IP behind the NAT device. Using similar concept, one can create an "address" (similar to Elastic IP) which can be mapped/remapped to an environment.

Going back to layers, an application design essentially looks like

where the visible layer contains the networking device where the TCP connection terminates for a service. The volatile layer represents all the load balancers, web servers, application servers, caching layer, queues etc. The persistent layer consists of servers which store the data, which in most cases will be database servers. However, if you want cache to persist across deployments, cache servers will also go into persistent layer. The detail diagram could look something like

Now, for the deployment, we can mix the basic principles of A/B testing and immutable servers to do the deployment. The high level concept is to create a completely new environment, add it to the visible layer device and switch the traffic over to new environment.

Here are the various stages of deployment in detail

Stage 1

This is the current setup, where 100% of the traffic is giving to existing environment.

Stage 2

Here, we are creating a new environment and setting it up. With the help of automated configuration management, we can speed up this process quite a bit.

Stage 3

We will be adding this environment to the visible address and send 0% of the traffic there. Many load balancers (like haproxy) can make this happen. We have added the new environment to production but we are not sending any traffic there yet. Note, that the new environment is talking to same persistent layer (databases, persistent cache servers) etc. This ensures data continuity from one environment to another.

Stage 4

Here, we are doing functional testing of the new environment while its hidden from the customers. Many load balancers like haproxy can be modified to have rules where certain kind of traffic can be forced to go to specific backend. Easiest way to do is by dropping a special cookie on browser of QA person who is doing functional testing.

Stage 5

Once we have done functional testing, we can do some real life testing here by sending a small % (10% here) traffic to the new environment. Borrowing the concept from A/B testing, we can send traffic in a sticky fashion to new environment. Again, load balancers like haproxy can do this easily using session persistence.

We can do some checks if required to make sure that new environment is not increasing error rates across the service.

Stage 6

Once we have confidence in the new environment, we can increase the throttle (in steps if desired) to 100% making the new environment completely live. We can disable the old environment so that existing sessions on old environment also switch over to new environment while giving requests in flight time to finish (think long http requests like upload, download).

Stage 7

Now, we can remove the old environment and call the deployment over. We can choose to run the old environment for a while. If things go wrong, we can switch the traffic back to old environment. That would work as an effective rollback strategy if needed.

Summary

Using rolling deployment, one can deploy to a completely new environment and simply "switch" the traffic from old environment to new environment. There is no redeployment since the code is always deployed to a fresh server. The server is built from scratch using configuration tools like Chef/Puppet. In fact, even the code (war files, php, whatever) can be placed into servers using configuration tools as well. Code deployment can be considered as an integral part of server configuration itself.

Such a deployment strategy has many benefits including an easy rollback strategy, test before switching traffic over. Using the financial features like pay per hour, the provisioning of new servers doesn't cost much since old servers are decommissioned after deployment. So, net cost of deployment is running an additional environment for say an hour which is minimal cost provided the benefits.

#deployment #haproxy #cloud computing

HTTP git server on nginx

Git project has recently launched smart http server which enables one to use git remotely over http instead of default ssh protocol. I can foresee http becoming the primary protocol to interact with remote git servers in future. Current documentation of git-http-backend only gives instructions for Apache.

However, I wanted to setup git over https using nginx, haproxy (for SSL offloading). Here are the working code snippets which enabled me to set up git remote server over https on Ubuntu 14.04 LTS.

Install nginx, git, fcgiwrap

sudo apt-get install nginx git fcgiwrap

Add the following lines to /etc/nginx/fastcgi_params

ubuntu@gitserver:~$ tail -2 /etc/nginx/fastcgi_params # Pass authenticated username to CGI app fastcgi_param REMOTE_USER $remote_user;

Create a new file in nginx configuration for git server

ubuntu@gitserver:~$ cat /etc/nginx/sites-available/git server { listen :80; server_name gitserver.example.com; auth_basic "Restricted"; auth_basic_user_file /etc/nginx/passwd; location ~ (/.*) { fastcgi_pass unix:/var/run/fcgiwrap.socket; fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend; fastcgi_param GIT_HTTP_EXPORT_ALL ""; fastcgi_param GIT_PROJECT_ROOT /srv/git; fastcgi_param PATH_INFO $1; include fastcgi_params; } }

Create a symlink

sudo ln -s /etc/nginx/sites-available/git /etc/nginx/sites-enabled/git

This enables a git server on http://gitserver.example.com with restricted authentication via nginx. Nginx supports authentication using htpasswd or even linux crypt authentication. I personally use 5,000 rounds of SHA-512 with random salt (default linux shadow password implementation).

If you want to go extra mile and move this http based git to https, we can set up haproxy to do SSL offloading. Working haproxy (using haproxy 1.5.4) config file

ubuntu@gitserver:~$ cat conf.haproxy.txt global log 127.0.0.1 local0 tune.ssl.default-dh-param 2048 maxconn 4096 user nobody group nogroup daemon pidfile /var/run/haproxy.pid stats socket /var/run/socket-haproxy level admin defaults log global mode http option httplog option dontlognull option httpclose option redispatch timeout client 30s timeout server 30s timeout connect 1s timeout http-keep-alive 60s timeout http-request 15s stats enable stats refresh 10s stats uri /stats frontend https bind 0.0.0.0:443 ssl crt /home/ubuntu/ssl.pem no-sslv3 ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK default_backend server backend server server 1 127.0.0.1:80

The above SSL configuration got me A on ssl labs with support for everything but IE 6 on XP, IE 8 on XP and Java 6.

The ssl.pem used above looks like

#haproxy #nginx #git #git-http-backend

Encrypting linux root disk in EC2/Cloud

I posted a blog earlier on the topic of encrypting a disk. That solution works great for secondary linux volumes. However, that solution doesn't extend to root device. This blog is about encrypting root disk in a AWS/Cloud environment (or any environment where console access is not available to enter the password).

One of the foremost requirement of encryption is protection of the encryption key. The key can't be embedded into the boot disk. Since we don't have console access, the key can't be typed during boot process. Certain solutions exist like embedding dropbear (a small ssh server) into boot process where someone is required to login via ssh and enter the password during boot process. However, that's a push model where a human or a process has to be aware that the server is waiting for password. Such a solution is not feasible for someone managing a large server fleet.

The solution mentioned here loads the encryption key into server memory only and doesn't store it anywhere on disk. There are 2 proposed solutions for 2 different use cases. The first solution is for volatile servers. As a part of this solution, the server generates a random encryption key and encrypts root device using that key. The key is only in server memory and never persisted. A reboot will make the server loose it memory and will cause loss of data. This solution is powerful but only usable for servers which don't persist data.

The second solution builds on top of first solution and requires a central keyserver which stores the keys. The central keyserver will be queried upon boot and will be required to provide the keys to the cloud server in need of encryption keys to mount its root volume.

The solution presented has some complexity as linux boot process can't do https. So, the solution is designed to protect the key in transit using authenticated encryption.

Before getting in technical details, you might want to get familiar with building an AMI from scratch blog post. Following are the technical code snippets used in Solution 1

# 150M for boot, rest for root device sudo sfdisk -uM /dev/xvdf << EOF ,150,L ,,L EOF sudo mkfs.ext4 /dev/xvdf1 sudo mkfs.ext4 /dev/xvdg sudo mount /dev/xvdg /mnt sudo mkdir /mnt/boot sudo mount /dev/xvdf1 /mnt/boot # Install Ubuntu 14.04 into /mnt sudo debootstrap --arch=amd64 trusty /mnt sudo chroot /mnt cat << EOF > /etc/fstab /dev/mapper/encfs / ext4 defaults 0 1 /dev/xvda1 /boot ext4 defaults,noatime 0 0 EOF apt-get -y install openssh-server cryptsetup-bin apt-get -y install linux-image-virtual apt-get -y purge grub2 grub-pc grub-common cd /boot kernel=$(ls -1 vmlinuz-* | head -1) initrd=$(ls -1 initrd.img-* | head -1) mkdir /boot/grub cat << EOF > /boot/grub/menu.lst timeout 0 hiddenmenu title Linux root (hd0,0) kernel /$kernel root=/dev/mapper/encfs ro console=hvc0 initrd /$initrd EOF # add kernel modules required by cryptsetup cat << EOF >> /etc/initramfs-tools/modules dm_crypt sha256 aes_x86_64 EOF cat << EOF > /etc/initramfs-tools/hooks/crypt #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case \$1 in prereqs) prereqs exit 0 ;; esac if [ ! -x /sbin/cryptsetup ]; then exit 0 fi . /usr/share/initramfs-tools/hook-functions copy_exec /sbin/cryptsetup /sbin copy_exec /usr/bin/head /bin EOF chmod +x /etc/initramfs-tools/hooks/crypt cat << EOF > /etc/initramfs-tools/scripts/local-top/crypt #!/bin/sh PREREQ="udev" prereqs() { echo "$PREREQ" } case \$1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac # create a random 256 bit key and use it for encryption head -c32 /dev/urandom | cryptsetup -c aes-cbc-essiv:sha256 --key-file - create encfs /dev/xvda2 # copy base OS install from unencrypted to encrypted volume dd if=/dev/xvdb of=/dev/mapper/encfs bs=1024k EOF chmod +x /etc/initramfs-tools/scripts/local-top/crypt update-initramfs -k all -c ec2reg -O KEY -W SECRET --kernel aki-919dcaf8 -a x86_64 -n encrypting-ami --root-device-name /dev/xvda -b "/dev/xvda=SDA_SNAPSHOT" -b "/dev/xvdb=BASE_OS_SNAPSHOT"

Following are the code snippets which extend Solution 1 to Solution 2

openssl genrsa -out server-sk.pem 2048 # ami-sk is placed in OS install openssl genrsa -out ami-sk.pem 2048 # server-pk is placed in OS install openssl rsa -in server-sk.pem -pubout -out server-pk.pem openssl rsa -in ami-sk.pem -pubout -out ami-pk.pem # Key generation on keyserver end head -c32 /dev/urandom > message # encrypt the key using RSA. Note that message has to be less than 2048 bits for this openssl rsautl -encrypt -inkey ami-pk.pem -pubin -in message > cipher # using encrypt-than-mac to provide Authenticated Encryption sha256sum -b cipher | head -c64 > hash # signing the hash using server secret key openssl rsautl -sign -inkey server-sk.pem -in hash > signature # key verification on instance end signedHash=$(echo $signature | base64 -d | openssl rsautl -inkey /etc/server-pk.pem -pubin) foundHash=$(echo $cipher | sha256sum -b | head -c64) message=$(echo $cipher | base64 -d | openssl rsautl -decrypt -inkey /etc/ami-sk.pem) # Some changes to initramfs scripts to pack openssl & base64 cat << EOF > /etc/initramfs-tools/hooks/crypt #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case \$1 in prereqs) prereqs exit 0 ;; esac if [ ! -x /sbin/cryptsetup ]; then exit 0 fi . /usr/share/initramfs-tools/hook-functions copy_exec /sbin/cryptsetup /sbin copy_exec /usr/bin/base64 /bin copy_exec /usr/bin/head /bin copy_exec /usr/bin/openssl /bin copy_exec /usr/bin/sha256sum /bin copy_exec /root/ami-sk.pem /etc copy_exec /root/server-pk.pem /etc EOF

#encryption #cloud computing

Path to Cloud Computing

Cloud Computing promises many attractive things to companies. It's the buzzword of the decade and attracts a considerable investment in hope for future returns. However, there is a lot of chaos in industry where every vendor is promising a turnkey cloud solution. The pot of gold lies at the intersection of 3 roads

Software defined network

Infrastructure provisioning service

Resilient Application design

To realize the benefits of cloud computing, a company has to invest in all 3 areas. Most large companies have opened their coffers to vendors who promise instant cloud. Organizations often fail to realize that cloud computing is nothing but a resilient distributed architecture supported by agile infrastructure and network. Cloud Computing benefits can't be realized without (re)building an application in a resilient fashion. Organizations look for turnkey solutions which can turn their rigid application architectures into cloud applications, a magic which is not possible.

Small companies/startups on the other hand start with a blank slate when it comes to application design. They are able to buy Infrastructure provisioning service & Software defined network as a package when they use Off premise cloud. Small companies are able to (re)design their software using the best practices suggested by providers like AWS and reap the benefits of Cloud Computing.

For an organization trying to build on-premise Cloud computing, here are some milestones along the way

Software defined network

Dynamic Firewall - Application servers will change often for a dynamic distributed application. The firewall needs to adapt to allow new servers and remove old servers.

Dynamic DNS - DNS updates is critical part of certain application failovers, moving traffic to another DC (when needed)

Floating IP - Floating IP allows a dynamic application to heal itself in situations where traffic can't be load balanced.

Infrastructure Provisioning service

Virtualization - VmWare, Xen, LXC (Linux Containers). This is just one of the milestones to cloud and not an end goal. Vendors would love to convince you otherwise.

Load Balancer - Ability to scale application without adding more IPs to DNS records. (since DNS updates are slow to propagate)

Resilient Application Design

Horizontally Scalable - Make the application scalable while reducing bottlenecks as they are identified.

Reconfigure - Adapt to the dynamic nature of cloud. Application should reconfigure itself based on health of other servers. Try Apache Zookeeper.

Dynamic LB pool - Health checks to weed out bad application servers. Add/Remove servers to match load

Configure - Configure new servers to install packages/deploy application before adding them to LB pool. Try Chef/Puppet.

#cloud #cloud computing

Presentation on AWS VPC technology

I recently prepared slides for a presentation on AWS VPC technology. Attached is the presentation which I prepared.

#aws #vpc #sdn

Create AMI/Image from scratch for EC2/Xen

This blog captures the steps required to create an image from scratch which can be used on Xen virtualization platform using PvGrub boot manager. The later half of the blog also highlights steps which can be used to convert this image into an EC2 AMI and can then be used to boot an EC2 instance.

The instructions below have been tested on Ubuntu 13.10 and builds a Ubuntu 13.10 image/AMI.

1) Create image file and mount it

# creating 1GB image here # image size should be bigger than disk required for AMI # size of root device is chosen when launching instance and not here dd if=/dev/zero of=linux.img bs=1M count=1024 sudo losetup /dev/loop0 linux.img sudo mkfs.ext4 /dev/loop0 sudo mount /dev/loop0 /mnt

2) Install base system

sudo apt-get -y install debootstrap # Installing 64-bit Ubuntu saucy (13.10) here # Modify for your use case sudo debootstrap --arch=amd64 saucy /mnt

3) Chroot into new installed system to configure it

sudo chroot /mnt

4) Configure basic system

mount none /proc -t proc mount none /sys -t sysfs # Adding root mount point cat << EOF > /etc/fstab /dev/xvda1 / ext4 defaults 0 1 EOF # Adding saucy specific apt sources here cat << EOF > /etc/apt/sources.list deb http://us.archive.ubuntu.com/ubuntu/ saucy main deb-src http://us.archive.ubuntu.com/ubuntu/ saucy main deb http://us.archive.ubuntu.com/ubuntu/ saucy-updates main deb-src http://us.archive.ubuntu.com/ubuntu/ saucy-updates main deb http://security.ubuntu.com/ubuntu saucy-security main deb-src http://security.ubuntu.com/ubuntu saucy-security main EOF # setting eth0 for dhcp cat << EOF >> /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp EOF apt-get update # installing ssh-server for headless setup apt-get -y install openssh-server

5) Setup kernel and system to boot - This was the most tricky part of the setup to learn/find out/get right. PvGrub essentially reads /boot/grub/menu.lst file and find the kernel and initrd information from the file. There is no need to create boot record or boot sectors or anything similar when booting in Xen with PvGrub manager. However, one does have control over what kernel will be running and a kernel needs to be installed.

# install linux kernel # don't install grub on any disk/device when prompted # commands requires manual prompt, haven't scripted next step apt-get -y install linux-image-virtual # remove grub which was installed during kernel install # choose to remove grub from /boot/grub when prompted # commands requires manual prompt, haven't scripted next step apt-get -y purge grub2 grub-pc grub-common # install grub-legacy-ec2 which is NOT a ec2 specific package # this package creates /boot/grub/menu.lst file # this package applies to all PvGrub guests, even outside ec2 apt-get -y install grub-legacy-ec2

6) Do custom configuration

# set up root password passwd # or create users with sudo access

7) unmount filesystems & loop devices

# exit out of chroot /mnt exit sudo umount /mnt/sys sudo umount /mnt/proc # if umount fails, you might have to force it by umount -l # umount will fail if any daemon processes were started in chroot sudo umount /mnt # losetup will fail till daemon process are still running # kill any chroot daemon processes if needed sudo losetup -d /dev/loop0

Your new linux.img file is ready for use. The next set of commands are specific to converting this image file into an AMI which can be used on EC2.

a) Transfer the image to a running EC2 instance

b) Mount a new EBS volume on the EC2 instance. Size of the EBS volume should be greater than size of image file chosen at step (1).

c) Copy image file contents into EBS volume

# assuming that the empty EBS drive is on /dev/xvdf sudo dd if=linux.img of=/dev/xvdf bs=1M

All the steps listed below can also be done in AWS console

d) Create a snapshot of the EC2 instance

ec2addsnap -O AWS_KEY -W AWS_SECRET -d `date +"%Y%m%d%H%M%Z"`

e) Register a new AMI from the snapshot

# Find the latest PvGrub kernel offered by Amazon # hd0 and hd00 are the same thing # choose 32-bit kernel or 64 bit kernel based on image # kernel-id looks like "aki-919dcaf8" (1.04-x86_64) ec2-describe-images -O AWS_KEY -W AWS_SECRET -o amazon --filter "name=pv-grub-*.gz" # registering a 64 bit AMI here ec2reg -O AWS_KEY -W AWS_SECRET --kernel KERNEL_ID -a x86_64 -n `date +"%Y%m%d%H%M%Z"` -b "/dev/sda1=SNAPSHOT_ID"

f) Test your new AMI by starting an instance with it

ec2run -O AWS_KEY -W AWS_SECRET -g VPC_SECURITY_GROUP -t INSTANCE_SIZE -s VPC_SUBNET_ID NEW_AMI_ID

#xen #pvgrub #ami #image #virtualization #ec2 #ebs

Secure data at rest with encryption

Encryption is critical to protecting data. Data should be encrypted both at rest and in transit. Data in transit can be encrypted using TLS/HTTPS. This blog talks about storing data securely using encryption. The solution listed below uses dm-crypt kernel module.

1) First, we will randomize the data on the disk to ensure that if someone gets access to the actual disk, is unable to determine the length/size of the encrypted data. The disk sectors can be randomized by overwriting them with random data. This step is important and shouldn't be overlooked. Without randomization of device blocks, someone can assert that the disk is encrypted. Once the disk sectors have been randomized, it gives you plausible deniable encryption.

Your disk might have been initialized with a specific pattern before you got it (say with zeros). Once your start using the disk, you disk will look like this

However, if you randomize the disk space before storing data, no-one will be able to tell the difference between encrypted/filled space and randomized/empty space.

dd if=/dev/urandom of=/dev/sdb bs=1M

2) First decision related to encryption is the selection of the algorithm and the key length. For this setup, we will use AES with 256 bit key.

In the next step, we are generating a 256 bit (32 bytes) key from /dev/random. We are using /dev/random instead of /dev/unrandom for more security. However, on headless servers, random data generation can be slow and hence the next command may take some time.

head -c 32 /dev/random > 256bitKey

The key generated here should be kept secure, safe and away from the computer where the encrypted disk is mounted. If this key is lost, there is no way to recover the data stored on the encrypted disk.

3) We will use the key generated and create a mapped device such that data is encrypted when stored on /dev/sdb but decrypted when viewed from /dev/mapper/secretFs

sudo cryptsetup -c aes-cbc-essiv:sha256 --key-file 256bitKey create secretFs /dev/sdb

The mapped device can be used like any disk device and can be used directly or as part of LVM etc.

4) Let's format the device and create an ext4 filesystem on it

sudo mkfs.ext4 /dev/mapper/secretFs

5) The formatted disk can now be mounted so that the applications can write data to the disk

sudo mount /dev/mapper/secretFs /mnt/secure

Summary

Here are the consolidated steps to prepare a new disk, set up encryption and mount it

dd if=/dev/urandom of=/dev/sdb bs=1M head -c 32 /dev/random > 256bitKey sudo cryptsetup -c aes-cbc-essiv:sha256 --key-file 256bitKey create secretFs /dev/sdb sudo mkfs.ext4 /dev/mapper/secretFs sudo mount /dev/mapper/secretFs /mnt/secure

If you reboot the server or unmount the disk, the disk can be re-mounted using the following steps

sudo cryptsetup -c aes-cbc-essiv:sha256 --key-file 256bitKey create secretFs /dev/sdb sudo mount /dev/mapper/secretFs /mnt/secure

#encryption

SSH Host Identification and Verification

When connecting to a SSH server, you would have often come across a prompt which asks you to confirm the host fingerprint before connecting. Many of the users might have picked up a bad habit of saying yes without understanding the implications of the decision. This blog is to highlight how ssh setup can be strengthened to include host identification (similar to HTTPS certificate signing). The benefit of such a setup is that the client can be confident that it is connecting to the right server and not becoming victim of MITM (man in the middle) attack.

Often when connecting to a SSH server, you would see a prompt like the following

user@client:~$ ssh user@remoteserver The authenticity of host 'remoteserver (10.20.42.54)' can't be established. ECDSA key fingerprint is dd:30:96:8a:46:78:76:0a:53:7d:9d:0d:23:d6:89:ce. Are you sure you want to continue connecting (yes/no)?

The presence of the prompt indicates the client can't confirm the authenticity of the host. This problem can be solved by creating a certificate authority, which will sign the host keys (similar to CA concept in HTTPS). A certificate authority can be created for ssh using the following

user@ca-server:~$ ssh-keygen -f ca

The step above will create 2 files, ca and ca.pub. Contents of file ca are private and should be kept secret. The contents of ca.pub are public and can be distributed freely.

As a part of creating new server, each server should have a unique set of host identification keys. They keys are usually automatically generated as a part of install. If not, such keys can be generated using the following steps.

root@ssh-server:~$ ssh-keygen -t rsa -q -N '' -f /etc/ssh/ssh_host_rsa_key root@ssh-server:~$ ssh-keygen -t dsa -q -N '' -f /etc/ssh/ssh_host_dsa_key root@ssh-server:~$ ssh-keygen -t ecdsa -q -N '' -f /etc/ssh/ssh_host_ecdsa_key

The above step generates private/public key pair for rsa, dsa and ecdsa algorithms. The public host keys can be copied over to ca-server for signing. ca-server is the server where ca private key is stored securely. The CA can sign the host keys using ca private key with the following command

user@ca-server:~$ ssh-keygen -s ca -I "remoteserver" -n remoteserver,remoteserver.domain,10.20.42.54 -h ssh_host_rsa_key.pub ssh_host_dsa_key.pub ssh_host_ecdsa_key.pub

The command is signing the host to be able to represent server remoteserver or remoteserver.domain or 10.20.42.54. The certificate will not match if this certificate is presented for any other hostname. The above command will generate files *-cert.pub which should be copied back to the ssh server. Further sshd configuration (usually /etc/ssh/sshd_config) should be modified to present HostCertificates to client during ssh handshake. SSH configuration file on ssh server looks like the following

user@ssh-server:~$ cat /etc/ssh/sshd_config # ssh daemon configuration HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_dsa_key HostKey /etc/ssh/ssh_host_ecdsa_key # new lines added HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub HostCertificate /etc/ssh/ssh_host_dsa_key-cert.pub HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub # further SSH configuration follows ...

Now the ssh server has its keys signed by CA. In the case of HTTPS, we have certain root certs which the browser trusts by default. However, there are no root certs in case of SSH. SSH client needs to explicitly configured to trust the CA. All ssh clients can be configured to trust the CA by putting the CA public key in SSH known hosts (/etc/ssh/ssh_known_hosts) configuration file

user@client:~$ cat /etc/ssh/ssh_known_hosts @cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCm+Bdq1eYGvddoWPRmJ43id7MioLeyRlOPNIJeuScHGMQro6jUYU4JyKx9dpKlQrZmn+hZeDxgx4fbxAQFKfdfgaLrFX3N06dR8uAFk7g+oimNJITWnaUgOuHGJrGEKIpNUqeLboOXm5aaYkiCH1ixx4r8hVIT4J+OM66oUZZmYTwWmxkxjj2Cu+Iuil7rpNzhjz9IVEzJrQA0KdpnfGQqv2KuaAhCCq6reZMoutE60HBX1Cww7Y3O26psp2AnL+xV5BzfhWYEdt98+Bz+WR/3Mt2u3NSv/ABwHZD3qseRFcWXnJGj9PbUAWAO6klMDqk9ok1nlmT0FjLbNk/R/gfh

Once done with all the steps above, you should be able to ssh from client machine to any server without facing the ssh host identification warning. The client trusts the CA and trusts the cert presented by the host when the cert is signed by CA and the cert name matches the hostname client is trying to connect to.

Like with any CA, SSH also has provision to revoke host keys when needed. However, there is no provision for a central revocation list (like in HTTPS). Such revocation information needs to be present in all client machines. The following snippet shows how to revoke public key of a server in a client.

#ssh #sshd_config #ssh_host_key

Secure SSH with Multi Factor authentication

Logging into a machine is essentially about proving your identity to the server. The human logging in needs to prove to the server that he/she is the holder of the account the session is trying to access.

Identity in general can be proved using the following methods

Something you have (like hardware token device, access card, public/private keys)

Something you know (like password, social security number, birth date)

Something you are (physically defines you, like fingerprint)

Most of the linux servers I deal with are headless and don't really have a bio-metric device attached to them to assert something you are. So, I will be focusing on the first 2 assertions.

Multi factor authentication essentially means asking user to verify his/her identity using two different methods like something you have and something you know.

The rest of the blog focuses on enabling various security mechanisms in a Linux server

a) Public/Private Key Authentication

Public key authentication falls in something you have but they have a weakness. Something you have should be tamper proof and hard to copy. However, private key files can be copied and can often be done without the user knowing that his/her key has been copied away. However, they offer better protection than password in attacks since we typically 2048 bit keys.

A private key can be generated easily using the following linux command

ssh-keygen -f file.rsa -t rsa -b 2048 -q -N ''

The command will produce a 2048 bit private RSA key and store that in a text file (file.rsa here). The private key can be potentially protected with a passphrase by either specifying it at command (-N '<passphrase>') or dropping -N parameter and specifying the passphrase as input in the tty session. The file created above is the private key and must be kept secure. The corresponding public key can be generated using

ssh-keygen -y -f file.rsa

The output is the public key which can be distributed freely. The public key is added/appended to ~<user>/.ssh/authorized_keys file on the server. This directs SSH server to accept the person in possession of the private key to log in as <user>.

Public Key Authentication can be enabled in a linux server by ensuring that the following is present in /etc/ssh/sshd_config

PubkeyAuthentication yes

b) Password Authentication

Passwords are the oldest security mechanism. However, people choose simple passwords. Even a random 8 character case sensitive alphanumeric (a-zA-Z0-9) password has only 48 bits of entropy which isn't too hard (1 week) for a botnet to crack. However, most of us don't use random passwords and easy to remember passwords have much less entropy.

Standard password can be enabled in linux server by making sure that sshd config (/etc/ssh/sshd_config) contains the following

PasswordAuthentication yes

c) TOTP Authentication

A better something you have example is a TOTP hardware device. A phone running TOTP app (like Google Authenticator) is not 100% tamper proof if running on a rooted Android device or if the underlying OS has security related bugs where other application might be able to steal the secret. A hardware key is more tamper proof and is a better choice but phone running TOTP app is also good.

Time based One Time Password (TOTP) algorithm generates a rolling password every X seconds (default 30). The password is based on 2 factors - a secret key which is known to both server and user and time. Given that time is a factor, it is critical that both server and client are using a synchronized clock.

While TOTP is a good something you have example, it has some flaws when it comes to a company wide deployment a) The algorithm uses a shared secret. This means that the server knows the secret and hence it must be guarded on the server. Anyone who gains root access to the server can find out the secret for all the other users on the system. b) The one time password is good for X seconds (default 30). This makes a man in the middle (MITM) attack easy. Anyone who intercepts or sees this password can use it for the next 30 seconds.

The flaws can be fixed by having a central TOTP authentication server which all the servers connect for TOTP validation. The central server can prevent MITM attack by enabling only login per verification code. This however limits the user to be able to log into a single server per X seconds.

TOTP authentication can be enabled on a linux server by running the following commands

# install pam module developed by google # which enables TOTP validation apt-get install libpam-google-authenticator # Run the google authenticator command to create new secret google-authenticator # copy the QR code or secret to a smartphone running a TOTP app # (like Google Authenticator)

Further, ssh pam configuration (/etc/pam.d/sshd) should be modified to add the following line

auth required pam_google_authenticator.so

This will require user to enter TOTP token (something you have) along with password (something you know)

Enabling combination of these methods

Openssh 6.2 added a new feature where user can be required to pass multiple validations before successful login. Here are some scenarios with the configuration

1) PublicKey + TOTP

Its not a desirable configuration as both public key and TOTP fall under something you have. So, this is not a multi factor authentication. Imagine a user storing both of these on a phone and loses the phone. This configuration can be enabled by doing the following

Add the following line to sshd config (/etc/ssh/sshd_config)

AuthenticationMethods publickey,keyboard-interactive

The configuration above requires user to pass public key check and keyboard-interactive check. keyboard-interactive check passes the control to pam module. SSH pam configuration file should have the following changes

# Require TOTP auth required pam_google_authenticator.so

Default pam ssh configuration requires a password. That can be disabled by removing/commenting out the line indicated below

# @include common-auth

2) PublicKey + Password

This is a good configuration since it mixes something you have and something you know. This can be enabled by adding the following to sshd config

AuthenticationMethods publickey,password

AuthenticationMethods publickey,keyboard-interactive

password authentication method is handled by sshd itself. While keyboard-interactive is handled by pam. So, in a default setting, they both appear to be the same to user but they are different under the surface. keyboard-interactive enables complex mechanisms possible via pam module. If the intention is use just password along with public key, its desirable to use "AuthenticationMethods publickey,password"

3) PublicKey + Password + TOTP

All 3 authentication methods listed here can be enabled by adding the following line to sshd config file

AuthenticationMethods publickey,keyboard-interactive

Further, pam ssh module should be modified to require totp code like mentioned in (c) above.

#sshd_config #sshd #totp #security

Running a process inside a network namespace

I have been reading and playing with Linux containers. I previously covered cgroups which enables process group level cpu/memory allocation. This blog entry is about my understanding of network namespace. And about running a process inside an isolated network namespace. Running a process there allows us to set specific filters on the specific process.

Network namespace allows Linux to clone the network stack and make the new stack available to a limited set of processes. This is used primarily with Linux containers such that each container has a different network stack altogether. There are multiple options for adding network interfaces to a newly created network namespace

veth

venet

vlan

Out of the three options above, I haven't been able to find a lot about venet. I think venet is part of OpenVZ kernel changes and is not available in mainstream linux kernel.

A new network namspace can be created using the following command

ip netns add myspace

Now, we will create a new pair of type veth network interfaces. veth interfaces come in pair and act like a pipe of data. Each packet sent to veth0 shows up at veth1 and each packet to veth1 shows up at veth0.

ip link add veth0 type veth peer name veth1

Now, let's move veth1 to our newly created namespace

ip link set veth1 netns myspace

Now, we will bring up veth0 (in original namespace) and assign IP address and subnet to it

ifconfig veth0 192.168.42.1 netmask 255.255.255.0 up

Assigning ip address and netmask to veth0 (inside myspace namespace)

ip netns exec myspace ifconfig veth1 192.168.42.2 netmask 255.255.255.0 up

The command above is important to look at again. The format of the above command is ip netns exec myspace <command>. The <command> executed here will be running in the myspace network namespace. The command will only see the interfaces, the route table configured inside the myspace network namespace.

Setting up gateway for myspace namespace

ip netns exec myspace route add default gw 192.168.42.1

Now, we have a namespace myspace which can send network packets to veth0 which will reach host at veth1 interface. Here, we can multiple options to connect veth1 to the outside world (via eth0).

Bridging

NAT

I will be covering NAT setup here. NAT can be enabled by running the following commands

# Enable kernel to forward packets from one interface to another (veth0 <-> eth0) echo 1 > /proc/sys/net/ipv4/ip_forward # Each packet coming via 192.168.42.* address space should be sent # via eth0 after changing the source ip to eth0 ip. iptables -t nat -A POSTROUTING -s 192.168.42.0/24 -o eth0 -j MASQUERADE

Now, our network namespace is ready for use. A process can be run inside the network namespace by using the following

ip netns exec myspace <command>

The command will run inside the network namespace. If you want to run the command inside network namespace as an unprivileged user, use the following

# first sudo is optional and only needed if running this command as non-root sudo ip netns exec myspace sudo -u <user> <command>

This way, all the traffic generated by the process will be inside the network namespace myspace. The traffic can be inspected and accepted or dropped in the parent namespace by using the filter table, forward chain.

#os virtualization #network namespaces

Is your application Cloud Ready?

Cloud computing delivers a quick to market development platform which application teams can use to translate business requirements into applications faster than ever. However, the application needs to have certain features to be able to benefit from such a development platform.

Cloud enabled Architecture - The application should be able to distribute workload across multiple workers, be able to scale out by adding more resources, have no single point of failure, be able to recover from certain infrastructure failure scenarios like disk failure, datacenter failure, network degradation, database failure. This enables the application to be resilient and scale out when required.

Automated Configuration Management - The server configuration required to host the application should be automated to the last mile. This enables the infrastructure provider to patch the Operating System, move application across shared servers in a reliable fashion.

Automated deployment - The application should provide an automated deployment workflow for each release. Ideally, each deployment should have an automated rollback workflow defined too.

Support shutdown, start, moving - The application should be able to take in commands/signals/notifications when the application is required to shut down, start on the existing server or move to a new server. This enables the infrastructure provider to reduce servers, add servers, move servers when required.

Automated Build - The application should work with automated build tools and should be enabled to use automated build on check-in, nightly etc. This enables a quick turnaround for a developer to test their changes, makes the build process clean.

Automated QA - The application should have automated test cases which are run after every build. The regression results should be notified to developer making the changes so that either the code or the test cases can be fixed. This increases the speed of development while keeping quality high. It also avoids last minute QA surprises at the end of the sprint.

Support health checks - Health checks are important to make sure that the service is working as intended. Health checks can be at machine level, at load balancer, at cluster level and even at a datacenter level. Health checks are required to make decisions for self healing infrastructure.

Metrics & Monitoring - You can't improve what you don't measure. Whether its uptime, average response time, average failure rate, average pageload time, the metrics should be measured and observed consistently. Obsession with metrics will help us deliver better products with each sprint/release. Metrics should be monitored to observe any problems which would need either automated self healing or manual fixing.

#cloud computing

There is an API for that

The API revolution is upon us. Many new age companies are embracing APIs and enabling ecosystem of apps/plugins/enhancement made by partners using the APIs. While there are benefits in exposing APIs to external customers, there are numerous benefits to exposing APIs to internal customers as well. Services Oriented Architecture (SOA) is a well understood design pattern. Some of the benefits of SOA design pattern include

Services can be measured. What can be measured can be improved. SLAs on uptime and response times can be defined which service must adhere to.

Services can be vendor independent. Services can hide the vendor complexity behind the scenes.

Services are reusable. Reusable services foster better integration between teams and products.

Ideally, (almost) every technical team should be delivering their product as a web service. There should be a service catalog where all services in the organization can be discovered. Each service should be accompanied with documentation, sample client code and maybe even some client libraries. The services should have well defined SLAs and the performance graphs should be transparent to the organization. Services should have throttling to protect themselves about getting abused by a single (internal) client. However, it should be noticed that there should not be too many services as well. Ideally, each service should be supported by at least 8-10 person team to ensure that there is business continuity in the team.

Going for a services oriented architecture also enables teams to automate their processes and tasks. Human based processes are error prone and susceptible to failure to follow steps correctly. Automation enables repeatable results and quick turnaround on requests. If each team were to expose their product as a web service, the clients can invoke the business flow at any time of the day without scheduling a meeting or filling a long form to get attention to another team.

Here are some rules to consider adapting when looking at Services Oriented Architecture across the organization

API is more important than UI. Each product to be procured should be required to have an API interface. UI matters but less than API.

API should support client authentication, client throttling.

APIs should be used for all technical teams including infrastructure.

Teams should publish SLAs for their APIs and ensure that their APIs meet or exceed their SLA

We should be working towards a goal, where for most of the tasks inside the organization, we can say "There is an API for that".

#software development #api

Intro to websockets

Websocket is an application level protocol providing full duplex communication over a single TCP connection. Websocket is related to HTTP as it uses a HTTP/1.1 compliant handshake. The protocol was primarily designed to enable a continuous interaction between browsers and servers which is not natively supported in HTTP/1.0 or HTTP/1.1 specifications. HTTP/2.0 specification which is currently in progress will enable request/response pipelining and will natively solve the need which led to Websockets development.

To understand the need for websocket, one needs to understand the requirement first. The requirement primarily arises from a chatty client which wants to get regular updates from web server (like a stock ticker) and/or wants to regular messages to web server (like chat). HTTP request/response model enables clients to send messages to servers, but doesn't make it easy for servers to send messages back to client. Comet solution was devised as a work around where client makes a call to the server and the server just holds the request till it has something to say or till a predefined time passes.

Websocket support is inbuilt into current browsers and that enables developers to write applications where data can be streamed easily between browser and web server in both directions. Given that websockets uses HTTP/1.1 upgrade header, they should work nicely with HTTP/1.1 compliant proxy servers. However, for greater success, one should use websockets over HTTPS, since that solution works with HTTP/1.0 proxy servers as well. Many client and server libaries allows application to fall back on Comet/Request-Response model in case websocket connection attempt is not successful or not supported due to a overzealous proxy server or unsupported browser.

Websockets specification enables ping/pong frames which can be used for keeping the underlying TCP connection alive, allow both text and binary frames which enables using websockets for both text (say json) and binary data transfer. Various load balancers like Haproxy, F5 have inbuilt support for Websockets which enables the application team to load balance websocket connections as well. However, AWS Elastic load balancer (ELB) doesn't support Websockets as of today.

#websocket

NoSQL Landscape

NoSQL like Cloud Computing is a jargon term with no specific meaning. The industry can't agree upon whether NoSQL stands for "No SQL" or "Not only SQL". SQL database traditionally have been great at consistency and availability but have performed poorly at partition tolerance. SQL solutions have traditionally been slow and nosql effort is essentially providing speed and scalability at the cost of consistency/referential integrity. CAP Theorem proved that no system can be consistent, available and partition tolerant at the same time. Most nosql solutions start with being partition tolerant and provide tunable consistency and availability depending on application needs. Further, many nosql solutions provides enough low latency that caching layer can be dropped off from the architecture providing lower cost of development and operations.

Since the popularity of NoSQL, SQL databases have also made effort to improve their latency. MySQL in version 5.6 release APIs to access InnoDB storage engine directly bypassing SQL interface which provides low latency reads and writes.

Here is an introduction to some of the popular NoSQL solutions in the market. The list below doesn't capture all of them as there are way too many solutions coming up. Each solution provides different features and interface since there is no standardization on the nosql front.

Redis - Redis is a fast caching solution which can store data structures like string, list, hash, sorted set. It allows operations like set union, interesection etc which makes is win over traditional key-value caches. Redis doesn't have cluster capability as of now (work in progress). Redis can be used on multiple nodes using proxy solution like Twemproxy. Redis allows taking snapshots of data in cache on disk. Redis has some (basic) disk persistence options as well which enables one to use Redis as a database as well.

Couchbase - Couchbase at a very high level is Membase (for caching) + CouchDB (for persistence). Couchbase has a pretty interesting architecture as it combines in-memory caching and database in a single product. Couchbase has 2 distinct modes - memcache compatible and couchbase mode. Memcache compatible doesn't have all the features and should be avoided in favor of couchbase mode. Couchbase claims to provide consistent reads even in case of failure but the implementation falls short in practical tests. Couchbase with its cache+disk model gives very low latency. The solution can give high write throughput when write is appended to in-memory queue instead of pushed synchronously to disk. The solution ideally should scale well with number of nodes, but practical tests show performance degradation with high number of nodes. Couchbase has the potential to become a very strong NoSQL solution. But its implementation as of today falls behind the promises made by the design architecture.

Cassandra - Cassandra uses a append only architecture which is crash safe. Cassandra is optimized for writes and delivers a predictable write throughput. Cassandra scales very well horizontally. Cassandra read latency is not as low as Couchbase but is fast enough. Cassandra is eventually consistent database and provides tunable consistency for read requests.

DynamoDB - DynamoDB is database-as-a-service provided by AWS. DynamoDB is eventually consistent. Single key consistent reads are available at higher cost. DynamoDB is highly available (across multiple zones in a single region) and can be scaled up or down using AWS APIs. The database works at low latency with scalable throughput. Given its pricing and features, it gives the lowest total cost of ownership. Find more details about DynamoDB architecture in a research paper.

HBase - HBase is a strongly consistent database and is optimized for reads. HBase is part of Hadoop family and is ideal for storing large amount of sparse data. The solution is not as easy to administer as Cassandra or Couchbase. Its doesn't work well when used for small amount of data or on small number of nodes.

MongoDB - MongoDB was the first popular NoSQL solution. The product became quite popular among early adopters as they compared the MongoDB solution to the SQL world they were migrating from. However, MongoDB is not the best NoSQL solution out in the market in terms of scalability, partitioning, server usage. If you want to follow the herd, choose this solution. If you can think for yourself, look at other solutions.

Here is a very basic dumbed down flowchart to help you choose your next database. Use this flowchart only to guide you to the best possible option. Database migration is costly. So read a lot more, test and benchmark before making the final selection.

[Click on image to see non compressed version]

#nosql #database

Automated Configuration - Blueprint, Chef, Puppet

Configuration of a server is the step which ensures that the server is ready for hosting the application. Configuration of server takes place after provisioning and before deployment of the application. The server lifecycle looks like the following.

In real life, a server once provisioned lives for a long time and sees multiple reconfigurations and redeployments. So, the server lifecycle actually looks more like

Configuring a server involves setting up the server in the best possible manner. This includes workflows like creating optimal apache configuration, setting up best mysql settings, setting up correct sudo permissions, logging formats, log rotations etc. However, best/optimal/standard configuration is a living beast which evolves over time.

When using a script based approach to configuration ie divergent configuration, the configuration drift is high. Older servers are farther away from standard configuration than newer servers. Its pretty difficult to keep the servers in sync with the standard configuration.

However, when using a configuration management tool like Puppet/Chef, it is easier to keep server configuration in sync with optimal standard. The configuration drift still happens, but it is limited. Configuration management tool can also monitor a limited set of files and directories. Any manual changes made outside the scope of managed resources remain as a drift from enterprise standard. Sometimes, a server might be pulled out of configuration management solution (like server 3 in diagram) which leads to large configuration drift like in divergent configuration solutions.

Using Puppet or Chef as a configuration manager requires setting up a central server which maintains the configuration. Both Chef and Puppet are similar in the following manners

They use state based approach where the client takes the node into a desired final configuration

They monitor a set of resources continuously.

They can't do any orchestration which includes multiple machines.

They can be used both for initial provisioning as well as configuration change propagation.

Both Puppet and Chef are limited in following ways

They both lack a good high availability story. Chef is a little better than Puppet here as it provides a failover solution

They both have limited amount of documentation of administration of the central servers.

They run all commands as root on agent. They don't drop privileges to support running commands as different user based on author/class/recipe.

Chef has the following advantages over Puppet

API integration - Chef Server exposes API which makes it easier to integrate Chef with other products in a workflow

Higher scalability - Chef runs all recipe on nodes. Chef can split front end and back end into different server providing higher scalability and nodes per server.

As a DevOps team matures, their choice of configuration solution looks like the following

I would like to end the blog quoting DevopsBorat tweet

To make error is human. To propagate error to all server in automatic way is devops.

#puppet #chef #configuration

Trending Blogs

Recently Viewed Blogs

Tech Talk