Bootstrapping WebSphere Liberty in Docker with confd
I've been working with WebSphere Liberty inside Docker for some time now. When I first started, there was no official Liberty Docker image. I had a fairly simple solution to dockerize Liberty, but now there is an official repository websphere-liberty. As a side note, I usually preferable to use a managed repo so I can focus on the primary goal of building working software. As an application developer, I don't want to worry about patches and fixpacks. I would like to focus as much as possible on development.
As nice as it is to have an official WebSphere Liberty image, it still has its warts. This article describes the limitations I've run into and my solution to those limitations. I'd love any and all feedback the community can give.
When I read through the instructions for WebSphere Liberty, I see there are several approaches they give. One is to add the war file to the dropins directory and run it that way. Another is to mount an entire directory that contains your application and configuration. Both solutions are fine, but don't allow for good, flexible environment handling.
WebSphere Liberty depends on several configuration files. Most are optional and have sensible defaults; however, in many cases there is need to update those defaults. Let's look quickly at the basic configuration files Liberty requires.
The application will need to run under a directory in /opt/ibm/wlp/usr/servers/yourserver. I won't get into the nitty gritty of each of the relevant files, but let me provide a quick summary of why anyone might want to touch any one of these.
This file contains the default port numbers along with customizing WebSphere's trace files.
Any -X or -D option passed to the java command, is specified in this file. To increase memory sizes, then specify the desired options in this file.
This contains any environment variable for the Java application to use. More on this later.
This file contains the definition of the application, the OSGI server features, references to key stores, and several other items.
Problem: Environment Variable Handling
Liberty's answer for environment variables is through the server.env file. It does not allow for system level environment variables such as an exported bash variable. The implication: A server.env file must be provided for each different deployment. Stop and think about this for a second. Is a project team to check in multiple server.env files, one for each environment or variation? How many combinations or variations would there eventually be? Are all these checked into source control? Are there passwords in there? Maybe WebSphere Liberty's approach technically satisfies factor number three, config, of The Twelve-Factor App, but it is too rigid in my opinion. I don't believe this approach is in the spirit of 12-factorness.
My Liberty Solution: confd
confd is a simple, yet powerful utility written in Go to write configuration files. It can technically write any kind of file, but its primary use case is configuration files.
Its design is very simple and only requires two files for each file to create.
Template File - Written using Golang text templates
Template Resource Configuration File - Declares the template file name, final file path, file permissions, expected env vars, etc. and is written in TOML format.
Let's look at some examples:
server.env: /etc/confd/conf.d/server.env.toml
[template] src = "server.env.tmpl" dest = "/webapp/server.env" mode = "0644" keys = [ "/db/url", "/db/user", "/db/pass", "/ldap/url", "/ldap/user", "/ldap/pass" ]
server.env: /etc/confd/templates/server.env.tmpl
database_url={{getv "/db/url"}} database_username={{getv "/db/user"}} database_password={{getv "/db/pass"}} ldap_url={{getv "/ldap/url"}} ldap_user={{getv "/ldap/user"}} ldap_password={{getv "/ldap/pass"}}
server.xml: /etc/confd/conf.d/server.xml.toml
[template] src = "server.xml" dest = "/webapp/server.xml" mode = "0644"
server.xml: /etc/confd/templates/server.xml
<server description="myapp"> <featureManager> <feature>servlet-3.0</feature> <feature>jsp-2.2</feature> <feature>localConnector-1.0</feature> <feature>ssl-1.0</feature> </featureManager> <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="${default.http.port}" httpsPort="${default.https.port}" /> <webApplication contextRoot="/myapp" id="myapp" location="/webapp/my-app.war" name="myapp" type="war"> <classloader apiTypeVisibility="spec, ibm-api, api, third-party" delegation="parentLast"/> </webApplication> <keyStore id="defaultKeyStore" password="{xor}Jh3qaafiLTs=" /> </server>
confd is very powerful. I use it for both configuration files that need variable substitution and configuration files that don't. Notice that there is no environment substitution in my server.xml file. I could opt to copy these files the docker way using COPY/ADD or volume mounting, but I like to keep things as consistent as possible. So since I am using confd for the other files that are all basically grouped together, I have chosen to use confd for all my configuration files. One benefit is if requirements change, my approach does not. For example, my jvm.options file was simple at first with no variable substitution. It didn't take long until I needed to pass in a Spring profile as a -D option. Since I was already using confd for all my files, all I had to do was make some minor updates to the template and the template resource config. Using the same approach for all files reduces long term maintenance because the mental model is simpler to understand - both now and in the future.
Another benefit of confd is its ability of using either environment variables or a configuration server. At the time of this writing, confd is compatible with both etcd and consul configuration servers. This makes deployment very flexible. I currently pass in environment variables for both local development and Jenkins; however, for production, I use etcd.
Lastly, although I've not used this feature, confd is also capable of monitoring the configuration server for changes and then running a command when it detects a change.
Docker, WebSphere Liberty & confd
I extended the official websphere-liberty Docker image with my own that includes confd. I named this mkboudreau/liberty-confd
That Docker image is automatically built on DockerHub when I push changes to my GitHub repo
For my project, I extend my mkboudreau/liberty-confd and bake in the confd files and call my own startup script following that is very similar to the one the websphere-liberty folks are using.
The Dockerfile for mkboudreau/liberty-confd
FROM websphere-liberty:8.5.5 MAINTAINER Michael Boudreau <[email protected]> ### INSTALL CONFD RUN wget https://github.com/kelseyhightower/confd/releases/download/v0.7.1/confd-0.7.1-linux-amd64 RUN mv confd-0.7.1-linux-amd64 /usr/local/bin/confd RUN chmod +x /usr/local/bin/confd RUN mkdir -p /etc/confd/conf.d /etc/confd/templates
Sample Application Dockerfile
FROM mkboudreau/liberty-confd RUN mkdir -p /opt/ibm/wlp/usr/servers/defaultServer/resources/security RUN ln -s /opt/ibm/wlp/usr/servers/defaultServer /webapp COPY liberty/confd /etc/confd COPY liberty/liberty-run /webapp/liberty-run COPY build/libs/my-war.war /webapp/my-war.war WORKDIR /webapp ENV LICENSE accept ENTRYPOINT ["./liberty-run"]
Sample Application Script for liberty-run
#!/bin/bash # # RUN CONFD # echo "--- RUNNING CONFD ---" if [ -n "${CONFIG_SERVER}" ]; then echo "Using config server at ${CONFIG_SERVER}" /usr/local/bin/confd \ -backend etcd \ -node ${CONFIG_SERVER} \ -onetime else echo "No config server... using current environment" /usr/local/bin/confd -backend env -onetime fi # # RUN LIBERTY # echo "--- STARTING LIBERTY ---" /opt/ibm/wlp/bin/server run defaultServer --clean
Using the above approach, the application's Dockerfile is checked into source control and Jenkins builds and tests the application each time there is a checkin. If it passes those steps, it proceeds to build the docker image and starts it up. If it builds and starts successfully (i.e. Spring context loads successfully), I tag the docker image and push it to a docker repo. Now any server, anywhere can simply issue docker run -e CONFIG_SERVER=http://my.config.server.com:4001 myimage:mynewtag and the application is running.
I think WebSphere Liberty is far better than full blown WebSphere from a DevOps perspective; however, Liberty is still limited due to their decision to use server.env for environment variables. Hopefully, this article will be obsolete soon and Liberty will handle variables the way it should. In the meantime, injecting confd into the official WebSphere Liberty docker image is a very simple, yet very powerful solution.
GitHib Repo for mkboudreau/liberty-confd
DockerHub Repo for mkboudreau/liberty-confd
GitHub Repo for WebSphere Liberty
DockerHub Repo for WebSphere Liberty
Announcing: Liberty profile on Docker Hub
WebSphere under Docker: from Development to Production
WebSphere Liberty Configuration Directories
