If you are familiar with Docker, and understand the concept of a 'network container', you can directly dive into the mechanics and go to the Network Containers Github repository.
Audience
This article is written for people who have substantial experience with Docker. Dockerfiles, images, containers... it should all sound familiar. If not, this is a great pointer to get you started.
Docker Networking
If you have been working with Docker you know that Docker maps the container's private ports to a -sort of random- unique public port on the host machine. This way you can access the container's services from anywhere on your network. If you only expose one webserver, you can as well expose port 80 directly and this will work well. But if you want to expose two (or more) webservers from one host? Only one will be able to use port 80 directly. The other webservers will be available at awkard ports unless you use some kind of reverse proxy mechanism.
Wouldn't it be nice...
If a container can get its own IP on the regular network?
If a container could get this IP from the regular DHCP server?
that all of this would happen without the container being aware?
Network Containers: An Agnostic Approach
The goal should be clear by now: Starting multiple services on one Docker host that all can expose the same standard ports (for instance, port 80 for http). In order to achieve this each container should have its own IP on the network assigned by DHCP.
But how to assign an IP to a container dynamically without modifying the Docker image and having to install a DHCP client, let alone connecting the container to the network? It would be great if the containers would be unaware of the networking that's going on. This way images from Docker Hub could be used directly without modification.
What is a Network Container?
The sole purpose of a Network Container is to add 'public' network capabilities to an existing service container. This is essentially what the network container does:
Get an IP from DHCP
Route public ports (e.g. 80) to the service container
Basically the Network Container is a simpel NAT-router.
Example
Suppose we have two containers called 'web1' and 'web2' running on a Docker host with IP 192.168.1.1. Both services run a webservice on port 80. Docker would route these ports to unique ports on the host so that both services are accessible from the host IP. Probably the following table makes it all clear.
Service Service Port Accessible on web1 80 192.168.1.1:5601 web2 80 192.168.1.1:5602
It is clear that these services cannot be reached on their standard port. If we put a Network Container in front of each service it is assigned its own IP from the network DHCP server. See the following table.
Service Service Port Accessible on Network Container Access web1 80 192.168.1.1:5601 192.168.1.50:80 web2 80 192.168.1.1:5602 192.168.1.51:80
And here we have it, the services are both available directly on the host IP and from a DHCP assigned IP. This also opens the possibility of adding DNS capabilities to the set-up.
Network Containers: A Scripted Approach
To automatically create a Network Container for a given Docker container I've created a script that performs all of the steps outlined previously. The script checks a running container for the exposed ports, and generates a simple iptables script.
The script takes two parameters, the name of a running container and the name of the network interfaces. The latter is needed to correctly acquire an IP adress from DHCP. The script is simply called as following:
create-network-container.sh web1 eth1
The script will block until the Network Container is fully up and has acquired an IP from DHCP. The output will look like this:
creating iptables route for port 80 containerid=63f967c4cc1e0cd166fffc6b469cc03190e9fd3b2ea86d290304b227459f5202 waiting for public ip to be bound ip=192.168.1.50
The script, along with a more detailed explanation of its inner workings and installation requirements is available on the Network Containers Github repository.
Whats next?
So now all your containers can expose standard ports to the network. That's great I hear you say, but what's next? This opens a world of possibilities, of which DNS is one. In my daily job I use it to run services within our department and have them available at their own hostnames. For this I use SkyDNS, I'm planning to write an article about that soon.
Migrating from SVN to Git can be a tedious and nerve-racking process, especially in environments where there is resistance or inability to change.
However, as a developer you should not suffer from this. In this post I will show how you can use Git locally and still be able to push your changes to the corporate’s central SVN repository.
For setting up and managing a local Git repository you have two options:
Using command line tools (difficult)
Using SmartGit (easy)
After you’ve setup the Git repository, you can simply use it like any other Git repository. Commits will be done on the Git repository and you can choose to push them to SVN at a later time. When you push to SVN, each Git commit will result in an SVN commit with the same commit message.
Since changes will be pushed to SVN there are some caveats regarding .gitignore files that you need to take into account. The last chapter of this post explains it all.
For this post I assume that you are a bit familiar with Git, have used it before and know how to do basic things like committing.
Create a Git – SVN repository: The command line approach
On Linux you can install git-svn through your favorite package manager.
sudo apt get install git-svn
Create an empty directory that will host your Git repo.
mkdir ~/code/myProject && cd ~/code/myProject
Now initialize the Git repo with the SVN repository and fetch the contents from it.
Because of Git’s distributed and decentral nature it will fetch the complete SVN history.
Depending on the size of your repo this will take some time.
git svn init URL_OF_SVN_REPO
git svn fetch
When finished, your Git repository is ready. You can just use it like you would with any other Git repo.
To grab new revisions from SVN you do
git svn rebase
To push local Git commits to SVN you do
git svn dcommit
Create a Git – SVN repository: SmartGit
SmartGit is a visual tool for managing Git repositories and is available for Linux, Mac and Windows. SmartGit supports Git repositories with SVN as a remote repository. SmartGit is very useful when merging remote changes to your local repository since it offers visual diff tooling.
To clone the SVN repo open the Clone dialog (Project -> Clone).
On the next screen select the directory in which the Git repo will be created.
After completing these steps SmartGit will download the SVN repository. After a short time your repository is already usable. SmartGit continues to download all revisions on the background and will notify you once it’s done.
Just use the repository as you normally would (in your IDE for instance). When you’re ready to push your commits to SVN, or want to fetch the latest revisions from SVN use the Push and Pull buttons from SmartGit.
Gitignore and svn ignore
SmartGit will automatically convert your gitignore directives into SVN ignore properties when you push the changes to the SVN repository. One thing to take into account: SVN doesn't support recursive ignores like Git does. A recursive gitignore directive will be translated to an svn ignore directive on every sub-directory. So don't be surprised if you see a lot of propset changes in SVN, it's because of this feature.
As part of my thesis research I had to find out which Java projects have a high usage frequency. As more and more projects are hosted on Github, it is an important and vast source for data mining and software statistics.
To answer the question, which projects are depended upon the most, I queried GitHub for active mature projects and aggregated all available dependency data.
The Approach
Github exposes an API through which you can search for projects with a certain language, rating, etc. Furthermore it is possible to query a project’s tree structure and obtain file data.
Because I needed a representative data set I choose to include mature and active projects only. For this purpose a project is considered active if it had at least one commit in the last year. Secondly a project is considered mature if it is older than at least one year.
The Java world has three major build and dependency management tools; Maven, Gradle and Ivy. I simply downloaded the according build files to obtain dependency related information.
For each Github project each dependency is only counted once. This means that if a project contains multiple modules each having its own dependency management, duplicated dependencies are counted as one occurrence. Furthermore, build tools specific dependencies (such as maven-compiler-plugin) are omitted from the results. This is because depending on these artifacts is a consequence of using the build tool. They would thus occur frequently and cloud the results.
The Results
From the years 2008 to 2012 I was able to retrieve 3.029 projects with dependency management files (of which 2502 (82%) Maven projects, 430 (14%) Gradle projects and 97 (3%) Ant+Ivy projects).
From these figures it is not difficult to conclude that the majority of Java projects use Maven as a build and dependency management tool.
These projects had a total of 26.235 unique dependencies. The following list details the top 5 projects on which others depend:
junit – 1883
slf4j-api - 764
oss-parent – 700
log4j - 671
commons-io – 543
The following graphic shows the top 25 most depended on projects. We observe that JUnit is by far the most depended on artifact, followed by sfl4j-api and oss-parent.
We can see that a large portion of these top projects are related to testing (junit, mockito-all, spring-test, mockito-core) and logging (sfl4j-api, log4j, slf4j-log4j12, commong-logging, logback-classic).
The following graphic shows the top 100 projects as a word-cloud:
The code
To obtain and analyze the Github data I had to implement two relatively small programs. Both of them are freely available under the GNU GPL from, off course, Github.
Raw and analyzed data can also be obtained from the above mentioned Github repository. I’ve also compiled an ODF spreadsheet which you can download here directly.
Today a colleague and I ran into the problem of finding commented out sections in xhtml files. Particularly, we were looking for commented out sections that contained the keyword ‘.xhtml’. In this post I present you how to do this.
Regular expression
Because we use Eclipse to perform the search, the presented regular expression is in the Java flavour. See the end of the post for a Perl-style regex.
Listing of the complete regular expression to find xml comments containing the keyword ‘.xhtml’:
(?s)<!--((?!-->).)*\.xhtml((?!-->).)*-->
Breakdown of the regular expression
(?s) is the Java-flavour modifier to tell the regex runner that a dot should also match new lines. This way we discover comments spanning more than a single line.
<!- - is our start symbol.
(?!- ->) is a negative lookahead. This matches any sequence, except - ->. We need this because we want to restrict the regular expression to finding single comments.
Consequently ((?!- ->).)* matches any character except the sequence after the negative lookahead.
So until now, we match text that starts with <- -, doesn’t contain - -> but may contain every other character sequence.
Burried inside the comment we expect to find our keyword. Therefore we match on \.xhtml.
Any sequence of characters may come after that, but we should stop as soon as we find the closing tag. We repeat the same negative lookahead match as before to achieve that.
At last, we expect to find the closing sequence of the comment (- ->).
Comments in other languages
It should be fairly trivial to adapt the regular expression to find comments in other languages.
The general pattern is as follows (keyword is optional):
From the Attic: The Ten Commandments of Egoless Programming
Already in 1971 Jerry Weinberg established a list of ten commandments of egoless programming in his book The Psychology of Computer Programming.
The goal is to minimize personal factors, be cooperative and respectful. This way quality can be achieved.
Even though it was some time ago, it does not hurt to repeat the commandments once in a while and to recognize that code is written for humans in the first place, and secondly for computers.
Computers can very well execute code that is beyond comprehensibility for us mortal humans.
Now please write code and take the following into account:
Understand and accept that you will make mistakes. The point is to find them early, before they make it into production. Fortunately, except for the few of us developing rocket guidance software at JPL, mistakes are rarely fatal in our industry, so we can, and should, learn, laugh, and move on.
You are not your code. Remember that the entire point of a review is to find problems, and problems will be found. Don’t take it personally when one is uncovered.
No matter how much “karate” you know, someone else will always know more. Such an individual can teach you some new moves if you ask. Seek and accept input from others, especially when you think it’s not needed.
Don’t rewrite code without consultation. There’s a fine line between “fixing code” and “rewriting code.” Know the difference, and pursue stylistic changes within the framework of a code review, not as a lone enforcer.
Treat people who know less than you with respect, deference, and patience. Nontechnical people who deal with developers on a regular basis almost universally hold the opinion that we are prima donnas at best and crybabies at worst. Don’t reinforce this stereotype with anger and impatience.
The only constant in the world is change. Be open to it and accept it with a smile. Look at each change to your requirements, platform, or tool as a new challenge, not as some serious inconvenience to be fought.
The only true authority stems from knowledge, not from position. Knowledge engenders authority, and authority engenders respect – so if you want respect in an egoless environment, cultivate knowledge.
Fight for what you believe, but gracefully accept defeat. Understand that sometimes your ideas will be overruled. Even if you do turn out to be right, don’t take revenge or say, “I told you so” more than a few times at most, and don’t make your dearly departed idea a martyr or rallying cry.
Don’t be “the guy in the room.” Don’t be the guy coding in the dark office emerging only to buy cola. The guy in the room is out of touch, out of sight, and out of control and has no place in an open, collaborative environment.
Critique code instead of people – be kind to the coder, not to the code. As much as possible, make all of your comments positive and oriented to improving the code. Relate comments to local standards, program specs, increased performance, etc.
When writing unit tests it is important to understand which parts of your code are covered by the tests. In general software development companies have quality control systems in place which keep track of, amongst other things, unit test coverage. Normally a developer needs to commit code to the version control system before the code is analyzed by the quality system.
I like to stay in control of my code and knowing the quality of my code before committing it is a must for me! I’m using the code coverage tool JaCoCo for over a year now and I am very happy with the detailed coverage reports that it generates.
If you ever wondered what the coverage of your unit tests is I would recommend using JaCoCo. You can use it as a standalone tool or as a plugin in Eclipse.
EclEmma is an Eclipse plugin for the popular code coverage tool JaCoCo. Previous versions of EclEmma used Emma as code coverage tool, hence the name. Active development of Emma seized some years ago. The authors of EclEmma developed JaCoCo to be a drop-in replacement of Emma, with functional enhancements like branch coverage. At time of writing JaCoCo is the only coverage tool with full support for Java 7.
