#gsoc 2017

[bumblebeesky]

Wrapping Things Up

It has been a wonderful journey so far. I feel blessed to have worked with such experienced people at The NetBSD Foundation. I would like to thank all the people I’ve interacted to during the entire period for helping me out and teaching me wonderful new things about how the release engineering team operates and how important the automation process is.

This post pretty much wraps everything up and contains all the relevant links related to the project. Here’s a list:

Here’s a link to my clone of the original anita repository. It contains two main branches.

The master branch is for testing an my own bookkeeping purposes. It contains many commits that were later reverted.

The origin/Merge-safe branch is the one that is regularly being merged to the master of the the original anita repository. It basically contains squashed commits of the finalised changes that are ready to be merged.

Here’s a link to the pull requests created by me in the original anita repository. As of now, all, but the last one have been accepted. The last one is still pending, as the owner of the repository is travelling and is on a rather tight schedule. Do note that, it shows that two of these were closed, but they were actually merged. This is because the owner had done it from the command line and for some reason github didn’t reflect the desired status.

Here’s a link to the documentation for anita. A part of it has been written by me.

Here’s a link the The NetBSD release engineering page. I feel really honoured to know that my work is already being utilised by The NetBSD Foundation.

Here’s a link to the official anita website, just in case you’d like to know a bit more about it. I’ve written a lot about it in my earlier blog posts. The two blog posts prior to this one have are related to this project.

Finally, to conclude, I would like to let you know that we have successfully added support for the following ports to anita:

NetBSD/pmax

NetBSD/hpcmips

NetBSD/evbarm

NetBSD/landisk

NetBSD/sparc64

NetBSD/vax

With this, we have achieved what we had planned when we started off. Since, we had enough time to spare in the end, we decided to try out another one, NetBSD/amiga, which is almost about to be merged. The only thing left as of now is the structure of the functions involved for adding support and this will be looked into, after Andreas, the owner of original anita repository, is free. Everything else has been tested and rectified to make the procedure suitable to be used by anita.

Most of the things that we have done, that is testing and debugging, cannot be linked as I didn’t have commit rights to any of those repositories. Thus, only the links associated to anita have been provided.

I’ll continue to contribute to NetBSD and anita in any way possible.

[anascarlet-blog1]

Coding Week 2 & 3

I am not sure where I missed a post, but apparently I did. Sorry about that. I was planning on posting every week during the coding period.

So far, I would say my project is doing well on time and progressing according to schedule. It really looks more complex now than it did during the first week because I implemented Maven and added a few more smaller files. The small files are mainly tests and their results. Last week (#3) I mainly worked on tests and I also ventured ahead into the next part of my project, which I am supposed to work on after evaluations according to my schedule. I think that did me good in terms of thinking ahead to the next part, however. I am also probably going to end up starting on the second part a bit earlier than planned, which is, of course, a good thing.

At the start of fourth week / end of third week, I have been testing the main bulk of code that is behind my simulation. I ran into several problems, which I have been able to fix so far. I have also had to make some structural decisions and I am facing one more big one.

The above-mentioned decision is about trying to achieve thread immutability and abstraction. Right now, I have three threads managing Oxygen and CO2 consumption and production in my fish tank simulation. They are Fish, Plants, and Decomposers. I am facing with the decision of making separate threads for everything dealing with Oxygen and CO2. I may combine Oxygen and CO2 in one thread because the conditions for changing both of them are quite similar. This would give me threads that are immutable. It would also give me less messy code. However, it means making more threads, which is expensive, and it would also make slightly less sense because in reality, plants manage their own O2 and CO2 and fish manage their own and so on. So, that is the decision I have to make... and I am leaning towards the option of change.

On another note, I have noticed something curious about the GSoC student population. 43.17 percent of all GSoC students this year are from India, which is surprisingly not proportional to India’s population compared to the rest of the world. The number of Indian students is 11 times greater than the number of Chinese students. If this was population-proportionate, the numbers would be roughly the same.

Some more fun stats. There are 0.39 enrolled American students per million US residents. There are 0.42 Indian students per million Indian residents. 0.68 German students per million German citizens. There are 0.73 Canadian participants per million Canadian residents. 2.6 Sri Lankan students per million Sri Lankan citizens. Only 0.035 Chinese students per million Chinese citizens. Stats calculated from this GSoC blog post.

Now I should mention that these are just statistics and they do not mean to generalize or offend anyone. Although some conclusions may be drawn from these stats.

Until next time!

[bumblebeesky]

My experience with The NetBSD Foundation as a part of Google Summer of Code, 2017

As the first evaluation is approaching, I’d like to share my experience, so far, with The NetBSD Foundation. My project is related to adding support for other virtual machine systems, like GXemul, Virtualbox etc. to Anita (a tool for automated testing of NetBSD installations on various platforms.) as well as integrating support for cpu architectures it doesn’t currently support.

Anita is a set of a library and a few scripts. Although, its main purpose is to test sysinst installation procedure and quickly detecting the errors encountered in the process, it is also used for testing the whole NetBSD system by running ATF test suite. In future, we also have plans for supporting kyua, which would ultimately replace ATF. The problem with kyua right now is that it doesn’t get installed properly when a NetBSD release is built with MKKYUA=yes build option. Although, it can be installed manually, that’s not really what Anita should be doing. Both, the library part and the controlling script for Anita are written in python. Additionally the package includes a bourne shell script for testing older NetBSD releases too. This has been done to ensure back compatibility.

So far, throughout the coding period, I have build lots of realeases (and fixed many of them), done lots of testing and also contributed to the Anita library and the controlling script.

Building releases

Building releases is as easy as cloning the repository and executing build.sh in the root of source directory along with all the options. However, it takes a lot of time for the build process to complete. On a laptop with intel core-i5 6300hq, building the amd64 release takes about an hour and ten minutes with all the 4 cores enabled. (On a laptop that was purchased for web browsing and using spreadsheet, this will take much much longer than that.) This may look like an easy task and you might be thinking “Bro! I’ll just let it build for an hour, leave it like that and enjoy all this while”, but it’s not that easy. While testing random realeses or trying to get a build to work, it will fail to compile several times. Let’s suppose, it fails 5 times, just when the build was about to complete. On my system, you’d end up wasting 5 hours building and let’s say, an hour, in total, fixing the errors. So, that’s a solid 6 hours. Not only that, while all this is happening, you’ll get up and leave several times and wouldn’t know that the build has failed. This will lead to extra loss of time. The only way to stop this is to keep sitting in front of your screen.

Let’s add up an hour for it too. So now, it’s 7 hours. This was just for one release. While trying to figure out which release broke the build process, you’ll have to test multiple releases. In my case, this was almost always 15 – 16 releases. So lets multiply that too. 7*16 = 112 hours. Now imagine, on a machine running exactly 4 times slower, how long would it take? 112*4 = 448 hours. Now let’s suppose, you work for 10 hours a day. In my case, it would take me just 11.2 days to finish off the work whereas on a 4X slower machine, it’ll take 44.8 days. I hope you got the point! And mind you, the entire calculation assumes that you’ll be using your processor to the fullest possible extent, so don’t expect a long battery life. It’ll probably get reduced to 1/3rd or 1/4th. So even if you get 5 hours of extra usage with a slower machine, the difference will now be merely 1 hour 15 minutes.

The system libraries a generally written in C and sometimes in assembly language. So, you need have at least basic debugging skills for C. Although, the system files are quite long (in length) and contain a lot of macros and pointer usage etc. that a regular college student might not be familiar with, you won’t face any difficulty in debugging them if you are good at understanding what the code is trying to do, i.e. your skill for interpreting new code base is strong enough.

Testing

The testing part is actually a bit better than the building part. All you need is a virtual machine system, like GXemul, an empty (or populated with a working filesystem) raw disk image. In some cases, it requires us to have a kernel and an iso image for the first boot too. Basically, you just need to be familiar with the command line for that particular virtual machine system. The main objective of testing is to make sure there are no errors during the boot process and you’re able to log in and then halt the system properly. In case of any errors, you’ll have to fix that and go through the build process once again.

Apart from that, the entire NetBSD system can be tested with ATF test suite and kyua. But, that is not really a part of my project. Anita does that for me.

Contribution to Anita library

This is something that required the least amount of effort until now. Now that we have a few more ports working (by going through the build process, testing and then fixing errors.), the code contribution to this and possibly the controlling script is going to increase. A special emphasis was given on finding the simplest possible solution with least amount of additional code. I ensured that I took full advantage of the classes and funtions that already existed in the library. Of course, this means that I had to go through lots of code muliple number of times. The Anita library was already over 1700 lines of code, when I started.

Currently, we’re working on adding support for NetBSD/evbearmv7hf-el. Now this port contains a lot of kernels. The first kernel to be supported would be VEXPRESS_A15.

Future plans

In future (during the GSoC period), we may even have a working NetBSD/macppc. This is yet to be tested successfully. 

Work is already on for getting a mips port supported on Anita. I (with the help of my mentors and various other members of The NetBSD Foundation) have succesfully tested the NetBSD/pmax port. If time permits, we will likely have more arm and mips ports supported on Anita.

Conclusion

For me, Google Summer of Code was the single best experience I’ve had in years. Until now, it has been quite adventurous and I’ve learnt a lot more than any college could have taught me.