At the heart of this project are VPN tunnels. Since this is an overlay network, the physical links for the network are through several NAT isolated links to the Internet. As such, it is critical that network tunnels be used to establish a virtual cable between the various nodes in the network.
However, tunnels are always a bit of a weird experience. I’ve done a lot of work over the year with small-scale OpenVPN tunnels for connecting my own PC to my home network while roaming, but I knew there were several other tunneling technologies out there.
After researching a few of the other technologies I discovered that the big technologies out there were PPTP (an old tech I’d known about for over a decade,) L2TP/IPSEC, SSTP, IKE, and OpenVPN. The first two are supported directly in Windows, which would have been nice. However, each of them had their own problems.
PPTP is insecure. The initial tunnel’s authentication process is relatively easy to defeat and the PPP headers are unencrypted/unsigned.
L2TP/IPSEC isn’t well suited for NAT. Yes, there’s things you can do to enable NAT traversal by tunneling your tunnel, but at some point you realize that you’re sacrificing a ton of your MTU to headers. Also, L2TP/IPSEC is a pain to set up.
SSTP and IKEv2 aren’t well supported on Linux.
OpenVPN is fully encrypted and uses SSL/TLS authentication, but is somewhat proprietary.
At the end of the day, Oni and I had the most experience with OpenVPN coming into the project, and none of the other alternatives were compelling enough to make us learn a new server/client system. As such, we opted to use OpenVPN.
VPN tunnels come up in two places in the network. The first is in the router-to-router links. These are not setup as client-server relationships since they’re always one-to-one; essentially like the old PPP links used to connect routers. These were pretty easy to set up and debug.
The second place that VPN links come in is allowing clients to join the network. In Oni’s network, many of the computers participate directly on DotNQIG. However, for people who are roaming, or in my case where direct participation isn’t possible due to a quirk in my Internet gateway router (my router refuses to route packets back out the port they came in on) it allows me to join the network as necessary.
That said, even though I did have experience with OpenVPN before, one thing I hadn’t done before with it was mix it with OSPF/Quagga. That turned out to a be a bit more interesting.
In the context of the router-to-router links, it was pretty straightforward. I assigned a network that covered our peer-to-peer routing region (10.255/16) and quagga did a good job of inferring the necessary routes. However, the client/server links turned out to require a bit more work. The reason was that quagga inferred the available links from the tunnel interface’s PtP target. In client/server mode, this address isn’t likely to be valid. But beyond that, it would normally mean that routing entries would be propagated to EVERY ROUTER for each VPN client. As such, it was time to learn a bit about area summarization.
After adding an area summarization line to the quagga config, the end result is that so long as there were any routes in my 10.10.255/24 client VPN subnet, the whole network would get wrapped up into a single entry. First, this made the connections work, but secondly, this allowed me to have one or one hundred clients connected to the VPN server at the same time, and the OSPF routing table would see all of them as a single entry.
DNS in DotNQIG is a bit of an interesting challenge. The reason is that we still want domain resolution to work for the Internet, but we want systems participating in DotNQIG to be aware of the .NQIG top level domain (TLD). In the real internet, this would be done by registering a new TLD with the ICANN root servers.
However, doing that is actually not possible. Although a few were added recently, like .photography, the reality of the matter is that ICANN does not add TLDs. But more than that, we wouldn’t really want them to; this is intended to be a private LAN, and none of the addresses we’re using are routable from the Internet anyway. So we had a problem: how do we overlay our domain system onto the real one?
The initial answer was forwarding. In both Oni’s and my network, we set up a forwarding server and some basic zones. The forwarder was simply pointed at our ISP’s nameserver. This worked well when we were the main nameserver on the network. However, when we merged the networks and slaved his .nqig and wikis.nqig records on my system, things started getting a bit more difficult.
You see, in Bind 9, if a forwarder is configured, any request that cannot be addressed authoritatively by the server is sent to the forwarder address. When the systems were separate, there was only one server in DotNQIG that could answer these queries. However, when we bound the two together, each with it’s own nameserver, a problem presented itself. se.wikis.nqig was a CNAME to web-server.jimthecactus.nqig. The wikis.nqig zone lived on Oni’s nameserver and was slaved on mine, and the jimthecactus.nqig zone lived on my nameserver. Resolving SubsequentErrors.wikis.nqig failed on oni’s system.
The reason was a bit of nuance in the above statement. If Oni asked his nameserver for the address, it would know to check it’s own zone for wikis.nqig to get the address for SubsequentErrors, which it would happily return as being a CNAME for web-server.jimthecactus.nqig. Since it didn’t have zone information for the jimthecactus.nqig zone, this request couldn’t be answered authoritatively. As such, even though it knew what nameserver to ask for authoritative answers, it instead would send the request on to the ISP per the rules of forwarding.
This meant that my zones couldn’t be resolved. The same held true in the reverse direction about the oni.nqig zone. The only reason I was able to access the SubsequentErrors.wiki.nqig address on my system was because I had a slaved (and therefore authoritative) copy of both the .nqig and the .wikis.nqig domains.
The correction then was to use a full, caching server, rather than a simple forwarding server. This is a bit more resource intensive since the server is now keeping a cache of all queries and has to make queries against the real root servers rather than running through the ISP’s caching servers, but in exchange, it will fully work to resolve queries, even if it has to reach out to other servers to do it.
Okay, so I said I'd cover DNS next, but I wanted to get this out first.
First, though I kinda hinted at it rather loosely, I got my friend Oni to do their bit of the network on Friday. This really helped me work out some stuff that wasn't immediately apparant with my fully private setup; DNS turned out to be a big thing I needed an external peer to see all of the problems with. So yay for having people to connect to! But that also means it was time for a new network diagram.
On that front, I did a bit of playing around with Dia, a FOSS tool for drawing diagrams. With a bit of work, I was able to get a network schematic put together. I was even able to keep my notation where hostnames are italic and bridges and networks are done in bold.
The DNS writeup will take a bit, but look for that either today or Wednesday, depending on how things go over the next few days.
When we decided to start this project, one parameter that was defined early on was that routing tables had to be dynamic; i.e. it had to be possible for new nodes to be added to the network without every person in the network having to hand add the new nodes. On the scale we’re working it’s probably not a big issue, but if nothing this would give us a chance to learn a bit about dynamic routing protocols.
We settled on OSPF as we were making a network that most resembled an inter-office network and OSPF is well suited for internal networks. As such we needed to lay down areas and setup an OSPF server on the routers to get them to share routing tables. The first step was to determine which software to run.
Turns out that there have been several competing servers out there for a few years, and some politics along the way that have ultimately lead to several projects ending up dead. Our initial search turned up with gated and zebra. Zebra seemed to be more full featured than gated. However, some research determined that zebra was no longer maintained. A new FOSS project had taken up the reins called quagga, amusingly named after an extinct zebra-like critter.
Ignoring that that doesn’t bode well for their project, we started getting things setup. In my initial tests I had set up a few areas and defined privbridge as the area 0. However, since I now had my friend’s network coming online and up to speed, it became apparent that doing that had broken some stuff for reasons that weren’t initially clear to me, so to get things moving I crammed all of the routers into area 0.
Upon further study into OSPF though, the reason became a bit more clear to me; OSPF requires that all areas are directly connected to area 0. This meant that either every router had to be added to area 0 where dynamic routes could exist,or a “virtual link” had to be made between them so that they were effectively on area 0. Because our network is small, the correct answer is that every router that participates in the backbone links between systems will be in area 0, and the router attached to leafbr will be in two areas. This isn’t strictly necessary as that complexity of the network only exists to ensure that we always have a thing two steps away to test against, but since this is about the journey and not entirely about functionality, a bit of synthetic complexity can be seen as a good thing.
Next time I’ll cover DNS. That was another one I’d done a million times before that ended up taking me 4 hours to figure out because of a quirk in the way overlay networks work.
So the first step in this project turned out to be both easier, and harder than I had originally expected. The reason is that initially I had expected to have a lot of trouble getting the XEN systems prototyped and built. In the end, however, it turned out that XEN-Tools made it surprisingly easy to establish a template system and the build out from there.
What was less easy was making (and routing) the necessary bridges. As you can see from the network architecture diagram, the goal was to have two bridges so that from the most distant virtual machine to my main network was at least two hops. On those same lines, the router on my main network that handles my gateway to the internet isn’t smart enough to route packets between subnets on the same LAN port.
Ya, I know it’s not good practice to run two subnets over the same network, but this is a project about learning routing, it would have worked well enough.
In any case, this means that I needed to do masquerading at the final hop before it hit the physical wire. Masquerading is something I’ve done plenty of, and expected to not have to think too hard about, but boy was I wrong.
See, it happened that as long as packets originated from node3, everything was fine; the packets would be masqueraded with no fuss, and everything would work fine. However, any packets that came from web-server would come out of blocky without any source mangling at all! I added debugging counters and traced packets around, and did everything to try and figure out where my rules had gone wrong, and found myself running in circles trying to hunt this down.
In the end, it turned out that by default, IPTables inspects every packet that traverses a bridge. Since, at the time, I was using blocky to both provide the bridge, and also do the masquerading (something that node1 is now responsible for,) this meant that packets had to traverse leafbr and then privbridge to get to their destination. This normally wouldn’t be a big issue, it should just be like going across a link from one router to the next.
However, IPTables flags every packet it inspects, and once it’s routed a packet, it will not route it again. This may not seem like an issue (and actually would be great to prevent routing loops and minimize CPU load) but in this case, it meant that when the packet traversed leafbr, it was flagged as having been routed. Normally in the router-to-router traversal, this wouldn’t be an issue because as it moved from leafbr to privbridge it would have moved between two different routers and the flag would have been dropped.
But this is a virtual link, and leafbr, privbridge, and my physical ethernet link are all on the same computer, and by extension, IPTables is shared between them. Digging around online turned up a valuable nugget; that the bridge inspections could be disabled. Since I was fine with bridges not having control lists (I don’t need a smart switch or vLANs) I simply disabled it via adding “net.bridge.bridge-nf-call-iptables = 0″ to my /etc/sysctl.conf file. “net.bridge.bridge-nf-call-ip6tables = 0″ does the same thing for IP6Tables.
That said, with the shift to using node1 for routing packets to the physical LAN, effectively isolating the dom0 from the whole process, this change became moot and I have since removed it. But learning that IPTables is a touch-once mechanism, and also finding a bunch of other neat things like the IPTables Flow Chart, fully justified the experience.
Short update. I've modified the network topology from the one in the chart and added node1. node1 does a lot of the routing blocky used to do. Now, blocky has no ip on privbridge, and instead node1 has an interface on both the physical bridge and also on privbridge.
The reason for the change is that right now, if I’m working on the various routing components remotely and make an iptables rule that makes me unroutable, I can’t recover until I get back to a physical keyboard. By taking all of those rules away from blocky and assigning them to node1, I make it so that in a worst case scenario, I can connect directly to blocky and use a Xen console command to recover, either by rejoining node1′s terminal, or by rebooting node1 altogether.
Presented here is the network topology for the DotNQIG project test bed. Chao is my main PC, and it played console for everything. blocky is the aforementioned XEN VM server and primary router. node2, node3, and web-server are all VMs running as DomUs under blocky. privbridge is the main virtual bridge for the VMs and has a connection to blocky to allow for traffic to be routed. leafbr is a pure virtual bridge that should (in theory) have no connection to blocky. I’ll cover why that “in theory” was necessary in a later article as that nuance cost me several hours.
Source NAT is used on all traffic routed through blocky. This is done because my wireless router will not let me create static routes pointing at anything other than the WAN interface. As such, traffic destined for the internet would not be able to return to the computer on the source network. The SNAT on blocky then allows for all traffic to appear to come from blocky’s physical ethernet port, regardless of where it actually came from inside the VM.
Getting Chao to talk with web-server then was the true test as it’s only accessible via node3. I could have used DNAT to connect things, but the goal here is that I want to have full NAT-less communication with the darknet. As such, this meant I needed a more rugged solution, and I’ll cover the VPN link I used to do that in a later article.
I’ll see if I can get more written up in the coming days so you can see the rest of the project, along with any additional things that come up as I get my friends’ nodes added to the system and fully debugged.
Over 10 years ago I went through Cisco’s CCNA program and learned the (hilariously flawed) basics of how the internet worked. However, I never took the test and ended up going into electrical engineering instead. However, I’ve worked to keep the more common of those skills up, but I never really went all the way into making a fully fledged internet with proper boundary routers.
However, some conversations over the last few years with some friends suggested that we put together an overlay network and create a darknet-like network where we can play LAN games without worrying about NAT. However, over those years, the real challenge has been getting them involved and active. Making a network means having more than one node, and my friends aren’t terribly good about actually moving on projects
Well after something like 4 years, I’m finally getting around to doing the project anyway; with or without them. The secret is working around my friend’s participation using a computer I’ve called Blocky; a XEN VM server that can host a number of VMs. Using Blocky. I laid out a number of nodes and a complete network with multiple paths, some unroutable except through other nodes. This allowed for me to play with all of the pieces I needed to get things working.
The end goal is to make a network that would be isolated from my main network, be able to access the internet itself, be able to host services (namely a web server with a Wiki on it that used to be hosted on a public server), and to automatically advertise and discover new routes.
I’ll be writing a series of articles about that experience and the weird things I learned (and am still learning) in the upcoming days. Hopefully I’ll get the first one written up sometime today and show you guys the network topology.
