donut2D

[donut2d]

My experiences building a URL shortener with OCaml and ReasonML

This is not a guide, just a summary of my experiences with a few, hopefully helpful, tips scattered throughout.

I recently published two projects to Github: a URL shortener redirection server and GraphQL API and a web client for the API. The server is written in OCaml and the client in ReasonML.

ReasonML is basically a dialect of OCaml with a strong focus on interop with JavaScript, especially React. With that in mind and since I started this project with OCaml, I'll write first about OCaml and then finish with Reason. But since they are so similar, most of what I write about OCaml that doesn't involve tooling or JS, is also applicable to Reason.

Why OCaml

I wanted to build a URL shortener. I looked at several open source solutions, but all of them were written in dynamic languages (PHP, Python, NodeJS) and were often setup in a way that wouldn't exactly fit our needs (mainly a lack of analytics and pattern matching). At least the redirection server of a URL shortener should be fast. It's hard to beat a native binary.

My goto strictly-typed, compiled language has been Swift. I've found it to be very practical and enjoyable to write in. But there definitely have been issues with maturity when using it to build services to run on Ubuntu servers. There's also been a lot of churn in third-party libraries that I had relied on in the past. It's left a bit of a bad taste in my mouth, and I'm kind of giving it a bit more time to bake before going back to server-side Swift.

If I wasn't going to use Swift, then I wanted to use something functional. Out of curiosity I read through Learn You a Haskell for Great Good! a couple years ago. I enjoyed a lot of the concepts and brought them into my work with PHP, JavaScript, and Swift. But I did have trouble actually being productive in Haskell. The package management story was confusing, and even to accomplish a simple project, I found I needed to copy and paste a lot of things I didn't understand.

In the time since then, I've seen OCaml come up in discussions on Hacker News and other places frequently, always in expression of positive experiences. It's functional, statically typed, compiled, and mature. It hit what I was looking for and I didn't see a reason not to jump in.

Learning OCaml

...was difficult. There really needs to be a beginners guide that runs through creating a small project, that is kept up-to-date with modern practices. Even figuring out the right, and modern way to build projects was tricky. There are definitely some resources out there, but the learn page on ocaml.org doesn't do a good job directing newbies on a path to get started. It's just a list of resources and it's not clear where to start and what to start with.

Resources

Real World OCaml. This is an excellent book and has probably been the biggest help in learning the concepts of the language. It's a bit dated, especially around tooling and compilation, but otherwise very solid. The authors are currently working on the 2nd Edition which includes modern tooling.

#ocaml channel on the ReasonML Discord. This is an active and welcoming community, a great place to ask for help when you're having troubles with concepts or tooling.

jbuilder Quickstart Guide. Learn how to compile your OCaml projects with jbuilder. Wish I had found this much earlier. Seems to be the best compilation story, at least until bsb-native starts working for OCaml > 4.02.3 and allowing for sub-modules in separate files (as of writing, there can be namespacing issues since each file represents its own project-global module, regardless of directory structure).

OCaml main forum. A good and active community, willing to help.

Using OPAM. The official documentation for the OCaml Pacakge Manger (OPAM). The documentation is clear and concise. Be sure to read up on opam switch (--alias-of flag is good for separating OCaml version and libs between projects) and opam pin (pin a specific version of library so it doesn't upgrade; can be used with local paths and remote git repos as well as version numbers).

vscode-reasonml. A great plugin that also supports OCaml for Visual Studio Code. If you're not using Vim or Emacs, the editor/IDE story can be weak. This VSCode plugin has been mostly great for me.

The Good

I found that when I write something in OCaml, it generally works. The type system is very strong and basically forces my code to be correct. I've experienced this to some extent with Swift, but OCaml seems to be even stronger in this regard.

OCaml has just felt easy and natural for me. I'm not sure why exactly this is. The language feels relatively simple, with a few basic tools that can be arranged in many different ways. It didn't take me long to feel like I understood most of what was going on. Reading the source of third-party libraries I was using came much easier than it has in other languages. With Haskell it felt like I needed to understand a lot of the functional programming concepts before I could really use the language. With OCaml, I feel like I am able to use it before I fully understand it, and that in the use I come to better understanding.

The package management story with opam is pretty great. It's not yet at the ease of use of yarn, but it sure beats my experience with Haskell. Documentation is pretty clear and if there's something wrong, opam's error messaging has been really helpful. When installing packages on Ubuntu, it even told me what libraries I needed to install via apt-get.

The Bad

The community is relatively small. This really hit me when I started looking for a library to work with PostgreSQL. Most of what's out there is either outdated with modern OCaml/tooling or it's not asynchronous. It turned out that what libraries were available and up-to-date decided what DB I was going to use (MariaDB). A forum discussion that sums this up. I ended up using ocaml-mariadb which has worked well, but it was nearly my only viable option at the time. Now, there is also ocaml-caqti which currently supports MariaDB, PostgreSQL, and SQLite3.

That said, with the popularity of Reason increasing comes an increased interest in OCaml. Also, many of the contributions to Reason also benefit OCaml.

Take Away

OCaml has been wonderful. I've come away from this project wanting to write everything I can in it. It seems to fit me better than object oriented programming ever did. Everything feels simple and transparent.

ReasonML

I knew that I wanted to build the web UI client of the URL shortener in React. I assumed I'd do this with TypeScript as I have with other projects. But as I got deeper into OCaml and its community, the more I learned about Reason. I wasn't thrilled about the syntax changes from OCaml, but I had really warmed up to the language and wanted more.

Some have asked why I didn't just use BuckleScript, allowing me to avoid Reason's syntax changes. Reason is designed with React in mind and has great interop with it. I think those advantages outweigh the minor annoyance of the syntax changes.

What really sealed the deal for me was the account of Facebook's experience rewriting 50% of Messenger in ReasonML. Way faster build times, 93% fewer bugs, improved iteration. This was enough to get my attention and suggest I should check it out.

Resources

In addition to the OCaml resources listed earlier, which mostly apply to Reason as well. These are pretty obvious, but I sometimes have found that I missed what many thought were obvious.

Try Reason. Does what it says on the tin. Write some Reason and it'll show you the OCaml and JavaScript equivalents.

Official ReasonML Docs. They're well written and cover the basics.

Official BuckleScript Docs. BuckleScript is basically straight OCaml with some syntax helpers for interop with JavaScript. Its compiler is used to compile both Reason and OCaml to JavaScript as well as to native binaries. These docs have some more detail on using non-Reason/OCaml JS libraries in Reason. This helps fill in gaps in the official Reason docs.

ReasonReact. If you're going to be working with React in Reason, this is how to do it and it's pretty great.

reasonml-community Github Org. There are a lot of helpful projects that live here. If you think something is missing, this is a decent place to start.

bs-webapi library. This has a lot of bindings to browser DOM and JS APIs that don't come out-of-the-box with Reason.

ReasonML Editor Plugins.

Reason Package Index. Want to know if a JS library you want to use already has Reason bindings available? Search here. Also, most Reason compatible projects have bs- or reason- prefixes on GitHub.

Take Away

I found almost the same ease of use and "it compiles, it works" experience in Reason as I did with OCaml. The language and tooling are definitely experiencing some growing pains. As I was beginning with it, I'd get a lot of <unknown syntax error> messages from the compiler. But with some time I was able to learn the typical culprits.

I found the JS interop to be much easier to grasp than what I experienced with TypeScript. BuckleScript/Reason let you define just the types you want to use and takes your word for it. It makes it really easy to idiomatically solve most interop issues. The syntax felt a little weird at first, but after some time I've learned to appreciate it.

The team developing Reason seem to be pretty smart. If somethings not in the language yet, it's because they're trying to get it right the first time. The Promise story isn't great yet, but it's workable for now and I expect something much slicker to come out sooner than later.

[donut2d]

Basic Firewall Setup and Securing SSH

Firewall Setup with ufw

ufw or Uncomplicated Firewall is a simple interface for configuring iptables and ip6tables. It also has the benefit of configuring both IPv4 and IPv6 at the same time. This is the main reason I ended up choosing it over manual configuration of iptables and ip6tables.

Getting Started

If you don't have ufw installed, you can get it by running sudo apt-get update and sudo apt-get install ufw.

Go ahead and run sudo ufw status verbose. You'll either get a message saying "Status: inactive" if ufw hasn't been enabled yet, or a list of rules currently implemented.

If you need IPv6 support, open up [/etc/default/ufw] in a text editor and be sure IPV6 is set to "yes", so IPV6=yes.

If it wasn't already set to "yes", then restart the firewall using sudo ufw disable and then sudo ufw enable to make sure it sets rules for both in the following steps.

Base Rules

Typically you'll want to allow all outgoing connections and default to denying all incoming connections. If you want to whitelist outgoing connections as well, this post isn't for you.

Run sudo ufw default deny incoming to block all incoming connections that don't match any allow rules.

Run sudo ufw default allow outgoing to allow all outgoing connections that weren't blocked by any deny rules.

Specific Rules

The first thing you'll probably want to do is allow SSH access: sudo ufw allow ssh. This will open up port 22 on TCP and UDP. ufw can automatically setup access/blocking by name for any service listed in [/etc/services].

If you don't want to use the service name, you can also add the same rule for SSH by running sudo ufw allow 22/tcp and sudo ufw allow 22/udp.

To get an idea of what ports need to be opened, you can get a list of ports and the processes listening to them by running sudo netstat -plunt:

↪ sudo netstat -plunt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 928/sshd tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1022/mysqld tcp6 0 0 :::22 :::* LISTEN 928/sshd tcp6 0 0 :::443 :::* LISTEN 1057/apache2 tcp6 0 0 :::80 :::* LISTEN 1057/apache2

You don't need to worry about anything listening on 127.0.0.1 since that's the loopback interface and is only used for internal communication.

Using netstat's output as a guide, go ahead and allow ports using sudo ufw allow <port>/<protocol>, replacing the placeholders appropriately. Note that you don't need to separately specify "tcp6". Just using "tcp" will also generate the rule for "tcp6".

For example, for the above output I'd run the following:

sudo ufw allow 443/tcp sudo ufw allow 80/tcp

You can also allow incoming requests to a port from only one IP address:

sudo ufw allow from <ip> proto <protocol> to any port <port>

Finishing Up

Once you have all your rules in place, go ahead and run sudo ufw disable and sudo ufw enable. Once ufw is active, all rules will take affect immediately.

If the rules become a mess, feel free to run sudo ufw reset. This will clear out all of the rules and disable ufw.

You can always check the status by running sudo ufw status verbose.

Like I mentioned before, ufw is just a simple interface to iptables and ip6tables. With that in mind, you can check out the rules that have been added by running iptables -S and ip6tables -S.

Sources

DigitalOcean Community: How To Setup a Firewall with UFW on an Ubuntu and Debian Cloud Server

DigitalOcean Community: 7 Security Measures to Protect your Servers

Securing SSH with Fail2Ban

fail2ban watches log files for failed authorization attempts and then bans the IP address. By default, does this by adding rules to iptables. This works fine, but may cause confusion since we are managing iptables indirectly through ufw and in the default state, fail2ban's rules will not show up in ufw.

Install fail2ban: sudo apt-get update and sudo apt-get install fail2ban

Copy default config to local copy that wont be overwritten by updates: sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Add ufw action configuration:

sudo curl https://raw.githubusercontent.com/fail2ban/fail2ban/master/config/action.d/ufw.conf | sudo tee /etc/fail2ban/action.d/ufw.conf > /dev/null

Edit local config: sudo vim /etc/fail2ban/jail.local

Some settings to review:

bantime = 600 The number of seconds to ban an IP address.

findtime = 600 The amount of time maxretry must be exceeded in for an IP to be banned.

maxretry = 3 Maximum number of retries within findtime before an IP is banned.

destemail = [email protected] Where to send emails of ban notifications (you'll probably want to setup a filter so they don't flood your inbox).

banaction = ufw What action (see [/etc/fail2ban/action.d/]) to run to ban an IP. The default is iptables-multiport. I've changed this to the ufw action added in the previous step.

action = %(action_mw)s What actions to take when banning. This is different than banaction in that you can also include emailing destemail. See the comments around this option for details. I changed it from the default of %(action_)s so that emails would be sent out with the whois details of the IP that was banned.

Restart fail2ban: sudo service fail2ban restart

Notes

The default way fail2ban sends emails is using sendmail. Be sure it's installed: sudo apt-get install sendmail. If you want to use something else, look for the mta = sendmail line in our [jail.local].

Since fail2ban works for SSH by default, you don't need to enable it. However, fail2ban supports monitoring several other services. You'll see configurations for them in your [jail.local] file with headings surrounded by square brackets. For example, ssh is [ssh]. You can also override default settings under these headers. It's a good place to increase bantime and maxretry for only one service.

Sources

DigitalOcean Community: How To Protect SSH with Fail2Ban on Ubuntu 14.04

Ask Ubuntu: potential ufw and fail2ban conflicts

[donut2d]

Getting a never-expiring page access token on Facebook

My answer to this StackOverflow question.

It seems that Vlasec's answer is a bit out of date. Following the instructions laid out in Facebook's Extending page tokens documentation I was able to get a page access token that does not expire.

I suggest using the Graph API Explorer for all of these steps except where otherwise stated.

0. Create Facebook App

If you already have an app, skip to step 1.

Go to My Apps.

Click "+ Add a New App".

Setup a website app.

You don't need to change its permissions or anything. You just need an app that wont go away before you're done with your access token.

1. Get User Short-Lived Access Token

Go to the Graph API Explorer.

Select the application you want to get the access token for.

Click "Get Access Token".

In the pop-up, under the "Extended Permissions" tab, check "manage_pages".

Click "Get Access Token".

Grant access from a Facebook account that has access to manage the target page. Note that if this user loses access the final, never-expiring access token will likely stop working.

Token that appears in the "Access Token" field is your short-lived access token.

2. Generate Long-Lived Access Token

Following these instructions from the Facebook docs, make a GET request to

https://graph.facebook.com/v2.2/oauth/access_token?grant_type=fb_exchange_token&client_id={app_id}&client_secret={app_secret}&fb_exchange_token={short_lived_token}

entering in your app's ID and secret and the short-lived token generated in the previous step.

You cannot use the Graph API Explorer. For some reason it gets stuck on this request. I think it's because the response isn't JSON, but a query string. Since it's a GET request, you can just go to the URL in your browser.

The response should look like this:

access_token=ABC123&expires=5182959

"ABC123" will be your long-lived access token. You can put it into the Access Token Debugger to verify. Under "Expires" it should have something like "2 months".

4. Get User ID

Using the long-lived access token, make a GET request to

https://graph.facebook.com/v2.2/me?access_token={long_lived_access_token}

The id field is your account ID. You'll need it for the next step.

5. Get Permanent Page Access Token

Make a GET request to

https://graph.facebook.com/v2.2/{account_id}/accounts?access_token={long_lived_access_token}

The JSON response should have a data field under which is an array of items the user has access to. Find the item for the page you want the permanent access token from. The access_token field should have your permanent access token. Copy it and test it in the Access Token Debugger. Under "Expires it should say "Never".

[donut2d]

Vagrant: "mount.nfs: requested NFS version or transport protocol is not supported"

I ran into the following error when changing an existing Vagrant to use NFS:

The following SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed! mount -o 'vers=3,udp' 192.168.7.1:'/Users/donut2d/example.com' /vagrant Stdout from the command: Stderr from the command: mount.nfs: requested NFS version or transport protocol is not supported

I at least temporarily resolved this by removing the /etc/exports file. I dunno the problem will show up again, but this solved it for me.

I found this answer in this Github issue.

[donut2d]

APC and "redeclared class" PHP issue

With a Drupal site I ran into this error message:

PHP Fatal error: include_once(): Cannot redeclare class insertquery_mysql in /usr/local/www/example.com/web/includes/bootstrap.inc on line 1100

This happened after I pushed an update. Eventually, I found this help page from Pantheon. It suggests wrapping the troublesome class definition in a if (!class_exists('InsertQuery_mysql')) {} block. That solved my problem initially and I thought I was good.

Then I pushed another update and the problem returned. I then went and used their suggestion of clearing APC's opcode cache. This can be done by running apc_clear_cache() in a PHP script. They recommend uploading a file with just that to your server for this purpose. Gave that a shot and it didn't help a bit.

Finally, I came across When APC appears to fail Drupal on the Malone Thought Repository. This article links to a lot of other people reporting this issue and possible solutions, mostly just to do what I tried above. But, he also noted that this is sometimes called by the use of call time pass-by-reference (which was removed in PHP 5.4).

I had stopped using call time pass-by-reference years ago and so I thought surely this is not the case. But I had run out of options and there is a lot of third-party code in my site so I thought, it can't hurt. Like he recommends, I installed the PHP Code Sniffer package using Homebrew (you have to first install a version of PHP using Homebrew, it didn't like the version that comes with OS X) and ran the following command:

phpcs --extensions=php,module,inc,install --standard=Generic --sniffs=Generic.Functions.CallTimePassByReference /path/to/modules

Sure enough, I had accidentally added an ampersand before a variable I passed to a function call. I removed the ampersand but still no good. After more messing around, I finally restarted MySQL and Apache on all of the servers related to this site and I haven't had any issues in the 3 or so hours since.

I'm not sure what will happen the next time I push an update, but hopefully it will be nothing.

Also, I read some thing about this being fixed in later versions of APC but my servers were running the latest stable version at the time of writing (3.1.9).

[donut2d]

A simple library for creating a progress circle using SVG.

[donut2d]

SSH Tunnelling (on Mac OS X)

Tunnel all traffic through a server supporting SSH

Based off of this post by Drew Moris.

Run the following command in your terminal with the correct server details:

ssh -C -N -D 1357 [email protected]

-C says that all transfered data between you and the server should be compressed.

-N says that you don't intend to actually work on (log in to) the server, just forward the ports.

-D basically says that you want to setup a socks proxy on the following port, "1357" in this case. This is a dynamic (hence the "D") in that it will forward all traffic over the socket instead just specific ports.

Run man ssh and to get a more complete understanding of these options (hint: use / to search for them directly). These options can be combined so that the command reads ssh -CND 1357 [email protected]. The D must be at the end so that you can specify the port.

Now, you can just setup one of your browsers to use this new proxy through its proxy settings. But, that would only cover the traffic from your browser. If you're like me, you'll want all your traffic to go through this server.

In OS X (I'm running 10.9):

Go to [ > System Preferences...] and select "Network".

Select your active network connection (has a green icon next to it and says "Connected", most likely "WiFi") and click "Advanced...".

Go to the "Proxies" tab and check the "SOCKS Proxy" checkbox.

Set the "SOCKS Proxy Server" to "127.0.0.1" and set the port to "1357" or whatever you used when running the command above.

Click "Ok" and then "Apply".

Now if you check your external IP it should be the same as the server you connected to.

Forwarding a single port

This is simpler:

ssh -NL <local port>:<remote address>:<remote port> [email protected]

After running this, any attempt to connect to "localhost:\" will be forwarded to "\:\". So, if I ran

ssh -NL 5022:another-example.com:22 [email protected]

and then ran

ssh -P 5022 [email protected]

I would be sshed into "another-exmample.com".

[donut2d]

Setting up M/Monit & Monit

Installation

I used the official installation instructions for the initial install. Specifically, Used the Fedora method with yum install monit. It installed the somewhat old 5.2.X version, but that is new enough to work with the latest version of M/Monit.

monitrc Control/Config File

Monit looks for the "monitrc" file in [~/.monitrc] but I haven't figured out for which user this should be. Presumably, the user it runs under (root), but the offical monit docs say that it can be in several places, including [/etc/monitrc].

However, I install this on Fedora using yum and the configuration it uses is [/etc/monit.conf].

Connecting Monit to M/Monit

The M/Monit documentation PDF has details on this under the "Configuration" section.

[donut2d]

What I Learned Today

A list of some useful apps. Including the one I'm using to write this. The article these Hacker News comments are for is good, as well. But there is a lot of good in those comments.

CoffeeScript class proprties are static by default.

I always forget this, but if properties defined as part of the class definition then they are effectively static.

So,

class Goomba awesomes: [] constructor: (awesome, jawesome) -> @awesomes.push awesome @jawesomes = [jawesome] puts: -> console.log @awesomes, @jawesomes thing1 = new Goomba 'hello', 'anyong' thing1.puts() # Outputs: ['hello'], ['anyong'] thing2 = new Goomba 'world', 'bulgogi' thing2.puts() # Outputs: ['hello', 'world'], ['bulgogi'] thing1.puts() # Outputs: ['hello', 'world'], ['anyong']

Hopefully I'll remember this now.

Shell scripting stuff

A function to exit with a message and error code.

[donut2d]

Auto install/update VirtualBox Guest Additions for Vagrant VMs

vagrant-vbguest - A Vagrant plugin to keep your VirtualBox Guest Additions up to date

Works really well.

[donut2d]

Benefits US border guards could offer travellers

Every time a non-citizen enters the US they must have their finger prints scanned and their photo taken. Possible features the US border guards could add to help their image:

Every 5 years they send you a photo album with all the photos they took. This could include copies of the finger prints.

Add some basic health checks to the process so they can let you know if you should have something checked out.

[donut2d]

I ended up using this voxel tutorial from Alexandros Stavrinou. It was very helpful and I feel like I finally have a good enough understanding that I can really start getting into creating my game.

I did the tutorial all in C# but have since moved to Boo and have no regrets, yet. The docs are decent and I've received good, in depth answers on the Google group and I am just tired of the C-style languages with all their unnecessary characters. I like languages like Boo, Python and CoffeeScript. I feel like my brain has an easier time understanding what's happening when there's less noise.

My initial fear of not using the more popular C# or UnityScript (Unity's weird JavaScript thing) was that I would not be able to access plugins or libraries written in anything except for Boo. It looks like this wont be much of a problem: I just have to make sure I put things written in other languages in folders that compile before they are referenced. See Unity's Special Folders and Script Compilation Order doc.

I'm happy with my progress so far. Things always feel like they're going slower than they should. But I almost always feel that way when getting into new things. I'm excited for what's to come.

[donut2d]

I'm looking into building a block based game. I've only really programmed with web technologies so far. The engineer in me wants to build a proper voxel system before jumping in. But I think it would be wise to create a prototype of my idea with just the basics of Unity before going in too deep. Know what I want to build I'm fairly certain it wouldn't require too many blocks to hurt performance, at least not for the prototype.

But, once I do want to get more serious about voxels, it looks like this CoreDev tutorial by Myhijim would be a good start. Well, assuming it gets further along.

I've found a lot of articles about the ideas and advantage of a voxel based system but very little on the actual, technical implementation in Unity.