Sourcegraph @ GopherCon 2014

We're blogging live from the first annual GopherCon in Denver, Colorado!

[sourcegraph-blog]

Elliott Stoneham's lightning talk slides on Mutant Go Compilers

Elliott Stoneham @ElliotStoneham spoke about transpiling Go to other languages such as JavaScript, Java, and C++. See Elliott's previous blog posts on GopherCon here:

Reflections on Russ Cox's keynote about converting C to Go

Other Go runtime targets

Running untrusted Go code as part of your application

[sourcegraph-blog]

Reflections on Russ Cox's C-to-Go talk

Guest post by Elliott Stoneham @ElliottStoneham. Elliott is author of TARDIS Go, a cross-language Go transpiler. Also check out his previous 2 guest posts during GopherCon on other Go runtime targets and running untrusted Go code as part of your application.

Here is our liveblog writeup of Russ Cox  @_rsc's talk yesterday at GopherCon.

I’ve spent the last few months championing the idea of “mutant”  Go-to-X transpilers, so Russ Cox’s C-to-Go talk was of particular interest to me.

Russ Cox spoke about the practical issues of source-to-source compilation when moving the existing Go compiler code, which is written in C, into Go. Which everyone reading these words will think is a wonderful idea.

Knowing the extent of the code you want to transpile has some significant advantages:

First, it gives you the fallback option of re-writing the source code itself in order to solve the problem, rather than forcing the translation code to deal with ever more unusual cases. A good example of this was Russ’ description of handing C goto statements.

Second, it allows you to know how source language features are actually used, and only code for those use-cases. Russ’ description of how to handle C union (by mapping it to Go struct) was a great example of this.

But for me, the real jewel in the crown is how type mismatches between C an Go are solved. Russ gave a great example of how the int type in C is actually a Go bool in some usage cases. The translation process therefore infers what Go type to use based on a variable’s usage pattern in the C code. Some of the techniques he described for working this out sounded positively magical.

Most impressive though, is that it is being done at all. A lesser programmer, like me, would have created a new type in Go that mimicked the operation of the C type to be replaced, and suffered the runtime overhead of that decision for a potentially very long time.

So a consummately professional job is going to be done on this C to Go conversion process. 

Russ himself is very clear that his code will not be a general purpose tool. Once complete however, I have no doubt that the Go community will take what Russ has done and seek to generalise it into an all-purpose C-to-Go transpiler, which will be an impressive tool to see.

So what lessons can we in the small “mutant” Go compiler community learn from Russ’ approach?

Firstly, it validates that source-to-source compilation is a viable approach, which is great news for those of us trying to convince the skeptical.

But secondly, and perhaps more importantly for the success of our projects, it demonstrates that compromises have to be made in the code of the source language in order successfully translate it into the target environment.

Take GopherJS (github.com/gopherjs/gopherjs) for example, it generates very fast JavaScript from Go, but currently does not permit goroutines or channels. Is that a price worth paying for a usable JavaScript implementation of most of Go?

Similarly, my own TARDIS Go (github.com/tardisgo/tardisgo) does not have very good support for the Go standard library. But it does give access to the cross-platform Haxe (haxe.org) standard library, which is available across all 7 target languages (JavaScript, Flash, PHP, C++, C#, Java and NekoVM). Is restricted access to the Go standard library a price worth paying for wide portability?

You can find the slides for my lightning talk on “mutant Go compilers” at 

https://speakerdeck.com/elliott5/mutant-go-compilers-gophercon-2014-short-talk.

[sourcegraph-blog]

Official repository for GopherCon 2014 talks and slides

Check out the official repository for talks and slides from GopherCon: https://github.com/gophercon/2014-talks.

[sourcegraph-blog]

Andrew Gerrand: GopherCon 2014 closing keynote

Andrew Gerrand @enneff joined Google and the Go team in Feb 2010. After joining, he had to rethink some of his preconceptions about programming from his prior work in Python, PHP, JavaScript, etc. He talked about what he has learned in his 4 years on the Go team as well as some general insights about programming.

(Ed. note: This was a fantastic talk that our note-taking wouldn’t do justice to, because it contained a lot of spoken insights, pithy bullets, and small code snippets. Also, it’s getting near dinnertime for your editors. We’ll link to the slides once they’re up.)

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Interfaces

Andrew gave demonstrations of all the ways in which interfaces are profound and core to Go.

Concurrency

Andrew’s first exposure to concurrency was in C, Java, and Python. Later, he encountered the event-driven concurrency model in Python and JavaScript. His initial reaction to Go’s concurrency model was, “Why can’t I wait on or kill a goroutine?”

He walked through the Go tour’s module on tree equality. http://tour.golang.org/#72

Why can’t Go code kill goroutines? Because that will encourage programmers to rely on thread-local storage. And that means that all code in the goroutine needs to be aware of which goroutine it’s in.

Andrew: "Concurrency is not just for writing faster code. It’s for writing better code."

(Ed. note: Again, we recommend checking his slides, which have much more content, once they’re posted.)

Syntax

Initially, Andrew thought Go’s syntax was confusing and inflexible. 

Error handing

Go makes error handling as important as your other code.

Packages

Packages are probably Andrew’s favorite feature of Go.

Documentation

Godoc makes it easy to write documentation as you write code, and doing so makes your code better.

Conclusion

Andrew’s closing message: “Let’s build small, simple, and beautiful things together.”

[sourcegraph-blog]

Building Web Services in Go

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Richard Crowley @rcrowley, of Slack, talks about building web services in Go.

Richard's definition of a web service: "Data and a set of operations on that data networked using the http protocol; structured and machine-parseable requests and responses (otherwise, it's more of a web application, not a service)."

Hello, www example

Pretty straightforward, but let's zoom in on the handler function signature. It corresponds to this interface:

type Handler interface {     ServeHTTP(ResponseWriter, *Request) }

This is the lowest common denominator of a web service in Go. Embrace it, and use it as much as possible. Let's rewrite "Hello, www" to use the interface.

JSON & Error handling

We want our web service to read and write JSON, and handle errors.

To parse and handle JSON, we can wrap the initial handler in another handler that handles the marshaling and unmarshaling of JSON to Go structs.

Go's philosophy is that errors are not exceptional. You deal explicitly with errors. Error handling in the web service should be the same. The client should get as much information as possible.

You can make your JSON handler wrapper handle errors from the underlying handler and automatically marshal and unmarshal these errors into JSON. This way of handling errors conveys all the information to the web service client that you would see inside Go code.

Request multiplexing / build your own mux example

Now we want our web service to support (potentially very many) multiple API endpoints (request multiplexing). In Go, it's all about wrapping/composing handlers ("handlers all the way up").

Richard walks through building our own multiplexer (similar to the standard library's request.ServeMux() handler).

You could do a linear scan through individual path handlers, but this is tedious and slow. It's better to use a trie, so you can prefix match URLs quickly. You might also want to add wildcards.

How not to repeat yourself

In a given web service, there are some operations you want to do in many handlers (e.g., auth, rate limit checking). The solution, again, is to compose handlers.

You can implement middleware chains as a handler that calls a list of sub-handlers. The first handlers can handle rate limit checking, authorization enforcement. The last handler does the "actual work." You can detect the first responder that writes a response (and therefore should prevent subsequent handlers from being called) by setting a written=true in the ResponseWriter header.

Visibility

What we've covered so far is functionality. Now Richard talks about getting visibility into the status of our running service (e.g., logs and metrics).

Logging

Again, you can wrap your handler in another handler that automatically logs requests and responses to wherever we want. So the underlying handler is completely agnostic of logging.

Metrics

Wrap the handler with something that records the time before running the underlying handler and records the finish time after it completes.

Who calls the first (parent) handler?

http.Server is what reads a raw request off the wire and passes it onto the top-level handler.

Gracefully stopping

Gracefully stopping a server is hard. There is a tradeoff between safety and liveness. You could wait an arbitrary amount of time for currently running handlers to complete or kill everything now and leave it in a potentially corrupt state. Neither is ideal.

In Go 1.3, there is more support for gracefully dying via the ConnState callback and SetKeepAlivesEnabled method.

Now, in server.Close(), the server will use these to signal to handlers that they need to die, and there is a grace period for this to happen.

Parting words

Embrace http.Handler interface as the fundamental building block

Provide abstractions within handlers, not as one giant handler to handle everything.

Compose bottom-up

Errors are not exceptional

Links

https://github.com/rcrowley/go-metrics

[sourcegraph-blog]

Build Your Developer Tools in Go

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Alan Shreve (@inconshreveable) is the author of ngrok and gave us a breakdown on living outside the Cloud and building distributable production applications in Go. It’s a scary world out there when you leave the walled garden that is very much in your control with web-services. Alan’s talk focused on deploying to user-land and the challenges associated with these types of applications.

Motivation

ngrok is a tool that runs on your laptop. A thought experiment to the audience: Imagine you have some software that you need to deploy 100/times a day, on wildly different platforms/hardware, and every deployment continues to run forever. This is a fundamental challenge, and very different from server deployment. Alan wanted to try to solve this problem using Go and make it easy to make quality software.

Embedding Assets

How do you embed all of your static assets (js/css/etc.) that you need for your program to run? In the server-space, you have full control and can ensure that the assets remain in the right place. Unfortunately, when dealing with end-users, you have to worry about them breaking things (for example, moving things on the filesystem).

Wouldn’t it be nice to have one deployment artifact? 

This is more than just a deployment nicety, it really makes it much easier to manage down the road. A helpful tool for this in Go is go-bindata. It’s a command-line tool that automatically generates source files with variables containing the raw bytes of files in a static assets folder. Also handles gzipping the contents for you!

Example usage:

bin/go-bindata     -o=src/ngrok/client/assets/assets_release.go     -pkg=assets     -tags=release     -debug=false     assets/client/…

Now, you can build with go build -tags “release”.

Cross-compiling

Simple answer, don’t do it. Just compile natively. Most practical solution is to automate using a VM tool like vagrant.

Why don’t we want to cross-compile? Answer: Cgo. More like Cno! Lot’s of problems like:

user.Current() doesn’t work when cross-compiling to Linux/Darwin

native resolver not used in cross-compiled builds, bgrok ignores resolve.conf

…but if you must cross-compile, gox is worth looking into.

 Terminal UX

Alan recommended checking out termbox-go. Its cross platform, pretty low-level, and totally awesome.

A specific pain point is dealing with windows. Users would just double-click the executable and expect it to install. They didn’t realize you need to run in it’s own terminal session.

To handle this, Alan built a tool called mousetrap that can capture mouse-events and detect a double click and raise a message to the user.

Tracking Crashes

Another problem is when your software crashes. When this happens server-side, you have lots of ways to track this. But when it happens on a users machine, there is no easy way to get this reported back. Alan wanted to be able to capture this information and be able to do something about it. 

Initially he tried a non-automated solution that asked users to drop the stack trace to a github issue. Guess how many people took advantage of this? Exactly one, which told Alan that ngrok NEVER CRASHES! 

Obviously people don’t want to manually report things. So Alan is looking to get auto-reporting built-in through an atexit() func. Alan is playing with the ast ast to wrap everything and auto-send when a crash occurs. Watch his twitter account to see how this pans out. The jury is still out.

Auto-update

How do you get new versions out to your users? Simple three-step process:

Download new binary

Move current binary to a different spot on the filesystem

Move the downloaded binary into the old place.

Alan has created a library to help with this: go-update. Just updated today. New API changes out updating and checking into separate. Also has binary diffing and signature verification built-in.

Now that you have auto-updating, how do you know what URL to download from? Alan is working on equinox.io. A site which will handle a lot of this for you. Includes Stable vs. Beta vs. Custom release types. Has a REST API that allows you to POST up your releases. Definitely worth checking out!

—————-

Dennis Coldwell is an engineer at Junction and loves playing with shiny auto-updating things.

[sourcegraph-blog]

Building Database Applications with database/sql

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Baron Schwartz @xaprb, CEO of VividCortex, shows us how to properly use Go's database/sql package (using MySQL as an example).

An excellent tutorial here: http://go-database-sql.org/

Schwartz's initial impression of the database/sql package was that it repeated some common mistakes in database connection interface libraries. But upon closer inspection, he changed his mind. [He shouts out to Brad Fitzpatrick for making the package so awesome.]

Basic concepts

Open() – creates a DB

Close() - closes the DB

Query()

QueryRow()

Exec()

Row - A row is not a hash map, but an abstraction of a cursor

Next()

Scan()

Note: a DB is not a connection. 

Some patterns and code snippets

He shows a simple, but complete, program that demonstrates all the basic concepts of how to properly use the database/sql package. Note that at "// What do we have here?", you do not have a connection to the database, because you haven't done anything. database/sql creates and destroys connections for you underneath the hood.

A DB is not meant to be opened and closed at function boundaries. It should essentially be global and stick around for the entire duration of the program. Connection pooling is done for you under the hood.

QueryRow("select ...") - Use this when you expect the query to return a max of 1 row from the database. A row is not a higher-level abstraction (e.g., hashmap) over the row. It is a cursor (which is more memory efficient, puts the power in your hands), so you need to use Scan().

Query("select ...") - Don't concat strings together to avoid SQL injection attacks. Be safe. Use '?' parameters and let the driver prepare statements. Returns a Rows (rows set), again, not a blob of memory with arrays/hashes, etc. It's just a cursor. Use rows.Next() and rows.Scan() to advance the cursor and scan the value out of the row and copy them into destination values (interfaces or pointers).

Afterward, remember to check for an error. Remember to close the Rows object (defer rows.Close()). Otherwise you will leak a database connection.

Explicit preparation of statements - the prepared statement in Go is NOT necessarily bound to a single database connection. If the connection used to prepare the statement is busy, the library will try to re-prepare the statement and use another connection. This is a little bit of magic.

Use stmt.Exec() for insert and delete, since they don't return rows. If you use Query(), it will return a Rows (which you probably won't close) and you will unnecessarily hog a connection until the Rows object is garbage collected.

Further advice and good-to-knows

Worth reading the documentation carefully (good, but dense; doesn't repeat itself). E.g., there is a connection pool, so don't do this yourself. And you can't use uint64 with high bit set in parameters (#gotcha); this can bite you if you have auto-increment keys with on int64 column.

Don't defer() in long-running functions. Memory usage will keep going up. Don't use cxn state. A lot of people have a habit of setting session variables before executing queries, but here, the operations you execute are not bound to a specific connection. Use Tx to bind to a connection (call db.Begin()). And on that note, don't do db.Exec("BEGIN ...") or db.Exec("COMMIT ..."). Use db.Begin() and db.Commit().

db.Exec will retry a query 10 times if it fails. So if you have a bad query, you log into MySQL and kill the connection, it will rise from the dead 10 times.

There are some issues with leaking/not deallocating prepared statements.

Use provided Null types (e.g., NullString), because you can't scan a null into a string.

You can create custom types using the built-in interfaces. Use case: transparently encrypt/compress data stored in db. So if, God forbid, someone executes a successful SQL injection, they won't see anything useful.

Links

http://golang.org/pkg/database/sql http://go-database-sql.org https://github.com/go-sql-driver/mysql http://jmoiron.net/blog @VividCortex

[sourcegraph-blog]

Panel discussion with Go team members

Blake Mizerany’s scheduled talk, “Little Known Tricks in the Go Standard” was made redundant by other talks, which covered all of Blake’s material. Instead, Blake hosted a panel discussion with 4 members of the Go team: Rob Pike, Andrew Gerrand, Brad Fitzpatrick, and Robert Griesemer.

(Ed. note: We’ve paraphrased most of the responses, and we’ve omitted the answers that referenced personal stories or jokes that wouldn’t come through in writing. Watch the video (which will be posted later) if you want to see the full Q&A. Let us know if we’ve missed or mistranscribed anything.)

Q: Rob (Pike), what was the most important thing you learned in C++ that affected Go?

A: Build time matters more than anything else. There are so many other things I could think about, but they’re all “what not to do.” The amount of extra typing you must do is awful. One afternoon, I typed the same function signature 56 times. 

Q: Robert (Griesemer), you worked on V8 before coming to the Go team. What have you brought to Go from that experience?

A: I worked mostly on the V8 internals. So, I wasn’t necessarily an expert on JavaScript, just as you can write the JVM without knowing Java. I also didn’t grow up with a C and Unix background, as the rest of the Go team did.

Q: What were the first Go programs you wrote that genuinely surprised you?

A from Andrew Gerrand: Probably the http.HandlerFunc type, which implements the http.Handler interface, but it’s a function type. When I first saw this, it was like “woah, that’s weird.”

A from Rob Pike: http.HandlerFunc was going to be my answer. Another thing was the encoding/xml package, where you just write the struct with tags and encoding/decoding just happens.

A from Brad Fitzpatrick: When there’s a named type that’s not a struct.

A from Robert Griesemer: When I realized you can have a non-nil interface that you assign a nil value to. (Brad Fitzpatrick agreed. Rob Pike: “nil is a value, too!”)

Q: What was the most awkward thing you’ve seen written in Go?

A from Andrew: Some of Brad Fitzpatrick’s tests. And bad concurrent code.

A from Robert: I still see a lot of “if boolVar == true”. And for loop index variables that have long names.

Q: What contribution you made to Go are you most proud of?

A from Brad: net/http. (Rob Pike: “I’d like to thank Brad for the net/http and os/exec packages. I think they’re just marvelous.”)

A from Andrew: map deletion syntax.

Q: What do you wish you hadn’t contributed to Go?

A from Rob: netchan. Also, variable declarations. They could be cleaner and less confusing.

A from Brad: C constants in the os package that are named like C, like os.O_RDONLY.

Q: The implementation of sync.Pool uses all kinds of stuff in the runtime that we don’t have access to (such as garbage collector internals). Why?

A from Rob: It’s the job of the stdlib to give you controlled, clean access to certain things. The behavior of the garbage collector is something that should probably be hidden from user code.

Q: What was awkward about var decls? [Referring to Rob’s statement above.]

A from Rob: Shadowing. Colon-equals syntax is good, but it looks like an assignment (not declaration). There are so many ways one can declare a variable. Also, you can’t declare the same things with “var” that you can with colon-equals, so it’s not clean. It all works pretty well in practice, but there are some corner cases that’d be nice to clean up.

Q: Are there any plans to make the garbage collection in Go pluggable?

A from Rob: It’s open source. If you make a patch to improve it, we’ll accept it. There are no plans to make it pluggable, but we’re always trying to make it better. Note that Go’s garbage collector doesn’t have a lot of flags (like Java’s garbage collector). That’s on purpose.

Q: Why are build tags in comments?

A from Andrew: Because they’re not part of the language, they’re part of the build system. The language is interpreted by 6g, and the comments are interpreted by the go tool.

Q: What is the next thing that Go can do to increase the rate of its growth?

A from Andrew: I’m really excited about Go moving into new spaces. Most people use Go to write servers now, but it’s exciting to see Go being used in the embedded (ARM) space, and Gustavo Niemeyer’s graphics stuff. Also, desktop apps, mobile apps. We want to help improve support in these areas in the coming years.

A from Robert Griesemer: I’m looking forward to a Go compiler written in Go. I think it could be much faster than it is now. (Rob Pike: “Yeah, Go builds are way too slow. [laughter] I mean it.”)

Q: What about Go 2.0?

A from Andrew: We have a label in the issue tracker for Go 2, and we use it to label issues that we can’t address yet. We’ll address those issues when the next breaking release comes along. We’re learning from how other languages have made breaking transitions, such as Python 2 to 3. Overall, Go 2 is not top of mind for us right now.

Q: You blogged about the unexpected uptake of Go in the dynamic languages community (as opposed to the C/C++ community). Does that change your plans for the language?

A from Rob: It hasn’t changed the language, but it has changed my own perception of the way you can use it, what’s important, serialization, etc. We had no idea that dynamic language people would adopt Go this much when we designed the language; it happened well after the initial design. Even if we knew about it in advance, I don’t think it would have changed the language design much.

Q: What’s the best thing about your open source contribution process?

A from Rob: Windows support was entirely from open source contributors. It’s a complete vindication of the open source ideas.

Q: Have you looked into embedding Go in C, Rust, etc.?

A from Rob: We’ve thought about it a lot and want it to be possible. The challenge is that Go’s runtime is built in such a way that it must own everything: memory, signals, etc. All of this is dictated by Go’s concurrency model and implementation. E.g., stack switch when switching between C and Go in cgo. Embedding is a tricky thing, but we really want to see it.

A from Robert Griesemer: Also, since we have a garbage collector, we need to track pointers on either side of the language fence.

Q: Brad, how did you design the HTTP keepalive pool?

A from Brad: Someone from the Chrome team wrote a long blog post about their HTTP keepalive support. I read this blog post and heard about the performance benefits they got from this. Adding it to Go took just a few lines, but the Chrome implementation required many more lines of code.

Q: If you could add one thing to the Go stdlib, what would it be?

A from Andrew: A superb, universally cross-platform graphics toolkit that looks amazing.

A from Rob: We have a lot of good tools for static analysis. I’d like to see a good tool for dynamic analysis. I haven’t seen any languages with the kind of tool I want.

A from Brad: Android, iOS, generics. [audience laughter]

A from Robert Griesemer: I’d like the compiler to be a library so I can call it and get an executable that I can invoke from code.

Q: What’s the best way to work around the lack of generics and polymetric polymorphism?

A from Rob: If it’s not performance critical, just use reflection. Otherwise, use Robert Griesemer’s fantastic libraries for source code rewriting, and write a tool to automate the writing of source code. The whole computing community underappreciates the value of programs that write programs. This is a perfect use for them.

Q: What are the plans for Go Oracle?

A from Robert Griesemer: This is really Alan’s work. My contribution is the go/types package, which is essentially complete but probably has some bugs. The first true implementation of this is in Go 1.3’s godoc, which has the Oracle’s functionality built in.

A from Andrew: A lot of people want autocomplete in editors. My main issue is navigating the code I’m working with, and understanding that code. Once I understand the libraries, remembering what to type is not the biggest issue. I’d love tools that help you understand how code fits together in a foreign codebase.

Q: How did interfaces and structural typing come to exist?

A from Rob: In the first or second day, we realized we wanted a pure behavioral type, so you don’t conflate the data and methods.

A from Robert Griesemer: We debated for a while whether you should be able to have concrete methods on an interface. We concluded that not having concrete methods would keep the concept of interfaces orthogonal from the other concepts in the language.

Q: What is your favorite Go keyword?

A from Andrew: notwithstanding

A from Rob: go

A from Brad: defer

A from Robert Griesemer: go

Q: What is your least favorite Go keyword?

A from Andrew: thetruthofthematter

A from Rob: fallthrough

A from Brad: fallthrough

A from Robert Griesemer: goto

(Ed. note: Don't believe that notwithstanding and thetruthofthematter are real Go keywords? We didn't, at first. grep the Go source to prove it for yourself.)

[sourcegraph-blog]

Gobot: Go-powered robots

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Adrian Zankich and Ron Evans, members of the Hybrid Group and creators of KidsRuby, talk about Gobot, a library for programming robots in Go.

Ron leads with a quote from the Go FAQ, "By its design, Go proposes an approach for the construction of system software on multicore machines." He then follows immediately with a quote from William Gibson: "The street finds its own uses for things." Thus Go for robots.

Gobot uses the adapter pattern. Devices have drivers (go packages). Pub/sub system to communicate events from devices to robots (in code, you call goBot.on(<event>, <handler func>)).

API calls are routed through what they call the "Master Control Program" or MCP.

Gobot supports test-driven robotics (If you're designing, for instance, a sprinkler system that only goes off once a week).

Gobot-related libraries

There's a command-line interface (gort.io), because you don't want to use a GUI to configure hardware.

RobotOps is DevOps for robots (robotops.com)

Demo: Gobot and Arduino

There is a Gobot adapter package for Arduino. Ron shows a code example, which is quite simple. Gobot uses firmata, a protocol for communicating with microcontrollers from a host machine (there's a simple Go package that encapsulates this). The code is simple and clean – they create a firmata connection to a device and send messages over pins. The device in this case is an LED connected to the Arduino.

Demo: Gobot and Beaglebone Black

There's an adapter for Beaglebone Black, a single board on-chip ARM Linux computer. Much pins, such I/O. The code uses the same device drivers with a different adapter (Beaglebone instead of Arduino). This involves just swapping out the Beaglebone adapater package for that of the Arduino. They do the same demo with the LED now plugged into the Beaglebone.

In the Arduino, the PC was running code and remotely issuing instructions to Arduino. With Beaglebone, code is running on Beaglebone on its onboard chip. The code hasn't really changed, except for the adapter.

Robeaux.io is a web app (built with AngularJS) that allows you to issue input to your program, controlling the robot from your browser. [Adrian demos strobing the LED by clicking on an HTML button.]

[Free stuff giveaway interlude. Much tweeting ensues.]

Demo: Conway's game of life... with Robots (Spheros).

Rules of Conway's game of life here: http://en.wikipedia.org/wiki/Conway's_Game_of_Life.

With physical robots, they approximate the number of neighbors of a cell (i.e., Sphero) by taking the inverse Fourier transform of the approximate frequency of collisions.

The Gobot "application" code for this demo is about 100 lines long.

Demo: AR Drone + Digispark

Digispark is one of the smallest microcontrollers. Digispark is plugged into AR Drone via USB. They have 2 Gobot instances, one running on a laptop communicating with Digispark (to toggle the LED) and one running on the computer inside the drone itself (to fly the drone).

Demo: AR Drone + OpenCV

OpenCV is a powerful open-source computer vision library. Gobot has a Go wrapper library around OpenCV. The code loads training data to train a face detector. They hook up a Drone camera to OpenCV and laptop screen. The drone flies up to face level, hovers, runs face detection. Code is less than 100 lines long. Very slick.

Demo: AR Drone, Neurosky, OpenCV, PS3 Controller

The code, again, is not much more than 100 lines of code. They use Gobot to fly the drone while observing Adrian's (the pilot) brain state as a histogram of brain wave types.

The site is Gobot.io. Gobot is all open source, Apache 2.0 license.

Corrections: The original version of this post misstated Ron's name as "Rob" (perhaps short for "Robot"?) and incorrectly stated that Robeaux.io is a NodeJS app.

[sourcegraph-blog]

Running untrusted Go code as part of your application, an outline idea

Guest post by Elliott Stoneham @ElliottStoneham. Also check out his guest post from yesterday on other Go runtime targets.

I’m writing this blog entry as a reflection on the first day of GopherCon.

One of the great strengths of Go is the ease with which one can reference and use an existing package created by a third party, often on an open-source basis.

But there are inherent risks with this. The version and change control issues are well understood, with a number of approaches to managing them evolving, for example vending. However the security issues have been less well discussed.

When your application uses a third-party Go package, you currently give that software author unrestricted access to all of your data and systems. There are tremendous opportunities for badly written or malicious code to be unwittingly included inside production systems.

What we need, I think, is to implement security controls on un-trusted Go packages.

Luckily, as Rob Pike said at GopherCon yesterday, the package location string could contain additional elements besides the package URL. So some top-level trust-control instructions could sit here. Trust control should certainly be part of the programming code, rather than an external addition. So for example one might write:

import “github.com/coolhacker/usefulrepo,trust=none”

As with using Go on AppEngine at the moment, trust controls that might be implemented include access to:

multiple CPUs;

the network;

the filesystem; and

other packages.

I would expect the untrusted Go code to have no visibility of the trusted code’s namespace. 

Clearly rules also need to be put in place to match the named types at the trusted and untrusted code boundary.

There are two approaches to implementation, depending on the execution needs of the third-party package.

If the third-party package, or a package it relies on, needs to use standard package “unsafe” there is really no alternative but to use a separate execution space for that code, where there is no possibility of it accessing the main application’s memory space. 

But if, like code for use on Go App Engine, the third-party package does not use standard package “unsafe” an alternative is available. 

Automated code refactoring can be used to generate a trusted version of the un-trusted code, that can then be called using the normal Go approach. 

I know this can be done, because my mutant Go compiler, TARDIS Go, already does something very similar, only it currently emits Haxe rather than Go.

So I want to see a new tool emerge which would vendor third party packages, but with the addition of security controls added through automated code refactoring. 

Is anyone else interested in working on this? 

Catch me at GopherCon on the Hack Day, where I’ll be speaking, or @ElliottStoneham.

—

Posted by Elliott Stoneham @ElliottStoneham.

Elliott started programming in 1974, he exhibited at the first Internet World Show in London and wrote the website that sold the first insurance product online in the UK. After working as a systems programmer into his 30s, Elliott spent two decades in IT and general management. Outside of IT, he has two master’s degrees in business administration and Chinese (普通话). He is also holds qualifications in computing, coaching, finance and teaching. But recently he has been tempted back to his programming roots by the elegant Go language. His current personal mission is to write a Go compiler targeting the amazing Haxe language. Professionally, he is a freelance Go developer. He currently works for Countersoft, helping to develop their new Documize product.

[sourcegraph-blog]

Stephen McQuay and Fraser Graham: Why Killer Robots Are a Concurrency Problem

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Stephen McQuay @smcquay and Fraser Graham @twisted_weasel talked about building hackerbots, a cool and nerdy game for hackers written in Go.

One day, Stephen told Fraser he wanted to write a game in Go. Fraser initially thought that was an absurd idea: “Everybody knows you write games in C and C++, or maybe JavaScript, Objective-C, Flash, or C#, but not Go (or Java, or Python, or Ruby)!” One by one, Stephen wore down Fraser’s objections.

A long time ago, Fraser had made a game where you programmed killer robots using assembly code. They decided to make a modern adaptation of that game, and they named it hackerbots. Players program their robots’ behavior in any language; the server just sends JSON game state to the player, and the player sends back JSON actions.

Game implementation, timing, and event loop

Simulation logic should run at 60hz, but player management and serialization should run as fast as possible. The usual way to build a game like this is to use one big event loop, and running these various kinds of operations at different frequencies is complex. With Go, however, it’s very easy to make these various parts run concurrently at the correct speeds, and channels work well to communicate among the components. In fact, the Go language’s features and constraints steered them to make better design choices about their program.

Bandwidth management

There are 2 kinds of clients: a spectator and a player. Both clients receive updates of world states, and both clients can send instructions to the server (e.g., “move to this location”).

Hackerbots have a limited scanner radius, so clients only need to receive updates about world state inside of that radius. But spectators see everything all of the time, so the world state updates can’t be easily limited. This stream was tens of megabytes per second, which would not scale. How could they compact the data?

One common piece of data was a `type Point struct { X, Y int }`. The default JSON encoding of this struct was needlessly large. They used wrapper functions and different types to try to reduce the size of the JSON representation: using `[2]float64` and `[2]float32` types, and so on. Then they realized the power of Go interfaces and made the Point type have a custom json.Marshaler impl. They also added gob support. After these improvements, the size of world state updates and actions was about 10% to 50% of the previous size. Success!

Demo

Fraser and Stephen showed an awesome demo of hackerbots!

[sourcegraph-blog]

Making Docker Go

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Victor Vieux talks about making one of today's most popular Go open source projects – what is Docker and how do they use Go?

What is Docker?

Docker solves the common environment config problem of, "How to do I deploy service X (e.g., MySQL, Redis, Elasticsearch) to environment Y (e.g., dev machine, prod machine, CI server)?" A physical analog is how modern-day shipping containers solved the problem of shipping different physical things over different modes of transportation.

Docker is built on top of lxc (Linux containers) a new feature of the LInux kernel. You can think of it as a lightweight VM. You can quickly and easily spin up isolated environments (containers). Learn more here: https://www.docker.io/learn_more/.

Why Go?

They used Python prior to Go, but decided to rewrite their entire platform in Go. Go has broad appeal to OPS people – Ruby shops don't use Java and Python shops don't use node. The fact that you can cross-compile Go to a single binary is a big win. At the time they started to use Go, there was little hate and little hype (in contrast to Ruby, Python, and Java). Go is easy to read and go fmt is great.

Quote from Docker pull request: "I just completed the Tour of Go and felt competent enough to fix this issue." – Wow.

Docker uses CGo to bind with sqlite, devicemapper, btrfs. Go packages make it easy to modularize Docker codebase.

Go drawbacks

Dependency management (go get) doesn't allow you to fetch a specific revision of a dependency (this is a common refrain at GopherCon). Updating is not automatic (have to run go get -u) and building from depenndencies' master branches isn't reliable. They got around this by vendoring all dependencies – they have a bash script that handles git and mercurial.

They don't like the standard library's flag package (it doesn't support option grouping, didn't handle short/long options). They created the mflag package, a nearly drop-in replacement for flag.

Go's youth still shows. The syscall package isn't perfect. The sendmsg() wrapper is missing a return value. There were a few issues in the Go tar package. They've made patches upstream to Go. They switched from Go tip to Go 1.2.1 (but still vendor selected packages from tip).

Testing frustrations: Would like setup and cleanup functions; go test doesn't work when running individual files (doesn't get imports, so breaks).

GoCover.io

GoCover.io is a cool site that allows you to see coverage of *any* Go package. This presumably uses gocov, but saves you the trouble of running it yourself (which would involve running untrusted code on your machine).

[sourcegraph-blog]

William Kennedy: Building an Analytics Engine using MongoDB and Go

Update: Link to Bill's slides Update: Cute gopher video presented during Bill's talk: http://vimeo.com/92375131.

Other GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks

William Kennedy @GoingGoDotNet is one of the authors of Go in Action (from Manning) and talked about his experiences building an analytics pipeline using Go, MongoDB, mgo, beego, and Iron.io.

Bill led with a video establishing the motivation for the problem he was working on. It featured 2 cute gophers: http://vimeo.com/92375131.

A while ago, Bill’s company was hired to build PowerWallet (similar to Mint). They soon realized that this product also gave them a lot of interesting consumer data: they knew, in aggregate, where and what people were buying. How could they analyze this data? And how could they present this in an actionable feed?

The search for a solution

The first version that Bill built used a SQL database. But whenever they wanted to make a new feed, they’d have to build a new table, populate it with data, etc. That would take a long time. This solution wasn’t working out for them.

They needed a system that could easily and quickly create dynamic feeds based on all of the data. They also wanted to be able to write rules to alter the overall content of a user’s feed quickly (for example, to create custom Valentine’s Day suggestions).

Their system needed to allow them to:

Write rules that can be updated and applied at runtime

Pass variables to filter and pinpoint relevance

Use data aggregation technique to filter and group data

Build tests around aggregated datasets

Build tests against multiple aggregated datasets

Publish data from offer/deal feed and other internal feeds

After evaluating a number of other tools, they settled on Go, Linux, MongoDB, beego, mgo, and Iron.io. (They choose beego over other Go Web frameworks because they liked its MVC architecture.)

They used a denormalized schema for their feed data and kept it updated using workers running tasks on Iron.io.

Walkthrough

Bill walked through the system he made and showed how it performs incremental updates. (Ed. note: This portion of the talk was heavy on screenshots and code, and we couldn’t transcribe it in a way that’d be helpful to our readers. We’ll link to the slides if/when they’re up.)

Q&A

Q: How well do you think this will scale? A: I went with MongoDB because of its scalability. It is scaling much better than the previous SQL database solution. Iron.io is also super helpful in allowing this system to scale easily.

[sourcegraph-blog]

Best practices for production environments

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Peter Bourgon of SoundCloud (which he cheekily describes as a "site that helps you put your sounds in the cloud") talks about Go best practices.

SoundCloud uses Go in a lot of places – Bazooka, their traffic tier, media streaming, interface to search & explore infrastructure, Prometheus, a new backend for the stream feature (currently powered by Cassandra, being replaced by nearly pure-Go backend), an implementation of HLS, and many other small product features.

Special thanks to Dennis Coldwell @dencold for contributions to this writeup!

Dev environment

They have a single GOPATH for all projects. They experimented with having a GOPATH per project, but it was more hassle than was worth. Repositories are cloned to their canonical file path, e.g., $GOPATH/src/github.com/soundcloud/foo. They use vim, Sublime Text, and emacs to edit code – no IDEs.

Repo structure

Keep it simple. A lot of repositories are single packages and are flat and have just a few .go files. Have a main.go in each package that contains the core code. Notably, there's no src/ directories.

E.g.,

github.com/soundcloud/whatever/

    README.md

    Makefile

    main.go

    support.go

    foo/

        foo.go

        bar.go

Formatting style

Check out the code review and style guidelines from Google. Use go fmt. They avoid named returned parameters (but mostly just out of convention). They avoid make and new (unless you know the sizes), opting for the struct literal syntax. It's a good idea to use struct{} for signaling/sentinel values, instead of interface{}{} or bool, because struct{} conveys the complete absence of information.

Configuration

They have mostly 12-factor apps. They use flags from the flag package in the standard lib. Flags are great because they are explicit documentation of the program. Environment variables are convenient, but they have a tool that converts environment variables to flags. A nice idiom is to only define your flags in func main. That forces you to scope flags to the main function, so you're not tempted to access them from another package and break dependency injection.

Logging and telemetry

Use the standard library's package log. Logs should contain only actionable information – either tells a human what to do or will get consumed by another program.

For telemetry, there's 2 ways of doing it – push model (Graphite, Statsd, AirBrake) and pull model (expvar, Prometheus). Push means you emit your data to a known third party. Pull means you have a thing that publishes data and a third party consumer subscribes/scrapes the published data. Push is more intuitive, but the downside to push is that the volume of telemetry data will grow as you grow and eventually overwhelm communication channels. For that reason, they prefer the pull model.

Validation and testing

They tried a variety of testing libraries, but in the end, they settled on using the standard libraries package testing #YAGNI. Checkout reflect.DeepEqual for deep-comparing test values.

There's also the issue of handling both unit tests and longer running integration/smoke tests. They want to differentiate between the two. For instance, you might want to only define a set of flags if you're running the integration tests. A good solution is to use Go build flags, e.g., at the top of your integration test file, have:

// +build integration

Here are some things they do as part of the save-build-deploy process: On save - gofmt or goimports On build - go vet and golint and go test (your unit tests should be *fast*) On deploy - go test -tags=integration on CI server (should probably use gocov, too, but they don't right now)

Big takeaway: they tried many third party libraries for various things like testing, etc., but in most cases, ended up just using the standard library. With other languages/ecosystems, we're used to assuming we need all these third party libraries, but that's not the case with Go.

Dependency management

Ah yes, the gorilla in the room.

They follow this protocol: Ask how important is your project? if "eh", then go get -d (and hope) if "very" then vendor if "somewhere in between" then maybe there's a niche for goproxy? They don't do this yet.

Vendor means "copy your dependencies." They considered 3 options: Option 1: Git submodules – just don't do it. Option 2: Git subtrees – like submodules that make sense. Option 3: With a tool (e.g., godep from Keith Rarick)

They do option 2, but are considering changing to option 3. In option 2, they have a subdirectory (which they name _vendor) that is the root of another GOPATH that gets prepended to the GOPATH env variable. Option 3 might do the same, or do something like rewrite imports automatically (to obviate the need for modifying GOPATH).

They use go build when developing. But when cutting "official" releases, the have a Makefile. They are a polyglot company, so a language agnostic build mechanism is necessary.

Build & deploy

There are 2 kinds of services: stateless and stateful. Stateless services (e.g., request router): 12-factor model, scale up, typically deployed in container Stateful services (e.g., MySQL, Redis): any/no model, provisioned, should probably be deployed in containers now, but haven't gotten around to it

They mostly have stateless services. Their system is Heroku-like:

git push bazooka master bazooka scale -r <new> -n 4.... # validate bazooka scale -r <old> -n 0....

Concluding remarks

These are informed opinions, but still just opinions. Take them with a grain of salt. Go's greatest advantage is its structural simplicity – embrace it!

[sourcegraph-blog]

Gustavo Niemeyer and Steve Francia: Painless Data Storage with MongoDB and Go

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

Gustavo Niemeyer @gniemeyer and Steve Francia @spf13 gave this talk about MongoDB and mgo together. (Ed. note: We'll post slides for this talk as soon as we can obtain them. There were a lot of code snippets. Also, we're not super familiar with MongoDB concepts, so please send in corrections or clarifications.)

Gustavo began by announcing the results of the Go QML Contest. There were 3 entries, and the winner was Robert Nieren (the other entries were from Mark Saward and Maxim Kouprianov). Mark's submission was a multiplayer tiled game like Minecraft. Maxim's submission was an electronics diagram creator. Robert's winning submission was a Go implementation of 2048.

MongoDB + Go == mgo

Background: Steve works on client libraries for MongoDB, overseeing API implementations in 10+ different languages. Gustavo is the primary author of mgo, the Go MongoDB driver.

MongoDB uses BSON, which is a binary encoding format modeled after JSON. The Go gobson library provides support for BSON.

"You can think of MongoDB as a database for storing Go structs." - Steve (paraphrased)

Manipulating MongoDB using mgo

Gustavo described how to use mgo.Dial to connect to MongoDB. Once connected, the caller receives a session. Concurrent callers typically use copied sessions (obtained using the Copy() method).

He also showed code for fetching data (using Find), inserting data (using Insert), updating data (using Update and `$set`), and indexing data (using EnsureIndexKey).

Application-side schemas

MongoDB is a schemaless database, which is great for getting started. But once you've settled on a schema, you can define it in Go. The benefit of application-defined schemas (rather than database-defined schemas) is that your schema definition exists in only one place, in Go structs in this case. The mgo driver lets you serialize and deserialize Go structs to/from MongoDB.

Concurrency using separate (Copy'd) sessions

If you use a global session with the mgo driver, each query will wait for the previous query to return. The mgo driver lets you perform concurrent operations against the database by using separate, copied sessions. When you use a separate session, the query is issued without waiting.

Other features

Various replication topologies

GridFS (like an S3 that you run on your own infrastructure)

Client-side transactions

Aggregation pipelines

Full-text search

Geospatial support

Hadoop

How to understand MongoDB (quick start)

Install MongoDB

go get labix.org/v2/mgo

Build something!

Q&A

Q: What is the Hadoop support for MongoDB like? A: MongoDB can be the input or output to Hadoop jobs. There's an open source project named MongoD

Q: When using application-defined schemas, how do you manage migrations and versioning of schemas? A: MongoDB doesn't provide any built-in support for versioning schemas; it provides the primitives that let you implement it yourself (or using a library). There are 2 kinds of changes: when the data changes and when the schema changes. Both are very common and handled by many people who use MongoDB.

Q: Many people say that MongoDB struggles to scale. Is that a real problem? A: We don't struggle to scale. Some people to struggle to scale with MongoDB; there are a lot of users and some of them will do things that cause scaling issues. We have many clients and users who are scaling well.

Q: Who designed the mgo API? What inspired you? What would you change in the database/sql stdlib package? A: Gustavo's experience with Storm informed his design of mgo. Python presents different challenges when writing ORMs. The design of mgo was created in 2010 and has actually inspired later MongoDB drivers in other languages. Regarding database/sql, Gustavo doesn't use it and is unable to provide informed opinions.

[sourcegraph-blog]

Russ Cox – porting the Go compiler from C to Go

This is an unofficial liveblog post summarizing Russ Cox's Go from C to Go talk at GopherCon 2014.

(Update: GopherCon talk slides are being posted here by speakers: https://github.com/gophercon/2014-talks)

The Go compiler was originally written in C because 1) Go didn't exist, 2) Go was unstable, and 3) Go was originally intended for writing network/systems code and not for compilers. But now 1) Go exists, 2) Go is stable as of Go 1.x, and 3) Go has turned out to be a nice general purpose language and the compiler won't be an outsize influence on the language design.

They're deciding to automatically convert the Go compiler written in C to Go, because writing from scratch would be too much hassle. To understand the challenges of automatically converting from C to Go, we must first look at the history of C and some of its idiosyncrasies.

A History of C

Writen in 1972 at Bell Labs

"Quirky, flawed, and an enormous success" – Dennis Ritchie

The C Data Model

Original target hardware had 24KB memory

programmer in charge of memory(assembly)

type are there to help bu tnot enforced

dynamic storage is provided by library and mem management is burden on programmer

C conflates pointers with array types

C Control Flow

Similar to Go control flow primitives (e.g., do, while, for, switch)

C goto is similar to Go goto, but with some caveats.

C Program Model

Per-file compilation

Separate headers vs code

#define, #include

Challenges in converting C to Go

The goal is to convert *their* C code (not all C code). They want generated code to be human-readable and maintainable. They want automatic conversion to handle 99+% of code.

Unions

A Union is like a struct, but you're only supposed to use one value (they all occupy the same space in memory). It's up to the programmer to know which variable to use

There's a joke in some of the original C code:        #define struct union /* Great space saver */ This inspired a solution:        #define union struct /* keeps code correct, just wastes some space */

#define

Can't just expand during parsing. But in practice, there aren't many #define's in the Go compiler. So the solution is to extend the converter parser to handle special cases. Annotate some and rewrite others in existing code.

Comments

The converter records precise source locations and attaches comments to syntax immediately following the comment.

C Goto

In C, goto can't jump beyond function boundaries. In Go, goto has more restrictions than C goto. In Go, a goto cannot jump over a variable definition (otherwise, you could see uninitialized memory). This has an easy automatic fix. In Go, you can't jump into a new scope (curly-brace {} block). This is harder to fix.

There are currently 1032 goto statements in the Go compiler jumping to 241 labels. Russ breaks them down case by case. Turns out there are only a handful that require a manual rewrite. The others fall into more general categories to which auto-rewrite rules can be applied.

Note: There's a book written about converting goto code to code without goto in general, but this is a sledgehammer and not necessary here.

Types and Type Mapping

In C, you can do implicit conversion between int, long, char, bool etc. C pointers translate to Go pointers AND arrays. So the problem is converting C types that could map to multiple Go types.

The solution is to build a graph of "assigned" value flow, extract clusters of expressions around variables that should have the same type, and infer the correct variable type. This is like doing type inference applied to the special case of disambiguating between potentially ambiguous types in C (like pointer vs. array, int vs bool). For instance, if a function returns int and control flow analysis shows that the return value is always 0 or 1, you can infer that the type is actually bool.

State of the project

Still prototyping but outlook is good, shooting for Go 1.4. Please try the Go 1.3 beta!

Concluding thoughts

Automatically converting C to Go is possible. Same concepts can be applied to other trans-piler projects.

Q&A

How can we help?

File bugs

Did you find any bugs with this tool?

No, but I did find code that scared me! [Cue amusing anecdote about converting from longs to long longs, but it didn't matter in the end, because the values involved were never greater than 29 bits.]

Will it be slower?

It might be a little slower, because Go has bounds checking, etc. But it shouldn't be significantly slower.

Other communities have tried this in the past and they end up not getting any more compiler contributions. Will this happen with the Go compiler?

I think we will get more contributions, since the Go compiler should be easier to work with and understand. But even if not, this is still a big win for me.

Any plans to apply this to other C code in Go codebase?

I don't know for all of it, but we'll see. For the runtime, we definitely want to make it pure Go, but since it's a small amount of code, we might port by hand.

[sourcegraph-blog]

Day 2 has begun.

Expect more posts soon. Russ Cox's @_rsc's talk just began.