#google app engine endpoints

[redgreenrefactor-eu]

From Authenticating JSON APIs, through SNI, to pyenv

What would be a pragmatic approach to authenticate JSON APIs? I did some short googling just to see what the most popular options are. If you're are using Google App Engine endpoints, then you can take advantage of all good stuff from Google (I was even writing about Testing Google App Engine Endpoints in one of my previous entires). But what if you are on your own, let's say you're hosting your own Ruby on Rails app?

I really like Rails Cast Securing an API. It really gives a nice overview of the state of the art approaches one may take to give JSON APIs a bit more protection. The cast is from May 2012 - it would be nice to see an updated version (or there're is nothing really new on this topic?). In the end, though, OAuth comes as the most complete, mainstream solution and the cast will point you out to the right direction. A Ruby gem recommended to follow is intridea/oauth2.

What was stopping me recently from using OAuth on my own server though, was my lack of awareness (I would even call it ignorance) of Server Name Indication - an extension to the TLS protocol that effectively lets you use different SSL certificates for different virtual hosts. Without this extension, providing separate SSL certificates for each site required separate IP addresses. This is because the TLS handshake happens before the server sees any HTTP headers, which is where name based virtual hosting gets the information about the hostname so that it can serve you the requested site. Server Name Indication addresses this issue by sending the name of the virtual domain as part of the TLS negotiation. SNI is quite well supported, only if you use Python, the wikipedia link above says that you need to work with Python version 3.2 at least (for ssl, urllib, and httplib modules). If you are using Ubuntu with Apache, you may really like a tutorial from DigitalOcean by Etel Sverdlov: How To Set Up Multiple SSL Certificates on One IP with Apache on Ubuntu 12.04. I will be giving it a try on my multiple-site ruby-on-rails installation soon.

This last remark about SNI support on Python motivated me to take a look at one more issue I was somehow avoiding recently. I am using Python virtualenv through virtualenvwrapper and I was already wondering how can I make virtualenv wrapper to use a specific version of Python. As you can read on this StackOverflow question both virtualenv and mkvirtualenv commands have -p option where you can specify the python version to be used (I hope to give a more complete example of this soon). So the question reduces to how can I conveniently switch between different Python versions (I should add on Mac OS as I am still somehow in the mindset that Windows was never really meant to be used as development platform besides Visual Studio and embedded systems - I hope the latter to change in the foreseeable future)? I was searching for something similar to the famous Ruby Version Manager. And lucky me! There is something and it is called pyenv - Simple Python Version Management. Again, I hope to have a more detailed example on this blog soon.

[redgreenrefactor-eu]

Developer Diaries: Testing Google App Engine endpoints?

Google App Engine is continuously evolving. Always check out the latest documentation. Google has power and capability to provide first-class support for local testing of Google App Engine endpoints.

Introduction

Google App Engine endpoints provide an encapsulated, native approach to talk to your Google App Engine (GAE) web apps from either javascript client, Android, or iOS native apps. Before, you had to design your own "headless" API to talk to your backend. You also had to deal with all technicalities of OAuth if you wanted to protect some of your interfaces. GAE endpoints simplify both, even though you must be prepared to invest some time to learn these relatively new technology.

Learn more about GAE endpoints.

The problem

We need to be able to automate testing for Google App Engine endpoints. Especially, considered it may change in the future, we want to know that all dependent systems can talk to our endpoints without issues.

In the GAE endpoints documentation you will find a chapter devoted to Testing and Deploying an API Backend. It shows how to test API backends locally on the development server either by using the Google API Explorer (by navigating to http://localhost:8080/_ah/api/explorer) or by using curl. Unfortunately, currently it does not seem like we could take advantage of the webtest and testbed frameworks and test our APIs in a way similar to that described in the Google article on Handler Testing for Python. You may want to check the WSGI Service Library - in the end endpoints are just rpc services (endpoint APIs subclass remote.Service from protorpc). Hopefully, we will have some data on it very soon. For now, you can easily write Cucumber scenarios to validate your APIs by using GET and POST requests with JSON (as you see we get to JSON anyway :)).

In short, automating your backend tests, is quite easily achievable. What's the problem then? Well, the problem starts when you want to use authentication with your endpoints. At the moment of this writing and to my best knowledge, you can't use user authentication with endpoints on the development server. It uses the old good OAuth under the hood, but we should not mistake the OAuth endpoints with using OAuth on backend API endpoints. For the former, in OAuth for Python Overview we read:

On the local development server, oauth.get_current_user() always returns a User object with email set to [email protected] and user ID set to 0 regardless of whether or not a valid OAuth request was made.

This does not appear to be true with OAuth with backend API endpoints. Clone the TicTacToe example. It does not seem to work on the development server. This is something to be expected as the tutorial asks you to fill in your CLIENT_ID. Also the Using Auth with Endpoints document, when describing how to create your web CLIENT_ID instructs you to use https as the Web Origin protocol. Development server does not support https, which indicates that you may need to deploy your app before being able to use authentication. Indeed, in all articles from Google dealing with OAuth with endpoints, you are instructed to deploy the app before you can test the endpoints requiring authentication.

This is clearly a bad thing if you need a decent test automation. Assuming your endpoints do not change frequently, we could have a test version of our app deployed and then we could manually trigger the automated tests for the endpoints every time we change anything in the backend. This becomes less practical when your testing scenarios involve an iOS app. Taking into account that your iOS app needs Google generated files to make API calls, it won't be easy to mock the Google App Engine with some simple rpc service (I did not test it but it seems to me that it will take some effort to figure out). It also does not make too much sense to modify the client library from Google.

Two questions then: 1) How to test your backend API running locally on the development server if our endpoints involve user authentication? 2) Can we test an iOS app against a local instance of the backend API running on the development server when user authentication is in use?

The solution

The Using Auth with Endpoints document tells us what does an API backend need in order to support authentication?

Specify the client IDs (allowed_client_ids) allowed to make requests.

Add a User Check to all exposed methods to be protected by authorisation.

Re-generate the client libraries and redeploy the API backend.

We can take advantage of the second point on the list above. Take a look at the contents of the models.py file in the example TicTacToe backend API mentioned above. Let's target the get_endpoints_current_user function:

def get_endpoints_current_user(raise_unauthorized=True): current_user = endpoints.get_current_user() if raise_unauthorized and current_user is None: raise endpoints.UnauthorizedException('Invalid token.') return current_user

In testing, we do not want authentication to interfere too much . We would be just happy with a User object with email set to [email protected] and user ID set to 0 regardless of whether or not a valid OAuth request was made. So, imagine, that for the whole time of testing, you use simplified version of this method, say this one:

def get_endpoints_current_user(): return User(email='[email protected]')

Of course, you have to put some effort in writing a small script that will replace that function with the proper authentication code at the moment you decide to deploy. You may still want to have some automated tests on a deployed tests version, but that's perfectly fine. Sure, it is not ideal, but it is clear and simple and your backend can be run on the local development server without any further changes. This way we are still able to test all the functionality that is user specific - something we would lost if we decide to remove authentication completely for the purpose of testing.

Unfortunately, the above procedure will not be sufficient to test your iOS app against the backend API running on a development server. Actually, even if you do not use user authentication, you won't be able to successfully POST anything from iOS app until you deploy the backend. This must be https thing I guess.

This difficulties should not stop you from your automation efforts. We will tackle this problem in one of the next series of Developer Diaries.