App Engine Modules and Go / Golang

21 Mar
21/03/2014

Update: If you’re primarily interested into separating module code into its own, distinct Go compiled binaries and packages, you might want to follow along the discussion on Google+.

Recently Google deprecated App Engine Backends in favour of Modules.

In my case, I want to use Modules to bundle indefinitely running processes responsible for frequently persisting data from Dedicated Memcache to Datastore. These processes should live outside any request context and just run “forever”.

An important aspect for me is to keep the module code separate from my main application’s code. On App Engine for Python and PHP this is pretty obvious, as one can simply map URIs to scripts. On the contrary, App Engine for Go uses a single binary and all of the URI matching is handled from within the compiled program.

Unfortunately, the Golang Modules documentation is incomplete and not extremely helpful in getting you up and running using Modules. For example, the samples for Module YAML files, do not include the handlers tag – which not only seems to be required in app.yaml but also on the module level.

Disclaimer: While I am not at all sure whether what I describe here is the correct, best or even idiomatic way of doing it, I can confirm it is working. Any comments are appreciated. You might also want to get in touch with me on TwitterApp.net or Google+.

I hope, Google will amend their docs so we all have a chance to find out how exactly Mountain View intends us to use Modules.

Folder structure

My project structure looks something like this:

/ Project root containing the .yaml files
/main The main applications .go files
/config Configuration files
/services Go files for shared services
/etl The Go files for the ETL Module

YAML files

The main app.yaml file is absolutely straight forward:

app.yaml

 

Here is the etl.yaml file:

etl.yaml

A couple of important things to note:

  • I’m using the manual scaling instance type, which according to Google’s docs facilitate the following: “Requests can run indefinitely. A manually-scaled instance can choose to handle /_ah/start and execute a program or script for many hours without returning an HTTP response code.”
  • I have added a handlers section. Google does not illustrate this in their docs, but without a handlers section, I couldn’t get Modules to work at all. Also note that the modules registers for the /_ah/.* URI pattern. If you try to register the Module for any pattern, that gets registered by another Module or the default Module, you will see “panic: http: multiple registrations for / goroutine” errors in your logs.

The code

In my main application, I’m using the Martini web framework to route requests. The code lives in the /main folder and is not particularly important to understanding how to set up Modules.

The ETL Module lives in the /etl folder and does not make use of the Martini framework, as it essentially just needs to handle the start and stop triggers requested by App Engine’s runtime environment:

Again, a few things to note:

  • The Module belongs to the same package as the rest of my code. While the code for the Module is cleanly separated, we still get just one single binary.
  • The Module implements handlers for  /_ah/start and /_ah/stop. Once an instance configured for manual scaling is fired up, App Engine automatically sends an empty GET request to /_ah/start.
  • The rest of the code is straight forward: I’m using runtime.RunInBackground to start a background process which – in this case – will run forever.
  • The first thing I do in the handlers is returning a response. If a Module does not respond with an HTTP 200 to /_ah/start, App Engine, the runtime environment will shut down the Module. (You can use this to signal initialisation or other failures to App Engine, by responding with a 404).

goapp deploy and the development server

goapp is just a thin wrapper around appcfg. To deploy your app along with its modules just type: goapp deploy app.yaml etl.yaml. Though the goapp serve app.yaml etl.yaml command works, too, I couldn’t get the module working on my local development server. If anybody figures it out, let me know!

This is my take on Golang Modules for App Engine. I’m looking forward to getting your feedback and possible hints of how to improve the code and setup.


GoSublime code completion for App Engine and Go

16 Feb
16/02/2014

Go is a fascinating “new” language.

I’m doing most of my Go development in Sublime Text 2 using the superb GoSublime package for code completion and many more IDE like features. One thing that bothered me for a long time, though, was that I could not get code completion working for Google’s App Engine packages.

I finally solved the problem:

1. Create a symbolic link in /go_appengine/goroot/pkg named darwin_amd64 and point it to darwin_amd64_appengine/ like so: ln -s darwin_amd64_appengine/ darwin_amd64

2. Make sure that in Sublime Text 2 > Preferences > Package Settings > GoSublime > Settings – User the GOPATH environment variable includes the path to App Engines goroot folder. Mine is: “env”: {“GOPATH”: “$HOME/src/gocode/:$HOME/src/go_appengine/goroot/”}

I hope this helps and appreciate any feedback.


Making Ghost available beyond localhost

12 Jul
12/07/2013

Since many years, I’m using WordPress for my blogs and even some sites. As many other bloggers, I’ve ended up in a love/hate relationship with the software.

When John O’Nolan announced Ghost, a new kind of blogging platform, I immediately jumped over to Kickstarter and chose the VIP package.

Today, the Ghost folks shipped the first version, they call it an “alpha of an alpha”.

Ghost is built on top of Node.js and Express. The default configuration binds to the loopback interface (127.0.0.1) which makes it available only to localhost. So even if you try http://192.168.2.1:2368 or something, it won’t work.

In case you want to make your Ghost deployment available to other hosts on your network or even publicly, make sure you change the binding in config.js inside Ghost’s root directory and restart Node:

While this is not at all Ghost specific, I thought it might help users new to Ghost and without any Node specific experience.


I posted about the nonsense of agile fixed pricing

07 Jul
07/07/2013

Over at SVBTLE, I posted about the nonsense of agile fixed pricing. Hope you’ll find it worth reading!

 


I joined SVBTLE

30 May
30/05/2013

From the Svbtle homepage:

“SVBTLE is a new kind of magazine. We’re a network of great people mixed with a platform that takes the best things from publishing and combines them with the best parts of the web. We want to make it easier for people to share and discover new ideas.”

Find my SVBTLE contributions at ralfrottmann.net and feel free to subscribe to the feed.

I’m excited to become a part of this diverse, inspiring, motivating, thought-provoking, educating and eye-opening community. I will keep my personal blog for deep tech talk and use SVBTLE mostly for all other posts.


Unable to sign in for Hangouts with Google I/O 2013 Google+ upgrade

15 May
15/05/2013

If you are a Google Apps for Businesses customer you might see a red error messaging stating “Unable to sign in :(” in the Hangouts sidebar, detailing:

“Hangouts has not been enabled for your account. Please contact your administrator to enable this service.”

To get rid of it, in the classical admin panel to to Settings > Talk and activate the feature called “Enable the new Hangouts for messaging and video calls”.

The Google+ help might wrongly point you to the Services > Google+ section. I hope this helps others stopping the frustration. If you want to stay in touch, I’m on Google+.


Launch iTerm 2 on startup without opening a terminal window

14 Mar
14/03/2013

iTerm 2 is the best Terminal replacement for Mac OS X.

As somebody, spending most of his time on the console, I set up the Hotkey window so it opens fullscreen on my MacBook Air and fills half of the screen in my iMac. Obviously, I added iTerm 2 to the Login Items in Users & Groups to automatically launch it on startup.

One little thing always bothered me, though: On launch, iTerm 2 automatically opens a terminal window.
Read more →


How to stop posting public Google+ Communities posts to your profile

06 Mar
06/03/2013

I cannot believe I did not know this.

Ever since Google+ Communities became available, people complained about their Community posts automatically popping up on their profiles. This has been the default for public Communities. In fact, many Community owners on finding this out killed their public Communities and started over doing private ones.

Read more →


Stay hungry, don’t stop learning

11 Feb
11/02/2013

I have shamelessly stolen this entire post from Jeffrey Way, who is a fantastic teacher over at tuts+ premium. Of course, I asked for his permission and got it. I’ve been working with developers for most of my life and found this so to the point, that I just had to make it available to the rottmann.net audience. Thanks, Jeffrey, for nailing it.

If I could only offer one piece of advice to a new web development student, it would be: “don’t worry; we all feel overwhelmed.” I’d then remind him or her that I, too, am still a student, and so are all of my peers. This industry is an incredibly tough one that requires constant continued education throughout your career. You better be okay with this truth.

Read more →


Setting up Scalr with Ubuntu 12.04 Server and Predis

08 Feb
08/02/2013

Part of my work at acceleract is to build highly scalable web applications – mostly driving RESTful APIs for mobile clients these days.

The infrastructure powering these scalable PHP applications can get quite complex. For example, for a current client I’m running two load balancers which dispatch traffic to four app servers each using a powerful master-slave MySQL cluster and four master-slave Redis Key/Value stores for fast delivery of in-memory hashes. For all of this, I currently mostly use Ubuntu 12.04 server.

Read more →


© Copyright 2013 by Ralf Rottmann. rottmann.net is a work in progress by Ralf Rottmann. This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
If you would like to make use of any of the content you see here, please contact the author.