From Design To Production

Leveraging Google Cloud Endpoints to deploy goa services

Google recently announced the open beta release of the newest set of features in Google Cloud Endpoints. The part of the announcement that got me especially excited was:

We’re also announcing support for the OpenAPI Specification. We’re a founding member of the Open API Initiative (OAI), and recognize the value of standardizing how REST APIs are described.

In other words Google Cloud Endpoints can be completely configured using an OAI spec, one for example that has been generated by goa! All the goa services (goa.design, swagger.goa.design, cellar.goa.design and talks.goa.design) already run on top of the Google Cloud Platform so I had to give Google Cloud Endpoints a shot.

First Experiment

My first thought was to re-deploy goa-cellar on Appengine Flex so I could front it with Endpoints (it runs in Appengine Classic and Endpoints currently only supports Flex). So I just tweaked the app.yaml file used by the gcloud tool to switch to Flex and enable Endpoints:

runtime: go

# Use Appengine Flex
vm: true

beta_settings:
  # Enable Google Cloud Endpoints API management.
  use_endpoints_api_management: true
  # Specify the Swagger API specification.
  endpoints_swagger_spec_file: public/swagger/swagger.yaml

Next I deployed the service with:

aedeploy gcloud beta app deploy

aannnnd it almost worked… The tool complains that the spec is invalid which is strange because as far as I know goa generates valid specs. Digging further it appears Endpoints does not understand the file type that one may use in the parameter object. The goa-cellar example uses file servers in the design which generates file parameters in the spec. I decided to comment out the public resource from the design since that’s the only resource using file servers. I then re-ran goagen to produce a new spec:

goagen swagger -d github.com/goadesign/goa-cellar/design

Deployed again aannnnd it worked! I now have goa-cellar running in Appengine Flex and fronted by Endpoints, pretty awesome stuff!

Leveraging Endpoints Auth

OK now what? one of the main attraction of Endpoints is its ability to handle authentication. Endpoints supports many different authentication schemes including API keys and JWT.

Now that we know it’s possible to deploy to Endpoints using a goa generated OAI spec let’s create a new example that focuses on that. The service is based off of the adder example and adds a new auth resource that accepts requests secured by API keys or JWTs and returns a response detailing the information that Endpoints parsed out of the authentication token.

Endpoints requires using OAI spec extensions to specify the JWT issuer and token retrieval URI (x-issuer and x-jwks_uri respectively). It also adds a x-security field on path objects used for specifying the expected value of the JWT token aud field. Good timing as support for OAI extensions has just been added to goa. Here is what the design for the jwt action which is secured via JWT looks like:

Action("jwt", func() {
    Security(JWT, func() {
        // Swagger extensions as per https://cloud.google.com/endpoints/docs/authenticating-users
        Metadata("swagger:extension:x-issuer", "client.goa-endpoints.appspot.com")
        Metadata("swagger:extension:x-jwks_uri", "https://www.googleapis.com/service_accounts/v1/jwk/[email protected]")
    })
    Routing(GET("info/jwt", func() {
        Metadata("swagger:extension:x-security", `json:[{"google_jwt":{"audiences":["goa-endpoints.appspot.com"]}}]`)
    }))
    Response(OK)
})

The jwt security scheme is defined as:

// JWT defines a security scheme using Google Endpoints JWT.
var JWT = OAuth2Security("google_jwt", func() {
    // Dummy value to make OpenAPI spec valid, Endpoints take care of implementation.
    ImplicitFlow("/auth")
})

Note the use of the OAuth2Security DSL instead of the JWTSecurity DSL, that’s because the OAI spec consumed by Endpoints must make use of oauth2.

Security Middleware

Overall not bad at all! The last piece remaining is to implement the security middleware that the generated service code invokes upon handling a request made to a secured endpoint. goa generates functions for registering the security middleware, we just need to implement it. Since authentication is handled by Endpoints all the middleware has to do is extract the information from the header it initialized, the complete code for the middleware is thus fairly simple:

// Endpoints returns a goa middleware that extracts the user information initialized by
// Google Cloud Endpoints and stores it in the context.
// See https://cloud.google.com/endpoints/docs/authenticating-users
func Endpoints() goa.Middleware {
    return func(h goa.Handler) goa.Handler {
        return func(ctx context.Context, rw http.ResponseWriter, req *http.Request) error {
            var (
                logger = goa.ContextLogger(ctx)
                info   = req.Header.Get("X-Endpoint-API-UserInfo")
                user   = User{ID: "anonymous"}
            )
            if info != "" {
                // The X-Endpoint-API-UserInfo header is initialized
                // It consists of a Base64 encoded JSON payload.
                js, err := base64.StdEncoding.DecodeString(info)
                if err != nil {
                    logger.Error("invalid header Base64 encoding", "err", err)
                } else {
                    if err = json.Unmarshal(js, &user); err != nil {
                        logger.Error("invalid header JSON", "err", err)
                    }
                }
            }
            // Store the information in the context
            ctx = context.WithValue(ctx, userKey, &user)
            // Call the next handler
            return h(ctx, rw, req)
        }
    }
}

This loads the user information in a User struct:

// User is the type that matches the data serialized into the user info header by Google Endpoints.
type User struct {
    ID     string
    Email  *string
    Issuer *string
}

Which the middleware package makes available via a function that extracts it from the request context. Now all our controllers have to do is get that value and send it back:

func (c *AuthController) JWT(ctx *app.JWTAuthContext) error {
    res := app.Auth(*middleware.UserInfo(ctx.Context))
    return ctx.OK(&res)
}

And that’s it! a grand total of 2 lines of code for the controller and about 20 lines for the middleware and we now have a service that can be secured using any of the schemes supported by Endpoints!.

Running the Example

The code above is available in the new endpoints example. The repo README contains a set of step-by-step instructions to compile, run and deploy the example to Appengine Flex and Endpoints. The OAI spec for the endpoints example service is available at swagger.goa.design.

Once the service is deployed clients may make requests to it by using the proper security scheme depending on the endpoint: JWT for the /auth/jwt endpoint and API key for all other endpoints. API keys can be created in the console either via service accounts for server to server communication or simple project bound API keys.

Endpoints also makes it possible to share the API with developers in the Endpoints console settings. The users that the API has been shared with can create API keys in any project they have access to.

Using API Key Authentication

Back to goa, the generated client tool makes it possible to use API keys via the --key flag. Making a request to the basic endpoint thus looks like:

cd tool/adder-cli
go build
 ./adder-cli basic auth --key YOUR_API_KEY
2016/09/22 19:17:31 [INFO] started id=OEclkaX4 GET=http://goa-endpoints.appspot.com/auth/info/basic?key=XXX
2016/09/22 19:17:31 [INFO] completed id=OEclkaX4 status=200 time=103.953623ms
{"id":"anonymous"}

Omitting the key results in a 401 like you’d expect:

./adder-cli basic auth
2016/09/22 10:27:43 [INFO] started id=Ig/FDyXs GET=http://goa-endpoints.appspot.com/auth/info/basic?key=
2016/09/22 10:27:43 [INFO] completed id=Ig/FDyXs status=401 time=86.605517ms
error: 401: {
 "code": 16,
 "message": "Method doesn't allow unregistered callers (callers without established identity).  Please use API Key or other form of API consumer identity to call this API.",
 "details": [
  {
   "@type": "type.googleapis.com/google.rpc.DebugInfo",
   "stackEntries": [],
   "detail": "service_control"
  }
 ]
}

Using JWT Authentication

Let’s move on to more interesting stuff: API endpoints secured via Google JWT. Using JWT makes it possible for Endpoints to extract user information from the token. The JWT token iss and aud fields must match the values specified in the OAI spec. These values are set in the design using the Metadata DSL as shown above.

JWT tokens are usually created by the service and returned to clients so they may use them to perform authentication. Here we are going to use a service account JSON key file so we can create JWT tokens client side. The golang.org/x/oauth2 package contains the jws sub-package which provides helper functions for encoding valid JWT tokens. It also contains a google sub-package which contains functions for creating tokens from Google Developers service account JSON key file.

The JSON key file can be downloaded from the Google API console. The generated client tool main.go needs to be tweaked to read the file and use the aforementioned packages to create JWT tokens from the credentials in it. The code can be found in the endpoints examples repo. Note that editing the generated tool main.go file is OK because this file - like the files generated in the service main package - is only generated once.

The code in the file above simply creates a goa client signer which reads the service account JSON key file, extracts the private key from it and uses it to sign the JWT token it creates. The --jwt flag creates and registers the JWT signer.

Let’s try it:

./adder-cli jwt auth --jwt
2016/09/22 19:18:17 [INFO] started id=U/d+/6gG GET=http://goa-endpoints.appspot.com/auth/jwt
2016/09/22 19:18:17 [INFO] completed id=U/d+/6gG status=200 time=83.828797ms
{"id":"[email protected]","issuer":"client.goa-endpoints.appspot.com"}

The response contains the information extracted by Endpoints from the token. Our service could use that informationt to do authorization, very cool!

And just to make sure, not setting the JWT token does result in a Unauthorized response:

./adder-cli jwt auth
2016/09/22 19:26:24 [INFO] started id=ra3d2o5Y GET=http://goa-endpoints.appspot.com/auth/jwt
2016/09/22 19:26:24 [INFO] completed id=ra3d2o5Y status=401 time=89.411491ms
error: 401: {
 "code": 16,
 "message": "JWT validation failed: Missing or invalid credentials",
 "details": [
  {
   "@type": "type.googleapis.com/google.rpc.DebugInfo",
   "stackEntries": [],
   "detail": "auth"
  }
 ]
}

Open Standards FTW

Overall the whole thing turned out to be a lot easier than I was expecting. For the most part “it just works”. goa takes care of the boilerplate that comes with implementing a (micro)service, leveraging Endpoints does the same with its deployment. What’s left to do is what matters and cannot be automated: the actual domain logic.

This is a real testament to the power of open standards, certainly a success story for the Open API Initiative. It’s also a good validation of the “Design first” approach promoted by goa. I can’t wait to see what other cool integration it will enable in the future. In the mean time you get to reap the benefits: you can now deploy your goa services behind Google Cloud Endpoints with no extra work!

Hello, goa