As a web app becomes more mature, you often want to enable deeper customization than is possible just by making selections in a UI. For Reviewable, there was strong demand for customizing the completion condition that determines when a code review is done, but a quick survey of the interested users showed that I'd need to add a large number of toggles to satisfy a majority of them. Furthermore, since many use the review completion status check in GitHub to enforce business rules and prevent merging pull requests too early, “close enough” logic wouldn't really cut it.

My immediate thought was to expose the review state as a simple JSON structure and let repo admins write a snippet of JavaScript code to apply their completion rules. This would be very flexible, reasonably simple (especially if I supplied a bunch of examples to tweak), and pretty accessible since my target audience consists of developers and JavaScript is the closest we've got to a lingua franca.

The major issue, of course, was how to run user-supplied code safely, but also quickly and cheaply.

Some approaches that almost work

Initially I tried to find some way to run the code on my existing application servers. This would be simplest, fastest, and cheapest, but it turns out that achieving strong isolation this way is still an unsolved problem.

• Google Caja can transpile the user's code so that it's safe, but debugging becomes tricky, it's slow (you have to spin up a Java VM just to compile), and it's not clear if the tool is still being maintained.

• Jailed and SandCastle attempt to isolate the user's code in a separate Node.js process, but both are billed as “best effort”, which doesn't really cut it especially when running on the main server where the parent process has full read/write access to the datastore.

• Webtask and StackHut provide sandboxed code execution as a service, building on top of Docker containers to ensure security. However, webtasks get expensive quickly when you need to isolate many users from each other as well as from your app, and StackHut is focusing on self-hosted execution clusters that would get expensive to maintain (in ops time, if nothing else).

AWS Lambda to the rescue

I had originally discounted AWS Lambda because I remembered that it was a purely async service, with no built-in way to get the result of a task back to the caller promptly. Turns out that they added synchronous calls in early 2015, though! Running user code in Lambda would definitely isolate it from my server, and I trust Amazon to get the security right and isolate my tasks from other Lambda users. I could just use a Lambda function like this to run user code and return the result:

'use strict';
var vm = require('vm');

exports.handler = function(request, context) {
  if (!request.code) throw new Error('No code passed to executor');
  var code = '(function() {\n' + request.code + ';\n})();';
  try {
    context.succeed(vm.runInNewContext(
      code, request.env, {displayErrors: true, timeout: 3000}));
  } catch (e) {
    // Capture stack in the message, otherwise it gets lost in transmission.
    context.fail(new Error(e.stack));
  }
};

There was just one hitch: I also needed to isolate my own users from each other—specifically, a rule written for one repo must never get access to the review state in any other repo. This looked like it was going to be tricky since Lambda reserves the right to reuse containers for a given function, i.e. send further requests to a Node.js process once it's been started, which would be a problem if I had a single, generic “eval user code” function.

So instead, let's create a separate Lambda function for each isolation context (aka container), all running the same code! The container is just an arbitrary, distinct string (/[\w-]{1,64}/) that you can derive consistently for each desired isolation context—for Reviewable, this is the fully qualified, escaped repo name, with fallback to a SHA1 hash if too long.

var lambdaRole = 'arn:aws:iam::123456789012:role/lambda_basic_execution';
var AWS = require('aws-sdk');
var lambda = new AWS.Lambda();

function runSafely(code, env, container, cb) {
  // Try blindly calling function first, as it usually exists.
  lambda.invoke({
    FunctionName: container, Payload: JSON.stringify({code: code, env: env})
  }, function(error, data) {
    if (!error) {
      cb(null, JSON.parse(data.Payload));
    } else if (error.code === 'ResourceNotFoundException') {
      // If function is missing, try to create it.
      lambda.createFunction({
        FunctionName: container, Runtime: 'nodejs', Role: lambdaRole,
        Handler: 'index.handler', Timeout: 4, MemorySize: 128,
        Code: {ZipFile: require('fs').readFileSync('run_user_code.zip')}
      }, function(error, data) {
        if (error) {
          cb(error);
        } else {
          // Try calling the function again, now that it's been created.
          lambda.invoke({
            FunctionName: container,
            Payload: JSON.stringify({code: code, env: env})
          }, function(error, data) {
            if (error) cb(error); else cb(null, JSON.parse(data.Payload));
          });
        }
      });
    } else {
      cb(error);
    }
   }
  });
}

Creating a new Lambda function is fast enough that the additional delay is barely noticeable by the user, and it's a one-time cost. There appears to be no limit on the number of Lambda functions you can have, and the hard limit of 1.5GB of total code will take hundreds of thousands of containers to reach even if you bundle a library or two in your run_user_code.zip package. For bonus points, you can encode your library's version in the container name so that it'll be automatically lazily upgraded, and also automatically modify the retention of the log group corresponding to each container so they're not retained forever.

Pretty, pretty, pretty... good!

By using AWS Lambda in this way, Reviewable is now able to isolate users' code from the app and from each other, execute it quickly (usually within a couple hundred milliseconds, even from a cold container start), and do all this cheaply thanks to Lambda's generous free tier and low ongoing costs. Woot!