Failure Flags

Running Failure Flags experiments

This document will walk you through running your first experiment using Failure Flags.

Example: the HTTPHandler application

Throughout this document, we'll demonstrate examples use a simple application called "HTTPHandler". This application takes incoming web requests and returns the application's execution time. We'll provide examples for each of the available SDKs.

We've added a Failure Flag named http-ingress with two labels: one that tracks the request method, and one that tracks the URL path:

Node.js example

1const gremlin = require('@gremlin/failure-flags')
2
3module.exports.handler = async (event) => {
4  start = Date.now()
5
6  // If there is an experiment defined for this failure-flag, that is also
7  // targeting the HTTP method and or path then this will express the
8  // effects it describes.
9  await gremlin.invokeFailureFlag({
10    name: 'http-ingress',
11    labels: {
12      method: event.requestContext.http.method,
13      path: event.requestContext.http.path,
14    },
15  })
16
17  return {
18    statusCode: 200,
19    body: JSON.stringify(
20      {
21        processingTime: Date.now() - start,
22        timestamp: event.requestContext.time,
23      },
24      null,
25      2
26    ),
27  }
28}

Go example

1package main
2
3import (
4    "fmt"
5    "time"
6
7  "github.com/aws/aws-lambda-go/events"
8  "github.com/aws/aws-lambda-go/lambda"
9
10  gremlin "github.com/gremlin/failure-flags-go"
11)
12
13func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
14    start := time.Now()
15
16  // Add a failure flag
17    gremlin.Invoke(gremlin.FailureFlag{
18        Name: `http-ingress`, // The name of the failure flag
19        Labels: map[string]string{  // Additional metadata we can use for targeting
20              `method`: request.HTTPMethod,
21              `path`: request.Path,
22        }})
23
24    return events.APIGatewayProxyResponse{
25        Body:       fmt.Sprintf(`{"processingTime": %v, "timestamp": "%v"}`, time.Since(start), start),
26        StatusCode: 200,
27    }, nil
28}
29
30func main() {
31    lambda.Start(handler)
32}

Creating a new Failure Flags experiment

To create a new experiment:

Open the Gremlin web app and select Failure Flags in the left-hand nav menu.
Click Create an Experiment.
Enter an experiment name. This can be anything you wish.
Enter the Failure Flag selector name. This should match the value of the name attribute you gave when creating the Failure Flag in your code. For example, the name of the Failure Flag example here is http-ingress.
Enter the Failure Flag selector attributes to specify the types of traffic that the experiment will apply to.
Enter the Application selector name. This should match the name of the application that you want to run the experiment on. You can see your list of active applications in the Gremlin web app.
Enter the Application selector attributes to specify the application instances that the experiment will run on. See Selectors for more details.
Enter the experiment Effects in the Effects box. See Effects for more details.
Choose the percentage of applicable failure flags and applications to impact using the Impact Probability boxes. For example, if you choose 1%, then only 1% of the total failure flag instances matched by your selectors will be impacted by the experiment. This does not apply to code executions - the Failure Flag selector name determines that.
Specify how long the experiment will run for using Experiment Duration.
Click Save to save the experiment, or Save & Run to save and immediately execute the experiment.

Selectors

Selectors are JSON objects consisting of key-object pairs. These objects tell Gremlin which applications and Failure Flags to target for an experiment, as well as what effects to apply.

As an example, our HTTPHandler contains the following Node.js code:

1const gremlin = require('@gremlin/failure-flags');
2
3module.exports.handler = async (event) => {
4  await gremlin.invokeFailureFlag({
5    name: 'http-ingress',
6    labels: {
7      method: event.requestContext.http.method,
8      path: event.requestContext.http.path }});
9...
10};

This means that the the Failure Flag name is http-ingress, and the application name is HTTPHandler.

Application Attributes

Application attributes let you identify specific instances of an application to run an experiment on. For example, imagine our HTTPHandler application runs in AWS Lambda in several different regions. We can use the following application attribute to only impact instances in us-west-1:

json

1{ "region": ["us-west-1"] }

Flag Attributes

Flag attributes are selectors for targeting specific executions of the application's code. Our example HTTPHandler application has a method label containing the HTTP request method. If we only want to impact POST requests, we'd add the following flag attribute:

json

1{ "method": ["POST"] }

Experiments and Effects

The Effect parameter is where you define the details of the experiment and the impact it will have on your application.The Effect parameter is a simple JSON map that gets passed to the Failure Flags SDK when an application is targeted by a running experiment.

The SDK currently supports two types of effects: latency and error.

Latency

Latency introduces a constant delay into each invocation of the experiment. Specify latency for the key, and the number of milliseconds you want to delay as the value. For example, this effect introduces a 2000 millisecond delay:

json

1{ "latency": 2000 }

Minimum latency with jitter

Alternatively, you can add latency where the amount varies. For example, this effect introduces between 2000 and 2200 milliseconds of latency, where there is a pseudo-random uniform probability of the SDK applying any value within the jitter amount:

json

1{
2  "latency": {
3    "ms": 2000,
4    "jitter": 200
5  }
6}

Errors

The Error effect throws an error with the provided message. This is useful for triggering specific error-handling methods or simulating errors you might find in production. For example, this effect triggers an error with the message "Failure Flag error triggered":

json

1{ "exception": "Failure Flag error triggered" }

If your appliation uses custom error types or other error condition metadata, you can add this metadata to the error effect:

json

1{
2  "exception": {
3    "message": "Failure Flag error triggered",
4    "name": "CustomErrorType",
5    "someAdditionalProperty": "add important metadata here"
6  }
7}

Combining Latency and Error effects

You can combine the latency and error effect to cause a delay before throwing an exception. This is useful for recreating conditions like network connection failures, degraded connections, or timeouts.

For example, this effect will cause the Failure Flag to pause for 2 full seconds before throwing an exception with a custom message:

json

1{
2  "latency": 2000,
3  "exception": "Failure Flag delayed error triggered"
4}

Changing application data

Note

This feature is currently only available in the Node.js SDK.

Failure Flags are also capable of modifying application data. This is an advanced effect that requires additional setup using the Failure Flags SDK.

In your application's call to invokeFailureFlag, add a new dataPrototype property and assign it a variable like a network request or response. You could also pass in an object literal.

1let myData = {name: 'HTTPResponse'}; // this is just example data, it could be anything
2
3myData = await failureflags.invokeFailureFlag({
4  name: 'flagname',       // the name of your failure flag
5  labels: {},             // additional attibutes about this invocation
6  dataPrototype: myData); // "myData" is some variable like a request or response. You could also pass in an object literal.

Once the dataPrototype property is set, you can add a data object to the effect statement. Any properties in the data object will be copied into a new object created from the prototype you provided.

json

1{
2  "data": {
3    "statusCode": 404,
4    "statusMessage": "Not Found"
5  }
6}

While this experiment is active, myData will be changed to the following:

json

1{
2  "name": "HTTPResponse",
3  "statusCode": 404,
4  "statusMessage": "Not Found"
5}

Note

If the experiment is not running, then myData will remain unaltered.

Customizing an experiment's impact

You can customize the impact of the experiment by adding a behavior function. For example, the following snippet writes data about the experiment to the console instead of applying the experiment to your code:

Node.js example

javascript

1await gremlin.invokeFailureFlag({
2  name: 'http-ingress',
3  labels: {
4    method: event.requestContext.http.method,
5    path: event.requestContext.http.path,
6  },
7
8  // Log the experiment after it's complete
9  behavior: async (experiment) => {
10    console.log('handling the experiment', experiment)
11  },
12})

Go example

1gremlin.Invoke(gremlin.FailureFlag{
2  Name: `http-ingress`,
3  Labels: map[string]string{
4    `method`: request.HTTPMethod,
5    `path`: request.Path,
6  },
7
8  // the following line provides an implementation of the failureflags.Behavior type
9  Behavior: func(ff FailureFlag, exps []Experiment) (impacted bool, err error) {
10    // write the experiments to standard out
11    fmt.Fprintf(os.Stdout, `processing experiments: %v`, exps)
12    // continue processing using the default behavior chain
13    return failureFlags.DelayedPanicOrError(ff, exps)
14  }
15})

If you want even more manual control, the SDK can detect whether an experiment is currently active. For example, during an experiment, you might want to prevent making certain API calls, or rollback a transaction. In most cases the Exception effect can help, but you can also create branches in your code. For example:

Node.js example

javascript

1if (await failureflags.invokeFailureFlag({ name: 'myFailureFlag' })) {
2  // If there is a running experiment then run this branch
3} else {
4  // If there is no experiment, or it had no impact, then run this branch
5}

Go example

1if active, impacted, err := FailureFlag{Name: `myFailureFlag`}.Invoke(); active && impacted {
2  // If there is a running experiment then run this branch
3} else {
4  // If there is no experiment, or it had no impact, then run this branch
5}

Language-specific features

This section is for features unique to specific SDKs.

Go

Panic

The Go SDK offers a unique fault called panic. This causes Failure Flags to panic with the provided message. This is useful when validating that either your application handles Go panics correctly, or when assessing the impact to other parts of the system when your code panics:

1{ "panic": "this message will be used in an error provided to panic" }

More information and examples are available on the project's GitHub repo.