Search documentation
Dashboard
Failure Flags

Deploying Failure Flags on AWS ECS

This document will walk you through setting up Failure-Flags-Sidecar for your ECS Tasks. Failure-Flags-Sidecar runs alongside your application and is responsible for managing Chaos Engineering experiments and reliability tests.

Adding Failure-Flags-Sidecar to your ECS Tasks

Failure-Flags-Sidecar container images are available via DockerHub and support both AMD64/x86_64 and ARM64 architectures. These container images include a LICENSE file and a single binary program built for Linux. Alternatively, you can download archives directly: arm64, x86_64.

All versions are listed in a file at: https://assets.gremlin.com/packages/failure-flags-sidecar/VERSIONS.

Setting required environment variables

You can add Failure-Flags-Sidecar to any Task without impacting your application availability or performance. But you do need to add configuration to your environment variables before Failure-Flags-Sidecar will add any value. Configuration comes in via environment variables and or configuration files.

Get started quickly with environment variables only:

  • GREMLIN_SIDECAR_ENABLED must be set to either true or yes or 1 to enable Failure-Flags-Sidecar. If unset or set to any other value Failure-Flags-Sidecar will operate in NOOP mode.
  • GREMLIN_TEAM_ID must be set to your Gremlin Team ID. This and other credential material is available through the Gremlin UI.
  • GREMLIN_TEAM_CERTIFICATE must be set to your Gremlin Team certificate. Newlines may be preserved using the \n escape characters or omited entirely. This and other credential material is available through the Gremlin UI.
  • GREMLIN_TEAM_PRIVATE_KEY must be set to your Gremlin Team private key. Newlines may be preserved using the \n escape characters or omited entirely. This and other credential material is available through the Gremlin UI.

Setting Targeting Environment Variables

You will want to set custom targeting labels to uniquely identify deployments of your software. Setting custom labels is done through environment variables with a prefix, GREMLIN_LABEL_. Any environment variable set on the sidecar with that prefix will be included as labels on the service. For example:

1An environment variable `GREMLIN_LABEL_CUSTOM` with the value `custom value` will result in the label: "CUSTOM: custom value".

Individual Configuration Values from Files or ARNs

You can configure individual configuration values like GREMLIN_TEAM_CERTIFICATE, GREMLIN_TEAM_PRIVATE_KEY, and GREMLIN_CUSTOM_ROOT_CERTIFICATE to retrieve values from files in the sidecar container or from AWS services using their ARNs. Instead of setting those environment values directly, use their _FILE or _ARN counterparts. Files must be fully qualified paths from the filesystem root. This project currently supports secretsmanager secret and ssm paramter ARNs.

When you add the Failure-Flags-Sidecar container to your ECS Task and configure the environment variables correctly, your service will be able to consult that extension for Gremlin experiment configuration. You will be able to find your service in the Gremlin UI under Failure Flags > Services after you launch your app with the layer configured and you exercise the integration.

Once you've added Failure-Flags-Sidecar to your project you can use the Failure Flags library (Node, Python, Java, Go) from your code!

Example Pod Spec with Failure Flags Sidecar

Adding the sidecar means including an additional task in any ECS application where you want to use Failure Flags.

json
1{
2    "containerDefinitions": [
3        {
4            "name": "app",
5            "image": "samw/ff-nodejs-demoapp:amd64",
6            "cpu": 0,
7            "portMappings": [
8                {
9                    "name": "app-3000-tcp",
10                    "containerPort": 3000,
11                    "hostPort": 3000,
12                    "protocol": "tcp",
13                    "appProtocol": "http"
14                }
15            ],
16            "essential": true,
17            "environment": [
18                {
19                    "name": "FAILURE_FLAGS_ENABLED",
20                    "value": "true"
21                }
22            ],
23            "logConfiguration": {
24                "logDriver": "awslogs",
25                "options": {
26                    "awslogs-create-group": "true",
27                    "awslogs-group": "/ecs/nodejs-ff-demoapp",
28                    "awslogs-region": "us-east-2",
29                    "awslogs-stream-prefix": "ecs"
30                },
31                "secretOptions": []
32            }
33        },
34        {
35            "name": "gremlin-sidecar",
36            "image": "gremlin/failure-flags-sidecar:latest",
37            "cpu": 0,
38            "portMappings": [],
39            "essential": false,
40            "environment": [
41                {
42                    "name": "SERVICE_NAME",
43                    "value": "my-fargate-sidecar-demo"
44                },
45                {
46                    "name": "GREMLIN_DEBUG",
47                    "value": "true"
48                },
49                {
50                    "name": "GREMLIN_TEAM_CERTIFICATE",
51                    "value": "-----BEGIN CERTIFICATE-----ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-----END CERTIFICATE-----"
52                },
53                {
54                    "name": "GREMLIN_SIDECAR_ENABLED",
55                    "value": "true"
56                },
57                {
58                    "name": "GREMLIN_API_ENDPOINT_URL",
59                    "value": "https://beta.gremlin.com/v1"
60                },
61                {
62                    "name": "GREMLIN_TEAM_ID",
63                    "value": "ffffffff-ffff-ffff-ffff-ffffffffffff"
64                },
65                {
66                    "name": "GREMLIN_TEAM_PRIVATE_KEY",
67                    "value": "-----BEGIN EC PRIVATE KEY-----ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==-----END EC PRIVATE KEY-----"
68                }
69            ],
70            "logConfiguration": {
71                "logDriver": "awslogs",
72                "options": {
73                    "awslogs-create-group": "true",
74                    "awslogs-group": "/ecs/nodejs-ff-demoapp",
75                    "awslogs-region": "us-east-2",
76                    "awslogs-stream-prefix": "ecs"
77                },
78                "secretOptions": []
79            }
80        }
81    ],
82    "family": "nodejs-ff-demoapp",
83    "executionRoleArn": "arn:aws:iam::999999999999:role/ecsTaskExecutionRole",
84    "networkMode": "awsvpc",
85    "volumes": [],
86    "requiresAttributes": [
87        {
88            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
89        },
90        {
91            "name": "ecs.capability.execution-role-awslogs"
92        },
93        {
94            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
95        },
96        {
97            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
98        },
99        {
100            "name": "ecs.capability.task-eni"
101        },
102        {
103            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
104        }
105    ],
106    "requiresCompatibilities": [
107        "FARGATE"
108    ],
109    "cpu": "1024",
110    "memory": "3072",
111    "runtimePlatform": {
112        "cpuArchitecture": "X86_64",
113        "operatingSystemFamily": "LINUX"
114    }
115}