Failure Flags

Deploying Failure Flags on Kubernetes

This document will walk you through setting up Failure-Flags-Sidecar, a small per-process sidecar agent. Failure-Flags-Sidecar runs alongside your application and is responsible for managing Chaos Engineering experiments and reliability tests.

Note

The Failure Flags agents are not in critical path for your application logic or network. They are never exposed to sensitive customer data (encrypted or otherwise). They do not act as network proxies. They do periodically reach out to Gremlin to determine if there are any experiments targeting the attached application and cache those results for a short time.

Adding Failure-Flags-Sidecar to your Pod or Deployment

Failure-Flags-Sidecar container images are available via DockerHub and support both AMD64/x86_64 and ARM64 architectures. These container images include a LICENSE file and a single binary program built for Linux. Alternatively, you can download archives directly: arm64, x86_64.

All versions are listed in a file at: https://assets.gremlin.com/packages/failure-flags-sidecar/VERSIONS.

Setting required environment variables

You can add Failure-Flags-Sidecar to any pod without impacting your application availability or performance. But you do need to add configuration to your environment variables before Failure-Flags-Sidecar will add any value. Configuration comes in via environment variables and or configuration files.

Get started quickly with environment variables only:

FAILURE_FLAGS_ENABLED must be set to either true or yes or 1 to enable the Failure Flags SDK in your application.
GREMLIN_SIDECAR_ENABLED must be set to either true or yes or 1 to enable Failure-Flags-Sidecar. If unset or set to any other value Failure-Flags-Sidecar will operate in NOOP mode.
GREMLIN_TEAM_ID must be set to your Gremlin Team ID. This and other credential material is available through the Gremlin UI.
GREMLIN_TEAM_CERTIFICATE must be set to your Gremlin Team certificate. Newlines may be preserved using the \n escape characters or omited entirely. This and other credential material is available through the Gremlin UI.
GREMLIN_TEAM_PRIVATE_KEY must be set to your Gremlin Team private key. Newlines may be preserved using the \n escape characters or omited entirely. This and other credential material is available through the Gremlin UI.

Setting Targeting Environment Variables

You will want to set custom targeting labels to uniquely identify deployments of your software. Setting custom labels is done through environment variables with a prefix, GREMLIN_LABEL_. Any environment variable set on the sidecar with that prefix will be included as labels on the service. For example:

1An environment variable `GREMLIN_LABEL_CUSTOM` with the value `custom value` will result in the label: "CUSTOM: custom value".

Individual Configuration Values from Files or ARNs

You can configure individual configuration values like GREMLIN_TEAM_CERTIFICATE, GREMLIN_TEAM_PRIVATE_KEY, and GREMLIN_CUSTOM_ROOT_CERTIFICATE to retrieve values from files in the sidecar container or from AWS services using their ARNs. Instead of setting those environment values directly, use their _FILE or _ARN counterparts. Files must be fully qualified paths from the filesystem root. This project currently supports secretsmanager secret and ssm paramter ARNs.

When you add the Failure-Flags-Sidecar to your pod spec and configure the environment variables correctly, your application will be able to consult that extension for Gremlin experiment configuration. You will be able to find your Function in the Gremlin UI under Failure Flags > Services after you launch your app with the layer configured and you exercise the integration.

Once you've added Failure-Flags-Sidecar to your project you can use the Failure Flags library (Node, Python, Java, Go) from your code!

Example Pod Spec with Failure Flags Sidecar

Adding the sidecar means including an additional container definition in the pod spec of any application where you want to use Failure Flags. This example includes a Kubernetes secret to store sensitive team credentials.

yaml

1apiVersion: v1
2kind: Secret
3metadata:
4  name: example-gremlin-secret
5type: Opaque
6data:
7  ## Base64 Encoded Gremlin Team Id
8  team_id: ZmZmZmZmZmYtZmZmZi1mZmZmLWZmZmYtZmZmZmZmZmZmZmZmCg==
9  ## Base64 Encoded Gremlin Team Certificate
10  team_certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCkV4YW1wbGVYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWApYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFgKWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYClhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWApYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFgKWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYClhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWApYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFgKWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYClhYWFhYWFhYCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
11  ## Gremlin Team Certificate
12  team_private_key: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCkV4YW1wbGVYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWApYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFgKWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWD09Ci0tLS0tRU5EIEVDIFBSSVZBVEUgS0VZLS0tLS0K
13---
14apiVersion: app/v1
15kind: Deployment
16metadata:
17  name: sidecar-demo
18  labels:
19    app: sidecar-demo
20spec:
21  replicas: 1
22  selector:
23    matchLabels:
24      app: sidecar-demo
25  template:
26    metadata:
27      labels:
28        app: sidecar-demo
29    spec:
30      containers:
31       - name: demo-application
32         image: YOUR IMAGE HERE
33         env:
34          ## FAILURE_FLAGS_ENABLED
35          - name: FAILURE_FLAGS_ENABLED
36            value: "true"
37
38       ## THIS CONTAINER IS THE SIDECAR
39       - name: gremlin
40         image: gremlin/failure-flags-sidecar:latest
41         imagePullPolicy: Always
42         env:
43          ## GREMLIN_SIDECAR_ENABLED
44          - name: GREMLIN_SIDECAR_ENABLED
45            value: "true"
46          ## GREMLIN_API_ENDPOINT_URL
47          - name: GREMLIN_API_ENDPOINT_URL
48            value: "https://beta.gremlin.com/v1"
49          ## GREMLIN_TEAM_ID
50          - name: GREMLIN_TEAM_ID
51            valueFrom:
52              secretKeyRef:
53                name: example-gremlin-secret
54                key: team_id
55          ## GREMLIN_TEAM_CERTIFICATE
56          - name: GREMLIN_TEAM_CERTIFICATE
57            valueFrom:
58              secretKeyRef:
59                name: example-gremlin-secret
60                key: team_certificate
61          ## GREMLIN_TEAM_PRIVATE_KEY
62          - name: GREMLIN_TEAM_PRIVATE_KEY
63            valueFrom:
64              secretKeyRef:
65                name: example-gremlin-secret
66                key: team_private_key
67          ## GREMLIN_DEBUG will enable debug logging to standard out of the sidecar
68          - name: GREMLIN_DEBUG
69            value: "true"
70          ## SERVICE_NAME is the name of the application you're connecting to Gremlin
71          - name: SERVICE_NAME
72            value: "demo-application"
73          ## REGION is the name of the region or data center you're deploying into (for targeting)
74          - name: REGION
75            value: "demo"
76---
77apiVersion: v1
78kind: Service
79metadata:
80  name: demo-entrypoint
81spec:
82  type: NodePort
83  selector:
84    app: sidecar-demo
85  ports:
86   - port: 3000
87     targetPort: 3000
88     nodePort: 30001