Chaos Monkey Alternatives

Private Cloud

  • 2 min read
  • Last Updated October 17, 2018

GomJabbar

GomJabbar is an open-source implementation of Chaos Monkey written in Java and designed to perform attacks within a private cloud architecture. Attacks are defined through the YAML configuration file and are executed as plain shell commands (e.g. sudo service ${module} stop). It also integrates with Ansible and Rundeck.

Attacks are defined in the config.yaml within the commands block. The fail value is the fault command to be executed, while the (optional) revert value is executed to attempt reversion. For example, here the shutdown_service command calls sudo service <service-name> stop as a fault and the opposite to revert.

yaml
1commands:
2 kill_service:
3 description: 'Kills services.'
4 fail: 'sudo pkill -s 9 -f ${module}'
5 revert: 'sudo service ${module} start'
6
7 shutdown_service:
8 description: 'Shuts down services.'
9 fail: 'sudo service ${module} stop'
10 revert: 'sudo service ${module} start'

The filters block in config.yaml defines a list of clusters, modules, and tags to either be included or excluded via include or exclude, respectively.

yaml
1filters:
2 clusters:
3 include:
4 - gremlin-chaos
5 exclude:
6 - grandmas-cluster
7
8 modules:
9 include:
10 - nginx
11 - redis-server
12 exclude:
13 - critical
14
15 tags:
16 include:
17 - development
18 - production
19 exclude:
20 - test

As we saw in the commands block above ${module} can be referenced in fault commands and will automatically be replaced with the relevant service name. Once configured you can start the server by exporting the generated GJ_OPTIONS and calling the gomjabbar.sh script.

bash
1export GJ_OPTIONS="-Dcom.outbrain.gomjabbar.configFileUrl=<config file url> ..."
2./gomjabbar.sh

You can then use the REST API to trigger attacks and revert faults.

Failure Injection on Your Private Cloud with Gremlin

Gremlin locates weaknesses in your private cloud architecture before they cause problems. Gremlin makes Chaos Engineering simple, safe, and secure, improving your private cloud's stability and resilience. You can begin executing Chaos Experiments in just a few minutes by signing up for an account and installing Gremlin. Gremlin can run experiments on every major type of private cloud infrastructure including Linux, Docker, Kubernetes, OpenStack, VMware, and many more.

Check out these tutorials to learn how to install Gremlin and start Chaos Engineering your private cloud today.

Muxy

Muxy is an open-source tool written in Go that allows you to tamper with network traffic at the transport, TCP, or HTTP protocol layers of your systems. In can simulate real-world network connectivity problems and can also be extended with plugins via the Plugo interface.

Muxy is configured through a YAML file that defines a number of proxy and middleware blocks.

  • Start by installing Muxy with go get.

    bash
    1go get github.com/mefellows/muxy
  • Create a YAML file and paste the following test configuration in it.

    yaml
    1# gremlin.yml
    2proxy:
    3 - name: http_proxy
    4 config:
    5 host: 0.0.0.0
    6 port: 8181
    7 proxy_host: www.gremlin.com
    8 proxy_port: 443
    9
    10# Proxy plugins
    11middleware:
    12 - name: http_tamperer
    13 config:
    14 request:
    15 host: www.gremlin.com
    16 port: 443
    17
    18 # Message Delay request/response plugin
    19 - name: delay
    20 config:
    21 request_delay: 2500
    22 response_delay: 2500
    23
    24 # Log in/out messages
    25 - name: logger

    This configuration creates a localhost:8181 proxy that targets www.gremlin.com:443. It also creates an http_tamperer middleware that catches requests made to www.gremlin.com:443 and adds a 2.5-second delay to both requests and responses. Finally, it outputs Muxy messages to the default terminal logger.

  • Start Muxy with the following command, ensuring that you specify the location of the YAML configuration file.

    bash
    1muxy proxy --config ./gremlin.yml
    bash
    1# OUTPUT
    22018/09/18 22:47:39.179395 [INFO] Loading plugin http_tamperer
    32018/09/18 22:47:39.179415 [INFO] Loading plugin delay
    42018/09/18 22:47:39.179419 [INFO] Loading plugin logger
    52018/09/18 22:47:39.179451 [INFO] Loading proxy http_proxy
    62018/09/18 22:47:39.179465 [DEBUG] HTTP Tamperer Setup()
    72018/09/18 22:47:39.179476 [DEBUG] Delay Symptom - Setup()
    82018/09/18 22:47:39.179578 [INFO] HTTP proxy listening on http://0.0.0.0:8181
  • Now that Muxy is listening at http://0.0.0.0:8181 you can make a request to www.gremlin.com through the proxy to test it out. Muxy should add a delay of approximately 5 seconds to the request.

    bash
    1time curl -H"Host: www.gremlin.com" http://localhost:8181/
    bash
    1# OUTPUT
    2real 0m5.070s
    3user 0m0.000s
    4sys 0m0.004s
  • Now try the same timed test but without passing through the localhost:8181 proxy, and you'll see Muxy doesn't catch or muck with this request.

    bash
    1time curl -o /dev/null https://www.gremlin.com
    bash
    1# OUTPUT
    2real 0m0.257s
    3user 0m0.036s
    4sys 0m0.012s

Check out the official Muxy documentation for more information on integrating Muxy into your private cloud Chaos Engineering.

© 2023 Gremlin Inc.All rights reserved.Privacy Policy
Download PDF