Awesome Open Source
Awesome Open Source

Build Status Maven Central Javadoc

AWSSSMChaosRunner

AWSSSMChaosRunner is a library which simplifies failure injection testing and chaos engineering for EC2 and ECS (with EC2 launch type). It uses the AWS Systems Manager SendCommand for failure injection.

An in-depth introduction to this library and how Prime Video uses it can be found here - https://aws.amazon.com/blogs/opensource/building-resilient-services-at-prime-video-with-chaos-engineering/

Usage for EC2

  1. Setup permissions for calling SSM from tests package

    This can be done in many different ways. The approach described here generates temporary credentials for AWS SSM on each run of the tests. To enable this the following are needed

    • An IAM role with the following permissions. (JSON snippet)
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Action": [
                      "sts:AssumeRole",
                      "ssm:CancelCommand",
                      "ssm:CreateDocument",
                      "ssm:DeleteDocument",
                      "ssm:DescribeDocument",
                      "ssm:DescribeInstanceInformation",
                      "ssm:DescribeDocumentParameters",
                      "ssm:DescribeInstanceProperties",
                      "ssm:GetDocument",
                      "ssm:ListTagsForResource",
                      "ssm:ListDocuments",
                      "ssm:ListDocumentVersions",
                      "ssm:SendCommand"
                  ],
                  "Resource": [
                      "*"
                  ],
                  "Effect": "Allow"
              },
              {
                  "Action": [
                      "ec2:DescribeInstances",
                      "iam:PassRole",
                      "iam:ListRoles"
                  ],
                  "Resource": [
                      "*"
                  ],
                  "Effect": "Allow"
              },
              {
                  "Action": [
                      "ssm:StopAutomationExecution",
                      "ssm:StartAutomationExecution",
                      "ssm:DescribeAutomationExecutions",
                      "ssm:GetAutomationExecution"
                  ],
                  "Resource": [
                      "*"
                  ],
                  "Effect": "Allow"
              }
          ]
      }
      
    • An IAM user which can assume the above role.
  2. Add AWSSSMChaosRunner maven dependency to your tests package

    <dependency>
      <groupId>software.amazon.awsssmchaosrunner</groupId>
      <artifactId>awsssmchaosrunner</artifactId>
      <version>1.3.0</version>
    </dependency> 
    
  3. Initialise the SSM Client (Kotlin snippet)

    @Bean
    open fun awsSecurityTokenService(
       credentialsProvider: AWSCredentialsProvider, 
       awsRegion: String
       ): AWSSecurityTokenService {
        return AWSSecurityTokenServiceClientBuilder.standard()
            .withCredentials(credentialsProvider)
            .withRegion(awsRegion)
            .build()
    }
    
    @Bean
    open fun awsSimpleSystemsManagement(
       securityTokenService: AWSSecurityTokenService,
       awsAccountId: String,
       chaosRunnerRoleName: String
       ): AWSSimpleSystemsManagement {
        val chaosRunnerRoleArn = "arn:aws:iam::$awsAccountId:role/$chaosRunnerRoleName"
        val credentialsProvider = STSAssumeRoleSessionCredentialsProvider
            .Builder(chaosRunnerRoleArn, "ChaosRunnerSession")
            .withStsClient(securityTokenService).build()
    
        return AWSSimpleSystemsManagementClientBuilder.standard()
            .withCredentials(credentialsProvider)
            .build()
    }
    
  4. Start the fault injection attack before starting the test and stop it after the test (Kotlin snippet)

    import software.amazon.awsssmchaosrunner.attacks.SSMAttack
    import software.amazon.awsssmchaosrunner.attacks.SSMAttack.Companion.getAttack
    ...
    
    @Before
    override fun initialise(args: Array<String>) {
        if (shouldExecuteChaosRunner()) {
            ssm = applicationContext.getBean(AWSSimpleSystemsManagement::class.java)
            ssmAttack = getAttack(ssm, attackConfiguration)
            command = ssmAttack.start()
        }
    }
    
    @After
    override fun destroy() {
        ssmAttack.stop(command)
    }
    
  5. Run the test

FAQs

  • What about Chaos-SSM-Documents (github repo) ?

    The idea for AWSSSMChaosRunner came from Chaos-SSM-Documents (and from medium post).

  • Why use AWS SSM ?

    In most cases EC2 fleets are already using the SSM Agent for OS patching, this library leverages this existing agent and reduces setup work needed for fault injection.

  • What failure injections are available ?

  • What about other failure injections ?

    You're welcome to send pull requests for other failure injections.

  • How is the failure injection rolled back ? / What if AWS SSM fails to stop the failure injection ?

    SSM is not actually used to stop/roll back the failure injection. The failure injection scripts first schedule the failure rollback (with at command) and then start the actual failure injection. This ensures that, barring special cases, the failure injection will be rolled back at a specified time in the future.

  • What languages does AWSSSMChaosRunner support ?

    AWSSSMChaosRunner can be used as a dependency from Kotlin, Java or Scala.

  • Is there a complete working demo of using this library ?

    A demonstration can be found in Demo.kt. To run the demo:

    • Clone this project.

    • Build the gradle project successfully (via gradle CLI or IDE).

    • Modify the awsProfile value to the awsProfile name for your AWS account.

    • Comment @Disabled() annotation.

    • Run the gradle test target.

  • Can AWSSSMChaosRunner be used for Amazon Elastic Container Service (ECS) ?

    Yes. The above EC2 usage steps should be followed after the SSM agent setups listed below.

    • ECS + EC2 launch type
      • SSM Agent setup

        The SSM Agent is required for using SSM SendCommand API and thus, for using AWSSSMChaosRunner. The base EC2 images include the SSM Agent, but the base ECS images do not. It can be installed directly at the host level. This can be achieved with the following CloudFormation snippet (YAML):

        # Adapted from https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ecs.html
          LaunchConfiguration0:
            Type: AWS::AutoScaling::LaunchConfiguration
            Metadata:
              # This is processed by cfn-init in the Properties.UserData script below. It installs a
              # service that monitors for changes in the Metadata just below, causing a configuration
              # update.
              #
              # CloudFormation updates to the LaunchConfiguration's Properties won't take effect on
              # existing instances. Consequently, any CloudFormation field that could change should go in
              # the Metadata.
              AWS::CloudFormation::Init:
                config:
                  # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html
                  packages:
                    rpm:
                      # The SSM (Systems Systems Manager) agent is necessary to use `aws ssm send-command`
                      # or 'Run Command' in the AWS-EC2 console. It's also required by InfoSec for our
                      # exception. The base EC2 images include it, but the base ECS images do not.
                      # https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-startup-linux.html
                      amazon-ssm-agent: !Sub https://s3.${AWS::Region}.amazonaws.com/amazon-ssm-${AWS::Region}/latest/linux_amd64/amazon-ssm-agent.rpm
        
      • Possible failure injections

        SSM SendCommand API will run the underlying failure injection commands directly on the EC2 host. This will affect all tasks running on these hosts. The EC2 + ECS host does not impose any additional restrictions regarding what resources can or can't be accessed. Thus, all AWSSSMChaosRunner attacks can be run on EC2 + ECS.

  • Can AWSSSMChaosRunner be used for AWS Lambda ?

    No.


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
kotlin (3,673
aws (1,104
ecs (131
ec2 (54
ssm (52
chaos-engineering (29
fargate (26