Adding JSON Parameter support using CloudFormation Macro Base64Loop

Page content

This time we cover how to add advanced functions to CloudFormation using a Macro. A macro is effectively a Lambda function that is used to process the template prior to CloudFormation deploying it. The macro can make changes to the resources and other objects, however Parameters cannot be changed when outputing the new template.

We can however use a passed in parameter to alter the Resources. As we are using a Lambda function, the possibilities are almost endless in what we can do.

After attempting to pass in a json string in different formats, (escaped, csv, and a couple of other weird methods), I remembered that AWS uses the Base64 encoding a lot. Passing userdata to EC2 and receiving detailed error messages are just some. So why not use this format to pass in our list, something CloudFormation does not support.

The cloudFormation Macro - base64loop

The macro started out using the ideas and format from the AWS CloudFormation Macro examples here: https://github.com/aws-cloudformation/aws-cloudformation-macros/ It is worth checking out this repo as well, as there are some other macros like StringFunctions that are really handy too!

This macro ha a few key functions

Decode a string parameter from Base64 back to a JSON dictionary
Iterate over the objects in the dictionary using Base64loop into new Resources.
Provide a list of the resources made using either Ref: or Gn::GetAtt.Arn

Encoding/Decoding the Base64 Parameter

Before encoding the string must be a valid JSON dict, otherwise there will be errors when processing the object. Use jq and base64 commands in linux or variants thereof to encode your string.

Example:

BASE64PARAM=$(cat base64-json-example.json | jq -c '.' | base64 -w0)

Once encoded, simply pass the base64 string to Cloudformation as a parameter. String parameters have a limt of 4096 bytes, so you can fit a fair bit of information.

In the Macro lambda function, the decoding process is quite simple, we use the python base64 library to decode the string that has been passed in, then load it using the json loads function.

Iterating over the objects

A CloudFormation Macro is simply a lambda function, in this case written in python.

To loop through our objects, we need to look for our Function keys in the resources of the template. For the loop function we look for a key called Base64loop: <ParameterName>. Following this is the name of the parameter storing your Base64 encoded data.

Using the objects in the JSON object, we now iterate over the resource, replacing string values starting with !Base64loop <keyname> with the values with those from the Parameter. Examples are on Github: (https://github.com/steven-geo/cloudformation-macro-base64loop)

Example:

Source Json Object:

{
    "databucket": {
        "name": "zzzht1-databucket"
    },"metabucket": {
        "name": "zzzht1-metabucket"
    }
}

CloudFormation:

Parameters:
  RuleBase64:
    Type: String

Resources:
  TestingBucket:
    Type: AWS::S3::Bucket
    Base64loop: RuleBase64
    Properties:
      BucketName: "!Base64loop name"

For more information on AWS CloudFormation Macros see here: AWS CloudFormation - Perform custom processing on CloudFormation templates with template macros

Resource Ref Id’s and Arn’s

Once I completed the code to iterate over the resources, I needed to obtain the list of resources created. Some common uses would be to add them to an IAM Policy or to add to a Resource Share (RAM).

Using the Base64loopArn: <ParameterName> or Base64loopRef: <ParameterName> in another resource, then the string !Base64loopArn or !Base64loopRef respectively to obtain the list of parameters.

NOTE: This will return a list, so when addign the above to a key, do not define the list, as this will then place the list within the first list object. Simply add the string as the value.

Example:

  TestPolicy:
    Type: "AWS::IAM::Policy"
    Base64loopArn: RuleBase64
    Properties:
      Roles:
        - Ref: TestRole
      PolicyName: S3BucketTesting
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - "s3:*"
            Resource: "!Base64loopArn"

Testing

To test on the commandline what the resulting template is, you can use the below commandline. As we currently don’t support passing parameters to the base64loop.py function, you will need to ensure a Default: value is defined in the template before running, otherwise you will get a ```KeyError: ‘Default’```` error when running.

python3 -u base64loop.py base64loop-test-s3.yml

If you wish to test in AWS, there are two test CloudFormation Templates in the github repo with matching json object files you can encode and use to test the macro. Refer to the GitHub Repo.

Macro Test Deployed

Examples and more Information

The full examples, and CloudFormation Macro template you can deploy directly into your account is here: (https://github.com/steven-geo/cloudformation-macro-base64loop)

Please feel welcome to create issues/PRs to fix any issues you have. I would like to improve the error handling of the Base64 decoding with JSON Objects, and add the ability to pass Parameters on the command line when testing