Fetching secret based on environment variable

[NickIannelli]

Problem

I am configuring my deploy pipeline so that the target environment can be selected from a list. That environment is then written into a variable ($$deploy_environment for example). What I am trying to do is fetch the relevant secret for said environment, following the path of /app-name/$$deploy_environment.

Code

  - input: "Deploy?"
    key: start-deploy
    fields:
      - select: "Environment"
        key: "deploy_environment"
        options:
          - label: "Dev"
            value: "dev"
          - label: "Production"
            value: "prod"
  - command: # Populate deploy_environment into the global env
      - deploy_environment=buildkite-agent meta-data get "deploy_environment"
    depends_on: start-deploy

  - wait

  - command:
      - echo "$ENV_SECRETS" # Obviously would never do this, but for example...
    plugins:
      - "seek-oss/aws-sm#v2.3.1":
          "env":
            "ENV_SECRETS":
              "secret-id": "my-app/$$deploy_environment/env"

Expected: This should fetch either the my-app/dev/env secret or the my-app/prod/env secret depending on which is selected.

Actual: When fetching, the below error is returned from SecretsManager

An error occurred (ValidationException) when calling the GetSecretValue operation: Invalid name. Must be a valid name containing alphanumeric characters, or any of the following: -/_+=.@!

Error appears to be because it's not evaluating the expression before fetching it from secrets manager.

Questions

• Is this expected?
• If so - what would the recommended pattern be for fetching different variables from secrets manager based on a variable?

Many thanks!

Nick.

[72636c]

Hi @NickIannelli!

The likely reason why the example code does not work is because Buildkite steps are independent; environment variables and previous command outputs are not carried across to subsequent steps.

The aws-sm plugin runs on the post-checkout hook, so you'll need to set any dynamic environment variables prior to that point. Per the Buildkite documentation you can write a repository hook in .buildkite/hooks (unfortunately there isn't a convenient pipeline.yml syntax like command to do this) or use a plugin hook like https://github.com/chronotc/metadata-env-buildkite-plugin.

Feel free to reopen this if you have other questions or concerns.

[NickIannelli]

Hi @NickIannelli!

The likely reason why the example code does not work is because Buildkite steps are independent; environment variables and previous command outputs are not carried across to subsequent steps.

The aws-sm plugin runs on the post-checkout hook, so you'll need to set any dynamic environment variables prior to that point. Per the Buildkite documentation you can write a repository hook in .buildkite/hooks (unfortunately there isn't a convenient pipeline.yml syntax like command to do this) or use a plugin hook like https://github.com/chronotc/metadata-env-buildkite-plugin.

Feel free to reopen this if you have other questions or concerns.

HI @72636c

I was (and have) been using the metadata-env-buildkite-plugin, still to no avail. I stripped it out for the sample code so as to show the isolated issue.

The original code for the plugins is:

    plugins:
      - chronotc/metadata-env#v1.0.0:
          keys:
            - DEPLOY_ENV=deploy_environment
      - "seek-oss/aws-sm#v2.3.1":
          "env":
            "SECRET_ENV":
              "secret-id": "my-app/$$DEPLOY_ENV/env"
      - docker#v3.8.0:
          image: "node:14"
          environment:
            - SECRET_ENV
            - deploy_environment

It appears that the secret attempting to be fetched is the raw string my-app/$DEPLOY_ENV/env - not the evaluated one.

Any help would be greatly appreciated.

[hateonion]

Hi @NickIannelli!

The likely reason why the example code does not work is because Buildkite steps are independent; environment variables and previous command outputs are not carried across to subsequent steps.

The aws-sm plugin runs on the post-checkout hook, so you'll need to set any dynamic environment variables prior to that point. Per the Buildkite documentation you can write a repository hook in .buildkite/hooks (unfortunately there isn't a convenient pipeline.yml syntax like command to do this) or use a plugin hook like https://github.com/chronotc/metadata-env-buildkite-plugin.

Feel free to reopen this if you have other questions or concerns.

I think it

Is anyone following this?

Hi @NickIannelli!
The likely reason why the example code does not work is because Buildkite steps are independent; environment variables and previous command outputs are not carried across to subsequent steps.
The aws-sm plugin runs on the post-checkout hook, so you'll need to set any dynamic environment variables prior to that point. Per the Buildkite documentation you can write a repository hook in .buildkite/hooks (unfortunately there isn't a convenient pipeline.yml syntax like command to do this) or use a plugin hook like https://github.com/chronotc/metadata-env-buildkite-plugin.
Feel free to reopen this if you have other questions or concerns.

HI @72636c

I was (and have) been using the metadata-env-buildkite-plugin, still to no avail. I stripped it out for the sample code so as to show the isolated issue.

The original code for the plugins is:

    plugins:
      - chronotc/metadata-env#v1.0.0:
          keys:
            - DEPLOY_ENV=deploy_environment
      - "seek-oss/aws-sm#v2.3.1":
          "env":
            "SECRET_ENV":
              "secret-id": "my-app/$$DEPLOY_ENV/env"
      - docker#v3.8.0:
          image: "node:14"
          environment:
            - SECRET_ENV
            - deploy_environment

It appears that the secret attempting to be fetched is the raw string my-app/$DEPLOY_ENV/env - not the evaluated one.

Any help would be greatly appreciated.

I think it won't work because the plugin load the secret into environment variable when running environment hook. And at that time the meta-to-env plugin is not evaluated..