Using Secrets manager in pipeline
One of the major challenges in a multi-team organisation is securing and managing secrets. In this blog, we will see how we use the secrets from AWS Secrets manager in configuration management tools.
Why migrate to AWS Secrets Manager?
- Our pipeline relies on ansible vault and shared among teams. It causes cross team secret exposure and also merge conflicts as git can’t see the value changes.
- Sole ownership to application team who will manage their own secrets.
- Reduce dependency on OPS team.
How this works?
let's take an example of microservices playbook screenshot above, and how we declare vars and vault. These vault will be replaced with:
Jenkins-credentials and ninjaops-credentials are owned by ninjaops team. vault secrets from vars/vault-* and vars/{{ profile }}/vault-env-region.yml can be seen in them.
Third secret, service_name (variable declared in jenkins pipeline) is an application secret (if its using one) owned by app team. Respective playbook can be declared with necessary secrets to use them in config.
Jenkins pipeline using secrets
Few stages within the pipeline uses ansible vault file to decrypt secrets and assign value to variables defined in vars yaml. So, these secrets have been migrated to AWS Secrets manager and secrets values are fetched from there.
Above is a example block declared in the ansible playbook, where it fetches values from secrets_name with defined region mentioned using the appropriate profile.
So, assuming secrets_name had username/password information as key/value,
It can be retrieved to be declared as variable in vars.yml like:
Secrets bifurcation
Secrets fetched from playbook will be divided between application, infrastructure level and others as explained earlier. Application level secret (service_name) will be solely owned by app/project teams.
Secrets (aws_secretsmanager_secret) shall be created by using terraform without terraform version (aws_secretsmanager_secret_version) i.e., not handle the key/values within the secrets resource. These secrets shall be created by invoking the secrets module in their respective application config in terraform.
Tags Rules
And with right tags, respective teams will be able to view only their project secrets alone which is access controlled via permission sets predefined with ResourceTag conditions as below.
This way, any update to secrets (service_name) can be done by app team, not having to rely on ops team.
This way we can reduce dependency on ansible vault from the repo further down the lane.
Trust, but verify.