AWS Secrets Manager

Integrate AWS Secrets Manager with POB.

Last Updated: 26 May 2022 • Page Author: Jillur Quddus

Overview

AWS Secrets Manager is the native managed secrets management engine of the AWS cloud computing platform, enabling applications, services and IT resources to secure manage sensitive credentials, API keys and other secrets throughout their lifecycle. This page provides instructions on how to integrate AWS Secrets Manager with POB.

For further information regarding AWS Secrets Manager, please visit https://aws.amazon.com/secrets-manager.

Setup

There are no explicit setup processes to follow in order to provision AWS Secrets Manager. As long as you have an AWS account, you can access AWS Secrets Manager via the AWS Management Console.

IAM Policies

As described in Bootstrap Configuration, the POB Spring bootstrap context is responsible for loading and decrypting configuration properties from external sources, such as HashiCorp Vault, Azure Key Vault or AWS Secrets Manager. This enables developers and platform engineers to avoid writing secrets (such as usernames, passwords, API keys, access tokens and database connectivity strings) directly in properties files. Instead they can define and provide placeholder variables and the Spring bootstrap context will load the relevant secrets from the relevant secrets engine.

In order to integrate the Spring bootstrap context with AWS Secrets Manager, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables must be set in the relevant AWS deployment environments associated with an IAM user that has the SecretsManagerReadWrite AWS managed policy attached to it (or equivalent custom policy), as illustrated in the following screenshot:

IAM User provisioned with the SecretsManagerReadWrite AWS managed policy

To configure an IAM user provisioned with the SecretsManagerReadWrite AWS managed policy, please visit the IAM Management Console via the AWS Management Console.

Integration

Bootstrap Context

As described in Bootstrap Configuration, the configuration for the POB bootstrap context may be found in the pob-configuration Maven module, at src/main/resources/bootstrap.yml.

To integrate the POB Spring bootstrap context with AWS Secrets Manager, explicitly disable integration with HashiCorp Vault and Azure Key Vault before enabling integration with AWS Secrets Manager as follows:

spring:
    application:
        name: pob
    cloud:
        vault:
            enabled: false
            host:
            port:
            scheme:
            authentication:
            token:
            kv:
                enabled:
                backend:
                default-context:
azure:
    keyvault:
        enabled: false
        client-id:
        client-key:
        tenant-id:
        uri:
aws:
    secretsmanager:
        enabled: true
        name: pob
        prefix: /secret
        defaultContext: application
        failFast: true
        region: eu-west-2

The aws.secretsmanager configuration namespace includes the following properties that must be completed:

Property	Description	Example Value
enabled	Whether to enable externalised configuration from AWS Secrets Manager.	true
name	The name of the secret to lookup containing the externalised configuration as key-value pairs.	pob
prefix	The name of the prefix indicating the first level for every property.	/secret
defaultContext	The context name used by all applications.	application
failFast	Whether to throw exceptions during configuration lookup, otherwise log warnings.	true
region	The AWS region of the AWS Secrets Manager.	eu-west-2

In the example configuration above, we inform the POB Spring bootstrap context that externalised configuration can be found in our AWS Secrets Manager instance in a secret named /secret/pob. Thus all placeholder variables defined in the application configuration MUST be defined as key-value pairs in the /secret/pob secret managed by AWS Secrets Manager.

If you define placeholder variables in the application context which are not also defined in the /secret/pob secret managed by AWS Secrets Manager, then an exception will be thrown when attempting to start the relevant Spring Boot application.

For further information and properties related to loading externalised configuration from AWS Secrets Manager, please refer to https://docs.spring.io/spring-cloud-aws/docs/2.2.4.RELEASE/reference/html/appendix.html.

Application Context

As described in Application Configuration, the configuration for the POB application context may be found in the pob-configuration Maven module, at src/main/resources/application.yml.

As described above, assuming that the POB Spring bootstrap context has been integrated with AWS Secrets Manager, then all placeholder variables defined in the POB Spring application configuration MUST be defined as key-value pairs in the relevant secret managed by AWS Secrets Manager (for example /secret/pob). For example the following POB application configuration YAML file has been configured with placeholder variables (for example rdbms-jdbcUrl) that will be loaded from AWS Secrets Manager:

name: pob
frameworks:
    - id: dos
      name: 'Digital Outcomes and Specialists Framework'
      enabled: true
      baseUrl: https://www.digitalmarketplace.service.gov.uk
      parserClass: ai.hyperlearning.pob.data.parsers.dos.DosParser 
      filter: false
      keywords: ''
      properties:
        opportunitiesUrl: https://www.digitalmarketplace.service.gov.uk/digital-outcomes-and-specialists/opportunities?q=&statusOpenClosed=open
    - id: cf
      name: 'Contracts Finder'
      enabled: true
      baseUrl: https://www.contractsfinder.service.gov.uk
      parserClass: ai.hyperlearning.pob.data.parsers.cf.ContractsFinderParser
      filter: true
      keywords: 'data software java python artificial intelligence machine learning training architecture engineering digital transformation computing language'
      properties:
        opportunitiesUrl: https://www.contractsfinder.service.gov.uk/Search/Results
publishers:
    - id: csv
      enabled: true
      publisherClass: ai.hyperlearning.pob.data.publishers.csv.CsvPublisher
      properties:
        path: ${java.io.tmpdir}/pob.csv
    - id: slack
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.slack.SlackPublisher
      properties:
        channel: ${slack-channel}
        webhook: ${slack-webhook}
    - id: microsoft-teams
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.microsoft.MicrosoftTeamsPublisher
      properties:
        webhook: ${microsoft-teams-webhook}
    - id: google-chat
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.google.GoogleChatPublisher
      properties:
        webhook: ${google-chat-webhook}
    - id: elasticsearch
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.elastic.ElasticsearchPublisher
      properties:
        url: ${elasticsearch-url}
        username: ${elasticsearch-username}
        password: ${elasticsearch-password}
        index: pob
        ssl: true
pipelines:
    main:
        enabled: true
        scheduler:
            enabled: false
            cron: "0 0/20 * * * *"
        bulkPublicationDelay:
            enabled: true
            duration: 10
storage:
    rdbms:
        driverClassName: ${rdbms-driverClassName}
        jdbcUrl: ${rdbms-jdbcUrl}
        username: ${rdbms-username}
        password: ${rdbms-password}

All the placeholder variables defined in the example application configuration file above, for example ${slack-channel}, ${slack-webhook}, ${microsoft-teams-webhook}, ${google-chat-webhook} and ${rdbms-jdbcUrl} (to highlight just a few), must also be defined as key-value pairs in the relevant secret managed by AWS Secrets Manager (for example /secret/pob). This can be achieved via the AWS management console as illustrated in the following screenshot:

POB Application Context Secrets