Azure Key Vault

Integrate Azure Key Vault with POB.

Last Updated: 26 May 2022 • Page Author: Jillur Quddus

Overview

Azure Key Vault is the native managed secrets management engine of the Azure cloud computing platform, enabling applications, services and IT resources to secure manage sensitive credentials, API keys and other secrets throughout their lifecycle. This page provides instructions on how to integrate Azure Key Vault with POB.

For further information regarding Azure Key Vault, please visit https://docs.microsoft.com/en-us/azure/key-vault/general/.

Setup

To provision an Azure Key Vault instance, visit the Key Vaults service via the Azure Portal, select "Create" and provide the required information including the desired Azure subscription and resource group to use, a custom name for the new Key Vault instance (for example myvault), the region and pricing tier. Once created, navigate to the newly created Key Vault instance via the Azure Portal, select "Properties" and make a note of its Vault URI (of the format https://myvault.vault.azure.net) and Directory ID / Tenant ID (similar to abcd1234-987a-1234-9b87-987654abcdef).

Service Principal

Next we need to create a new service principal or identity (or use an existing service principal object) and grant it access to the newly created Key Vault instance, provisioned with relevant read and write privileges. To do this, install the Azure Command Line (Azure CLI) and execute the following commands:

# Explicitly set the Azure subscription to use
$ az account set --subscription <Subscription Name>

# Create a new service principal object for role-based access
$ az ad sp create-for-rbac -n <Custom Service Principal Name> --skip-assignment

    {
        "appId": "wxyz1234-e987-19az-99ab-192837qwerty",
	"displayName": "mysp",
	"name": "wxyz1234-e987-19az-99ab-192837qwerty",
	"password": "C-r.kP~abcdefGH12345IJklmn.abc98",
	"tenant": "abcd1234-987a-1234-9b87-987654abcdef"
    }
    
# Grant the service principal object relevant privileges to the Key Vault instance
# As a minimum POB requires GET, LIST and SET privileges
$ az keyvault set-policy --name <Name of your Key Vault Instance> --spn <Service Principal App ID> --secret-permissions backup delete get list set

Make a note of the appID (client ID) and password (client secret) returned when creating the new service principal object as these will be required when integrating POB with Azure Key Vault as described below.

Integration

Bootstrap Context

As described in Bootstrap Configuration, the configuration for the POB bootstrap context may be found in the pob-configuration Maven module, at src/main/resources/bootstrap.yml.

To integrate the POB Spring bootstrap context with Azure Key Vault, explicitly disable integration with HashiCorp Vault and AWS Secrets Manager before enabling integration with Azure Key Vault as follows:

spring:
    application:
        name: pob
    cloud:
        vault:
            enabled: false
            host:
            port:
            scheme:
            authentication:
            token:
            kv:
                enabled:
                backend:
                default-context:
azure:
    keyvault:
        enabled: true
        client-id: ${AZURE_KEYVAULT_CLIENT_ID}
        client-key: ${AZURE_KEYVAULT_CLIENT_SECRET}
        tenant-id: ${AZURE_KEYVAULT_TENANT_ID}
        uri: ${AZURE_KEYVAULT_URI}
aws:
    secretsmanager:
        enabled: false
        name:
        prefix:
        defaultContext:
        failFast:
        region:

The azure.keyvault configuration namespace includes the following properties that must be completed:

Property	Description	Example Value
enabled	Whether to enable externalised configuration from Azure Key Vault.	true
client-id	The Azure Key Vault Client ID (i.e. the app ID of the service principal object with privileges to read secrets from the relevant Azure Key Vault instance). Note that the Client ID should be set as an environment variable and NOT stored as plaintext in bootstrap.yml.	wxyz1234-e987-19az-99ab-192837qwerty
client-key	The Azure Key Vault Client Secret (i.e. the password of the service principal object to privileges to read secrets from the relevant Azure Key Vault instance). Note that the Client Secret should be set as an environment variable and NOT stored as plaintext in bootstrap.yml.	C-r.kP~abcdefGH12345IJkl mn.abc98
tenant-id	The Azure Key Vault Tenant ID (this can be identified using the Azure Portal or via the Azure CLI). Note that the Tenant ID should be set as an environment variable and NOT stored as plaintext in bootstrap.yml.	abcd1234-987a-1234-9b87-987654abcdef
uri	The Azure Key Vault URI (this can be identified using the Azure Portal or via the Azure CLI). Note that the URI should be set as an environment variable and NOT stored as plaintext in bootstrap.yml.	https://myvault.vault.azure.net

If you define placeholder variables in the application context which are not also defined in Azure Key Vault, then an exception will be thrown when attempting to start the relevant Spring Boot application.

For further information and properties related to loading externalised configuration from Azure Key Vault, please refer to https://docs.microsoft.com/en-us/azure/developer/java/spring-framework/configure-spring-boot-starter-java-app-with-azure-key-vault.

Application Context

As described in Application Configuration, the configuration for the POB application context may be found in the pob-configuration Maven module, at src/main/resources/application.yml.

As described above, assuming that the POB Spring bootstrap context has been integrated with Azure Key Vault, then all placeholder variables defined in the POB Spring application context MUST be defined as a secret managed by Azure Key Vault. For example, the following POB application configuration YAML file has been configured with placeholder variables (for example rdbms-jdbcUrl) that will be loaded from Azure Key Vault:

name: pob
frameworks:
    - id: dos
      name: 'Digital Outcomes and Specialists Framework'
      enabled: true
      baseUrl: https://www.digitalmarketplace.service.gov.uk
      parserClass: ai.hyperlearning.pob.data.parsers.dos.DosParser 
      filter: false
      keywords: ''
      properties:
        opportunitiesUrl: https://www.digitalmarketplace.service.gov.uk/digital-outcomes-and-specialists/opportunities?q=&statusOpenClosed=open
    - id: cf
      name: 'Contracts Finder'
      enabled: true
      baseUrl: https://www.contractsfinder.service.gov.uk
      parserClass: ai.hyperlearning.pob.data.parsers.cf.ContractsFinderParser
      filter: true
      keywords: 'data software java python artificial intelligence machine learning training architecture engineering digital transformation computing language'
      properties:
        opportunitiesUrl: https://www.contractsfinder.service.gov.uk/Search/Results
publishers:
    - id: csv
      enabled: true
      publisherClass: ai.hyperlearning.pob.data.publishers.csv.CsvPublisher
      properties:
        path: ${java.io.tmpdir}/pob.csv
    - id: slack
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.slack.SlackPublisher
      properties:
        channel: ${slack-channel}
        webhook: ${slack-webhook}
    - id: microsoft-teams
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.microsoft.MicrosoftTeamsPublisher
      properties:
        webhook: ${microsoft-teams-webhook}
    - id: google-chat
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.google.GoogleChatPublisher
      properties:
        webhook: ${google-chat-webhook}
    - id: elasticsearch
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.elastic.ElasticsearchPublisher
      properties:
        url: ${elasticsearch-url}
        username: ${elasticsearch-username}
        password: ${elasticsearch-password}
        index: pob
        ssl: true
pipelines:
    main:
        enabled: true
        scheduler:
            enabled: false
            cron: "0 0/20 * * * *"
        bulkPublicationDelay:
            enabled: true
            duration: 10
storage:
    rdbms:
        driverClassName: ${rdbms-driverClassName}
        jdbcUrl: ${rdbms-jdbcUrl}
        username: ${rdbms-username}
        password: ${rdbms-password}

All the placeholder variables defined in the example application configuration file above, for example ${slack-channel}, ${slack-webhook}, ${microsoft-teams-webhook}, ${google-chat-webhook} and ${rdbms-jdbcUrl} (to highlight just a few), must also be defined as secrets managed by Azure Key Vault. This can be achieved via the Azure Portal as illustrated in the following screenshot:

POB Application Context Secrets