Azure Functions

Provision an Azure Function app via the Azure Portal and deploy the main end-to-end POB data pipeline Spring Boot application to it.

Last Updated: 26 May 2022 • Page Author: Jillur Quddus

Overview

Azure Functions is the native serverless, event-driven compute service offered by the Microsoft Azure cloud computing platform, enabling applications and backend services to be run without provisioning or managing any servers. This page provides instructions on how to provision an Azure Function app and then deploy the main end-to-end POB data pipeline Spring Boot application to it. The main POB data pipeline instantiates and executes all the enabled procurement framework parser classes and publisher classes that have been registered in application.yml.

For further information regarding Azure Functions, please visit https://azure.microsoft.com/en-gb/services/functions.

It is recommended that you configure and integrate the steps described in this page into a CI/CD pipeline to automate build, testing and deployment.

Main Data Pipeline

POB provides an out-of-the-box Azure Function Spring Boot application that wraps around the main POB data pipeline, enabling it to be deployed to an Azure Function app and triggered via a CRON schedule. The Azure Function Spring Boot application for the main POB data pipeline may be found in the $POB_BASE/pob-apps/pob-apps-azure/pob-azure-function-app-data-pipelines-scheduler Maven module.

CRON Schedule

The CRON schedule that determines how frequently the main POB data pipeline will run when deployed to an Azure Function app is defined in the Spring Cloud Azure Function handler Java class which may be found in the $POB_BASE/pob-apps/pob-apps-azure/pob-azure-function-app-data-pipelines-scheduler Maven module and in a class named MainPipelineSchedulerAzureFunctionHandler. By default it is configured to run every 30 minutes.

Setup

Build from Source

In order to compile and build the main POB data pipeline Azure Function Spring Boot application in preparation for deployment to an Azure Function app, please follow the instructions detailed in Build from Source.

Azure CLI

We shall use the Azure Command Line Interface (CLI) to deploy the main POB data pipeline Azure Function Spring Boot JAR file, that was created in the Build from Source stage above, to an Azure Function app. To install the Azure CLI, please follow the instructions below:

The instructions below are for Ubuntu 20.04. Installation instructions for other Linux distributions and other operating systems such as Windows may be found at https://docs.microsoft.com/en-us/cli/azure.

# Remove any old installations of the Azure CLI
$ sudo apt remove azure-cli -y && sudo apt autoremove -y

# Install the Azure CLI
$ curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Sign into your Azure account
$ az login

Azure Functions Core Tools

We shall use Azure Functions Core Tools to deploy a function app directly to an Azure subscription (note that Azure Functions Core Tools also enables developers to develop and test functions locally before remote deployment). To install Azure Functions Core Tools, please follow the instructions below:

The instructions below are for Ubuntu 20.04. Installation instructions for other Linux distributions and other operating systems such as Windows may be found at https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local.

We shall be installing the version of Azure Functions Core Tools that supports version 4.x of the Azure Functions runtime which is the recommended version (for Java and indeed all languages) at the time of writing. For further information regarding Azure Functions runtime versions, please refer to https://docs.microsoft.com/en-us/azure/azure-functions/functions-versions.

# Install the Microsoft package repository GPG key
$ curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
$ sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
$ sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-$(lsb_release -cs)-prod $(lsb_release -cs) main" > /etc/apt/sources.list.d/dotnetdev.list'

# Install Azure Functions Core Tools
$ sudo apt-get update
$ sudo apt-get install azure-functions-core-tools-4

Azure Function App

We shall use the Azure Portal to provision Azure Function apps. To do so, navigate to the Function App service via the Azure Portal, select "Create" and follow the instructions below:

1. Subscription and Resource Group - select the Azure subscription and resource group to use for the new function app.

2. Function App Name - enter a custom function name that describes the purpose of the function, for example pob-main-data-pipeline.

3. Publish - select "Code".

4. Runtime Stack - the main POB data pipeline is a Java Spring Boot application. Thus please select "Java" and version "11.0" as the runtime stack.

5. Storage Account - select (or create) an Azure Storage account for this function app.

6. Operating System - select "Linux".

7. Plan Type - dependent on your deployment architecture, select a plan that will dictate how the function app will scale. For most use cases, the "Consumption (Serverless)" plan should suffice.

Once configured, select "Review + Create" to create the new Azure Function app.

Deployment

Please follow the instructions below to deploy the main POB data pipeline Spring Boot application to an Azure Function app:

1) Create a new Azure Function app configured with the Java 11 runtime via the Azure Portal as detailed in the Setup section above. We shall call this Azure Function app pob-main-data-pipeline for the purposes of these instructions.

2) As detailed in Azure Key Vault, and assuming that you are integrating POB with the Azure Key Vault secrets engine, to integrate the POB Spring bootstrap context with Azure Key Vault we configure bootstrap.yml as follows:

spring:
    application:
        name: pob
    cloud:
        vault:
            enabled: false
            host:
            port:
            scheme:
            authentication:
            token:
            kv:
                enabled:
                backend:
                default-context:
azure:
    keyvault:
        enabled: true
        client-id: ${AZURE_KEYVAULT_CLIENT_ID}
        client-key: ${AZURE_KEYVAULT_CLIENT_SECRET}
        tenant-id: ${AZURE_KEYVAULT_TENANT_ID}
        uri: ${AZURE_KEYVAULT_URI}
aws:
    secretsmanager:
        enabled: false
        name:
        prefix:
        defaultContext:
        failFast:
        region:

The Azure Key Vault client ID (i.e. the app ID of the service principal object with privileges to read secrets from the relevant Azure Key Vault instance), client secret (i.e. the password of the service principal object to privileges to read secrets from the relevant Azure Key Vault instance), tenant ID and URI properties should be set as environment variables and NOT stored as plaintext in bootstrap.yml. Thus these environment variables should be set in our Azure Function app as application settings. To do this, navigate to the relevant Azure Function app via the Azure Portal, then select "Configuration" and "Application Settings". Select "New Application Setting" and create all the environment variables defined in bootstrap.yml. For further information regarding the values of these environment variables, please refer to Azure Key Vault.

3) When deploying Java-based functions to Azure Function apps, Azure requires us to explicitly provide the main Java class to invoke as an environment variable called MAIN_CLASS. To do this, navigate to the Azure Function app via the Azure Portal, then select "Configuration" and "Application Settings". Select "New Application Setting" and set the following environment variable:

Environment Variable Name	Value
MAIN_CLASS	ai.hyperlearning.pob.apps.azure.functions.data.pipelines.scheduler.MainPipelineSchedulerAzureFunctionApp

4) Next we need to configure another environment variable called JAVA_OPTS configured with the JVM memory parameters for our Azure Function. To do this, navigate to the relevant Azure Function app via the Azure Portal, then select "Configuration" and "Application Settings". Select "New Application Setting" and create a new application setting called JAVA_OPTS with the value -Xms512m -Xmx2g.

5) We have now set all the required Azure Function app environment variables. Press save to persist the new application settings (which will restart the Azure Function app).

6) We are now ready to deploy the packaged Java Spring application artifact to the Azure Function app. Assuming that you have followed the instructions detailed in the Setup section above, navigate to $POB_BASE/pob-apps/pob-apps-azure/pob-azure-function-app-data-pipelines-scheduler . Next open the pom.xml file inside this directory with your preferred text editor or IDE. In order to deploy the compiled Java Spring application artifact to the remote Azure Function app using the Azure Maven Plugin, we need to define four (4) environment variables in our deployment environment that are used in the pom.xml file, namely:

Environment Variable Name	Description	Example Value
POB_AZURE_FUNCTION_APP_DATA_PIPELINES_SCHEDULER_SUBSCRIPTION_ID	The ID of the Azure subscription in which this Azure Function app is provisioned.	12345678-1234-abcd-wxyz-987654321a
POB_AZURE_FUNCTION_APP_DATA_PIPELINES_SCHEDULER_RESOURCE_GROUP_NAME	The name of the resource group in which this Azure Function app is provisioned.	my-resource-group
POB_AZURE_FUNCTION_APP_DATA_PIPELINES_SCHEDULER_APP_NAME	The name of the Azure Function app.	pob-main-data-pipeline
POB_AZURE_FUNCTION_APP_DATA_PIPELINES_SCHEDULER_REGION	The region in which this Azure Function app is provisioned. For a list of current region names, please refer to https://azuretracks.com/2021/04/current-azure-region-names-reference/.	uksouth

7) Once these environment variables are set in our deployment environment, we can upload and deploy the packaged Java Spring application artifact to the Azure Function app by executing the following commands via our command line:

# Login to the relevant Azure account
$ az login

# Explicitly set the name of the Azure subscription to use
$ az account set --subscription my-subscription

# Navigate to the relevant project folder
$ $POB_BASE/pob-apps/pob-apps-azure/pob-azure-function-app-data-pipelines-scheduler

# Deploy the packaged artifact to the remote Azure Function app
$ mvn azure-functions:deploy

The main POB data pipeline will now be periodically triggered based on the CRON schedule defined in the Spring Cloud Azure Function handler Java class.