Application Configuration

Application configuration including the list of registered framework parsers, publishers and associated properties.

Last Updated: 26 May 2022 • Page Author: Jillur Quddus

Overview

The POB Spring application configuration includes the list of registered framework parsers and publishers executed by the POB data pipeline, along with associated connectivity properties.

Location

The configuration for the POB Spring application context may be found in the pob-configuration Maven module at src/main/resources/application.yml. The complete configuration file is as follows:

name: pob
frameworks:
    - id: dos
      name: 'Digital Outcomes and Specialists Framework'
      enabled: true
      baseUrl: https://www.digitalmarketplace.service.gov.uk
      parserClass: ai.hyperlearning.pob.data.parsers.dos.DosParser 
      filter: false
      keywords: ''
      properties:
        opportunitiesUrl: https://www.digitalmarketplace.service.gov.uk/digital-outcomes-and-specialists/opportunities?q=&statusOpenClosed=open
    - id: cf
      name: 'Contracts Finder'
      enabled: true
      baseUrl: https://www.contractsfinder.service.gov.uk
      parserClass: ai.hyperlearning.pob.data.parsers.cf.ContractsFinderParser
      filter: true
      keywords: 'data software java python artificial intelligence machine learning training architecture engineering digital transformation computing language'
      properties:
        opportunitiesUrl: https://www.contractsfinder.service.gov.uk/Search/Results
publishers:
    - id: csv
      enabled: true
      publisherClass: ai.hyperlearning.pob.data.publishers.csv.CsvPublisher
      properties:
        path: ${java.io.tmpdir}/pob.csv
    - id: slack
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.slack.SlackPublisher
      properties:
        channel: ${slack-channel}
        webhook: ${slack-webhook}
    - id: microsoft-teams
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.microsoft.MicrosoftTeamsPublisher
      properties:
        webhook: ${microsoft-teams-webhook}
    - id: google-chat
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.google.GoogleChatPublisher
      properties:
        webhook: ${google-chat-webhook}
    - id: elasticsearch
      enabled: false
      publisherClass: ai.hyperlearning.pob.data.publishers.elastic.ElasticsearchPublisher
      properties:
        url: ${elasticsearch-url}
        username: ${elasticsearch-username}
        password: ${elasticsearch-password}
        index: pob
        ssl: true
pipelines:
    main:
        enabled: true
        scheduler:
            enabled: false
            cron: "0 0/20 * * * *"
        bulkPublicationDelay:
            enabled: true
            duration: 10
storage:
    rdbms:
        driverClassName: ${rdbms-driverClassName}
        jdbcUrl: ${rdbms-jdbcUrl}
        username: ${rdbms-username}
        password: ${rdbms-password}

Namespaces

frameworks.*

This namespace contains the exhaustive list of procurement frameworks and associated Java-based parser classes that will be executed by the POB data pipeline. If you are extending POB and developing a custom procurement framework parser, then you must add your custom procurement framework to the list in this namespace.

Property	Description	Example Value
id	An unique string identifier for this procurement framework.	dos
name	A descriptive name for this procurement framework.	Digital Outcomes and Specialists Framework
enabled	Whether to enable the parser for this procurement framework during execution of the POB data pipeline.	true
baseUrl	The base URL to this procurement framework.	https://www.digitalmarketplace.service.gov.uk
parserClass	The fully qualified class name of the Java-based parser class that will be instantiated and executed during execution of the POB data pipeline.	ai.hyperlearning.pob.data.parsers.dos.DosParser
filter	Whether to apply a keyword filter to opportunities that have been parsed and extracted from this procurement framework.	true
keywords	If the keyword filter is enabled, then a space-delimited list of keywords that will be used to filter opportunities.	java python artificial intelligence digital
properties	An arbitrary list of key-value properties, such as the URL that will be used by the parser to extract procurement opportunities, that will be automatically injected and made available to the Java-based parser class. Note that sensitive properties such as usernames and passwords should be set as externalised variables (for example in HashiCorp Vault or AWS Secrets Manager which are loaded by the POB bootstrap context) and NOT stored as plaintext in application.yml.	opportunitiesUrl: https://www.digitalmarketplace.service.gov.uk/digital-outcomes-and-specialists/opportunities?q=&statusOpenClosed=open

publishers.*

This namespace contains the exhaustive list of Java-based publisher classes and associated properties that will be executed by the POB data pipeline. If you are extending POB and developing a custom publisher, then you must add your custom publisher to the list in this namespace.

Property	Description	Example Value
id	An unique string identifier for this publisher.	slack
enabled	Whether to enable this publisher during execution of the POB data pipeline.	true
publisherClass	The fully qualified class name of the Java-based publisher class that will be instantiated and executed during execution of the POB data pipeline.	ai.hyperlearning.pob.data.publishers.slack.SlackPublisher
properties	An arbitrary list of key-value properties, such as webhook URL, that will be automatically injected and made available to the Java-based publisher class. Note that sensitive properties such as webhook URLs, usernames and passwords should be set as externalised variables (for example in HashiCorp Vault or AWS Secrets Manager which are loaded by the POB bootstrap context) and NOT stored as plaintext in application.yml.	webhook: https://hooks.slack.com/services/TSW12345

pipelines.main

This namespace contains the configuration for the main end-to-end POB data pipeline which is responsible for executing all procurement framework parsers and publishers registered in application.yml.

Property	Description	Example Value
enabled	Whether the main POB data pipeline is enabled.	true
scheduler.enabled	Whether to enable a scheduler that will execute the main POB data pipeline based on a defined Spring CRON schedule. Note that this setting is only applicable when running POB in a self-managed environment. If POB is deployed as an AWS lambda application, then you should configure an AWS EventBridge trigger. If POB is deployed as a Microsoft Azure function app, then the CRON schedule defined in the Azure Handler class will be used instead.	true
scheduler.cron	If POB is to be deployed to a self-managed environment and the scheduler is enabled, then this setting defines the Spring-based CRON schedule.	0 0/20 * * * *
bulkPublicationDelay.enabled	Whether to enable a delay between publishing opportunities in bulk.	true
bulkPublicationDelay.duration	The duration (in seconds) of the delay between publishing opportunities in bulk.	10

storage.rdbms

This namespace contains the connectivity properties to a relational database management system (RDBMS) that POB will use to act as the master source for all opportunities that have been extracted from the registered procurement frameworks.

Property	Description	Example Value
driverClassName	The JDBC driver to use when connecting to the RDBMS.	com.mysql.cj.jdbc.Driver
jdbcUrl	The JDBC connection string that POB will use to connect to the RDBMS. Note that the JDBC URL should be set as an externalised variable (for example in HashiCorp Vault or AWS Secrets Manager which is loaded by the POB bootstrap context) and NOT stored as plaintext in application.yml.	jdbc:mysql://example.com:3306/pob
username	The username to authenticate the connection to the RDBMS. Note that the username should be set as an externalised variable (for example in HashiCorp Vault or AWS Secrets Manager which is loaded by the POB bootstrap context) and NOT stored as plaintext in application.yml.	pob
password	The password to authenticate the connection to the RDBMS. Note that the password should be set as an externalised variable (for example in HashiCorp Vault or AWS Secrets Manager which is loaded by the POB bootstrap context) and NOT stored as plaintext in application.yml.	password123

POB utilises the Java Spring Framework which provides extensive support for working with SQL databases, including but not limited to HSQLDB, MySQL, MariaDB, PostgreSQL, SQLite and Microsoft SQL Server.