Elasticsearch

Provision an Elastic Cloud managed Elasticsearch cluster and index procurement opportunities as they are collated by POB.

Last Updated: 27 May 2022 • Page Author: Jillur Quddus

Overview

Elastic Cloud is a public cloud managed service available on major cloud computing platforms (including Amazon Web Services, Microsoft Azure and Google Cloud Platform) providing fully-managed SaaS offerings from the Elastic stack including Elasticsearch and Kibana. This page provides instructions on how to provision an Elastic Cloud managed Elasticsearch cluster and integrate it with POB so that procurement opportunities are indexed in Elasticsearch as they are collated by POB.

For further information regarding Elastic Cloud and its features, please visit https://www.elastic.co/subscriptions/cloud.

Setup

Elastic Deployment

To provision a fully-managed Elastic Cloud Elasticsearch cluster, navigate to https://cloud.elastic.co and sign up for an Elastic Cloud account. Once signed up and logged in, select "Create Deployment" to create a fully-managed Elastic deployment consisting of a managed Elasticsearch cluster along with Kibana (as a minimum). You will be asked for the following information:

1. Name - enter a custom name for your deployment, for example mydeployment.

2. Cloud Provider - select a cloud provider (for example Amazon Web Services) which will be used to host the Elastic deployment. Note that administration of the Elastic deployment is performed entirely via Elastic Cloud and there is no need to create an account in the selected cloud computing platform.

3. Region - select a region, for example London (eu-west-2).

4. Hardware Profile - select a hardware profile. It is recommended to select "General Purpose" and an Elasticsearch hardware profile of 20GB storage with 2GB RAM and up to 4.3 vCPU (in the case of available Amazon Web Services hardware profiles after selecting "Advanced Settings") should suffice for the purposes of POB. In regards to Kibana, unless you wish to use Kibana as an additional data visualisation tool, then a hardware profile of 1GB with up to 8.5 vCPU (in the case of available AWS hardware profiles) will suffice.

5. Availability Zones - select the number of availability zones based on your deployment and fault tolerance requirements. For development and testing purposes, one (1) availability zone will suffice for both Elasticsearch and Kibana.

6. Version - POB currently only supports version 7.x of Elasticsearch.

Once configured, select "Create Deployment" to create the deployment (it may take Elastic Cloud between 5 - 10 minutes to fully deploy).

Elasticsearch Endpoint

Once created and deployed, navigate to https://cloud.elastic.co/deployments, select the newly provisioned deployment and make a note of the Elasticsearch endpoint (of the format https://mydeployment.es.eu-west-2.aws.cloud.es.io:9243). This will be required when we integrate the managed Elasticsearch cluster with POB below.

Authorisation

In order to securely integrate with POB, we must create a new user with a custom role profile limited to only those privileges required by POB. To do this, first we shall create and configure a custom role with selected privileges via Elastic Cloud. Navigate to https://cloud.elastic.co/deployments and select the newly created deployment. In the main menu on the left of the screen, select "Security" and then "Make security changes in Kibana". In Kibana, select "Roles" from the main menu on the left of the screen, then "Create Role" and enter the following information, as illustrated in the screenshot below:

1. Role Name - enter a custom role name, for example pob-read-write.

2. Index Privileges - enter pob in the "Indices" text box (note the wildcard character). Then in the "Privileges" dropdown, select the following privileges: auto_configure, create, create_doc, create_index, delete, delete_index, index, maintenance, read, view_index_metadata and write. Once configured, select "Create Role" to create the new role.

Authentication

Finally we must create the new user and attach to it the custom role configured above. To do this, in Kibana select "Users" from the main menu on the left of the screen and then select "Create User". Provide the required information including username (for example myuser) and password. Finally map the custom role created above to this new user in the "Privileges" section. Once configured, select "Create User" to create the new user whilst making a note of the provided username and password for integration with POB below.

Integration

The final step is to update POB's Application Configuration with the URL of your newly created Elasticsearch cluster along with the username and password created above. To do this, create the relevant secrets in your chosen secrets management service (for example HashiCorp Vault, AWS Secrets Manager or Azure Key Vault). Then open src/main/resources/application.yml in the pob-configuration Maven module in your preferred text editor or IDE, enable the native Elasticsearch publisher and finally enter placeholder variables corresponding to the name of your secrets, as follows:

publishers:
    - id: elasticsearch
      enabled: true
      publisherClass: ai.hyperlearning.pob.data.publishers.elastic.ElasticsearchPublisher
      properties:
        url: ${elasticsearch-url}
        username: ${elasticsearch-username}
        password: ${elasticsearch-password}
        index: pob
        ssl: true

Do NOT enter the Elasticsearch URL, username nor password directly in application.yml as plaintext as this could expose your Elasticsearch cluster to unwanted or malicious communications.

Deployment

You are now ready to deploy POB to your chosen target environment (for example a self-managed environment, or to the Amazon Web Services (AWS) or Microsoft Azure cloud computing platforms respectively). Once successfully deployed, your Elasticsearch cluster will automatically receive and index procurement opportunities from POB thus offering extensive search and analytical capabilities via the Elasticsearch API, and data visualisation, search and analysis via Kibana.