Elastic Stack on Openshift

This section outlines how to setup and configure an Openshift project to get the Elastic Stack to a deployable state. This document assumes a working knowledge of Kubernetes/Openshift container orchestration concepts (i.e. buildconfigs, deployconfigs, imagestreams, secrets, configmaps, routes, etc)

Our builds and deployments can be orchestrated with Jenkins. However at this time, all manifests are to be manually invoked.

Environment Setup - ConfigMaps and Secrets

There are some requirements in the target Openshift namespace/project which are outside of the CI/CD pipeline process. This application requires that a few Secrets as well as Config Maps be already present in the environment before it is able to function as intended. Otherwise the deployments fail due to missing data.

In order to prepare an environment, you will need to ensure that all of the following configmaps and secrets are populated. This is achieved by executing the following commands as a project administrator of the targeted environment. Note that this must be repeated on each of the target deployment namespace/projects (i.e. dev, test and prod) as that they are independent of each other. Deployment will fail otherwise.

Config Maps

At this time, all config maps are defined in the deploymentconfig manifest templates. As the Elastic Stack components are mainly configured through yaml files, the majority of the config maps will end up being mounted onto the containers at specific directory locations. Refer to the Official Elastic Stack documentation for more details on how to manage your Elastic Stack components.

Secrets

Elastic Stack secrets involves the use of a combination of TLS Certificates and user/password secrets. In order to generate Elastic Stack compatible certificates, they must be generated by an elasticsearch instance. To achieve this, we temporarily spin up an elasticsearch container on OCP in order to generate and extract the certificates. This is then pushed up as secrets.

Certificate Generation

Ensure your OpenShift CLI tool is logged in. Make sure you change your namespace to the correct project with oc project <projectname>.

Start Temporary Elasticsearch Pod

export ELK_VERSION=7.5.1
export NAMESPACE=ixhmbm-dev

oc run -n $NAMESPACE cert --image=elastic/elasticsearch:$ELK_VERSION --command -it --rm --restart=Never -- bash

Generate Certificates

Open a new terminal and ensure your OpenShift CLI tool is logged in. Make sure you change your namespace to the correct project with oc project <projectname>. You will create certs for desired elasticsearch host ${APP_NAME}-${INSTANCE}.

export APP_NAME=elasticsearch
export INSTANCE=master
export NAMESPACE=ixhmbm-dev

oc rsh -n $NAMESPACE cert elasticsearch-certutil ca --out /tmp/$APP_NAME-ca.p12 --pass ''
oc rsh -n $NAMESPACE cert elasticsearch-certutil cert --name $APP_NAME --dns $APP_NAME-$INSTANCE --ca /tmp/$APP_NAME-ca.p12 --pass '' --ca-pass '' --out /tmp/$APP_NAME-certificates.p12
oc cp -n $NAMESPACE cert:/tmp/$APP_NAME-ca.p12 ./
oc cp -n $NAMESPACE cert:/tmp/$APP_NAME-certificates.p12 ./
openssl pkcs12 -nodes -passin pass:'' -in $APP_NAME-certificates.p12 -out $APP_NAME-certificate.pem
openssl pkcs12 -clcerts -nokeys -passin pass:'' -in $APP_NAME-ca.p12 -out $APP_NAME-ca.pem

Create Secrets

Once the four certificate files are created, we can add them to OpenShift secrets.

oc create -n $NAMESPACE secret generic $APP_NAME-$INSTANCE-certificates --from-file=$APP_NAME-certificates.p12
oc create -n $NAMESPACE secret generic $APP_NAME-$INSTANCE-certificate-pem --from-file=$APP_NAME-certificate.pem
oc create -n $NAMESPACE secret generic $APP_NAME-$INSTANCE-ca-pem --from-file=$APP_NAME-ca.pem

oc process -n $NAMESPACE -f elasticstack.secret.yaml -p INSTANCE=$INSTANCE | oc create -f -

Stop & Remove Temporary Elasticsearch Pod

Once the secrets are up, you need to dispose of your cert pod. This can be done by exiting out of the remote terminal shell, or by force deleting the specific cert pod.

oc delete -n $NAMESPACE pod/cert --force

Build Config & Deployment

The Elastic stack is made up of ElasticSearch, Kibana and Logstash. We are currently leveraging basic Openshift Routes to expose and foward incoming traffic to the right services. Most of the images we are using are the official published images from Elastic.co. However, as we apply certain plugin plugins to support specific functionality, some components will have custom build configurations which extend on top of those images.

Templates

We leverage Openshift Templates in order to ensure all environment variables, settings and contexts are pushed to Openshift correctly. Files ending with .bc.yaml specify the build configurations, while files ending with .dc.yaml specify the components required for deployment.

Build Configurations

Build configurations will emit and handle the chained builds or standard builds as necessary. Note that most parameters will have reasonable defaults specified if you do not explicitly configure them. Refer to the template parameter definitions for more details. They take in the following parameters:

The template can be manually invoked and deployed via Openshift CLI. For example:

export INSTANCE=master
export NAMESPACE=ixhmbm-dev

oc process -n $NAMESPACE -f logstash.bc.yaml -p INSTANCE=$INSTANCE -o yaml | oc -n $NAMESPACE apply -f -

Note that these build configurations do not have any triggers defined. They will be invoked by the Jenkins pipeline, started manually in the console, or by an equivalent oc command for example:

oc start-build -n $NAMESPACE <buildname> --follow

The build config should write the resultant image to <imagename>:$INSTANCE, where the image name is ${APP_NAME}-custom, and the tag is the specified INSTANCE. Depending on your build tooling, you may need to do the equivalent oc command for tag management:

oc tag -n $NAMESPACE <imagename>:$INSTANCE <imagename>:latest

Note: Remember to swap out the bracketed values with the appropriate values!

Deployment Configurations

Deployment configurations will emit and handle the deployment lifecycles of running containers based off of the previously built images. They generally contain a deploymentconfig, a service, and a route, but may also contain configmaps and statefulsets depending on the application. Most templates will take in the following parameters (consult the templates directly for exact details):

A Jenkins pipeline will be handle deployment invocation automatically. However should you need to run it manually, you can do so with the following for example:

export INSTANCE=master
export NAMESPACE=ixhmbm-dev

oc process -n $NAMESPACE -f elasticsearch.dc.yaml -p INSTANCE=$INSTANCE -o yaml | oc -n $NAMESPACE apply -f -

oc process -n $NAMESPACE -f kibana.dc.yaml -p INSTANCE=$INSTANCE -p NAMESPACE=$NAMESPACE -o yaml | oc -n $NAMESPACE apply -f -

oc process -n $NAMESPACE -f logstash.dc.yaml -p INSTANCE=$INSTANCE -p NAMESPACE=$NAMESPACE -o yaml | oc -n $NAMESPACE apply -f -

Due to the triggers that are set in the deploymentconfig, the deployment will begin automatically. However, you can deploy manually by use the following command for example:

oc rollout -n $NAMESPACE latest dc/<buildname>-$INSTANCE

Note: Remember to swap out the bracketed values with the appropriate values!

Instance Cleanup

In order to clear all resources for a specific instance, run the following three commands to delete all relevant resources from the Openshift project:

export INSTANCE=master
export NAMESPACE=ixhmbm-dev

oc delete all,cm,pvc -n $NAMESPACE --selector app=elasticsearch-$INSTANCE
oc delete all,cm -n $NAMESPACE --selector app=kibana-$INSTANCE
oc delete all,cm -n $NAMESPACE --selector app=logstash-$INSTANCE

If necessary, you can remove old secrets through the Openshift Web Console or its equivalent OC CLI command.

export APP_NAME=elasticsearch
export INSTANCE=master
export NAMESPACE=ixhmbm-dev

oc delete -n $NAMESPACE secret $APP_NAME-$INSTANCE-certificates
oc delete -n $NAMESPACE secret $APP_NAME-$INSTANCE-certificate-pem
oc delete -n $NAMESPACE secret $APP_NAME-$INSTANCE-ca-pem
oc delete -n $NAMESPACE secret $APP_NAME-$INSTANCE-credentials