Upgrade from 5.8

General recommendations

Be sure to read the Everyware Cloud release notes and the planning the upgrade section.

It's recommended to upgrade to the latest version of the 5.8.x release before moving to 5.9.

Preparing to upgrade

Before upgrading to a patch release be sure to follow these steps:

  • backup the databases (relational and Elasticsearch)
  • backup your existing configuration exporting all project configurations

This is really important as in case of issues with the migration it would be possible to revert back the changes.

🚧

Elasticsearch Upgrade

  • This version of Everyware Cloud requires an upgrade of the Elasticsearch version from 6.x to 7.x. Supported Elasticsearch versions are 7.8+, but lower than 8.0.

  • This version of Everyware Cloud implements a tool that migrates data from the index structure used until EC version 5.8.x to the new index structure used by EC version 5.9.x.

  • The migration tool executes a reindexing of the documents stored in the Elasticsearch engine. Depending on the amount of data stored in your Elasticsearch instance, the execution of the data migration step may take a variable time length.

  • The migration tool works only on indices created/used by Everyware Cloud. If your Elasticsearch instance contains other indices they will not be migrated.

  • The migration tool is available as a Docker Image. The image can be downloaded from the same repo of the Everyware Cloud component images.

Upgrade steps

The process to upgrade Everyware Cloud is the following:

  • Login to the Helm chart registry
HELM_EXPERIMENTAL_OCI=1
helm registry login registry.everyware-cloud.com
  • Download the new version of the Helm charts from the registry
cd charts

HELM_EXPERIMENTAL_OCI=1
helm chart pull registry.everyware-cloud.com/eurotech/charts/<chart-name>:<everyware-cloud-version>
helm chart export registry.everyware-cloud.com/eurotech/charts/<chart-name>:<everyware-cloud-version>
  • Scale down all Everyware Cloud components
# For Kubernetes
COMPONENTS=("ec-events-broker" "ec-api" "ec-broker" "ec-console" "ec-vpn")
for COMPONENT in "${COMPONENTS[@]}"; do
  kubectl scale deployment/${COMPONENT} --replicas=0
done

# For OpenShift
COMPONENTS=("ec-events-broker" "ec-api" "ec-broker" "ec-console" "ec-vpn")
for COMPONENT in "${COMPONENTS[@]}"; do
  oc scale deploymentconfig/${COMPONENT} --replicas=0
done
  • Upgrade the Elasticsearch cluster to version 7.8+ (<8.0)

  • Check all the default parameters of the data migration tool by looking at the readme. Parameters and defaults are contained in the values.yaml file. Check the Installation with Helm Charts section to see how to access the charts.

helm show readme charts/ec-es-migrator
  • Launch the Elasticsearch data migration tool using the provided Helm chart and follow the execution on the ec-es-migrator pod logs.

🚧

Data migration execution

During execution of data migration don't run any other operation, like snapshots, on the Elasticsearch cluster.

helm install -n everyware-cloud -f ec-es-migrator.yaml ec-es-migrator charts/ec-es-migrator
  • Follow the migrator tool execution on the pod logs using the following commands. In case of errors check the troubleshooting section at the end of this page.
# For Kubernetes
kubectl get pods -n everyware-cloud | grep ec-es-migrator
kubectl log -f -n everyware-cloud <ec-es-migrator-pod> 

# For OpenShift
oc get pods -n everyware-cloud | grep ec-es-migrator
oc log -f -n everyware-cloud <ec-es-migrator-pod>
  • Once the tool successfully completes it can be removed with
helm delete -n everyware-cloud ec-es-migrator
  • Set the database schema update property to true in the ec-configs values override file and run the Helm upgrade procedure
helm upgrade -n everyware-cloud -f ec-configs.yaml ec-configs charts/ec-configs
  • Update the standard EC components values override file changing the version of the container images and the exposed port configuration. With version 5.9 ports can be enabled/disabled singularly. Check the chart readme for the default configuration. The following example shows the difference between 5.8 and 5.9 service configuration.
# Service configuration in 5.8.x

service:
  type: LoadBalancer
  loadBalancer:
    ports:
      mqtts: 8883
      wss: 8443
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "900"

# Service configuration in 5.9.x

service:
  type: LoadBalancer
  ports:
    mqtt:
        enabled: false
    mqtts:
      enabled: true
      port: 8883
    ws:
      enabled: false
    wss:
      enabled: true
      port: 8443
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "900"
  • Upgrade the standard EC components using Helm, changing the version of the container images in the values override file. Before launching the upgrade command it's necessary to remove the deployment as described in the following snipped (use the oc command instead of kubectl for OpenShift installations)
# Deploy the Events Broker
kubectl delete -n everyware-cloud deployment/ec-events-broker
helm upgrade -n everyware-cloud -f ec-events-broker.yaml ec-events-broker charts/ec-events-broker

# Deploy the Message Broker
kubectl delete -n everyware-cloud deployment/ec-broker
helm upgrade -n everyware-cloud -f ec-broker.yaml ec-broker charts/ec-broker

# Deploy the RESTful API
kubectl delete -n everyware-cloud deployment/ec-api
helm upgrade -n everyware-cloud -f ec-api.yaml ec-api charts/ec-api

# Deploy the Web Console
kubectl delete -n everyware-cloud deployment/ec-console
helm upgrade -n everyware-cloud -f ec-console.yaml ec-console charts/ec-console
  • Install the new job-engine component. First create a new override file like the other components, then install the new Helm chart.
helm install -n everyware-cloud -f ec-job-engine.yaml ec-job-engine charts/ec-job-engine
  • Repeat the upgrade operation for the optional components ec-vpn and ec-monitoring if installed.
# Upgrade the VPN Service if installed
kubectl delete -n everyware-cloud deployment/ec-vpn
helm upgrade -n everyware-cloud -f ec-vpn.yaml ec-vpn charts/ec-vpn

# Upgrade the Monitoring Service if installed
kubectl delete -n everyware-cloud deployment/ec-monitoring
helm upgrade -n everyware-cloud -f ec-monitoring.yaml ec-monitoring charts/ec-monitoring
  • Set the database schema update property to false in the ec-configs values override file and run the Helm upgrade procedure
helm upgrade -n everyware-cloud -f ec-configs.yaml ec-configs charts/ec-configs

Rollback

In case of problems with the upgraded version it's possible to revert to a previous version.

The steps for the rollback are the following:

  • stop all the services scaling down the deployments to 0
  • restore the databases to the backed-up version
  • execute the Helm charts for the previous installed version with the old values override files
  • start the services scaling up the deployments to 1 and then, once the service is available again, to the original number of pods. Follow the order described in the planning section.

Elasticsearch Migration troubleshooting

If any error happened during the execution of the migration tool, it's possible to re-run the tool. This second run will skip the already migrated indices and it will try to migrate the rest.

Depending on the error you observe you may:

  • Check the list of launch parameters as explained above. Changing default values may allow to adapt the behavior of the tool to your operational conditions
  • Contact the support team