Camunda back up and restore

note

The Camunda 8.8 release introduces breaking changes for Operate and Tasklist.

note

If the Camunda application(s) cannot access Elasticsearch with cluster-level privileges, it is possible to run the backup of Operate and Tasklist indices (steps 2 and 4 from the backup procedure below) as a standalone application separate from the main application (see standalone backup application).

Use the backup feature to back up and restore your Camunda 8 Self-Managed components and cluster.

About this guide

This guide covers how to back up and restore your Camunda 8 Self-Managed components and cluster. You should automate the procedures in this guide, choosing tools that fulfil the requirements of your organization.

• Regularly back up the state of your Zeebe, Web Applications (Operate, Tasklist), and Optimize components without any downtime. You can also back up and restore Web Modeler data.

• Restore a cluster from a backup if any failures occur that cause data loss.

Prerequisites

Set up a snapshot repository in the secondary datastore and configure component backup storage.

Create a backup

Create a backup. This involves backing up the WebApps and the Zeebe Cluster.

Restore a backup

Restore a backup. This involves restoring Elasticsearch/OpenSearch and the Zeebe Cluster.

note

• The examples in this guide are based on using the following tools: curl, jq, and kubectl.

Why you should use backup and restore

The Camunda 8 components Zeebe, Web Applications (Operate, Tasklist), and Optimize store data in various formats and across multiple indices in Elasticsearch / OpenSearch. Because of this distributed and interdependent architecture, creating a consistent and reliable backup requires coordination between the components.

For example, using Elasticsearch / OpenSearch’s native snapshot capabilities directly does not produce a coherent backup. This is because Operate, Tasklist, and Optimize each manage their data across multiple indices, which cannot be reliably captured together without involvement from the components that understand their structure. For this reason, backups must be initiated through each component individually, using their built-in backup functionality.

The same principle applies to Zeebe. Backups must be scheduled through Zeebe to ensure a consistent snapshot of all partition data. Simply taking a disk-level snapshot of each Zeebe broker is not enough, as the brokers operate independently and data may not be aligned across them at the time of the snapshot. Since disk-level backups are not synchronized, this can lead to inconsistencies and invalid recovery points.

A complete backup of a Camunda 8 cluster includes:

• Backups of Web Applications (Operate, Tasklist), and Optimize (triggered through their APIs).
• Backup of indices from Elasticsearch/OpenSearch containing exported Zeebe records.
• A Zeebe broker partition backup (triggered through its API).

Because the data across these systems is interdependent, all components must be backed up as part of the same backup window. Backups taken independently at different times may not align and could result in an unreliable restore point.

warning

To ensure a consistent backup, you must follow the process outlined in this guide. Deviating from it can result in undetected data loss, as there is no reliable method to verify cross-component data integrity after backup.

Following the documented procedure results in a hot backup, meaning that:

• Zeebe continues to process and export data.
• Web Applications (Operate, Tasklist), and Optimize remain fully operational during the backup process.

This ensures high availability while preserving the integrity of the data snapshot.

Prerequisites

The following prerequisites are required before you can create and restore backups:

Prerequisite	Description
Set up a snapshot repository in the secondary datastore.	Depending on the choice of secondary datastore, you must configure the following on the datastore itself: Elasticsearch snapshot repository OpenSearch snapshot repository Note: For Elasticsearch configuration with the Camunda Helm chart on AWS EKS using IRSA, see configuration example.
Configure component backup storage.	Configure the backup storage for the components. This is also important for restoring a backup. Operate Optimize Elasticsearch / Optimize OpenSearch Tasklist Zeebe

note

You should keep the backup storage of the components configured at all times to ease the backup and restore process and avoid unnecessary restarts.

tip

You can use the same backup storage location for both Elasticsearch / OpenSearch snapshots and Zeebe partition backups, as long as different paths are configured:

• Set the basePath for Zeebe.
• Set the base_path for Elasticsearch / OpenSearch.

To learn more about how to configure these settings, refer to the prerequisites linked documentation above.

Considerations

The backup of each component and the backup of a Camunda 8 cluster is identified by an ID. This means a backup x of Camunda 8 consists of backup x of Zeebe, backup x of Optimize, backup x of Operate, and backup x of Tasklist. The backup ID must be an integer and greater than the previous backups.

note

We recommend using the unix timestamp as the backup ID.

The steps outlined on this page are generally applicable for any kind of deployment but might differ slightly depending on your setup.

Optimize is not part of the Web Applications backup API and needs to be executed separately to successfully make a backup. Depending on your deployment configuration, you may not have Optimize deployed. It is safe to ignore the backup instructions for Optimize if it is not deployed.

breaking change

As of Camunda 8.8, the indexPrefix of Operate and Takslist must match. By default it is set to "". If overriden, it must set consistently across Operate and Tasklist.

breaking change

As of Camunda 8.8, configuring Operate and Tasklist with different repository names will potentially create multiple backups in different repositories.

breaking changes

As of Camunda 8.8, the /actuator endpoints for backups have been moved to /actuator/backupHistory and /actuator/backupRuntime. The previous /actuator/backups endpoint is still active only if the applications are deployed standalone (each application is running in its own process).

Management API

The management API is an extension of the Spring Boot Actuator, typically used for monitoring and other operational purposes. This is not a public API and not exposed. You will need direct access to your Camunda cluster to be able to interact with these management APIs. This is why you'll often see the reference to localhost.

Direct access will depend on your deployment environment. For example, direct Kubernetes cluster access with port-forwarding or exec to execute commands directly on Kubernetes pods. In a manual deployment you will need to be able to reach the machines that host Camunda. Typically, the management port is on port 9600 but might differ on your setup and on the components. You can find the default for each component in their configuration page.

Component	Port
Optimize	8092
Orchestration Cluster	9600

Examples for Kubernetes approaches

Port Forwarding

Port-forwarding allows you to temporarily bind a remote Kubernetes cluster port of a service or pod directly to your local machine, allowing you to interact with it via localhost:PORT.

Since the services are bound to your local machine, you cannot reuse the same port for all port-forwards unless you start and stop each one based on usage. To avoid this limitation, the examples use different local ports for each service, allowing them to run simultaneously without conflict.

export CAMUNDA_RELEASE_NAME="camunda"
# kubectl port-forward services/$SERVICE_NAME $LOCAL_PORT:$REMOTE_PORT
kubectl port-forward services/$CAMUNDA_RELEASE_NAME-core 9600:9600 & \
kubectl port-forward services/$CAMUNDA_RELEASE_NAME-optimize 9620:8092 & \
kubectl port-forward services/$CAMUNDA_RELEASE_NAME-elasticsearch 9200:9200 &

Using the bash instruction & at the end of each line would run the command in a subshell allowing the use of a single terminal.

Exec

An alternative to port-forwarding is to run commands directly on Kubernetes pods. In this example we're going to spawn a temporary pod to execute a curl request. Alternatives are to use existing pods within the namespace. Camunda's pod includes different base images, each with a different feature set.

# following will create a temporary alias within your terminal to overwrite the normal curl
export CAMUNDA_NAMESPACE="camunda"
export CAMUNDA_RELEASE_NAME="camunda"
# temporary overwrite of curl, can be removed with `unalias curl` again
alias curl="kubectl run curl --rm -i -n $CAMUNDA_NAMESPACE --restart=Never --image=alpine/curl -- -sS"

curl $CAMUNDA_RELEASE_NAME-core:9600/actuator/health
curl $CAMUNDA_RELEASE_NAME-optimize:8092/actuator/health
curl $CAMUNDA_RELEASE_NAME-elasticsearch:9200/_cluster/health

This allows you to directly execute commands within the namespace and communicate with available services.

Cronjob

The examples in this guide showcase the backup process in a manual fashion to help you fully understand the process. You might want to use Kubernetes Cronjobs to automate the backup process for your own use case based on your own environment on a regular schedule.

Kubernetes Cronjobs will spawn a Job on a regular basis. The job will run a defined image within a given namespace, allowing you to run commands and interact with the environment.

note

You can see further examples from Camunda consultants in the Backup and Restore Workshop. You can use these examples to achieve similar automation.

ContextPath

If you are defining the contextPath in the Camunda Helm chart or the management.server.servlet.context-path in a standalone setup, your API requests must prepend the value specific to the contextPath for the individual component. If the management.server.port is defined this also applies to management.endpoints.web.base-path. You can learn more about this behavior in the Spring Boot documentation.

Optimize Helm chart Exception

Setting the contextPath in the Helm chart for Optimize will not overwrite the contextPath of the management API, it will remain as /.

Example

If you are defining the contextPath for Operate in the Camunda Helm chart:

core:
   contextPath: /core

A call to the management API of Operate would look like the following example:

ORCHESTRATION_CLUSTER_MANAGEMENT_API=http://localhost:9600

curl $ORCHESTRATION_CLUSTER_MANAGEMENT_API/core/health

Without the contextPath it would just be:

ORCHESTRATION_CLUSTER_MANAGEMENT_API=http://localhost:9600

curl $ORCHESTRATION_CLUSTER_MANAGEMENT_API/health