Skip to main content
Version: 8.8 (unreleased)

Components update 8.7 to 8.8

The following sections explain which adjustments must be made to migrate from Camunda 8.7.x to 8.8.x.

Camunda Orchestration Cluster Identity Migration​

Camunda 8.8 introduces Orchestration Cluster Identity as a separate component within the Orchestration Cluster, you can learn more about it here.

To ease the effort for migrating data from Management Identity to the new Orchestration Cluster Identity, a Identity Migration Application is provided.

note

The Identity migration is automatically setup and executed in the official Camunda Helmchart, when enabled.

Running the Identity Migration Application​

The application can be run locally for manual setups or from a docker container.

Running the migration natively​

Using the native distribution of Camunda 8, you can run the Identity Migration Application as follows:

./bin/identity-migration

Running the migration using the camunda/camunda:8.8.0 docker image.​

You can run the Identity Migration Application by overriding the default entrypoint of the official Camunda 8 Docker image to /usr/local/camunda/bin/identity-migration, e.g. as such:

docker run --entrypoint /usr/local/camunda/bin/identity-migration camunda/camunda:8.8.0

Configuration of the Identity Migration Application​

The Identity Migration Application supports two scenarios to migrate from, each of those requires different configuration properties and values to be set.

Configuration can generally be set by directly modifying the application.yaml The default location of the config within the Camunda 8 distribution is ./config/application.yaml and /usr/local/camunda/config/application.yaml in the docker image respectively.

Alternatively all properties can also be set via Environment variables.

Common Configuration for the Identity Migration Application​

The Identity Migration Application needs to join the Orchestraction Cluster in order to create Identity Access Management records, the values can be taken from equivalent configuration of the Orchestration Cluster nodes.

Furthermore the Identity Migration Application requires full access to the Management Identity API and thus valid client credentials for a client with read permissions to access it.

Environment variableDescriptionRequired value
CAMUNDA_MIGRATION_IDENTITY_MODEMode of the migration app.In case of the Camunda Identity 8.7 default (Keycloak) setup: KEYCLOAK.
In case of the Camunda Identity 8.7 with OIDC setup: OIDC
CAMUNDA_MIGRATION_IDENTITY_RESOURCEAUTHORIZATIONSENABLEDSignals whether Resource Authorizations were used in the 8.7, defaults to false.Must be true in case Resource Authorizations were used.
CAMUNDA_MIGRATION_IDENTITY_CLUSTER_CLUSTERNAMEName of the Orchestration cluster.Same value as used on the cluster for ZEEBE_BROKER_CLUSTER_CLUSTERNAME
CAMUNDA_MIGRATION_IDENTITY_CLUSTER_CLUSTERSIZESize of the Orchestration cluster.Same value as used on the cluster for ZEEBE_BROKER_CLUSTER_CLUSTERSIZE
CAMUNDA_MIGRATION_IDENTITY_CLUSTER_PARTITIONSCOUNTPartition count of the Orchestration cluster.Same value as used on the cluster for ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT
CAMUNDA_MIGRATION_IDENTITY_CLUSTER_REPLICATIONFACTORReplication factor of the Orchestration cluster.Same value as used on the cluster for ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR
CAMUNDA_MIGRATION_IDENTITY_CLUSTER_INITIALCONTACTPOINTSList of known cluster hosts within the Orchestration cluster.Same value as used on the cluster for ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS
CAMUNDA_MIGRATION_IDENTITY_MANAGEMENTIDENTITY_AUDIENCEThe audience for accessing the Management Identity APIIn case of a default Keycloak setup, it is camunda-identity-resource-server, in case of OIDC, it is user defined.
CAMUNDA_MIGRATION_IDENTITY_MANAGEMENTIDENTITY_BASEURLThe URL under which Management Identity can be accessed by the Identity Migration ApplicationMust be a valid absolute URL to the Management Identity server, e.g. http://management-identity:8080.
CAMUNDA_MIGRATION_IDENTITY_MANAGEMENTIDENTITY_CLIENTIDThe client id for accessing Management Identity.The clientId set requires to have the read permission granted for the Camunda Identity Resource Server API in Management Identity
CAMUNDA_MIGRATION_IDENTITY_MANAGEMENTIDENTITY_CLIENTSECRETThe client secret for accessing Management Identity.Matching secret for configured clientId.
CAMUNDA_MIGRATION_IDENTITY_MANAGEMENTIDENTITY_ISSUERBACKENDURLThe URL under which the Token Issuer can be accessed.Use the same as used by Management Identity for CAMUNDA_IDENTITY_ISSUER_BACKEND_URL.

Manual migration instructions​

Roles & Application Permissions​

Permissions formerly assigned to Management Identity Roles and Management Identity Applications require the following equivalent Orchestration Cluster Identity Authorizations setup.

You need to manually recreate the Roles you need and grant them the equivalent Authorizations for the former Management Identity Permission as follows:

Management Identity PermissionEquivalent Camunda Orchestration Cluster 8.8 Authorizations
ComponentPermissionResource TypeResource IDPermission(s)
OperatereadBATCH*READ
COMPONENTOperateACCESS
MESSAGE*READ
PROCESS_DEFINITION*READ_PROCESS_DEFINITION, READ_PROCESS_INSTANCE
DECISION_DEFINITION*READ_DECISION_DEFINITION, READ_DECISION_INSTANCE
DECISION_REQUIREMENTS_DEFINITION*READ
writeBATCH*CREATE, READ, UPDATE
COMPONENTOperateACCESS
RESOURCE*DELETE_FORM, DELETE_PROCESS, DELETE_DRD, DELETE_RESOURCE
PROCESS_DEFINITION*READ_PROCESS_DEFINITION, READ_PROCESS_INSTANCE, DELETE_PROCESS_INSTANCE, UPDATE_PROCESS_INSTANCE, MODIFY_PROCESS_INSTANCE CANCEL_PROCESS_INSTANCE
DECISION_DEFINITION*READ_DECISION_DEFINITION, READ_DECISION_INSTANCE, CREATE_DECISION_INSTANCE, DELETE_DECISION_INSTANCE
DECISION_REQUIREMENTS_DEFINITION*READ, UPDATE, DELETE
TasklistreadCOMPONENTTasklistACCESS
PROCESS_DEFINITION*READ_PROCESS_DEFINITION, READ_USER_TASK
writeCOMPONENTTasklistACCESS
PROCESS_DEFINITION*READ_PROCESS_DEFINITION, READ_USER_TASK, UPDATE_USER_TASK
IdentityreadAUTHORIZATIONS*READ
COMPONENTIdentityACCESS
GROUP*READ
ROLES*READ
TENANTS*READ
USERS*READ
read:usersCOMPONENTIdentityACCESS
USERS*READ
ROLES*READ
writeAUTHORIZATIONS*CREATE, READ, UPDATE, DELETE
COMPONENTIdentityACCESS
GROUP*CREATE, READ, UPDATE, DELETE
ROLES*CREATE, READ, UPDATE, DELETE
TENANTS*CREATE, READ, UPDATE, DELETE
USERS*READ
User Role membership​

You can recreate Management Identity Role memberships using the Orchestration Cluster Identity Role Management.

Groups​

You can recreate Groups setup in the Management Identity Groups using the Orchestration Cluster Identity Group Management.

Alernatively you can make use of the Bring your own Groups feature and just recreate the Orchestration Cluster Authorization Records granting permissions to those groups. This is what the Identity Migration Application defaults to.

Resource Authorizations​

The former Management Identity Resource Authorizations are replaced with a more fine grained Orchestration Cluster Authorization Management. The following table lists the equivalent authorization records to create, to represent for a former Management Identity Resource Authorizations.

Management Identity Resource AuthorizationsEquivalent Camunda Orchestration Cluster 8.8 Authorizations
Resource TypePermissionResource TypePermission(s)
PROCESSReadPROCESS_DEFINITIONREAD_PROCESS_DEFINITION, READ_PROCESS_INSTANCE
PROCESSDeleteRESOURCEDELETE_PROCESS
PROCESSUpdate process instancePROCESS_DEFINITIONUPDATE_PROCESS_INSTANCE
PROCESSDelete process instancePROCESS_DEFINITIONDELETE_PROCESS_INSTANCE
PROCESSStart process instancePROCESS_DEFINITIONCREATE_PROCESS_INSTANCE
DECISIONReadDECISION_DEFINITIONREAD_DECISION_DEFINITION, READ_DECISION_INSTANCE
DECISIONDeleteRESOURCEDELETE_DRD

Tenants​

You can recreate Management Identity Tenants using the Orchestration Cluster Identity Tenant Management.

Mapping Rules​

You can recreate Management Identity Mapping Rules using the Orchestration Cluster Identity Mapping Rule Management.

Camunda Exporter and harmonized data model​

The following Camunda 8.7 to 8.8 update guide covers changes related to the new harmonized data model, and the newly introduced Camunda Exporter. For additional details about the introduced exporter and related changes, see the 8.8 announcements.

important

With the Camunda Exporter introduction, Camunda is compatible with Elasticsearch 8.16+ and no longer supports older Elasticsearch versions. See supported environments.

Ensure the exporter/import backlog is small​

For data from 8.7 to be imported, the importers must run first and complete before the Camunda Exporter can start.

The importer reads data from existing Zeebe Elasticsearch indices, and when it detects data from newer versions, it marks itself as complete. The Camunda Exporter then starts exporting only when the importer is done.

important

To reduce the potential of building a backlog, the user needs to make sure to drain the backlog queue as much as possible before the update.

Otherwise, the backlog can increase and cause web application data to become out of date. Zeebe will be not able to clean up data, potentially running out of disc space. This will produce downtime.

The backlog can be validated via the following Prometheus metrics:

  • operate_import_time_seconds_sum
  • operate_import_time_seconds_count

For example, the following query shows the import latency. If the result is in a range of seconds (ideally below 10 seconds), it is fine to update.

sum(rate(operate_import_time_seconds_sum{namespace=~"$namespace", partition=~"$partition", pod=~"$pod"}[$__rate_interval]))
/ sum(rate(operate_import_time_seconds_count{namespace=~"$namespace", partition=~"$partition", pod=~"$pod"}[$__rate_interval]))

Data migration​

To enable Tasklist to utilize the new harmonized indices, Camunda 8.8 provides a migration application to enhance existing process models. This migration application runs asynchronously to the other applications until the importers are done, and ensures that all processes created in 8.7 are migrated and enhanced.

The migration application can be found in the camunda/bin/process-migration directory. It can be run locally, against the same Elasticsearch or OpenSearch instance, separate/standalone or together with the rest of the Camunda 8 application.

The migration application expects the following configuration:

camunda:
  migration:
    process:
      # How many processes are migrated at once
      batchSize: 5
      # The timeout/count down after importer has finished before the migration app should stop
      # Default PT1M
      importerFinishedTimeout: PT1M
      # Retry properties, exponential backoff
      retry:
        # How many times do we retry failing operations until we stop the migration process
        maxRetries: 5
        # The minimum retry delay applied to backoff
        # Default PT1S
        minRetryDelay: PT10S
        # The maximum retry delay applied to backoff
        # Default PT1M
        maxRetryDelay: PT1M
  database:
    connect:
      type: elasticsearch
      # URL of ES cluster
      url: <ES CLUSTER CONNECTION>
      # Cluster name of ES cluster
      clusterName: <CLUSTER NAME>

The migration application exposes prometheus metrics (as other components) under the :9600/actuator/prometheus endpoint. This allows the user following up the migration process.

Useful metrics are:

  • camunda_migration_processes_migrated - Number of migrated processes so far
  • camunda_migration_processes_rounds - Number of migration rounds performed so far
  • camunda_migration_processes_round_time - Time taken to complete one migration round
  • camunda_migration_processes_single_process_time - Time taken to process a single process definition

Standalone Migration (locally)​

When running the application separately (locally for example), the configuration must be made public to the migration application.

For example, the location of the application.yaml can be provided for local execution:

SPRING_CONFIG_ADDITIONALLOCATION=/path/to/application.yaml ./camunda/bin/process-migration

Integration in the standalone Camunda application​

When running the standalone Camunda application, the migration application can be executed inside. It can be enabled via the process-migration Spring profile.

Example Spring profile configuration:

spring:
  profiles:
    active: "operate,tasklist,broker,auth,process-migration"
note

The above configuration must be made in the related application.yaml of the standalone Camunda application.

Helm charts​

When using the Camunda 8 Helm charts, the migration application will automatically deploy with the importers if the user is enabling the update path.

Turn off importers after completion​

note

In 8.8, the Operate and Tasklist importers are disabled by default.
When upgrading, enable them by setting:

  • camunda.operate.importerEnabled=true
  • camunda.tasklist.importerEnabled=true

Importers are only necessary at the start of the migration. Once complete, they can be safely turned off.

To detect whether importers are done, users can validate the state in the Elasticsearch/OpenSearch index:

  • tasklist-import-position-8.2.0_
  • operate-import-position-8.3.0_

If all entries show the completed property set to true, the importing is done.

Alternatively, metrics can be validated via:

  • operate_import_completed
  • tasklist_import_completed

These properties can be used in the following example query:

sum(tasklist_import_completed{namespace="$namespace"}) by (importPositionAlias)

metrics-import-complete

Update any custom prefixes​

important

This step is required if you use an installation with custom prefixes.

The new harmonized index schema introduces one common index prefix. To update to Camunda 8.8, existing indices using custom prefixes must be migrated to a common prefix, so the Camunda Exporter, REST API, and other consumers can be configured correctly.

Prefix migration​

warning

Disk usage will massively increase while the migration is ongoing (and will return to normal when complete). Ensure you have at least double the disk space free.

note

It is recommended that you first back up the database, and test the migration script in a test environment to ensure there are no unexpected issues.

note

The prefix migration involves downtime, as all data in the secondary storage needs to be migrated to a different prefix, and writes need to be stopped in between.

Complete the following steps to migrate any custom prefixes:

  1. Shut down your Camunda 8.7 cluster (for example, reduce the replicas to zero). No traffic can be executed against your to-be-migrated ES/OS prefixes.

  2. In the camunda/bin directory, the prefix-migration script will execute the prefix migration, which can be executed locally. Pass the desired prefix using the CAMUNDA_DATABASE_INDEXPREFIX environment variable.

    1. Make sure that your configuration in camunda/config has set the previous custom prefixes, and your configuration is made public to the script (for example, using SPRING_CONFIG_ADDITIONALLOCATION=/path/to/application.yaml ./camunda/bin/prefix-migration).
    2. Alternatively, you can set the configuration via environment variables:
    export CAMUNDA_OPERATE_ELASTICSEARCH_INDEX_PREFIX=old-operate-prefix; \
    export CAMUNDA_TASKLIST_ELASTICSEARCH_INDEX_PREFIX=old-tasklist-prefix; \
    export CAMUNDA_DATABASE_INDEXPREFIX=some-new-prefix; \
    ./prefix-migration

With the migration finished, the following configuration changes must be made prior to updating to 8.8.

note

Without these configuration changes, the update to 8.8 will fail.

Given a new desired prefix of some-new-prefix:

  • Set CAMUNDA_OPERATE_ELASTICSEARCH_INDEX_PREFIX to some-new-prefix.
  • Set CAMUNDA_TASKLIST_ELASTICSEARCH_INDEX_PREFIX to some-new-prefix.
  • Set ZEEBE_BROKER_EXPORTERS_CAMUNDA_ARGS_CONNECT_INDEXPREFIX (the prefix for the new Camunda exporter) to some-new-prefix.

Single Elasticsearch/OpenSearch instance​

Using more than one isolated Elasticsearch/OpenSearch instance for exported Zeebe, Operate, and Tasklist data is no longer supported. If your environment uses multiple Elasticsearch/OpenSearch instances, you must manually migrate the data from each to a single Elasticsearch/OpenSearch cluster before updating to Camunda 8.8. The migration should target Zeebe, Operate, and Tasklist indices, index templates, aliases, and ILM policies.

important

This step must be performed before all other migration steps.

Orchestration Cluster REST API​

Removed deprecated OpenAPI objects​

warning

With the Camunda 8.8 release, deprecated API objects containing number keys have been removed, including the corresponding application/vnd.camunda.api.keys.number+json content type header.

In previous releases, entity keys were transitioned from integer (int64) to string types, and deprecated integer (int64) keys were still supported. As of the 8.8 release, support for integer (int64) keys has been removed.

To update to Camunda 8.8, API objects using integer (int64) keys must be updated to use string keys and the application/json header.

For more information about the key attribute type change, see the 8.7 API key attributes overview.

Streamlined variable OpenAPI objects​

The OpenAPI specification streamlines the objects used for variable filtering in search requests. The supported Camunda Java client handles those changes transparently. The affected API endpoints are alpha feature endpoints and thus already marked as subject to change in 8.6 and 8.7.

If you are generating a custom REST client using the specification, consider the following changes:

  • The unified VariableValueFilterProperty replaces the ProcessInstanceVariableFilterRequest and UserTaskVariableFilterRequest in the OpenAPI spec.
  • For naming consistency, the UserTaskVariableFilter replaces the VariableUserTaskFilterRequest.

Exported records​

USER_TASK records​

To support user task listeners, some backward incompatible changes were necessary to the exported USER_TASK records.

assignee no longer provided in CREATED event​

Previously, when a user task was activating with a specified assignee, we appended the following events of the USER_TASK value type:

  • CREATING with assignee property as provided
  • CREATED with assignee property as provided

The ASSIGNING and ASSIGNED events were not appended in this case.

To support the new User Task Listeners feature, the assignee value will not be filled in the CREATED event anymore.

With 8.8, the following events are now appended:

  • CREATING with assignee property as provided
  • CREATED with assignee always "" (empty string)
  • ASSIGNING with assignee property as provided
  • ASSIGNED with assignee property as provided

ASSIGNING has become CLAIMING for CLAIM operation​

When claiming a user task, we previously appended the following records of the USER_TASK value type:

  • CLAIM
  • ASSIGNING
  • ASSIGNED

A new CLAIMING intent was introduced to distinguish between claiming and regular assigning. We now append the following records when claiming a user task:

  • CLAIM
  • CLAIMING
  • ASSIGNED

The ASSIGNING event is still appended for assigning a user task. In that case, we append the following records:

  • ASSIGN
  • ASSIGNING
  • ASSIGNED

Zeebe record types​

With the introduction of the Camunda Exporter, the Elasticsearch and OpenSearch Exporter no longer export all record types by default. Therefore, not all zeebe-record indices will be populated.

note

This only affects records exported with the 8.8 version. During the migration, leftover 8.7 records are exported as normal.

The record types that continue to be exported by default are the following:

  • DEPLOYMENT
  • PROCESS
  • PROCESS_INSTANCE
  • VARIABLE
  • USER_TASK
  • INCIDENT
  • JOB

To export other record types, review includeEnabledRecords for Elasticsearch or OpenSearch exporter configuration.

Exporter deletion​

Exporter configuration removal no longer deletes the exporter from the system. Instead, the exporter enters a blocked state to safeguard against data loss caused by accidental or manual configuration changes. To permanently remove an exporter, use the Exporters API.

Connectors​

Email connector​

Starting with the 8.8 release, angle brackets (< and >) are no longer removed from the messageId.

Elasticsearch​

Starting in 8.8, the default replica count for Camunda indices in Elasticsearch and OpenSearch changes from 0 to 1. This ensures that if a node goes down, Camunda is not blocked by a temporary outage of the secondary data store.

If you already override this value, no action is required. Otherwise, review the guidance below to adjust capacity or revert the change.

Throughout this section, references to Elasticsearch (ES) also apply to OpenSearch (OS). For simplicity, ES is used as the default term.

Single-node cluster​

If the current ES cluster has only one node, setting the replica value to 1 will cause the cluster state to go yellow.
This means all primary shards are allocated, but some replicas are unassigned.

To resolve this issue:

note

If you install Elasticsearch using the Camunda Helm chart, the default master.replicaCount is 3, resulting in a three-node cluster.
For other setups, you can apply the same logic by setting elasticsearch.master.replicaCount.

Once complete, continue with Multi-node cluster.

Multi-node cluster​

Starting in 8.8, the default replica count for indices is 1. If your cluster previously used a replica count of 0, this will roughly double disk usage.

To ensure the ES cluster can reach a green status, make sure the disk capacity is at least 2.5× the previously used storage.
This calculation accounts for watermarks, overhead, and growth.

For example:

  • In 8.7, node disk usage is 30/50 GB, 35/50 GB, and 40/50 GB.
  • The most used node has 40 GB.
  • To ensure replicas fit on other nodes, resize all disks to at least 100 GB (40 × 2.5).

Resize Elasticsearch disk​

If the current disk size is too small for replicas, assign more disk space. Whether a Persistent Volume Claim (PVC) is expandable depends on the underlying StorageClass.

For information on expanding PVCs using the calculations above, see:

For managed services, see:

Roll back​

If you encounter issues after the upgrade and replica increase, revert the replica count to 0 in the Zeebe deployment and restart.
Zeebe pods will trigger the processes which update the indices back to 0 replicas.

Set one of the following:

  • YAML: camunda.database.index.numberOfReplicas: 0
  • Env var: CAMUNDA_DATABASE_INDEX_NUMBER_OF_REPLICAS=0