Operate stores data in Elasticsearch. On first start Operate will create all required indices and templates.
Operate uses several Elasticsearch indices that are mostly created by using templates.
Index names follow the defined pattern:
datatype defines which data is stored in the index, e.g.
schemaversion represents version of Operate,
date represents finished date of archived data (see Data retention).
Knowing index name pattern, it is possible to customize index settings by creating Elasticsearch templates (Example of an index template) E.g. to define desired number of shards and replicas, you can define following template:
Note: In order for these settings to work, template must be created before Operate first run.
Version of Operate is reflected in Elasticsearch object names, e.g.
operate-user-0.24.0_ index contains user data for Operate 0.24.0. When upgrading from one
version of Operate to another, migration of data must be performed. Operate distribution provides an application to perform data migration from previous versions.
Each version of Operate delivers set of migration steps needed to be applied for corresponding version of Operate.
When upgrading from one version to another necessary migration steps constitute the so-called migration plan.
All known migration steps (both applied and not) are persisted in dedicated Elasticsearch index:
Make sure that Elasticsearch which contains the Operate data is running. The migration script will connect to specified connection in Operate
<operate_home>/bin/migrate.bat for Windows).
What is expected to happen:
- Elasticsearch schema of new version is created
- previous version is detected
- migration plan is built and executed reindexing data for each index
- old indices are deleted
All known migration steps with metadata will be stored in
Note: The old indices will be deleted ONLY after successful migration. That might require more disk space during migration process.
Important! You should take care of data backup before performing migration.
When running newer version of Operate against older schema, it will perform data migration on startup. The migration will happen when exactly ONE previous schema version was detected.
- If migration fails, it is OK to retry it. All applied steps are stored and only those steps will be applied that hasn't been executed yet.
- Operate should not be running, while migration is happening
- In case version upgrade is performed in the cluster with several Operate nodes, only one node (Webapp module) must execute data migration, the others must be stopped and started only after migration is fully finished
Automatic migration is enabled by default. It can be disabled by setting the configuration key:
camunda.operate.migration.migrationEnabled = false
You can specify previous ("source") version with the configuration key:
If no sourceVersion is defined Operate tries to detect it from Elasticsearch indices.
To ensure that the migration will be executed before Operate will be started you can use
the init container feature of Kubernetes. It makes sure that the 'main' container will only be started
if the initContainer was successfully executed.
The following snippet of a pod description for Kubernetes shows the usage of
migrate script as initContainer.