Configure data retention

Data retention policies automatically delete old data from Elasticsearch or OpenSearch after a specified time period. This prevents unlimited data growth, reduces storage costs, and maintains system performance by using Index Lifecycle Management (ILM) for Elasticsearch or Index State Management (ISM) for OpenSearch.

Best practice

Configure data retention during initial installation. Adding retention configuration after deployment may require manual policy creation in Elasticsearch/OpenSearch. See Differences from previous versions for version-specific behavior.

Prerequisites

• Camunda 8.8+ Helm chart deployment
• Elasticsearch 7+ or OpenSearch 2.5+
• Access to modify your values.yaml file

Configuration

Configure retention policies in your values.yaml file under the orchestration section.

Retention policy types:

1. Elasticsearch/OpenSearch Exporter indices (Zeebe records) (orchestration.retention) – Retention for Zeebe record indices written by the legacy Elasticsearch/OpenSearch Exporter (for example, zeebe-record-*). These are not Orchestration Cluster indices.
2. Orchestration Cluster indices (historical data) (orchestration.history.retention) – Retention for archived Operate, Tasklist, and Camunda indices stored in secondary storage.

For index prefix requirements and examples when both index families share the same Elasticsearch/OpenSearch cluster, see Configure Elasticsearch and OpenSearch index prefixes.

warning

Zeebe records retention requirements orchestration.retention.* apply only to Elasticsearch/OpenSearch Exporter indices (Zeebe records), not to Orchestration Cluster indices managed by the Camunda Exporter.

The orchestration.retention configuration requires the legacy Zeebe Elasticsearch/OpenSearch Exporter to be enabled. The legacy exporter is automatically enabled when:

• orchestration.exporters.zeebe.enabled: true is set, OR
• Optimize is enabled (optimize.enabled: true), OR
• Data migration is enabled (orchestration.migration.data.enabled: true)

Starting in Camunda 8.8, orchestration.exporters.zeebe.enabled defaults to false. If you need Zeebe record retention without Optimize, you must explicitly enable it.

For full exporter configuration and retention options, see Zeebe Elasticsearch Exporter retention.

Parameters

Zeebe records retention parameters:

Key	Type	Default	Description
orchestration.retention.enabled	boolean	false	If true, creates and applies the ILM/ISM policy to Zeebe record indices. Requires the legacy Zeebe exporter to be enabled (see prerequisites above).
orchestration.retention.minimumAge	string	30d	How old the data must be before deletion. Uses Elasticsearch TimeUnit format (for example, 30d, 7d, 1h)
orchestration.retention.policyName	string	zeebe-record-retention-policy	Name of the ILM/ISM policy to create and apply

History archiving and retention parameters:

Key	Type	Default	Description
orchestration.history.waitPeriodBeforeArchiving	string	1h	Grace period before archiving completed processes. Processes finished within this window are not yet archived.
orchestration.history.rolloverInterval	string	1d	Time range for creating dated indices (for example, 1d creates daily indices).
orchestration.history.rolloverBatchSize	integer	100	Maximum number of process instances per archiving batch
orchestration.history.elsRolloverDateFormat	string	date	Date format for historical indices in Java DateTimeFormatter syntax
orchestration.history.delayBetweenRuns	integer	2000	Millisecond interval between archiver runs
orchestration.history.maxDelayBetweenRuns	integer	60000	Maximum millisecond interval between archiver runs due to failure backoffs
orchestration.history.retention.enabled	boolean	false	If true, applies ILM/ISM policy to archived orchestration indices (Operate, Tasklist, Camunda)
orchestration.history.retention.minimumAge	string	30d	How old archived data must be before deletion
orchestration.history.retention.policyName	string	camunda-history-retention-policy	Name of the ILM/ISM policy for historical data
orchestration.history.retention.usageMetricsMinimumAge	string	730d	Retention period for usage metrics indices (2 years by default)
orchestration.history.retention.usageMetricsPolicyName	string	camunda-usage-metrics-retention-policy	Name of the ILM/ISM policy for usage metrics

Example usage

Orchestration Cluster history retention (recommended)

In most deployments starting with Camunda 8.8, you only need retention for Orchestration Cluster indices (archived Operate, Tasklist, and Camunda data).

The example below sets most properties to their current default values and explicitly enables history retention (orchestration.history.retention.enabled: true). If defaults change in future versions, clusters that keep these explicit values will retain the behavior shown here.

orchestration:
  history:
    waitPeriodBeforeArchiving: 1h
    rolloverInterval: 1d
    rolloverBatchSize: 100
    elsRolloverDateFormat: date
    delayBetweenRuns: 2000
    maxDelayBetweenRuns: 60000
    retention:
      enabled: true
      minimumAge: 30d
      policyName: camunda-history-retention-policy
      usageMetricsMinimumAge: 730d
      usageMetricsPolicyName: camunda-usage-metrics-retention-policy

Zeebe records retention

Enable Zeebe records retention only if you still use the legacy Elasticsearch/OpenSearch Exporter (for example, when Optimize reads from zeebe-record-* indices).

Exporter configuration and full retention examples are documented in Zeebe Elasticsearch Exporter retention. The snippet below shows only the Helm values related to enabling retention:

# Either enable Optimize, which in turn enables the legacy Zeebe exporter:
optimize:
  enabled: true

# Or enable the legacy Zeebe exporter directly:
orchestration:
  exporters:
    zeebe:
      enabled: true

# Zeebe records retention (Elasticsearch/OpenSearch Exporter indices)
orchestration:
  retention:
    enabled: true
    minimumAge: 30d
    policyName: zeebe-record-retention-policy

Retention configuration

Both scenarios use the same retention configuration:

orchestration:
  # Zeebe records retention
  retention:
    enabled: true
    minimumAge: 30d
    policyName: zeebe-record-retention-policy

  # Historical data archiving and retention
  history:
    waitPeriodBeforeArchiving: 1h
    rolloverInterval: 1d
    rolloverBatchSize: 100
    elsRolloverDateFormat: date
    delayBetweenRuns: 2000
    maxDelayBetweenRuns: 60000
    retention:
      enabled: true
      minimumAge: 30d
      policyName: camunda-history-retention-policy
      usageMetricsMinimumAge: 730d
      usageMetricsPolicyName: camunda-usage-metrics-retention-policy

Troubleshooting

Verifying retention policies

After deploying with retention enabled, verify that the ILM/ISM policies were created successfully.

• Zeebe records retention (zeebe-record-retention-policy) applies to Elasticsearch/OpenSearch Exporter indices (Zeebe records) created by the legacy exporter (for example, indices matching the exporter prefix such as zeebe-record-*). The Zeebe Elasticsearch/OpenSearch Exporter creates this policy automatically during initialization, typically within a few minutes of deployment.

• History retention (camunda-history-retention-policy) applies to archived Orchestration Cluster indices used by Operate, Tasklist, and Camunda (for example, operate-process-*, tasklist-task-* with date suffixes).

The schema manager creates this ILM/ISM policy on application startup when orchestration.history.retention.enabled: true. After that, the archiver:

1. Waits for waitPeriodBeforeArchiving (default: 1 hour) after a process instance completes.
2. Archives completed instances to dated indices (for example, operate-process-8.3.0_2024-01-15, tasklist-task-8.8.0_2024-01-15).
3. Attaches the policy directly to each archived index as it creates it.

Set your database URL:

export DATABASE_URL="https://your-database-host:9200"

Replace your-database-host with your Elasticsearch or OpenSearch hostname.

Check if policies exist:

For Elasticsearch:

curl -X GET "${DATABASE_URL}/_ilm/policy/zeebe-record-retention-policy?pretty"
curl -X GET "${DATABASE_URL}/_ilm/policy/camunda-history-retention-policy?pretty"

For OpenSearch:

curl -X GET "${DATABASE_URL}/_plugins/_ism/policies/zeebe-record-retention-policy?pretty"
curl -X GET "${DATABASE_URL}/_plugins/_ism/policies/camunda-history-retention-policy?pretty"

Expected response for a policy with 30-day retention:

{
  "zeebe-record-retention-policy": {
    "version": 1,
    "modified_date": "2025-01-15T10:30:00.000Z",
    "policy": {
      "phases": {
        "delete": {
          "min_age": "30d",
          "actions": {
            "delete": {}
          }
        }
      }
    }
  }
}

Check if archived indices exist:

curl -X GET "${DATABASE_URL}/_cat/indices/operate-*_20*?v"
curl -X GET "${DATABASE_URL}/_cat/indices/tasklist-*_20*?v"

Archived indices follow the pattern: {component}-{type}-{schema-version}_{date}
Examples:

• operate-process-8.3.0_2024-01-15
• operate-variable-8.3.0_2024-01-15
• tasklist-task-8.8.0_2024-01-15

Index versioning

The version number in index names (for example, 8.3.0) represents the schema version, not necessarily the Camunda platform version. Schema versions evolve independently as index structures change. For details, see schema and migration documentation.

If no archived indices exist:

• No processes have completed and been archived yet
• The waitPeriodBeforeArchiving period hasn't elapsed
• Deploy and complete a test process, then wait for archiving to occur

Check if retention policy is applied to an archived index:

For Elasticsearch:

curl -X GET "${DATABASE_URL}/operate-process-8.3.0_2024-01-15/_settings?pretty" | grep -A 3 lifecycle

For OpenSearch:

curl -X GET "${DATABASE_URL}/_plugins/_ism/explain/operate-process-8.3.0_2024-01-15?pretty"

Expected output showing the policy is attached:

{
  "index.lifecycle.name": "camunda-history-retention-policy"
}

Manually creating or updating policies (8.7 and earlier)

For Camunda 8.8+, policies are created automatically by the retention tooling. If you need to manually create or update policies, use the policy names configured in your values.yaml with the commands in the Camunda 8.7 manual policy management guide.

Camunda 8.8 default policy names (customizable via Helm values):

• zeebe-record-retention-policy - For Zeebe record indices (configured via orchestration.retention.policyName)
• camunda-history-retention-policy - For historical Operate, Tasklist, and Camunda indices (configured via orchestration.history.retention.policyName)
• camunda-usage-metrics-retention-policy - For usage metrics indices (configured via orchestration.history.retention.usageMetricsPolicyName)

Camunda 8.7 default policy names (only Zeebe is customizable):

• zeebe-record-retention-policy - For Zeebe records (customizable via zeebe.retention.policyName)
• operate_delete_archived_indices - For Operate indices (hardcoded)
• tasklist_delete_archived_indices - For Tasklist indices (hardcoded)

The curl commands for creating and applying policies are the same across versions—only the policy names differ. See the 8.7 guide's manual policy section for complete ILM and ISM policy creation commands.

Known limitations

OpenSearch policy updates:

When using OpenSearch, updating an existing ISM policy's minimumAge may not take effect automatically. OpenSearch requires the seq_no and primary_term parameters during policy updates to ensure proper version control. Without these parameters, policy updates may fail or be skipped.

To update an OpenSearch policy:

1. Get the current policy with its version information
2. Update the policy using the seq_no and primary_term from the current version
3. Verify the updated policy reflects the new minimumAge

See the OpenSearch ISM API documentation for details on policy version parameters.

Elasticsearch bulk operations:

When applying retention policies to a large number of existing indices, the operation may fail due to HTTP line length limits in Elasticsearch. This typically occurs when using wildcard patterns to apply settings to many indices at once (for example, operate-*, tasklist-*).

Workarounds:

• Apply settings to smaller batches of indices using more specific patterns
• Apply settings to individual indices when the number of indices is very large
• Use index templates for future indices instead of retroactively applying to all indices

Policy timing considerations:

The camunda-history-retention-policy is created by Camunda's retention tooling as part of the archiving workflow. If you query for this policy immediately after deployment and before any archiving occurs, you may receive a 404 response. The policy is created when the first archived index is created by the archiver.

Index naming

Operate and Tasklist indices use schema-specific versioning in their names (for example, operate-process-8.3.0_, tasklist-task-8.8.0_). The version numbers represent schema versions, which may differ from the Camunda platform version. When archived, these indices receive a date suffix (for example, operate-process-8.3.0_2024-01-15).

References

Related Camunda documentation:

• Configure Helm chart components - How to use orchestration.configuration for advanced settings
• Upgrade from 8.7 to 8.8 - Version upgrade guidance
• Zeebe Elasticsearch Exporter retention - Legacy Zeebe exporter retention settings
• Zeebe Camunda Exporter - Camunda exporter configuration (controls orchestration.history.* settings)
  • History archiving settings - Archiving and rollover configuration
  • Retention settings - Historical data retention policies
• Operate data retention - Operate-specific retention behavior
• Tasklist data retention - Tasklist-specific retention behavior

External documentation:

• Elasticsearch ILM documentation - Official Elasticsearch ILM guide
• Elasticsearch TimeUnit format - Valid time unit values
• OpenSearch ISM documentation - Official OpenSearch ISM guide
• Helm values documentation - Working with Helm values files

Differences from previous versions

Camunda 8.8:

• Policy creation and attachment: Policies are created automatically by retention tooling and attached to archived indices as they are created
• Unified history retention: Historical data retention (Operate, Tasklist, Camunda indices) uses unified orchestration.history.retention configuration
• Zeebe record retention: Legacy zeebe-record indices remain under exporter-specific configuration (orchestration.retention)
• OpenSearch policy updates: Updating existing ISM policies requires seq_no and primary_term parameters; without them, updates may fail and require manual intervention
• Bulk operations: Applying settings to many indices may fail due to HTTP line length limits, requiring batching or manual workarounds

Camunda 8.7 and earlier:

• Operate: Creates ILM policy when enabled, but configuration updates (for example, changing minimumAge) are not applied automatically - manual policy updates required
• Zeebe: Creates policies on initial install and applies ILM configuration updates automatically
• Tasklist: Does not apply ILM configuration updates after deployment - manual policy updates required
• Versions 8.5-8.6: ILM policies sometimes missing after configuration; may require manual creation or new record export to trigger policy creation

info

For Camunda 8.7 and earlier, if you change retention configuration after initial deployment, you must manually update the policies in Elasticsearch/OpenSearch for Operate and Tasklist. Zeebe automatically applies configuration updates. See Manually creating or updating policies.