Skip to main content
Version: 8.3

Elasticsearch exporter

note

Please refer to Supported Environments to find out which versions of Elasticsearch are supported in a Camunda 8 Self-Managed setup.

The Zeebe Elasticsearch exporter acts as a bridge between Zeebe and Elasticsearch by exporting records written to Zeebe streams as documents into several indices.

Concept

The exporter operates on the idea that it should perform as little as possible on the Zeebe side of things. In other words, you can think of the indexes into which the records are exported as a staging data warehouse. Any enrichment or transformation on the exported data should be performed by your own ETL jobs.

When configured to do so, the exporter will automatically create an index per record value type (see the value type in the Zeebe protocol). Each of these indexes has a corresponding pre-defined mapping to facilitate data ingestion for your own ETL jobs. You can find those as templates in the resources folder of the exporter's source code.

note

The indexes are created as required, and will not be created twice if they already exist. However, once disabled, they will not be deleted (that is up to the administrator.) Similarly, data is never deleted by the exporter, and must be deleted by the administrator when it is safe to do so. A retention policy can be configured to automatically delete data after a certain number of days.

Configuration

note

As the exporter is packaged with Zeebe, it is not necessary to specify a jarPath.

The exporter can be enabled by configuring it with the classpath in the broker settings.

For example:

exporters:
elasticsearch:
className: io.camunda.zeebe.exporter.ElasticsearchExporter
args:
# Refer to the table below for the available args options

The exporter can be configured by providing args. The table below explains all the different options, and the default values for these options:

OptionDescriptionDefault
urlValid URLs as comma-separated stringhttp://localhost:9200
requestTimeoutMsRequest timeout (in ms) for the Elasticsearch. client30000
indexRefer to Index for the index configuration options.
bulkRefer to Bulk for the bulk configuration options.
retentionRefer to Retention for the retention configuration options
authenticationRefer to Authentication for the authentication configuration options.

Index

In most cases, you will not be interested in exporting every single record produced by a Zeebe cluster, but rather only a subset of them. This can also be configured to limit the kinds of records being exported (e.g. only events, no commands), and the value type of these records (e.g. only job and process values).

OptionDescriptionDefault
prefixThis prefix will be appended to every index created by the exporter; must not contain _ (underscore).zeebe-record
createTemplateIf true missing indexes will be created automatically.true
numberOfShardsThe number of shards used for each new record index created.3
numberOfReplicasThe number of shard replicas used for each new record index created.0
commandIf true command records will be exportedfalse
eventIf true event records will be exportedtrue
rejectionIf true rejection records will be exportedfalse
checkpointIf true records related to checkpoints will be exportedfalse
commandDistributionIf true records related to command distributions will be exportedtrue
decisionIf true records related to decisions will be exportedtrue
decisionEvaluationIf true records related to decision evaluations will be exportedtrue
decisionRequirementsIf true records related to decisionRequirements will be exportedtrue
deploymentIf true records related to deployments will be exportedtrue
deploymentDistributionIf true records related to deployment distributions will be exportedtrue
errorIf true records related to errors will be exportedtrue
escalationIf true records related to escalations will be exportedtrue
formIf true records related to forms will be exportedtrue
incidentIf true records related to incidents will be exportedtrue
jobIf true records related to jobs will be exportedtrue
jobBatchIf true records related to job batches will be exportedfalse
messageIf true records related to messages will be exportedtrue
messageSubscriptionIf true records related to message subscriptions will be exportedtrue
messageStartEventSubscriptionIf true records related to message start event subscriptions will be exportedtrue
processIf true records related to processes will be exportedtrue
processEventIf true records related to process events will be exportedfalse
processInstanceIf true records related to process instances will be exportedtrue
processInstanceBatchIf true records related to process instances batches will be exportedfalse
processInstanceCreationIf true records related to process instance creations will be exportedtrue
processInstanceModificationIf true records related to process instance modifications will be exportedtrue
processMessageSubscriptionIf true records related to process message subscriptions will be exportedtrue
resourceDeletionIf true records related to resource deletions will be exportedtrue
signalIf true records related to signals will be exportedtrue
signalSubscriptionIf true records related to signal subscriptions will be exportedtrue
timerIf true records related to timers will be exportedtrue
variableIf true records related to variables will be exportedtrue
variableDocumentIf true records related to variable documents will be exportedtrue

Bulk

To avoid too many expensive requests to the Elasticsearch cluster, the exporter performs batch updates by default. The size of the batch, along with how often it should be flushed (regardless of size) can be controlled by configuration.

OptionDescriptionDefault
delayDelay, in seconds, before force flush of the current batch. This ensures that even when we have low traffic of records, we still export every once in a while.5
sizeThe amount of records a batch should have before we flush the batch1000
memoryLimitThe size of the batch, in bytes, before we flush the batch10485760 (10 MB)

With the default configuration, the exporter will aggregate records and flush them to Elasticsearch:

  1. When it has aggregated 1000 records.
  2. When the batch memory size exceeds 10 MB.
  3. Five seconds have elapsed since the last flush (regardless of how many records were aggregated).

Retention

A retention policy can be set up to delete old data. When enabled, this creates an Index Lifecycle Management (ILM) Policy that deletes the data after the specified minimumAge. All index templates created by this exporter apply the created ILM Policy.

OptionDescriptionDefault
enabledIf true the ILM Policy is created and applied to the index templatesfalse
minimumAgeSpecifies how old the data must be, before the data is deleted as a duration30d
policyNameThe name of the created and applied ILM policyzeebe-record-retention-policy
note

The duration can be specified in days d, hours h, minutes m, seconds s, milliseconds ms, and/or nanoseconds nanos.

Authentication

Providing these authentication options will enable Basic Authentication on the exporter.

OptionDescriptionDefault
usernameUsername used to authenticateN/A
passwordPassword used to authenticateN/A

Example

Here is an example configuration of the exporter:

---
exporters:
elasticsearch:
# Elasticsearch Exporter ----------
# An example configuration for the elasticsearch exporter:
#
# These setting can also be overridden using the environment variables "ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_..."
#

className: io.camunda.zeebe.exporter.ElasticsearchExporter

args:
# A comma separated list of URLs pointing to the Elasticsearch instances you wish to export to.
# For example, if you want to connect to multiple nodes for redundancy:
# url: http://localhost:9200,http://localhost:9201
url: http://localhost:9200

bulk:
delay: 5
size: 1000
memoryLimit: 10485760

retention:
enabled: true
minimumAge: 30d
policyName: zeebe-records-retention-policy

authentication:
username: elastic
password: changeme

index:
prefix: zeebe-record
createTemplate: true

command: false
event: true
rejection: false

commandDistribution: true
decisionRequirements: true
decision: true
decisionEvaluation: true
deployment: true
deploymentDistribution: true
error: true
escalation: true
form: true
incident: true
job: true
jobBatch: false
message: true
messageStartSubscription: true
messageSubscription: true
process: true
processEvent: false
processInstance: true
processInstanceCreation: true
processInstanceModification: true
processMessageSubscription: true
resourceDeletion: true
signal: true
signalSubscription: true
timer: true
variable: true
variableDocument: true