Skip to main content
Version: 8.5

Elasticsearch exporter

note

Please refer to supported environments to find out which versions of Elasticsearch are supported in a Camunda 8 Self-Managed setup.

The Zeebe Elasticsearch exporter acts as a bridge between Zeebe and Elasticsearch by exporting records written to Zeebe streams as documents into several indices.

Concept

The exporter operates on the idea that it should perform as little as possible on the Zeebe side of things. In other words, you can think of the indexes into which the records are exported as a staging data warehouse. Any enrichment or transformation on the exported data should be performed by your own ETL jobs.

When configured to do so, the exporter will automatically create an index per record value type (see the value type in the Zeebe protocol). Each of these indexes has a corresponding pre-defined mapping to facilitate data ingestion for your own ETL jobs. You can find those as templates in the resources folder of the exporter's source code.

note

The indexes are created as required, and will not be created twice if they already exist. However, once disabled, they will not be deleted (that is up to the administrator.) Similarly, data is never deleted by the exporter, and must be deleted by the administrator when it is safe to do so. A retention policy can be configured to automatically delete data after a certain number of days.

Configuration

note

As the exporter is packaged with Zeebe, it is not necessary to specify a jarPath.

The exporter can be enabled by configuring it with the classpath in the broker settings.

For example:

exporters:
elasticsearch:
className: io.camunda.zeebe.exporter.ElasticsearchExporter
args:
# Refer to the table below for the available args options

The exporter can be configured by providing args. The table below explains all the different options, and the default values for these options:

OptionDescriptionDefault
urlValid URLs as comma-separated stringhttp://localhost:9200
requestTimeoutMsRequest timeout (in ms) for the Elasticsearch. client30000
indexRefer to Index for the index configuration options.
bulkRefer to Bulk for the bulk configuration options.
retentionRefer to Retention for the retention configuration options
authenticationRefer to Authentication for the authentication configuration options.

In most cases, you will not be interested in exporting every single record produced by a Zeebe cluster, but rather only a subset of them. This can also be configured to limit the kinds of records being exported (e.g. only events, no commands), and the value type of these records (e.g. only job and process values).

OptionDescriptionDefault
prefixThis prefix will be appended to every index created by the exporter; must not contain _ (underscore).zeebe-record
createTemplateIf true missing indexes will be created automatically.true
indexSuffixDatePatternThis suffix will be appended to every index created by the exporter; The pattern is based on the Java DateTimeFormater and supports the same syntax. This is useful when indexes should be created in a different interval, like hourly instead of daily."yyyy-MM-dd'"
numberOfShardsThe number of shards used for each new record index created.3
numberOfReplicasThe number of shard replicas used for each new record index created.0
commandIf true command records will be exportedfalse
eventIf true event records will be exportedtrue
rejectionIf true rejection records will be exportedfalse
checkpointIf true records related to checkpoints will be exportedfalse
commandDistributionIf true records related to command distributions will be exportedtrue
decisionIf true records related to decisions will be exportedtrue
decisionEvaluationIf true records related to decision evaluations will be exportedtrue
decisionRequirementsIf true records related to decisionRequirements will be exportedtrue
deploymentIf true records related to deployments will be exportedtrue
deploymentDistributionIf true records related to deployment distributions will be exportedtrue
errorIf true records related to errors will be exportedtrue
escalationIf true records related to escalations will be exportedtrue
formIf true records related to forms will be exportedtrue
incidentIf true records related to incidents will be exportedtrue
jobIf true records related to jobs will be exportedtrue
jobBatchIf true records related to job batches will be exportedfalse
messageIf true records related to messages will be exportedtrue
messageSubscriptionIf true records related to message subscriptions will be exportedtrue
messageStartEventSubscriptionIf true records related to message start event subscriptions will be exportedtrue
processIf true records related to processes will be exportedtrue
processEventIf true records related to process events will be exportedfalse
processInstanceIf true records related to process instances will be exportedtrue
processInstanceBatchIf true records related to process instances batches will be exportedfalse
processInstanceCreationIf true records related to process instance creations will be exportedtrue
processInstanceMigrationIf true records related to process instance migrations will be exportedtrue
processInstanceModificationIf true records related to process instance modifications will be exportedtrue
processMessageSubscriptionIf true records related to process message subscriptions will be exportedtrue
resourceDeletionIf true records related to resource deletions will be exportedtrue
signalIf true records related to signals will be exportedtrue
signalSubscriptionIf true records related to signal subscriptions will be exportedtrue
timerIf true records related to timers will be exportedtrue
userTaskIf true records related to user tasks will be exportedtrue
variableIf true records related to variables will be exportedtrue
variableDocumentIf true records related to variable documents will be exportedtrue

Example

Here is an example configuration of the exporter:

---
exporters:
elasticsearch:
# Elasticsearch Exporter ----------
# An example configuration for the elasticsearch exporter:
#
# These setting can also be overridden using the environment variables "ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_..."
#

className: io.camunda.zeebe.exporter.ElasticsearchExporter

args:
# A comma separated list of URLs pointing to the Elasticsearch instances you wish to export to.
# For example, if you want to connect to multiple nodes for redundancy:
# url: http://localhost:9200,http://localhost:9201
url: http://localhost:9200

bulk:
delay: 5
size: 1000
memoryLimit: 10485760

retention:
enabled: true
minimumAge: 30d
policyName: zeebe-records-retention-policy

authentication:
username: elastic
password: changeme

index:
prefix: zeebe-record
createTemplate: true

indexSuffixDatePattern: "yyyy-MM-dd"

command: false
event: true
rejection: false

commandDistribution: true
decisionRequirements: true
decision: true
decisionEvaluation: true
deployment: true
deploymentDistribution: true
error: true
escalation: true
form: true
incident: true
job: true
jobBatch: false
message: true
messageStartSubscription: true
messageSubscription: true
process: true
processEvent: false
processInstance: true
processInstanceCreation: true
processInstanceMigration: true
processInstanceModification: true
processMessageSubscription: true
resourceDeletion: true
signal: true
signalSubscription: true
timer: true
userTask: true
variable: true
variableDocument: true