The Zeebe Elasticsearch exporter acts as a bridge between Zeebe and Elasticsearch by exporting records written to Zeebe streams as documents into several indices.
The exporter operates on the idea that it should perform as little as possible on the Zeebe side of things. In other words, you can think of the indexes into which the records are exported as a staging data warehouse. Any enrichment or transformation on the exported data should be performed by your own ETL jobs.
When configured to do so, the exporter will automatically create an index per record value type (see the value type in the Zeebe protocol). Each of these indexes has a corresponding pre-defined mapping to facilitate data ingestion for your own ETL jobs. You can find those as templates in this module's resources folder.
The indexes are created as required, and will not be created twice if they already exist. However, once disabled, they will not be deleted (that is up to the administrator.) Similarly, data is never deleted by the exporter, and must be deleted by the administrator when it is safe to do so. A retention policy can be configured to automatically delete data after a certain number of days.
As the exporter is packaged with Zeebe, it is not necessary to specify a
The exporter can be enabled by configuring it with the
classpath in the broker settings.
# Refer to the table below for the available args options
The exporter can be configured by providing
args. The table below explains all the different
options, and the default values for these options:
|url||Valid URLs as comma-separated string|
|requestTimeoutMs||Request timeout (in ms) for the Elasticsearch. client|
|index||Refer to Index for the index configuration options.|
|bulk||Refer to Bulk for the bulk configuration options.|
|retention||Refer to Retention for the retention configuration options|
|authentication||Refer to Authentication for the authentication configuration options.|
In most cases, you will not be interested in exporting every single record produced by a Zeebe cluster, but rather only a subset of them. This can also be configured to limit the kinds of records being exported (e.g. only events, no commands), and the value type of these records (e.g. only job and process values).
|prefix||This prefix will be appended to every index created by the exporter; must not contain ||zeebe-record|
To avoid too many expensive requests to the Elasticsearch cluster, the exporter performs batch updates by default. The size of the batch, along with how often it should be flushed (regardless of size) can be controlled by configuration.
|delay||Delay, in seconds, before force flush of the current batch. This ensures that even when we have low traffic of records, we still export every once in a while.|
|size||The amount of records a batch should have before we flush the batch|
|memoryLimit||The size of the batch, in bytes, before we flush the batch|
With the default configuration, the exporter will aggregate records and flush them to Elasticsearch:
- When it has aggregated 1000 records.
- When the batch memory size exceeds 10 MB.
- Five seconds have elapsed since the last flush (regardless of how many records were aggregated).
A retention policy can be set up to delete old data.
When enabled, this creates an Index Lifecycle Management (ILM) Policy that deletes the data after the specified
All index templates created by this exporter apply the created ILM Policy.
|minimumAge||Specifies how old the data must be, before the data is deleted as a duration|
|policyName||The name of the created and applied ILM policy|
The duration can be specified in days
ms, and/or nanoseconds
Providing these authentication options will enable Basic Authentication on the exporter.
|username||Username used to authenticate||N/A|
|password||Password used to authenticate||N/A|
Here is an example configuration of the exporter:
# Elasticsearch Exporter ----------
# An example configuration for the elasticsearch exporter:
# These setting can also be overridden using the environment variables "ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_..."
# A comma separated list of URLs pointing to the Elasticsearch instances you wish to export to.
# For example, if you want to connect to multiple nodes for redundancy:
# url: http://localhost:9200,http://localhost:9201