The Zeebe OpenSearch Exporter acts as a bridge between Zeebe and OpenSearch by exporting records written to Zeebe streams as documents into several indices.
The exporter operates on the idea that it should perform as little as possible on the Zeebe side of things. In other words, you can think of the indexes into which the records are exported as a staging data warehouse. Any enrichment or transformation on the exported data should be performed by your own ETL jobs.
When configured to do so, the exporter will automatically create an index per record value type (see the value type in the Zeebe protocol). Each of these indexes has a corresponding pre-defined mapping to facilitate data ingestion for your own ETL jobs. You can find those as templates in this module's resources folder.
The indexes are created as required, and will not be created twice if they already exist. However, once disabled, they will not be deleted (that is up to the administrator). Similarly, data is never deleted by the exporter, and must be deleted by the administrator when it is safe to do so.
As the exporter is packaged with Zeebe, it is not necessary to specify a
The exporter can be enabled by configuring it with the classpath in the broker settings.
# Refer to the table below for the available args options
The exporter can be configured by providing
args. The table below explains all the different
options, and the default values for these options:
|url||Valid URLs as comma-separated string|
|requestTimeoutMs||Request timeout (in ms) for the OpenSearch client.|
|index||Refer to Index for the index configuration options.|
|bulk||Refer to Bulk for the bulk configuration options.|
|authentication||Refer to Authentication for the authentication configuration options.|
|aws||Refer to AWS for the AWS configuration options.|
In most cases, you will not be interested in exporting every single record produced by a Zeebe cluster, but rather only a subset of them. This can also be configured to limit the kinds of records exported (e.g. only events, no commands), and the value type of these records (e.g. only job and process values).
|prefix||This prefix will be appended to every index created by the exporter; must not contain ||zeebe-record|
To avoid too many expensive requests to the OpenSearch cluster, the exporter performs batch updates by default. The size of the batch, along with how often it should be flushed (regardless of size) can be controlled by configuration.
|delay||Delay, in seconds, before force flush of the current batch. This ensures that even when we have low traffic of records, we still export every once in a while.|
|size||The amount of records a batch should have before we flush the batch.|
|memoryLimit||The size of the batch, in bytes, before we flush the batch.|
With the default configuration, the exporter would aggregate records and flush them to OpenSearch either:
- When it has aggregated 1000 records.
- When the batch memory size exceeds 10 MB.
- Five seconds have elapsed since the last flush (regardless of how many records were aggregated).
Providing these authentication options will enable Basic Authentication on the exporter.
|username||Username used to authenticate||N/A|
|password||Password used to authenticate||N/A|
When running OpenSearch in AWS, you may require requests to be signed. By enabling AWS in the configurations, a request interceptor will be added to the exporter. This interceptor will take care of signing the requests.
Signing requests requires credentials. These credentials are not directly configurable in the exporter. Instead, they are resolved by following the Default Credential Provider Chain.
|enabled||Enables AWS request signing|
|serviceName||AWS' name of the service to where requests are made. For OpenSearch this should be |
|region||The region this exporter is running in||The value of the |
Here is an example configuration of the exporter:
# Opensearch Exporter ----------
# An example configuration for the opensearch exporter:
# These setting can also be overridden using the environment variables "ZEEBE_BROKER_EXPORTERS_OPENSEARCH_..."
# A comma separated list of URLs pointing to the Opensearch instances you wish to export to.
# For example, if you want to connect to multiple nodes for redundancy:
# url: http://localhost:9200,http://localhost:9201