Skip to main content
Version: Next

Connector SDK

note

The Connector SDK is in developer preview and subject to breaking changes. Use at your own risk.

The Connector SDK allows you to develop custom Connectors using Java code. You can focus on the logic of the Connector, test it locally, and reuse its runtime logic in multiple environments. The SDK achieves this by abstracting from Camunda Platform 8 internals that usually come with job workers.

The SDK provides APIs for common Connector operations, such as:

  • Fetching and deserializing input data
  • Validating input data
  • Replacing secrets in input data

Additionally, the SDK allows for convenient testing of your Connector behavior and executing it in the environments that suit your use cases best.

Creating a custom Connector

Using the Connector SDK, you can create environment-agnostic and reusable Connector runtime behavior. This section outlines how to set up a Connector project, test it, and run it locally.

Setup

When developing a new Connector, we recommend using our custom Connector template repository on GitHub. This template is a Maven-based Java project, and can be used in various ways such as:

  • Create your own GitHub repository: Click Use this template and follow the prompted steps. You can manage code changes in your new repository afterward.
  • Experiment locally: Check out the source code to your local machine using Git. You won't be able to check in code changes to the repository due to restricted write access.
  • Fetch the source: Download the source code as a ZIP archive using Code > Download ZIP. You can adjust and manage the code the way you like afterward, using your chosen source code management tools.

To manually set up your Connector project, include the following dependency to use the SDK. Ensure you adhere to the project outline detailed in the next section.

<dependency>
<groupId>io.camunda.connector</groupId>
<artifactId>connector-core</artifactId>
<version>0.3.0</version>
</dependency>

Project outline

There are multiple parts of a Connector that enables it for reuse, as a reusable building block, for modeling, and for the runtime behavior. The following parts make up a Connector:

my-connector
├── element-templates/
│ └── template-connector.json (1)
├── src/main
│ ├── java/io/camunda/connector (2)
│ │ ├── MyConnectorFunction.java (3)
│ │ ├── MyConnectorRequest.java (4)
│ │ └── MyConnectorResult.java (5)
│ └── resources/META-INF/services
│ └── io.camunda.connector.api.outbound.OutboundConnectorFunction (6)
└── pom.xml (7)

For the modeling building blocks, the Connector provides Connector templates with (1).

You provide the runtime logic as Java source code under a directory like (2). Typically, a Connector runtime logic consists of the following:

  • Exactly one implementation of a OutboundConnectorFunction with (3).
  • At least one input data object like (4).
  • At least one result object like (5).

For a detectable Connector function, you are required to expose your function class name in the OutboundConnectorFunction SPI implementation with (6).

A configuration file like (7) manages the project setup, including dependencies. In this example, we include a Maven project's POM file. Other build tools like Gradle can also be used.

Connector template

To create reusable building blocks for modeling, you are required to provide a domain-specific Connector template.

A Connector template defines the binding to your Connector runtime behavior via the following object:

{
"type": "Hidden",
"value": "io.camunda:template:1",
"binding": {
"type": "zeebe:taskDefinition:type"
}
}

This type definition io.camunda:template:1 is the connection configuring which version of your Connector runtime behavior to use. In technical terms, this defines the Type of jobs created for tasks in your process model that use this template. Consult the job worker guide to learn more.

Besides the type binding, Connector templates also define the input variables of your Connector as zeebe:input objects. For example, you can create the input variable message of your Connector in the element template as follows:

{
"label": "Message",
"type": "Text",
"feel": "optional",
"binding": {
"type": "zeebe:input",
"name": "message"
}
}

You can also define nested data structures to reflect domain objects that group attributes. For example, you can create the domain object authentication that contains the properties user and token as follows:

{
"label": "Username",
"description": "The username for authentication.",
"type": "String",
"binding": {
"type": "zeebe:input",
"name": "authentication.user"
}
},
{
"label": "Token",
"description": "The token for authentication.",
"type": "String",
"binding": {
"type": "zeebe:input",
"name": "authentication.token"
}
}

You can deserialize these authentication properties into a domain object using the SDK. Visit the input data section for further details.

Connectors that offer any kind of result from their invocation should allow users to configure how to map the result into their processes. Therefore, Connector templates can reuse the two recommended objects, Result Variable and Result Expression:

{
"label": "Result Variable",
"description": "Name of variable to store the response in",
"type": "String",
"binding": {
"type": "zeebe:taskHeader",
"key": "resultVariable"
}
},
{
"label": "Result Expression",
"description": "Expression to map the response into process variables",
"type": "Text",
"feel": "required",
"binding": {
"type": "zeebe:taskHeader",
"key": "resultExpression"
}
}

These objects create custom headers for the jobs created for the tasks that use this template. The Connector runtime environments pick up those two custom headers and translate them into process variables accordingly. You can see an example of how to use this in the out-of-the-box REST Connector.

All Connectors are recommended to offer exception handling to allow users to configure how to map results and technical errors into BPMN errors. To provide this, Connector templates can reuse the recommended object Result Expression:

{
"label": "Error Expression",
"description": "Expression to define BPMN Errors to throw",
"group": "errors",
"type": "Text",
"feel": "required",
"binding": {
"type": "zeebe:taskHeader",
"key": "errorExpression"
}
}

This object creates custom headers for the jobs created for the tasks that use this template. The Connector runtime environments pick up this custom header and translate it into BPMN errors accordingly. You can see an example of how to use this in the BPMN errors in Connectors guide.

Runtime logic

To create a reusable runtime behavior for your Connector, you are required to implement and expose an implementation of the OutboundConnectorFunction interface of the SDK. The Connector runtime environments will call this function; it handles input data, executes the Connector's business logic, and optionally returns a result. Exception handling is optional since the Connector runtime environments handle this as a fallback.

The OutboundConnectorFunction interface consists of exactly one execute method. A minimal recommended outline of a Connector function implementation looks as follows:

package io.camunda.connector;

import io.camunda.connector.api.annotation.OutboundConnector;
import io.camunda.connector.api.error.ConnectorException;
import io.camunda.connector.api.outbound.OutboundConnectorContext;
import io.camunda.connector.api.outbound.OutboundConnectorFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@OutboundConnector(
name = "MYCONNECTOR",
inputVariables = {"myProperty", "authentication"},
type = "io.camunda:template:1"
)
public class MyConnectorFunction implements OutboundConnectorFunction {

private static final Logger LOGGER = LoggerFactory.getLogger(MyConnectorFunction.class);

@Override
public Object execute(OutboundConnectorContext context) throws Exception {
// (1)
var connectorRequest = context.getVariablesAsType(MyConnectorRequest.class);

// (2)
context.validate(connectorRequest);

// (3)
context.replaceSecrets(connectorRequest);

return executeConnector(connectorRequest);
}

private MyConnectorResult executeConnector(final MyConnectorRequest connectorRequest) {
LOGGER.info("Executing my connector with request {}", connectorRequest);
String message = connectorRequest.getMessage();
// (4)
if (message != null && message.toLowerCase().startsWith("fail")) {
throw new ConnectorException("FAIL", "My property started with 'fail', was: " + message);
}
var result = new MyConnectorResult();

// (5)
result.setMyProperty("Message received: " + message);
return result;
}
}

The execute method receives all necessary environment data via the OutboundConnectorContext object. The Connector runtime environment initializes the context and allows the following to occur:

  • Fetch and deserialize the input data as shown in (1). See the input data section for details.
  • Validate the created request object as shown in (2). See the validation section for details.
  • Replace secrets in the request object as shown in (3). See the secrets section for details.

If the Connector handles exceptional cases, it can use any exception to express technical errors. If a technical error should be associated with a specific error code, the Connector can throw a ConnectorException and define a code as shown in (4). We recommend documenting the list of error codes as part of the Connector's API. Users can build on those codes by creating BPMN errors in their Connector configurations.

If the Connector has a result to return, it can create a new result data object and set its properties as shown in (5).

For best interoperability, Connector functions provide default meta-data via the @OutboundConnector annotation. Connector runtime environments can use this data to auto-discover provided Connector runtime behavior.

Using this outline, you start the business logic of your Connector in the executeConnector method and expand from there.

Input data

The input data of a Connector is provided by the process instance that executes the Connector. You can either fetch this data as a raw JSON string using the context's getVariables method, or deserialize the data into your own request object directly with the getVariablesAsType method shown in (1).

Using getVariablesAsType will attempt to deserialize the JSON string containing the input data into Java objects. This deserialization depends on the Connector runtime environment your Connector function runs in.

Thus, use this deserialization approach with caution. While it works reliably for many input data types like string, boolean, integer, and nested objects, you might want to consider deserializing your Connector's input data in a custom fashion using getVariables and a library like Gson.

The getVariablesAsType method and tools like Gson can properly reflect nested data objects. You can define nested structures by referencing other Java classes as attributes. Looking at the authentication data input example described in the Connector template, you can create the following input data objects to reflect the structure properly:

package io.camunda.connector;

public class MyConnectorRequest {

private String message;
private Authentication authentication;
}
package io.camunda.connector;

public class Authentication {

private String user;
private String token;
}

Validation

Validating input data is a common task in a Connector function. The SDK provides an API to help you ensure the data conforms to your Connector's input requirements. A default implementation of the SDK's core validation API is provided in a separate, optional artifact connector-validation. If you want to use validation in your Connector, add the following dependency to your project:

<dependency>
<groupId>io.camunda.connector</groupId>
<artifactId>connector-validation</artifactId>
<version>0.3.0</version>
</dependency>

To initiate the validation from the Connector function, use the OutboundConnectorContext object's validate method as shown in the runtime logic section:

...
@Override
public Object execute(OutboundConnectorContext context) throws Exception {
...
// (2)
context.validate(connectorRequest);
...
}
...

This instructs the context to prepare a validator that is provided by an implementation of the ValidationProvider interface. The connector-validation artifact brings along such an implementation. It uses the Jakarta Bean Validation API together with Hibernate Validator.

To validate your input object connectorRequest using the API, you need to annotate the input's attributes to define your requirements:

package io.camunda.connector;

import javax.validation.Valid;
import javax.validation.constraints.NotEmpty;
import javax.validation.constraints.NotNull;

public class MyConnectorRequest {

@NotEmpty private String message;
@NotNull @Valid private Authentication authentication;
}

The Jakarta Bean Validation API comes with a long list of supported constraints. It also allows to validate entire object graphs using the @Valid annotation. Thus, the authentication object will also be validated.

package io.camunda.connector;


import javax.validation.constraints.NotEmpty;

public class Authentication {

@NotEmpty private String user;

@NotEmpty @Pattern(regexp = "^xobx") private String token;
}

Using this approach, you can validate your whole input data structure with one initial call from the central Connector function.

Beyond that, the Jakarta Bean Validation API supports more advanced constructs like groups for conditional validation and constraints on different types, i.e., attributes, methods, and classes, to enable cross-parameter validation. You can use the built-in constraints and create custom ones to define requirements exactly as you need them.

If the validation approach that comes with connector-validation doesn't fit your needs, you can provide your own SPI implementing the SDK's ValidationProvider interface. Have a look at the connector validation code for a default implementation.

Conditional validation

Validating Connector input data can require to check different constraints, depending on the specific input data itself. As an example, the following authentication input object requires that oauthToken is only necessary when the type is oauth. If the type is basic, the attribute password is required instead.

public class Authentication {

private String type;
private String user;
private String password;
private String oauthToken;
}

Using the connector-validation module, there are three common options to achieve this conditional validation:

  1. Write a custom constraint that allows to validate one attribute in relation to another attribute. This appraoch yields a reusable constraint that you can use in other classes as well. This approach also comes with the highest implementation effort.
  2. Write manual, imperative validation logic in a method with a boolean return value and annotate it with @AssertTrue. You require less code to take this appraoch but the result is also specifc to the respective class. You cannot reuse the logic in other classes as is. This approach also comes without further constraint annotation support. You have to write all validation logic manually in the method.
  3. Define validation groups dynamically with Hibernate Validator's @DefaultGroupSequenceProvider. This appraoch allows to reuse existing constraint annotations and to only apply them for specific use cases. It has a higher complexity than an imperative validation method but allows to reuse existing constraints to avoid writing manual validation logic.

Each option has its own benefits and drawbacks, depending on what you need in your Connector. The following sections cover each of the options in more detail.

Custom constraint

The Bean Validation guide covers defining custom constraints extensively. For the use case described above, you could write a custom constraint like the following:

@Target({TYPE, ANNOTATION_TYPE})
@Retention(RUNTIME)
@Repeatable(NotNullIfAnotherFieldHasValue.List.class)
@Constraint(validatedBy = NotNullIfAnotherFieldHasValueValidator.class)
@Documented
public @interface NotNullIfAnotherFieldHasValue {

String fieldName();
String fieldValue();
String dependFieldName();

String message() default "{NotNullIfAnotherFieldHasValue.message}";
Class<?>[] groups() default {};
Class<? extends Payload>[] payload() default {};

@Target({TYPE, ANNOTATION_TYPE})
@Retention(RUNTIME)
@Documented
@interface List {
NotNullIfAnotherFieldHasValue[] value();
}

}

You can use this constraint on the Connector input object as follows:

@NotNullIfAnotherFieldHasValue(
fieldName = "type",
fieldValue = "oauth",
dependFieldName = "oauthToken")
@NotNullIfAnotherFieldHasValue(
fieldName = "type",
fieldValue = "basic",
dependFieldName = "password")
public class Authentication {

@NotEmpty
private String type;
@NotEmpty
private String user;
private String password;
private String oauthToken;
}

You can find more details and the NotNullIfAnotherFieldHasValueValidator implementation in this StackOverflow thread.

This approach is the most flexible and reusable one for writing conditional constraints. It is independent from the parameters and classes involved. However, for simple use cases, one of the following approaches might lead to more maintainable results that require less code.

Manual validation method

The Jakarta Bean Validation API comes with an AssertTrue constraint that you can use to ensure boolean attributes are enabled.

The nature of the bean validation API allows to also use this annotation on methods. Those usually are the getter methods for boolean attributes. However, there doesn't have to be a related boolean attribute in an object in order to validate a method constraint. Thus, you can use this constraint to also write manual validation logic in a method that returns a boolean value and starts with is.

For the example use case, you can write a method that verifies the requirements as follows:

public class Authentication {

@NotEmpty private String type;
@NotEmpty private String user;
private String password;
private String oauthToken;

@AssertTrue(message = "Authentication must contain 'oauthToken' for type 'oauth' and 'password' for type 'basic'")
public boolean isAuthValid() {
return ("basic".equals(type) && password != null) ||
("oauth".equals(type) && oauthToken != null);
}
}

This approach allows for concise conditional validation when the constraint logic is simple and does not justify creating more complex, reusable interfaces and validators.

Dynamic validation groups

The Jakarta Bean Validation API allows to statically define validation groups for conditional constraint evaluation. However, to use those groups you have to define the group to validate statically when starting the validation. To dynamically define the groups to validate, you can use Hibernate Validator's DefaultGroupSequenceProvider.

Given the following validation groups:

public interface BasicAuthValidation {}
public interface OAuthValidation {}

You can annotate the input object as follows:

@GroupSequenceProvider(AuthenticationSequenceProvider.class)
public class Authentication {

@NotEmpty private String type;
@NotEmpty private String user;
@NotEmpty(groups = BasicAuthValidation.class)
private String password;
@NotEmpty(groups = OAuthValidation.class)
private String oauthToken;

The AuthenticationSequenceProvider needs to implement the DefaultGroupSequenceProvider to dynamically add the validation groups you need:

public class AuthenticationSequenceProvider implements DefaultGroupSequenceProvider<Authentication> {

@Override
public List<Class<?>> getValidationGroups(Authentication authentication) {

List<Class<?>> sequence = new ArrayList<>();

// Apply all validation rules from Default group, e.g. ensuring type is not empty
sequence.add(Authentication.class);

if ("basic".equals(authentication.getType())) {
sequence.add(BasicAuthValidation.class);
} else if ("oauth".equals(authentication.getType())) {
sequence.add(OAuthValidation.class);
}

return sequence;
}
}

Using this approach, you can reuse existing constraint annotations in your input objects. The sequence provider is however bound to your specific input class and therefore less reusable than writing custom constraints.

Secrets

Connectors that require confidential information to connect to external systems need to be able to manage those securely. As described in the guide for creating secrets, secrets can be controlled in a secure location and referenced in a Connector's properties using a placeholder pattern secrets.*. To make this mechanism as robust as possible, secret handling comes with the Connector SDK out of the box. That way, all Connectors can use the same standard way of handling secrets in input data.

The SDK allows replacing secrets in input data as late as possible to avoid passing them around in the environments that handle Connector invocation. We do not pass secrets into the Connector function in clear text but only as placeholders that you can replace from within the Connector function.

To initiate the secret replacement from the Connector function, use the OutboundConnectorContext object's replaceSecrets method as shown in the runtime logic section:

...
@Override
public Object execute(OutboundConnectorContext context) throws Exception {
...
// (3)
context.replaceSecrets(connectorRequest);
...
}
...

This will instruct the context to search for and replace secret placeholders in the object passed to it. The secret store present in the Connector runtime environment that invokes the Connector function takes care of that. Every environment can define its own way of providing such a secret store. Consult the runtime environments section to learn more about them.

To replace secrets in the connectorRequest using the API, you need to annotate the object's attributes that can hold secrets with @Secret. This instructs the SDK to search for and replace secrets in the annotated properties. You can use the @Secret annotation on String fields and container types. Using the annotation on container objects will lead to graph traversal until either no further matching annotation is found or the annotated property is of type String. Annotating properties that are neither a String nor a container will lead to runtime exceptions.

package io.camunda.connector;

import io.camunda.connector.api.annotation.Secret;

public class MyConnectorRequest {

private String message;
@Secret private Authentication authentication;
}
package io.camunda.connector;

import io.camunda.connector.api.annotation.Secret;

public class Authentication {

private String user;
@Secret private String token;
}

Using this approach, you can replace secrets in your whole input data structure with one initial call from the central Connector function.

Testing

Ensuring your Connector's business logic works as expected is vital to develop the Connector. The SDK aims to make testing of Connectors convenient without imposing strict requirements on your test development flow. The SDK is not enforcing any testing libraries.

By abstracting from Camunda Platform 8 internals, the SDK provides a good starting ground for scoped testing. There is no need to test Camunda engine internals or provide related mocks. You can focus on testing the business logic of your Connector and the associated objects.

We recommend testing at least the following parts of your Connector project:

  • All data validation works as expected.
  • All expected attributes support secret replacement.
  • The core logic of your Connector works as expected until calling the external API or service.

The SDK provides a OutboundConnectorContextBuilder for test cases that lets you create a OutboundConnectorContext. You can conveniently use that test context to test the secret replacement and validation routines.

Writing secret replacement tests can look similar to the following test case. You can write one test case for each attribute that supports secret replacement:

@Test
void shouldReplaceTokenSecretWhenReplaceSecrets() {
// given
var input = new MyConnectorRequest();
var auth = new Authentication();
input.setMessage("Hello World!");
input.setAuthentication(auth);
auth.setToken("secrets.MY_TOKEN");
auth.setUser("testuser");

// (1)
var context = OutboundConnectorContextBuilder.create()
.secret("MY_TOKEN", "token value")
.build();
// when
context.replaceSecrets(input);
// then
assertThat(input)
.extracting("authentication")
.extracting("token")
.isEqualTo("token value");
}

Ensuring validation routines work as expected can be written similarly for every attribute that is required:

@Test
void shouldFailWhenValidate_NoAuthentication() {
// given
var input = new MyConnectorRequest();
input.setMessage("Hello World!");
var context = OutboundConnectorContextBuilder.create().build();
// when
assertThatThrownBy(() -> context.validate(input))
// then
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("authentication");
}

Testing custom validations works in the same way:

@Test
void shouldFailWhenValidate_TokenWrongPattern() {
// given
var input = new MyConnectorRequest();
var auth = new Authentication();
input.setMessage("foo");
input.setAuthentication(auth);
auth.setUser("testuser");
auth.setToken("test");
var context = OutboundConnectorContextBuilder.create().build();
// when
assertThatThrownBy(() -> context.validate(input))
// then
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("Token must start with \"xobx\"");
}

Testing the business logic of your Connector can vary widely depending on the functionality it provides. For our example logic, the following test would be a good start:

@Test
void shouldReturnReceivedMessageWhenExecute() throws Exception {
// given
var input = new MyConnectorRequest();
var auth = new Authentication();
input.setMessage("Hello World!");
input.setAuthentication(auth);
auth.setToken("xobx-test");
auth.setUser("testuser");
var function = new MyConnectorFunction();
var context = OutboundConnectorContextBuilder.create()
.variables(input)
.build();
// when
var result = function.execute(context);
// then
assertThat(result)
.isInstanceOf(MyConnectorResult.class)
.extracting("myProperty")
.isEqualTo("Message received: Hello World!");
}

Runtime environments

The Connector SDK enables you to write environment-agnostic runtime behavior for Connectors. This makes the Connector logic reusable in different setups without modifying your Connector code. To invoke this logic, you need a runtime environment that knows the Connector function and how to call it.

In Camunda Platform 8 SaaS, every cluster runs a component that knows the available out-of-the-box connectors and how to invoke them. This component is the runtime environment specific to Camunda's SaaS use case.

Regarding Self-Managed environments, you are responsible for providing the runtime environment that can invoke the Connectors. The Connector SDK provides a pre-packaged environment and means to create a custom environment to make this situation as convenient as possible.

Pre-packaged runtime environment

The SDK comes with a pre-packaged runtime environment that allows you to run select Connector runtimes as local job workers out-of-the-box. You can find this Java application on Maven Central.

Refer to the Self-Managed installation guide for details on how to set up this runtime environment.

Custom runtime environment

If using the pre-packaged runtime environment that comes with the SDK does not fit your use case, you can create a custom runtime environment. There are three options that come with the SDK:

  • Create a custom job worker using Spring Zeebe.
  • Wrap Connector functions as job workers using the ConnectorJobHandler.
  • Implement your own Connector function wrapper.

Spring Zeebe

Being a Spring-wrapper around the Zeebe Java client, Spring Zeebe supports running outbound Connectors out of the box. You can expose them as Spring beans in your Spring application and Spring Zeebe picks them up and runs them automatically. Using this approach, you can run multiple Connector functions in one Java application.

Spring Zeebe uses the ConnectorJobHandler and thus supports all functionality the pre-packaged environment provides as well. It allows you to reuse this functionality in your own Spring or Spring Boot-based setup.

Connector job handler

To wrap Connector functions as job workers, the SDK provides the wrapper class ConnectorJobHandler. Spring Zeebe uses this handler as detailed above and the pre-packaged environment packages a Spring Zeebe application to provide its functionality.

The job handler wrapper provides the following benefits:

  • Provides an OutboundConnectorContext that handles the Camunda-internal job worker API regarding variables.
  • Handles secret management by defaulting to an environment variables-based secret store and allowing to provide a custom secret provider via an SPI for io.camunda.connector.api.secret.SecretProvider.
  • Handles Connector result mapping for Result Variable and Result Expression as described in the Connector template section.
  • Provides flexible BPMN error handling via Error Expression as described in the Connector template section.

Using the wrapper class, you can create a custom Zeebe client. For example, you can spin up a custom client with the Zeebe Java client as follows:

import io.camunda.connector.MyConnectorFunction
import io.camunda.connector.runtime.jobworker.outbound.ConnectorJobHandler;
import io.camunda.zeebe.client.ZeebeClient;

public class Main {

public static void main(String[] args) {

var zeebeClient = ZeebeClient.newClientBuilder().build();

zeebeClient.newWorker()
.jobType("io.camunda:template:1")
.handler(new ConnectorJobHandler(new MyConnectorFunction()))
.name("MESSAGE")
.fetchVariables("authentication", "message")
.open();
}
}

Custom function wrapper

If the provided job handler wrapper does not fit your needs, you can extend or replace it with your job handler implementation that handles invoking the Connector functions.

Your custom job handler needs to create a OutboundConnectorContext that the Connector function can use to handle variables, secrets, and Connector results. You can extend the provided io.camunda.connector.impl.outbound.AbstractConnectorContext to quickly gain access to most of the common context operations.