Version: 8.5

IAM roles for service accounts

IAM roles for service accounts (IRSA) is a way within AWS to authenticate workloads in Amazon EKS (Kubernetes), for example, to execute signed requests against AWS services. This is a replacement for basic auth and is generally considered a best practice by AWS.

The following considers the managed services by AWS and provided examples are in Terraform syntax.

Aurora PostgreSQL

Aurora PostgreSQL is a managed AWS PostgreSQL–compatible service.

Setup

When using the Terraform provider of AWS with the resource aws_rds_cluster to create a new rational database (RDS) or Aurora cluster, supply the argument iam_database_authentication_enabled = true to enable the IAM roles functionality. See the AWS documentation for availability and limitations.

AWS policy

An AWS policy (later assigned to a role) is required to allow assuming a database user within a managed database. See the AWS documentation for policy details.

Create the policy via Terraform using the aws_iam_policy.

resource "aws_iam_policy" "rds_policy" {
  name = "rds-policy"

  policy = jsonencode({
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:region:account-id:dbuser:DbiResourceId/db-user-name"
         ]
      }
   ]
})
}

IAM to Kubernetes mapping

To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful.

module "aurora_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = "aurora-role"

  role_policy_arns = {
    policy = aws_iam_policy.rds_policy.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
      namespace_service_accounts = ["aurora-namespace:aurora-serviceaccount"]
    }
  }
}

These two Terraform snippets allow the service account aurora-serviceaccount within the aurora-namespace to assume the user db-user-name within the database DbiResourceId. The output of the module aurora_role has the output iam_role_arn to annotate a service account to make use of the mapping.

Annotate the service account with the iam_role_arn output of the aurora_role.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
  name: aurora-serviceaccount
  namespace: aurora-namespace

Database configuration

The setup required on the Aurora PostgreSQL side is to create the user and assign the required permissions to it. The following is an example when connected to the PostgreSQL database, and can also be realized by using a Terraform PostgreSQL Provider. See the AWS documentation for reference concerning Aurora specific configurations.

# create user and grant rds_iam role, which requires the user to login via IAM authentication over password
CREATE USER "db-user-name";
GRANT rds_iam TO "db-user-name";

# create some database and grant the user all privileges to it
CREATE DATABASE "some-db";
GRANT ALL privileges on database "some-db" to "db-user-name";

Keycloak

caution

IAM Roles for Service Accounts can only be implemented with Keycloak 21 onwards. This may require you to adjust the version used in the Camunda Helm Chart.

From Keycloak versions 21+, the default JDBC driver can be overwritten, allowing use of a custom wrapper like the aws-advanced-jdbc-wrapper to utilize the features of IRSA. This is a wrapper around the default JDBC driver, but takes care of signing the requests.

Furthermore, the official Keycloak documentation also provides detailed instructions for utilizing Amazon Aurora PostgreSQL.

A custom Keycloak container image containing necessary configurations is conveniently accessible on Docker Hub at camunda/keycloak. This image, built upon the base image bitnami/keycloak, incorporates the required wrapper for seamless integration.

Container image sources

The sources of the Camunda Keycloak images can be found on GitHub. In this repository, the aws-advanced-jdbc-wrapper is assembled in the Dockerfile.

Maintenance of these images is based on the upstream Bitnami Keycloak images, ensuring they are always up-to-date with the latest Keycloak releases. The lifecycle details for Keycloak can be found on endoflife.date.

Keycloak image configuration

Bitnami Keycloak container image configuration is available at hub.docker.com/bitnami/keycloak.

Kubernetes configuration

As an example, configure the following environment variables to enable IRSA:

# The AWS wrapper is not capable of XA transactions
- name: KEYCLOAK_EXTRA_ARGS
  value: "--db-driver=software.amazon.jdbc.Driver --transaction-xa-enabled=false --log-level=INFO,software.amazon.jdbc:INFO"

# Enable the AWS IAM plugin
- name: KEYCLOAK_JDBC_PARAMS
  value: "wrapperPlugins=iam"
- name: KEYCLOAK_JDBC_DRIVER
  value: "aws-wrapper:postgresql"

# Configure database
- name: KEYCLOAK_DATABASE_USER
  value: db-user-name
- name: KEYCLOAK_DATABASE_NAME
  value: db-name
- name: KEYCLOAK_DATABASE_HOST
  value: db-host
- name: KEYCLOAK_DATABASE_PORT
  value: 5432

# Ref: https://www.keycloak.org/server/configuration-metrics
- name: KEYCLOAK_ENABLE_STATISTICS
  value: "true"

# Needed to see if Keycloak is healthy: https://www.keycloak.org/server/health
- name: KEYCLOAK_ENABLE_HEALTH_ENDPOINTS
  value: "true"

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Helm chart

For a Helm-based deployment, you can directly configure these settings using Helm values. Below is an example of how you can incorporate these settings into your Helm chart deployment:

identityKeycloak:
  postgresql:
    enabled: false
  image: docker.io/camunda/keycloak:23 # use a supported and updated version listed at https://hub.docker.com/r/camunda/keycloak/tags
  extraEnvVars:
    - name: KEYCLOAK_EXTRA_ARGS
      value: "--db-driver=software.amazon.jdbc.Driver --transaction-xa-enabled=false --log-level=INFO,software.amazon.jdbc:INFO"
    - name: KEYCLOAK_JDBC_PARAMS
      value: "wrapperPlugins=iam"
    - name: KEYCLOAK_JDBC_DRIVER
      value: "aws-wrapper:postgresql"
  externalDatabase:
    host: "aurora.rds.your.domain"
    port: 5432
    user: keycloak
    database: keycloak

note

For additional details, refer to the Camunda 8 Helm deployment documentation.

Web Modeler

Since Web Modeler RestAPI uses PostgreSQL, configure the restapi to use IRSA with Amazon Aurora PostgreSQL. Check the Web Modeler database configuration for more details. Web Modeler already comes fitted with the aws-advanced-jdbc-wrapper within the Docker image.

Kubernetes configuration

As an example, configure the following environment variables

- name: SPRING_DATASOURCE_DRIVER_CLASS_NAME
  value: software.amazon.jdbc.Driver
- name: SPRING_DATASOURCE_URL
  value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: SPRING_DATASOURCE_USERNAME
  value: db-user-name

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Identity

Since Identity uses PostgreSQL, configure identity to use IRSA with Amazon Aurora PostgreSQL. Check the Identity database configuration for more details. Identity already comes fitted with the aws-advanced-jdbc-wrapper within the Docker image.

Kubernetes configuration

As an example, configure the following environment variables

- name: SPRING_DATASOURCE_DRIVER_CLASS_NAME
  value: software.amazon.jdbc.Driver
- name: SPRING_DATASOURCE_URL
  value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: SPRING_DATASOURCE_USERNAME
  value: db-user-name

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

OpenSearch

AWS OpenSearch is a managed OpenSearch service provided by AWS, which is a distributed search and analytics engine built on Apache Lucene.

note

As of the 8.4 release, Zeebe, Operate, and Tasklist are now compatible with Amazon OpenSearch 2.5.x. Note that using Amazon OpenSearch requires setting up a new Camunda installation. A migration from previous versions or Elasticsearch environments is currently not supported.

caution

Optimize is not supported using the IRSA method. However, Optimize can be utilized by supplying a username and password. The migration step must also be disabled. For more information, refer to using AWS managed OpenSearch.

Setup

For OpenSearch, the most common use case is the use of fine-grained access control.

When using the Terraform provider of AWS with the resource opensearch_domain to create a new OpenSearch cluster, supply the arguments:

advanced_security_options.enabled = true
advanced_security_options.anonymous_auth_enabled = false to activate fine-grained access control.

Without fine-grained access control, anonymous access is enabled and would be sufficient to supply an IAM role with the right policy to allow access. In our case, we'll have a look at fine-grained access control and the use without it can be derived from this more complex example.

AWS Policy

An AWS policy, which later is assigned to a role, is required to allow general access to OpenSearch. See the AWS documentation for the explanation of the policy.

Create the policy via Terraform using the aws_iam_policy.

resource "aws_iam_policy" "opensearch_policy" {
  name = "opensearch_policy"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
        "es:DescribeElasticsearchDomains",
        "es:DescribeElasticsearchInstanceTypeLimits",
        "es:DescribeReservedElasticsearchInstanceOfferings",
        "es:DescribeReservedElasticsearchInstances",
        "es:GetCompatibleElasticsearchVersions",
        "es:ListDomainNames",
        "es:ListElasticsearchInstanceTypes",
        "es:ListElasticsearchVersions",
        "es:DescribeElasticsearchDomain",
        "es:DescribeElasticsearchDomainConfig",
        "es:ESHttpGet",
        "es:ESHttpHead",
        "es:GetUpgradeHistory",
        "es:GetUpgradeStatus",
        "es:ListTags",
        "es:AddTags",
        "es:RemoveTags",
        "es:ESHttpDelete",
        "es:ESHttpPost",
        "es:ESHttpPut"
        ],
        "Resource" : [
            "arn:aws:es:region:account-id:domain/test-domain/*"
        ]
    }
    ]
  })
}

IAM to Kubernetes mapping

To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful:

module "opensearch_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = "opensearch-role"

  role_policy_arns = {
    policy = aws_iam_policy.opensearch_policy.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
      namespace_service_accounts = ["opensearch-namespace:opensearch-serviceaccount"]
    }
  }
}

These two Terraform snippets will allow the service account opensearch-serviceaccount within the opensearch-namespace to generally access the AWS OpenSearch service for the test-domain cluster.

The output of the module opensearch_role has the output iam_role_arn to annotate a service account to use the mapping.

Annotate the service account with the iam_role_arn output of the opensearch_role.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
  name: opensearch-serviceaccount
  namespace: opensearch-namespace

This step is required to be repeated for Tasklist and Zeebe, to grant their service accounts access to OpenSearch.

Database configuration

This setup is sufficient for OpenSearch clusters without fine-grained access control.

Fine-grained access control adds another layer of security to OpenSearch, requiring you to add a mapping between the IAM role and the internal OpenSearch role. Visit the AWS documentation on fine-grained access control.

There are different ways to configure the mapping within OpenSearch:

Via a Terraform module in case your OpenSearch instance is exposed.
Via the OpenSearch dashboard.
Via the REST API.

The important part is assigning the iam_role_arn of the previously created opensearch_role to an internal role within OpenSearch. For example, all_access on the OpenSearch side is a good candidate, or if required, extra roles can be created with more restrictive access.

Camunda 8 Self-Managed Helm chart configuration

The following is an example configuration that can be used to configure the Camunda 8 Self-Managed Helm chart to use the feature set of IRSA for the OpenSearch Exporter:

global:
  elasticsearch:
    enabled: false
  opensearch:
    enabled: true
    aws:
      enabled: true
    url:
      protocol: https
      host: aws.opensearch.example.com
      port: 443

elasticsearch:
  enabled: false

optimize:
  enabled: false

note

AWS OpenSearch listens on port 443 opposed to the usual port 9200.

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Troubleshooting

Versions used

This page was created based on the following versions available and may work with newer releases of mentioned software.

Software	Version
AWS Aurora PostgreSQL	13 / 14 / 15
AWS JDBC Driver Wrapper	2.3.1
AWS OpenSearch	2.5
AWS SDK Dependencies	2.21.x
KeyCloak	21.x / 22.x
Terraform AWS Provider	5.29.0
Terraform Amazon EKS Module	19.20.0
Terraform IAM Roles Module	5.32.0
Terraform PostgreSQL Provider	1.21.0

Instance Metadata Service (IMDS)

Instance Metadata Service is a default fallback for the AWS SDK due to the default credentials provider chain. Within the context of Amazon EKS, it means a pod will automatically assume the role of a node. This can hide many problems, including whether IRSA was set up correctly or not, since it will fall back to IMDS in case of failure and hide the actual error.

Thus, if nothing within your cluster relies on the implicit node role, we recommend disabling it by defining in Terraform the http_put_response_hop_limit, for example.

Using a Terraform module like the Amazon EKS module, one can define the following to decrease the default value of two to one, which results in pods not being allowed to assume the role of the node anymore.

eks_managed_node_group_defaults {
    metadata_options = {
        http_put_response_hop_limit = 1
    }
}

Overall, this will disable the role assumption of the node for the Kubernetes pod. Depending on the resulting error within Operate, Zeebe, and Web-Modeler, you'll get a clearer error, which is helpful to debug the error more easily.

Aurora PostgreSQL​

Setup​

AWS policy​

IAM to Kubernetes mapping​

Database configuration​

Keycloak​

Container image sources​

Keycloak image configuration​

Kubernetes configuration​

Helm chart​

Web Modeler​

Kubernetes configuration​

Identity​

Kubernetes configuration​

OpenSearch​

Setup​

AWS Policy​

IAM to Kubernetes mapping​

Database configuration​

Camunda 8 Self-Managed Helm chart configuration​

Troubleshooting​

Versions used​

Instance Metadata Service (IMDS)​

Aurora PostgreSQL

Setup

AWS policy

IAM to Kubernetes mapping

Database configuration

Keycloak

Container image sources

Keycloak image configuration

Kubernetes configuration

Helm chart

Web Modeler

Kubernetes configuration

Identity

Kubernetes configuration

OpenSearch

Setup

AWS Policy

IAM to Kubernetes mapping

Database configuration

Camunda 8 Self-Managed Helm chart configuration

Troubleshooting

Versions used

Instance Metadata Service (IMDS)