Version: 8.3

IAM roles for service accounts

IAM roles for service accounts (IRSA) is a way within AWS to authenticate workloads in Amazon EKS (Kubernetes), for example, to execute signed requests against AWS services. This is a replacement for basic auth and is generally considered a best practice by AWS.

The following considers the managed services by AWS and provided examples are in Terraform syntax.

Aurora PostgreSQL

Aurora PostgreSQL is a managed AWS PostgreSQL–compatible service.

Setup

When using the Terraform provider of AWS with the resource aws_rds_cluster to create a new rational database (RDS) or Aurora cluster, supply the argument iam_database_authentication_enabled = true to enable the IAM roles functionality. See the AWS documentation for availability and limitations.

AWS policy

An AWS policy (later assigned to a role) is required to allow assuming a database user within a managed database. See the AWS documentation for policy details.

Create the policy via Terraform using the aws_iam_policy.

resource "aws_iam_policy" "rds_policy" {
  name = "rds-policy"

  policy = jsonencode({
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:region:account-id:dbuser:DbiResourceId/db-user-name"
         ]
      }
   ]
})
}

IAM to Kubernetes mapping

To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful.

module "aurora_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = "aurora-role"

  role_policy_arns = {
    policy = aws_iam_policy.rds_policy.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
      namespace_service_accounts = ["aurora-namespace:aurora-serviceaccount"]
    }
  }
}

These two Terraform snippets allow the service account aurora-serviceaccount within the aurora-namespace to assume the user db-user-name within the database DbiResourceId. The output of the module aurora_role has the output iam_role_arn to annotate a service account to make use of the mapping.

Annotate the service account with the iam_role_arn output of the aurora_role.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
  name: aurora-serviceaccount
  namespace: aurora-namespace

Database configuration

The setup required on the Aurora PostgreSQL side is to create the user and assign the required permissions to it. The following is an example when connected to the PostgreSQL database, and can also be realized by using a Terraform PostgreSQL Provider. See the AWS documentation for reference concerning Aurora specific configurations.

# create user and grant rds_iam role, which requires the user to login via IAM authentication over password
CREATE USER "db-user-name";
GRANT rds_iam TO "db-user-name";

# create some database and grant the user all privileges to it
CREATE DATABASE "some-db";
GRANT ALL privileges on database "some-db" to "db-user-name";

Keycloak

caution

IAM Roles for Service Accounts can only be implemented with Keycloak 21 onwards. This may require you to adjust the version used in the Camunda Helm Chart.

For Keycloak versions 21+, the default JDBC driver can be overwritten, allowing use of a custom wrapper like the aws-advanced-jdbc-wrapper to utilize the features of IRSA. This is a wrapper around the default JDBC driver, but takes care of signing the requests.

The following example uses the mentioned aws-advanced-jdbc-wrapper. Additionally, see the Keycloak documentation on overwriting the default JDBC driver.

A custom Docker image is required as there's currently no upstream image with all the configurations required.

Dependencies

Required are the following software.amazon.awssdk artifacts for the aws-advanced-jdbc-wrapper to work:

The wrapper itself is available on GitHub.

Example usage

The following will use Gradle to retrieve the artifacts from Maven Central and build the final KeyCloak image with AWS IRSA support. This only requires Docker on your machine as everything is done within Docker by using multi-stage builds.

Create a file called build.gradle with the following content:

apply plugin: 'groovy'

repositories {
    mavenCentral()
}

def jdbcversion = '2.2.2'      // set to latest version of aws-advanced-jdbc-wrapper package
def awsSdkVersion = '2.20.107' // set to latest version of software.amazon.awssdk

dependencies {
    implementation group: 'software.amazon.jdbc', name: 'aws-advanced-jdbc-wrapper', version: jdbcversion
    implementation group: 'software.amazon.awssdk', name: 'apache-client', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'auth', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'aws-core', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'aws-json-protocol', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'aws-query-protocol', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'endpoints-spi', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'http-client-spi', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'json-utils', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'metrics-spi', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'profiles', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'protocol-core', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'rds', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'regions', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'sdk-core', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'sts', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'third-party-jackson-core', version: awsSdkVersion
    implementation group: 'software.amazon.awssdk', name: 'utils', version: awsSdkVersion
}

task copyDependencies(type: Copy) {
    from configurations.runtimeClasspath
    into "lib"
}

Create a file called Dockerfile with the following content in the same directory:

FROM gradle:jdk17-focal as lib

WORKDIR /home/gradle

COPY build.gradle /home/gradle

RUN gradle copyDependencies

FROM keycloak/keycloak:21.1 as builder

# Enable health and metrics support
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true

# Configure a database vendor
ENV KC_DB=postgres

COPY --from=lib /home/gradle/lib /opt/keycloak/providers

WORKDIR /opt/keycloak

RUN /opt/keycloak/bin/kc.sh build

FROM keycloak/keycloak:21.1

COPY --from=builder /opt/keycloak/ /opt/keycloak/

ENV KC_DB=postgres

ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]

Run the following command or variation with docker buildx to create and publish the required image.

docker build . --no-cache

Kubernetes configuration

As an example, configure the following environment variables

- name: KC_DB_DRIVER
  value: software.amazon.jdbc.Driver
- name: KC_DB_URL
  value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: KC_DB_USERNAME
  value: db-user-name
# The AWS wrapper is not capable of XA transactions
- name: KC_TRANSACTION_XA_ENABLED
  value: false

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Web Modeler

Since Web Modeler RestAPI uses PostgreSQL, configure the restapi to use IRSA with Amazon Aurora PostgreSQL. Check the Web Modeler database configuration for more details. Web Modeler already comes fitted with the aws-advanced-jdbc-wrapper within the Docker image.

Kubernetes configuration

As an example, configure the following environment variables

- name: SPRING_DATASOURCE_DRIVER_CLASS_NAME
  value: software.amazon.jdbc.Driver
- name: SPRING_DATASOURCE_URL
  value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: SPRING_DATASOURCE_USERNAME
  value: db-user-name

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

OpenSearch

AWS OpenSearch is a managed OpenSearch service provided by AWS, which is a distributed search and analytics engine built on Apache Lucene.

Setup

For OpenSearch, the most common use case is the use of fine-grained access control.

When using the Terraform provider of AWS with the resource opensearch_domain to create a new OpenSearch cluster, supply the argument advanced_security_options.enabled = true and set advanced_security_options.anonymous_auth_enabled = false to activate fine-grained access control.

Without fine-grained access control, anonymous access is enabled and would be sufficient to supply an IAM role with the right policy to allow access. In our case, we'll have a look at fine-grained access control and the use without it can be derived from this more complex example.

AWS Policy

An AWS policy, which later is assigned to a role, is required to allow general access to OpenSearch. See the AWS documentation for the explanation of the policy.

Create the policy via Terraform using the aws_iam_policy.

resource "aws_iam_policy" "opensearch_policy" {
  name = "opensearch_policy"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
        "es:DescribeElasticsearchDomains",
        "es:DescribeElasticsearchInstanceTypeLimits",
        "es:DescribeReservedElasticsearchInstanceOfferings",
        "es:DescribeReservedElasticsearchInstances",
        "es:GetCompatibleElasticsearchVersions",
        "es:ListDomainNames",
        "es:ListElasticsearchInstanceTypes",
        "es:ListElasticsearchVersions",
        "es:DescribeElasticsearchDomain",
        "es:DescribeElasticsearchDomainConfig",
        "es:ESHttpGet",
        "es:ESHttpHead",
        "es:GetUpgradeHistory",
        "es:GetUpgradeStatus",
        "es:ListTags",
        "es:AddTags",
        "es:RemoveTags",
        "es:ESHttpDelete",
        "es:ESHttpPost",
        "es:ESHttpPut"
        ],
        "Resource" : [
            "arn:aws:es:region:account-id:domain/test-domain/*"
        ]
    }
    ]
  })
}

IAM to Kubernetes mapping

To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful:

module "opensearch_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = "opensearch-role"

  role_policy_arns = {
    policy = aws_iam_policy.opensearch_policy.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
      namespace_service_accounts = ["opensearch-namespace:opensearch-serviceaccount"]
    }
  }
}

These two Terraform snippets will allow the service account opensearch-serviceaccount within the opensearch-namespace to generally access the AWS OpenSearch service for the test-domain cluster.

The output of the module opensearch_role has the output iam_role_arn to annotate a service account to use the mapping.

Annotate the service account with the iam_role_arn output of the opensearch_role.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
  name: opensearch-serviceaccount
  namespace: opensearch-namespace

Database configuration

This setup is sufficient for OpenSearch clusters without fine-grained access control.

Fine-grained access control adds another layer of security to OpenSearch, requiring you to add a mapping between the IAM role and the internal OpenSearch role. Visit the AWS documentation on fine-grained access control.

There are different ways to configure the mapping within OpenSearch:

Via a Terraform module in case your OpenSearch instance is exposed.
Via the OpenSearch dashboard.
Via the REST API.

The important part is assigning the iam_role_arn of the previously created opensearch_role to an internal role within OpenSearch. For example, all_access on the OpenSearch side is a good candidate, or if required, extra roles can be created with more restrictive access.

Operate

Configure Operate to use the feature set of IRSA for the OpenSearch Exporter. Check the Operate OpenSearch configuration.

Kubernetes configuration

As an example, configure the following environment variables:

- name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
  value: https://test-domain.region.es.amazonaws.com
- name: CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_URL
  value: https://test-domain.region.es.amazonaws.com

Where the value is whatever the endpoint of your OpenSearch cluster is.

note

AWS OpenSearch listens on port 443 opposed to the usual port 9200.

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Zeebe

Configure Zeebe to use the feature set of IRSA for the OpenSearch Exporter. Check the Zeebe OpenSearch exporter configuration.

Kubernetes configuration

As an example, configure the following environment variables:

- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_AWS_ENABLED
  value: true
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_CLASSNAME
  value: io.camunda.zeebe.exporter.opensearch.OpensearchExporter
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_URL
  value: https://test-domain.region.es.amazonaws.com

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Troubleshooting

Versions used

This page was created based on the following versions available and may work with newer releases of mentioned software.

Software	Version
AWS Aurora PostgreSQL	13 / 14 / 15
AWS JDBC Driver Wrapper	2.2.2
AWS OpenSearch	2.5
AWS SDK Dependencies	2.20.x
KeyCloak	21.x
Terraform AWS Provider	5.9.0
Terraform Amazon EKS Module	19.15.3
Terraform IAM Roles Module	5.28.0
Terraform PostgreSQL Provider	1.20.0

Instance Metadata Service (IMDS)

Instance Metadata Service is a default fallback for the AWS SDK. Within the context of Amazon EKS, it means a pod will automatically assume the role of a node. This can hide many problems, including whether IRSA was set up correctly or not, since it will fall back to IMDS in case of failure and hide the actual error.

Thus, if nothing within your cluster relies on the implicit node role, we recommend disabling it by defining in Terraform the http_put_response_hop_limit, for example.

Using a Terraform module like the Amazon EKS module, one can define the following to decrease the default value of two to one, which results in pods not being allowed to assume the role of the node anymore.

eks_managed_node_group_defaults {
    metadata_options = {
        http_put_response_hop_limit = 1
    }
}

Overall, this will disable the role assumption of the node for the Kubernetes pod. Depending on the resulting error within Operate, Zeebe, and Web-Modeler, you'll get a clearer error, which is helpful to debug the error more easily.

Aurora PostgreSQL​

Setup​

AWS policy​

IAM to Kubernetes mapping​

Database configuration​

Keycloak​

Dependencies​

Example usage​

Kubernetes configuration​

Web Modeler​

Kubernetes configuration​

OpenSearch​

Setup​

AWS Policy​

IAM to Kubernetes mapping​

Database configuration​

Operate​

Kubernetes configuration​

Zeebe​

Kubernetes configuration​

Troubleshooting​

Versions used​

Instance Metadata Service (IMDS)​

Aurora PostgreSQL

Setup

AWS policy

IAM to Kubernetes mapping

Database configuration

Keycloak

Dependencies

Example usage

Kubernetes configuration

Web Modeler

Kubernetes configuration

OpenSearch

Setup

AWS Policy

IAM to Kubernetes mapping

Database configuration

Operate

Kubernetes configuration

Zeebe

Kubernetes configuration

Troubleshooting

Versions used

Instance Metadata Service (IMDS)