Table of Contents

• Table of Contents
• CyberArk Secrets Provider for Kubernetes
  • Supported services
  • Using This Project With Conjur Open Source
  • Methods for Configuring Secrets Provider
  • Configuring Kubernetes Probes
  • Enabling Tracing
• Releases
  • Stable release definition
• Development
• Documentation
• Community
• License

CyberArk Secrets Provider for Kubernetes

The CyberArk Secrets Provider for Kubernetes provides Kubernetes-based applications with access to secrets that are stored and managed in CyberArk Secrets Manager.

Consuming Secrets from CyberArk Secrets Provider

Using the CyberArk Secrets Provider, your applications can easily consume secrets that have been retrieved from Secrets Manager in one of two ways:

• Using Kubernetes Secrets: The Secrets Provider can populate Kubernetes Secrets with secrets stored in Secrets Manager. This is sometimes referred to as "K8s Secrets" mode.
• Using Secrets files: The Secrets Provider can generate initialization or credentials files for your application based on secrets retrieved from Secrets Manager, and it can write those files to a volume that is shared with your application container. This is referred to as the Secrets Provider "Push to File" mode. For more information, see the Secrets Provider Push-to-File guide.

Deployment Modes

The Secrets Provider can be deployed into your Kubernetes cluster in one of two modes:

• As an init container: The Secrets Provider can be deployed as a Kubernetes init container for each of your application Pods that requires secrets to be retrieved from Secrets Manager. This configuration allows you to employ Secrets Manager policy that authorizes access to Secrets Manager secrets on a per-application-Pod basis.

• As a standalone application container (Kubernetes Job): The Secrets Provider can be deployed as a separate, application container that runs to completion as part of a Kubernetes Job. In this mode, the Secrets Provider can support delivery of Secrets Manager secrets to multiple application Pods. In this mode, you would use Secrets Manager policy that authorizes access to Secrets Manager secrets on a per-Secrets-Provider basis.

  The Secrets Provider Helm chart can be used to deploy the Secrets Provider in standalone application mode.

• As a sidecar to enable secrets rotation.

NOTE: If you are using the Secrets Provider "Push to file" mode, the Secrets Provider must be deployed as an init or sidecar container, since these modes makes use of shared volumes to deliver secrets to an application.

Supported Services

• Secrets Manager, Self-Hosted 11.1+

• Conjur Open Source v1.4.2+

Supported Platforms

• GKE

• K8s 1.11+

• Openshift v4.6-v4.8 (Secrets Manager, Self-Hosted only)

Using secrets-provider-for-k8s with Conjur Open Source

Are you using this project with Conjur Open Source? Then we strongly recommend choosing the version of this project to use from the latest Conjur OSS suite release. Conjur maintainers perform additional testing on the suite release versions to ensure compatibility. When possible, upgrade your Conjur version to match the latest suite release; when using integrations, choose the latest suite release that matches your Conjur version. For any questions, please contact us on Discourse.

Methods for Configuring CyberArk Secrets Provider

There are several methods available for configuring the CyberArk Secrets Provider:

• Using Pod Environment Variables: The Secrets Provider can be configured by setting environment variables in a Pod manifest. To see a description of the Secrets Provider environment variables, and an example manifest in the Set up Secrets Provider as an Init Container section of the Secrets Provider documentation (expand the collapsible section in Step 6 of this guide to see details).

• Using Pod Annotations: The Secrets Provider can be configured by setting Pod Annotations in a Pod manifest. For details on how Annotations can be used to configure the Secrets Provider, see the Secrets Provider Push-to-File guide.

• Using the Secrets Provider Helm chart (Standalone Application Mode Only) If you are using the Secrets Provider in standalone application mode, then you can configure the Secrets Provider by setting Helm chart values and deploying Secrets Provider using the Secrets Provider Helm chart.

Some notes about the different configuration methods:

1. For a setting that can be configured either by Pod Annotation or by environment variable, a Pod Annotation configuration takes precedence over the corresponding environment variable configuration.
2. If you are using the Secrets Provider in Push-to-File mode, then the Secrets Provider must be configured via Pod Annotations.
3. If you are using the Secrets Provider in Kubernetes Secrets mode, it is recommended that you use environment variable settings to configure the Secrets Provider.

Retry Behavior

The default retry count depends on container mode:

• Standalone mode: Unlimited retries (-1) by default. The provider keeps retrying until Conjur is available or configuration issues have been resolved. The readiness probe keeps the pod out of service until secrets are successfully provisioned.

• Init, application, and sidecar modes: Limited to 5 retries by default (DefaultRetryCountLimit).

• To change the retry behavior via environment variable, set RETRY_COUNT_LIMIT to an integer value.

• To change the retry behavior via Pod annotation, set conjur.org/retry-count-limit to an integer value.

• To configure via Helm, set environment.conjur.retryCountLimit in the chart values.

Note: set the retry count to -1 to indicate "unlimited" retries. Retries will continue indefinitely until success or process termination.

Configuring Kubernetes Probes

The Secrets Provider exposes the following probe endpoints:

• GET /healthz (liveness)
• GET /readyz (readiness)

When probe endpoints are enabled

Probe endpoints are created only when the Secrets Provider runs in standalone mode. Both k8s_secrets and push-to-file store types are supported.

In other modes (init, application, or sidecar), probe endpoints are not started.

Configure server bind address

To let Kubernetes reach the probe endpoints from outside the container process, set SERVER_ADDRESS (or annotation conjur.org/server-address) to a bindable address such as :8080 or 0.0.0.0:8080.

Avoid 127.0.0.1:8080 for Kubernetes probes, because kubelet probes the Pod IP and may not be able to reach a localhost-only bind.

Pod spec example

containers:
  - name: cyberark-secrets-provider-for-k8s
    image: cyberark/secrets-provider-for-k8s:latest
    env:
      - name: CONTAINER_MODE
        value: "standalone"
      - name: SECRETS_DESTINATION
        value: "k8s_secrets"
      - name: SERVER_ADDRESS
        value: ":8080"
    ports:
      - name: probes
        containerPort: 8080
    livenessProbe:
      httpGet:
        path: /healthz
        port: probes
      initialDelaySeconds: 5
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /readyz
        port: probes
      initialDelaySeconds: 5
      periodSeconds: 10

/healthz reports process health. /readyz reports provider readiness based on the latest secret provisioning results.

Label-based Secret Management

Beginning with v1.8.0, when running in Kubernetes Secrets mode, the Secrets Provider can automatically detect which secrets it should manage by checking for the label conjur.org/managed-by-provider: true on Kubernetes secrets in the same namespace. If a secret has this label with the value true, the Secrets Provider will automatically manage it.

This configuration only applies when both the K8S_SECRETS environment variable and the conjur.org/k8s-secrets annotation are not set. It requires list privileges on secrets within the namespace.

Note: It is strongly recommended to use this feature only with Secrets Manager versions that support the V2 Batch Retrieval API:

Secrets Manager Flavor	Version
SaaS	Latest
Self-Hosted	14.0+
Conjur OSS	1.25.0+

Without this API, a Kubernetes secret which is improperly configured or lacking necessary Secrets Manager privileges could cause the Secrets Provider to fail to update all secrets until resolved.

When using label-based secrets, if the Secrets Provider is running continuously (as a sidecar container) and has watch privileges on secrets within the namespace, it will detect when a labeled Kubernetes secret is added or updated and immediately provision the secret.

Sanitization Behavior in Label-based Mode

The Secrets Provider's behavior when keys are removed from the conjur-map depends on the conjur.org/remove-deleted-secrets-enabled annotation (or REMOVE_DELETED_SECRETS environment variable), which defaults to true.

When sanitization is enabled (true, default):

• If a key is removed from the conjur-map, the corresponding application secret will be deleted from the Kubernetes Secret's data.
• If the entire conjur-map is cleared or removed, all Conjur-managed application secrets will be deleted from the Kubernetes Secret.

When sanitization is disabled (false):

• Keys removed from the conjur-map will remain in the Kubernetes Secret's data.
• The Secrets Provider will not automatically clean up stale secrets.

Warning: When using label-based mode with sanitization enabled, if you move a secret from the conjur-map to the Kubernetes Secret's data field directly, the application secret will be removed on the first update after the key is removed from the conjur-map. This occurs because the Secrets Provider treats it as a stale Conjur-managed secret. This behavior only happens during the first update after the key is removed from the conjur-map.

To avoid this:

• Set conjur.org/remove-deleted-secrets-enabled: "false" before removing the key from the conjur-map, or
• If the secret has already been removed, reapply the Kubernetes Secret with the correct data to restore the value.

Enabling Tracing

Tracing of CyberArk Secrets Provider for Kubernetes is available using the OpenTelemetry standard. Tracing is disabled by default. You can enable tracing using either Pod Annotations or environment variables. To enable traces appended to the init container's logs, add the annoation conjur.org/log-traces: true to the Pod manifest, or set the LOG_TRACES environment variable to true. To instead export the traces to a Jaeger server, use the following annotation: conjur.org/jaeger-collector-url: http://<jaeger-collector-host>/api/traces or use the JAEGER_COLLECTOR_URL environment variable. Traces will include errors to assist in troubleshooting.

Releases

The primary source of CyberArk Secrets Provider for Kubernetes releases is our Dockerhub.

When we release a version, we push the following images to Dockerhub:

1. Latest
2. Major.Minor.Build
3. Major.Minor
4. Major

We also push the Major.Minor.Build image to our Red Hat registry.

Builds

We push the following tags to Dockerhub:

Latest - on every release the latest tag will be updated (cyberark/secrets-provider-for-k8s:latest). This tag means the Secrets Provider for Kubernetes meets the stability criteria detailed in the following section.

Semver - on every release a Semver tag will be pushed (cyberark/secrets-provider-for-k8s:1.1.0). This tag means the Secrets Provider for Kubernetes meets the stability criteria detailed in the following section.

Stable release definition

The CyberArk Secrets Provider for Kubernetes is considered stable when it meets the core acceptance criteria:

• Documentation exists that clearly explains how to set up and use the provider and includes troubleshooting information to resolve common issues.
• A suite of tests exist that provides excellent code coverage and possible use cases.
• The CyberArk Secrets Provider for Kubernetes has had a security review and all known high and critical issues have been addressed. Any low or medium issues that have not been addressed have been logged in the GitHub issue backlog with a label of the form security/X
• The CyberArk Secrets Provider for Kubernetes is easy to setup.
• The CyberArk Secrets Provider for Kubernetes is clear about known limitations and bugs, if they exist.

Development

We welcome contributions of all kinds to CyberArk Secrets Provider for Kubernetes. For instructions on how to get started and descriptions of our development workflows, see our contributing guide.

Documentation

You can find official documentation on our site.

Community

Interested in checking out more of our open source projects? See our open source repository!

License

The CyberArk Secrets Provider for Kubernetes is licensed under the Apache License 2.0 - see LICENSE for more details.