Integrating HCP Vault with Kubernetes v1.24 +

HashiCorp Vault is a powerful secret management tool that allows secure management and access control for secrets of all types, secret rotation, and lifecycle management.

Recently, while working on a demo showcasing the integration of Hashicorp HCP Vault and AWS EKS, I encountered some issues, even after following the available official documentation.

After tracing my steps, I realized that I was using Kubernetes version 1.24 instead of version 1.23, which I had previously used without any issues. Upon further research, I discovered that Kubernetes version 1.24 introduced many new features and changes, one of which was that service accounts are no longer automatically assigned a JWT secret and certificate.

This change affected the proper installation of the Vault agent, which is used to inject desired secrets into pods as a sidecar. The Vault agent also relies on a service account named “vault,” created by the Helm chart provided by Hashicorp specifically for this purpose. However, because the service account is no longer automatically assigned a secret, the Vault-K8s authorization backend does not have anything to bind to, and therefore does not work.

To resolve this issue, we need to create a secret and associate it with the Vault service account. We can achieve this by using the following YAML:

After creating the secret, we can fetch the required information to configure the Vault-K8s auth backend using the following commands:

Finally, we can configure the Vault-K8s auth backend using the following command:

Note that the token_reviewer_jwt parameter requires the token reviewer JWT, which only exists once a secret is created and assigned to the user account. All the other parameters are not affected by the version change and are still available.

In conclusion, while integrating Hashicorp Vault with Kubernetes 1.24+ can pose some challenges, following the steps outlined in this post should help you overcome them. For more information, please refer to the official Hashicorp Vault documentation.

Written by: David Gidony, DevSecOps Solution Architect