Full Report
KubeCon Europe is the largest open-source community conference in Europe with hundreds of talks. We picked our favorite Kubernetes security sessions available online.
Analysis Summary
# Best Practices: Kubernetes Security Hardening and Operations
## Overview
These practices are derived from key sessions at KubeCon + CloudNativeCon Europe 2023, focusing on practical security enhancements for Kubernetes environments. The recommendations cover secret management, workload identity, network segmentation, least privilege enforcement, and modern security control adoption, particularly the migration away from deprecated mechanisms like Pod Security Policies (PSP).
## Key Recommendations
### Immediate Actions
1. **Migrate from Pod Security Policies (PSP) to Pod Security Standards (PSS):** Immediately cease reliance on PSPs and implement PSS to define security levels for Pods.
2. **Audit Root Usage:** Identify and flag all running containers that execute as the `root` user. Prioritize addressing these containers as a high-risk finding.
3. **Review Secret Access Mechanisms:** Assess current methods for injecting secrets into Pods. Begin planning the adoption of a Secret Store CSI driver to decouple secrets from the Kubernetes API server definition.
### Short-term Improvements (1-3 months)
1. **Implement PSS Profiles:** Apply appropriate Pod Security Admission (PSA) labels (e.g., `enforce: restricted`) to namespaces based on workload requirements, ensuring workloads adhere to defined standards.
2. **Initiate Non-Root Migration:** Begin the process of reconfiguring high-priority or sensitive applications to run containers as non-root users, addressing migration difficulties identified in current deployments.
3. **Enforce Security Contexts:** For workloads requiring elevated privileges that cannot immediately move to non-root, strictly define `securityContext` settings to minimize capabilities and utilize immutable file systems where possible.
4. **Deploy Network Policy Auditing:** Utilize network visibility tools (like `inspektor gadget`) to map existing application traffic flows and begin drafting preliminary Kubernetes Network Policies based on observed necessities, moving toward granular segmentation.
### Long-term Strategy (3+ months)
1. **Deploy a Secret Management Solution:** Implement a robust Secret Store Provider integrated with the Kubernetes Secrets CSI driver to inject secrets directly into Pod memory/filesystem at runtime, improving auditability and reducing API exposure.
2. **Adopt User Namespaces:** Plan the adoption and rollout of Kubernetes User Namespaces to provide workload isolation by mapping the container's root user to a non-privileged user on the host node, significantly mitigating container escape risks.
3. **Establish Zero Trust Threat Modeling:** Incorporate regular Zero Trust threat modeling exercises into the CI/CD pipeline to proactively identify vulnerabilities in identity, network, and resource access controls.
4. **Secure Managed Cluster Middleware:** Conduct a formal risk assessment focused specifically on the middleware layer of managed Kubernetes services (EKS, GKE, AKS) to identify and mitigate risks associated with control plane interfaces and underlying components.
## Implementation Guidance
### For Small Organizations
- Focus initial efforts on applying the **Restricted** Pod Security Standard to all namespaces.
- Utilize basic Kubernetes Network Policies that enforce an "implicit deny" for namespaces containing sensitive data, exposing only necessary ingress/egress ports.
- Ensure all container images are scanned for vulnerabilities before deployment (shift-left).
### For Medium Organizations
- Prioritize the migration away from workloads that require root access, potentially adopting an initial tier of non-root workloads and observing performance/compatibility.
- Implement an identity solution (e.g., Spire) to enforce workload identity separate from network trust, supporting future Zero Trust architecture goals.
- Develop standardized **Seccomp profiles** for common application types (e.g., web servers, databases) using tooling demonstrated at KubeCon.
### For Large Enterprises
- Deploy and mandate the use of a **Secret Store CSI Driver** platform-wide, retiring direct Kubernetes Secret usage for highly sensitive data.
- Roll out **User Namespaces** incrementally across development and staging environments to prove scalability and operational stability before applying to production.
- Standardize a **Federated Identity Context** across the organization that integrates with OPA (Open Policy Agent) gatekeepers to enforce fine-grained authorization policies across cluster operations and API access.
## Configuration Examples
Specific technical configurations were not provided in the summary text, however, the following concepts should be implemented:
* **Pod Security Admission (PSA) Labeling:** Apply namespace labels such as:
yaml
metadata:
labels:
pod-security.kubernetes.io/enforce: restricted
* **Seccomp Profile Application:** Define and reference a customized Seccomp profile within the Pod spec to limit syscall access.
* **Network Policy Definition:** Create policies leveraging the principles of the "Amsterdam Canal" analogy, ensuring every workload's ingress/egress rules are explicitly defined (default-deny posture).
## Compliance Alignment
The practices strongly align with security frameworks emphasizing least privilege, network segmentation, and continuous monitoring:
* **NIST SP 800-53:** Focus on AC (Access Control), CM (Configuration Management), and SC (System and Communications Protection).
* **CIS Benchmarks for Kubernetes:** Direct alignment with hardening cluster components, applying least privilege, and utilizing admission controllers effectively.
* **ISO/IEC 27001/27017:** Addressing requirements related to access control and secure operations management.
## Common Pitfalls to Avoid
- **"Lift and Shift" of Root Containers:** Assuming existing container configurations requiring root privileges can be seamlessly migrated to PSS without modification. This leads to either workload failure or reverting to permissive security standards.
- **Ignoring Middleware Risks:** Focusing solely on the workload Pods while overlooking vulnerabilities present in the managed cluster control plane components (EKS/AKS/GKE middleware).
- **Incomplete Secret Migration:** Switching to a CSI driver but failing to remove the old, insecure direct Kubernetes Secret definitions, leading to dual storage of sensitive data.
- **Overly Broad Initial Policies:** Creating Network Policies that are too permissive during the initial auditing phase, defeating the purpose of segmentation by not tightening rules post-discovery.
## Resources
* **Pod Security Policy Migration Guide:** (Reference the linked migration guide for specific commands and mapping details).
* **Kubernetes Secrets CSI Documentation:** Seek official documentation for the chosen Secret Store Provider (e.g., AWS Secrets Manager CSI, Azure Key Vault CSI) to configure the driver.
* **Inspektor Gadget:** Utilize this tool for real-time system tracing to gather data for **Seccomp profile generation** and **Network Policy drafting**.
* **OPA Gatekeeper:** Explore documentation for utilizing OPA/Gatekeeper alongside security standards enforcement.