As modern infrastructure continues to shift toward Kubernetes and GitOps, the need for scalable and secure multi-tenant environments becomes increasingly critical.
This shift introduces complex problems and challenges, especially when dealing with time-sensitive tasks, such as synchronizing deployments, managing resource allocation across tenants, and ensuring consistency in a highly dynamic environment.
🚀 Overview of the Architecture
The main goal of the setup is to support multi-team operations within multiple Kubernetes clusters while ensuring proper isolation and separation of responsibilities.
Here’s a high-level summary of the architecture:
- Cluster bootstrap with GitOps: The entire cluster configuration is defined and managed from Git.
- Team-based namespaces: Each team gets its own namespace, Git repository, and Flux
Kustomization. - RBAC policies: Granular access control ensures that teams can manage their own resources without impacting others.
- Automation with Flux controllers: Leveraging the full power of Flux controllers (
source-controller,kustomize-controller,helm-controller, etc.).
🔑 Key Advantages
- Eliminated Redundancy: Common configs are centralized, reducing duplication, while teams manage their apps within a shared global chart.
- Scoped Configuration Impact: Cluster-specific changes remain isolated; shared updates can be selectively propagated.
- Team Autonomy & Modularity: Teams control app deployment per cluster, with the flexibility to toggle apps as needed.
- Self-Contained Deployments: Apps include their dependencies (e.g., secrets), making each deployment portable and reliable.
- Reliable Versioning: Strict version control ensures safe rollbacks and consistent deployment across environments.
🧱 Global Architecture
📚 Repositories and charts Overview:
Purpose, Functionality & Benefits ?
This document outlines the key roles and structural advantages of each repository in our GitOps-managed deployment framework.
Each repository/chart plays a specific and essential role in maintaining modularity, scalability, and clarity across cluster and application configurations.
A . Fleet Repository
Purpose:
Acts as the centralized configuration hub for managing Flux deployments across clusters.
Core Functionalities:
- Flux Bootstrap Manifests: Automates the GitOps-based deployment of Flux onto clusters, ensuring continuous reconciliation of manifests from Git.
- Dynamic Kustomization Generation: Automatically generates Kustomization files tailored to individual tenants and clusters, enabling dynamic and scalable configuration management.
Benefits:
- Central Source of Truth: Offers a single, consistent reference for cluster-level configuration.
- Standardization: Ensures uniform deployment and management of Flux across all environments.
- Automation-Driven Consistency: Enhances reliability by removing manual deployment steps and enabling dynamic targeting of tenant environments.
B . Tenant chart
Purpose:
Manages tenant-specific configuration and deployment logic across various domains (e.g., AI, BigData, Security).
Core Functionalities:
- Multi-Tenant Configuration Management: Hosts isolated configurations for each tenant using structured repositories tagged accordingly.
- DRY Principle Implementation: Avoids redundant application manifests by centralizing shared configurations.
- Secure Isolation: Guarantees tenant-level security, compliance, and independence in configuration.
Benefits:
- Strong Isolation: Reduces risk of cross-tenant configuration conflicts.
- Simplified Maintenance: Decouples applications from clusters, enhancing flexibility and reducing complexity.
- Scalability & Reusability: Supports modular expansion and reuse of tenant definitions across clusters.
C . Applications layer
Purpose:
Maintains the lifecycle, versioning, and structure of application deployments via Helm charts and manifests.
Core Functionalities:
- Contains the default application manifests and value files, providing a foundation shared across all clusters.
Benefits:
- Modular Architecture: Clean separation between base configuration and cluster-specific overlays.
- Version Control: Enables consistent, versioned deployment and easy rollback capabilities.
- Operational Flexibility: Supports customized configurations per cluster without duplicating base logic.
🔑 Functional and Technical key takeaways
We opt to do tenant as helm chart because we need to choose which app to be deployed from a long list of apps of each team, which is not possible with flux only with helm conditions because kustomize does not support conditions
Each of these repositories and charts contributes to a scalable, secure, and highly maintainable GitOps ecosystem by promoting isolation, reusability, and automation at every level of deployment.
Each application within a tenant is treated as a black box, encapsulating not only the application’s deployment logic but also all of its required dependencies. This includes references to the application code itself, as well as supporting resources such as secrets, configuration files, or even other dependent applications.
With this setup, any additions or modifications to applications are handled exclusively at the tenant level. There is no need to update the cluster-level configuration, since the Fleet repository is configured to watch and reconcile only the tenant Helm chart. This decouples application management from cluster definitions, streamlining the deployment process and reducing the risk of configuration drift.
