Design considerations for choosing the right Delegate Architecture:
Ownership and Team structure
Ownership of teams over Infrastructure decisions is the most important design consideration.
Having a centralized team to create, upgrade and maintain delegate hosting infrastructure enables centralized governance and distributed onboarding of Delegates
Lack of Centralized practice and different BUs controlling RBAC individually needs distributed patterns
Performance
Optimized build execution and deployment times are important design considerations .
For example Delegates hosted on Serverless has additional startup time
Operations
This covers high availability, upgrade and maintenance requirements .
Security
Delegates need to access applications hosted in different Cloud providers and Data centers. It is an important design consideration to contain blast radius at all circumstances
Cost
Some patterns allow efficient cost optimization strategies than others .
Common patterns adapted for Delegate Architecture, pros and cons:
Delegate Architecture Best Practices(Applicable to All modules CI/CD/CCM/STO/IACM):
Create a separate Application Identifier for Harness Delegate hosting Infrastructure in CMDB and Tag each Delegate with App identifier. This is extremely important and useful for efficient operations and telemetry
Naming standardization is crucial for efficient Delegate Onboarding and Governance automation
Leverage BU Naming standards from CMDB or already existing asset governance
Delegate naming convention should serve as self identifier of its scope and purpose
Harness recommends Auto update of base images of Delegates . Refer Doc . Also configure Delegate for Graceful Shutdown
In case “Auto update is not allowed because of security policies “
Setting up automation to create and update Base images is important for efficient operations
Scanning base image before upgrade using minimal base image ensures security
Setup Delegate Onboarding Automation and Governance to add Delegates for new BUs and Applications on demand.
Create a BootStrap Delegate through script to avoid circular Dependency and use ‘bootstrap delegate’ to onboard delegates through automation
Auto scaling is a must and leverage Managed Infrastructure like EKS or GKE with HPA/ KEDA/Karpenter/Cluster autoscaler
Resort on Active-Active Setup for resiliency and optimal SLA, also use same selectors and same naming convention for Primary and DR delegates to avoid manual intervention to switch forth and back in between Primary and Secondary environments
Single Kubernetes Cluster is recommended for hosting CI delegates since CI Delegate is a mere orchestrator. Build steps are executed as Kubernetes Jobs
Same pool of Delegates can be used for all Applications in any size of Enterprise given that artifactory is cloud agnostic, auto scaling is set up
If capturing cost incurred by each Application is a requirement and hindering from using common set of delegates for , Running CI Delegates in common namespace and moving Build step execution jobs to App Specific namespaces is a tip that provides optimal usage of Delegates and required insights
Always use separate set of Delegates for Non-prod and Prod Deployments
Highly recommend to host Non-prod & Prod Delegates on different clusters adhering to the network and firewall compliance requirements
Go with Distributed pattern for Apps residing in PCI compliance and Air-gapped cloud Accounts/ BUs
Highly recommend to maintain source of truth for both Delegate installs and init_script
setup standardized catalog with security vetted init_scripts available for delegate onboarding
Explicit deny of CLI capabilities( aws cli, kubectl etc) on Delegates is important to contain blast-radius impact of unintended consequences of script execution
Resort on just-in-time access with explicit approval if CLI access is must for any operations