Onboarding Kubernetes Clusters

​

Overview

1. Grant Controller access to the cluster’s Kubernetes API — the Controller must be able to reach the API server and authenticate with sufficient RBAC permissions.
2. Register the cluster — tell the Controller which cluster to watch by completing the onboarding workflow in the UI or via Terraform.

​

Onboarding Paths

​

Prerequisites

​

Platform Requirements

• DCF is enabled. See Enable Distributed Cloud Firewall.
• Kubernetes Resource Discovery is enabled under Groups > Settings. See Kubernetes Resource Discovery.
• The Kubernetes API server must be network-reachable from the Controller.
  • Public clusters: restrict API server access to the Controller’s IP address for security.
  • Private clusters: a private network path is required. See Onboarding Private Kubernetes Clusters.
• SNAT must be disabled on worker nodes so that pod source IPs are preserved for policy enforcement. See the AWS documentation on external SNAT.
• Flat networking is recommended (this is the default for EKS and AKS). Overlay networks have limited SmartGroup support.

​

IAM Permissions

​

Kubernetes RBAC

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view-nodes
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view-nodes-binding
subjects:
  - kind: Group
    name: view-nodes
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view-nodes
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f cluster-role.yaml

​

Kubeconfig Requirements

• Allowed verbs: GET, LIST, and WATCH on namespaces, services, pods, nodes, and endpointslices.
• No exec-based authentication — the kubeconfig must not rely on exec plugins (such as aws eks get-token) because the Controller cannot execute external binaries.
• No create, update, or delete permissions — the Controller only needs read access.
• Exactly one cluster, user, and context — multi-context kubeconfig files are not supported.
• Inline credential data only — use certificate-authority-data, client-certificate-data, and client-key-data (base64-encoded) rather than file path references (certificate-authority, client-certificate, client-key, tokenFile). The Controller cannot read files from the local filesystem.

apiVersion: v1
kind: Config
clusters:
  - name: my-cluster
    cluster:
      server: https://ABCDEF1234567890.gr7.us-east-1.eks.amazonaws.com
      certificate-authority-data: LS0tLS1CRUdJTi...base64-encoded-ca-cert...LS0tLS1FTkQ=
contexts:
  - name: my-cluster-context
    context:
      cluster: my-cluster
      user: my-cluster-user
users:
  - name: my-cluster-user
    user:
      client-certificate-data: LS0tLS1CRUdJTi...base64-encoded-client-cert...LS0tLS1FTkQ=
      client-key-data: LS0tLS1CRUdJTi...base64-encoded-client-key...LS0tLS1FTkQ=
current-context: my-cluster-context
preferences: {}

​

Understanding the Access Model

arn:aws:iam::123456789012:role/aviatrix-role-app

​

Onboarding via Terraform

variable "controller_role_arn" {
  description = "Aviatrix Controller IAM role ARN (aviatrix-role-app)"
  type        = string
}

variable "cluster_name" {
  description = "EKS cluster name"
  type        = string
}

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

# --- Grant Controller access ---

# EKS access entry — authenticates Controller IAM role to the cluster
resource "aws_eks_access_entry" "aviatrix" {
  cluster_name      = var.cluster_name
  principal_arn     = var.controller_role_arn
  kubernetes_groups = ["view-nodes"]
  type              = "STANDARD"
}

# Attach the EKS managed view policy for broad read access
resource "aws_eks_access_policy_association" "aviatrix" {
  cluster_name  = var.cluster_name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
  principal_arn = var.controller_role_arn

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.aviatrix]
}

# ClusterRole for node visibility (not covered by AmazonEKSViewPolicy)
resource "kubernetes_cluster_role" "view_nodes" {
  metadata {
    name = "view-nodes"
  }

  rule {
    verbs      = ["get", "list", "watch"]
    api_groups = [""]
    resources  = ["nodes"]
  }
}

# Bind the ClusterRole to the "view-nodes" group
resource "kubernetes_cluster_role_binding" "view_nodes" {
  metadata {
    name = "view-nodes"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.view_nodes.metadata[0].name
  }

  subject {
    kind      = "Group"
    name      = "view-nodes"
    api_group = "rbac.authorization.k8s.io"
  }
}

# --- Register the cluster ---

resource "aviatrix_kubernetes_cluster" "this" {
  cluster_id          = data.aws_eks_cluster.this.arn
  use_csp_credentials = true

  depends_on = [
    aws_eks_access_policy_association.aviatrix,
    kubernetes_cluster_role_binding.view_nodes,
  ]
}

data "azurerm_kubernetes_cluster" "this" {
  name                = "my-cluster"
  resource_group_name = "my-rg"
}

resource "aviatrix_kubernetes_cluster" "this" {
  cluster_id          = lower(data.azurerm_kubernetes_cluster.this.id)
  use_csp_credentials = true
}

data "google_container_cluster" "this" {
  name     = "my-cluster"
  location = "us-central1"
}

resource "aviatrix_kubernetes_cluster" "this" {
  cluster_id          = data.google_container_cluster.this.self_link
  use_csp_credentials = true
}

​

Custom or Self-Managed Clusters

resource "aviatrix_kubernetes_cluster" "custom" {
  cluster_id  = "my-cluster-id"
  kube_config = var.kubeconfig

  cluster_details {
    account_name           = data.aviatrix_account.aws.account_name
    account_id             = data.aviatrix_account.aws.aws_account_number
    name                   = "my-cluster"
    region                 = "us-east-2"
    vpc_id                 = data.aws_vpc.vpc.id
    is_publicly_accessible = true
    platform               = "kops"
    version                = "1.30"
    network_mode           = "FLAT"
    tags = {
      environment = "production"
    }
  }
}

​

Multi-Cluster Automation

variable "eks_clusters" {
  description = "Map of logical name to EKS cluster name"
  type        = map(string)
}

data "aws_eks_cluster" "clusters" {
  for_each = var.eks_clusters
  name     = each.value
}

resource "aviatrix_kubernetes_cluster" "clusters" {
  for_each            = data.aws_eks_cluster.clusters
  cluster_id          = each.value.arn
  use_csp_credentials = true
}

​

Onboarding via CoPilot UI

​

Manual Onboarding

​

Onboarding via CLI (AWS Only)

kind: ClusterConfig
apiVersion: eksctl.io/v1alpha5
metadata:
  name: CLUSTER_NAME
  region: REGION
accessConfig:
  accessEntries:
    - principalARN: "CONTROLLER_ROLE_ARN"
      kubernetesGroups:
        - view-nodes
      accessPolicies:
        - policyARN: arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy
          accessScope:
            type: cluster

eksctl create accessentry -f accessentry.yaml

kubectl config current-context
kubectl apply -f cluster-role.yaml

aws eks create-access-entry \
  --cluster-name CLUSTER_NAME \
  --principal-arn CONTROLLER_ROLE_ARN \
  --kubernetes-groups view-nodes \
  --type STANDARD

aws eks associate-access-policy \
  --cluster-name CLUSTER_NAME \
  --principal-arn CONTROLLER_ROLE_ARN \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \
  --access-scope type=cluster

kubectl apply -f cluster-role.yaml

AWSTemplateFormatVersion: '2010-09-09'
Description: Grant Aviatrix Controller read access to an EKS cluster
Parameters:
  ClusterName:
    Type: String
    Description: Name of the EKS cluster
  ControllerRoleArn:
    Type: String
    Description: IAM role ARN of the Aviatrix Controller (aviatrix-role-app)

Resources:
  AviatrixAccessEntry:
    Type: AWS::EKS::AccessEntry
    Properties:
      ClusterName: !Ref ClusterName
      PrincipalArn: !Ref ControllerRoleArn
      KubernetesGroups:
        - view-nodes
      Type: STANDARD

aws cloudformation deploy \
  --template-file aviatrix-eks-access.yaml \
  --stack-name aviatrix-eks-access-CLUSTER_NAME \
  --parameter-overrides \
    ClusterName=CLUSTER_NAME \
    ControllerRoleArn=CONTROLLER_ROLE_ARN

aws eks associate-access-policy \
  --cluster-name CLUSTER_NAME \
  --principal-arn CONTROLLER_ROLE_ARN \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \
  --access-scope type=cluster

kubectl apply -f cluster-role.yaml

​

Post-Onboarding

​

Verifying Onboarding Status

​

Creating Kubernetes SmartGroups

1. Click Create SmartGroup on an onboarded cluster.
2. Choose One SmartGroup per Namespace or One SmartGroup per Service.
3. Select the namespaces or services to include.
4. Click Create.

​

Editing and Offboarding

​

Editing a Cluster

​

Offboarding a Cluster

• CoPilot UI: Click Offboard next to the cluster and confirm.
• Terraform: Remove the aviatrix_kubernetes_cluster resource and run terraform apply.

​

Troubleshooting

​

Related Topics

• Aviatrix Kubernetes Firewall
• Kubernetes Prerequisites and Permissions
• Onboarding Private Kubernetes Clusters
• Distributed Cloud Firewall for Kubernetes
• Creating SmartGroups