Extensions: Networking

WARNING: The following options are provided as a reference for evaluation only, and may install software to the cluster that is not supported by or sourced from Oracle. These features should be enabled with caution as their operation is not guaranteed!

Multus CNI

Usage

multus_install       = true
multus_namespace     = "network"
multus_daemonset_url = null // determined automatically for version by default
multus_version       = "3.9.3"

References

Cilium CNI

Usage

cilium_install           = true
cilium_reapply           = false
cilium_namespace         = "kube-system"
cilium_helm_version      = "1.16.3"
cilium_helm_values       = {}
cilium_helm_values_files = []

Cillium is a eBPF based CNI for Kubernetes that can be configured on OKE clusters.

The OKE cluster should be initially configured to run flannel.

On enhanced clusters we can use the cluster-addons module to remove flannel extension and kube-proxy (Optional) at cluster creation.

cluster_addons_to_remove = {
  Flannel = {
    remove_k8s_resources = true
  },
  KubeProxy = {
    remove_k8s_resources = true
  }
}

If you want to use cilium as kube-proxy replacement, you can use the following helm_values:

cilium_helm_values      = {
  kubeProxyReplacement = true
}

For the basic clusters you can add the following label to the worker nodes to prevent flannel pods from being scheduled:

oci.oraclecloud.com/custom-k8s-networking=true

If you want to override and of the default values(listed below) you can use the cilium_helm_values variable:

"annotateK8sNode": true
"cluster":
  "id": 1
  "name": "oke-${var.state_id}"
"clustermesh":
  "apiserver":
    "kvstoremesh":
      "enabled": false
  "useAPIServer": false
"cni":
  "exclusive": true
  "install": true
"hubble":
  "metrics":
    "dashboards":
      "enabled": false
  "relay":
    "enabled": true
  "ui":
    "enabled": true
"installNoConntrackIptablesRules": false
"ipam":
  "mode": "kubernetes"
"k8s":
  "requireIPv4PodCIDR": true
"k8sServiceHost": "${var.cluster_private_endpoint}"
"k8sServicePort": "6443"
"kubeProxyReplacement": false
"operator":
  "prometheus":
    "enabled": false
"pmtuDiscovery":
  "enabled": true
"rollOutCiliumPods": true
"tunnelProtocol": "vxlan"

Notes:

  1. Tested with OKE version v1.29.1 and the worker nodes running: Oracle-Linux-8.9-2024.05.29-0-OKE-1.29.1-707.

  2. In case the hubble-relay and hubble-ui pods fail to start, run the following commands:

kubectl delete pod --namespace kube-system -l k8s-app=kube-dns
kubectl delete pod --namespace kube-system -l k8s-app=hubble-relay
kubectl delete pod --namespace kube-system -l k8s-app=hubble-ui
kubectl delete pod --namespace kube-system -l k8s-app=kube-dns-autoscaler

References

Whereabouts IPAM plugin

Usage

whereabouts_install       = true
whereabouts_namespace     = "network"
whereabouts_daemonset_url = null // determined automatically for version by default
whereabouts_version       = "master"

References

SR-IOV Device plugin

Usage

sriov_device_plugin_install       = true
sriov_device_plugin_namespace     = "network"
sriov_device_plugin_daemonset_url = null // determined automatically for version by default
sriov_device_plugin_version       = "master"

References

SR-IOV CNI plugin

Usage

sriov_cni_plugin_install       = true
sriov_cni_plugin_namespace     = "network"
sriov_cni_plugin_daemonset_url = null // determined automatically for version by default
sriov_cni_plugin_version       = "master"

References

RDMA CNI plugin

Usage

rdma_cni_plugin_install       = true
rdma_cni_plugin_namespace     = "network"
rdma_cni_plugin_daemonset_url = null // determined automatically for version by default
rdma_cni_plugin_version       = "master"

References