Workers: Node Cycle
Cycling nodes simplifies both the upgrading of the Kubernetes and host OS versions running on the managed worker nodes, and the updating of other worker node properties.
When you set node_cycling_enabled
to true
for a node pool, OKE will compare the properties of the existing nodes in the node pool with the properties of the node_pool. If any of the following attributes is not aligned, the node is marked for replacement:
kubernetes_version
node_labels
compute_shape
(shape
,ocpus
,memory
)boot_volume_size
image_id
node_metadata
ssh_public_key
cloud_init
nsg_ids
volume_kms_key_id
pv_transit_encryption
The node_cycling_max_surge
(default: 1
) and node_cycling_max_unavailable
(default: 0
) node_pool attributes can be configured with absolute values or percentage values, calculated relative to the node_pool size
. These attributes determine how OKE will replace the nodes with a stale config in the node_pool.
The node_cycling_mode
attribute supports two node cycling modes:
instance
- (default) - cycling deletes and recreates a new node with the changes applied.boot_volume
cycling swaps the boot volume on the same node.
Notes:
- Only a subset of fields (
kubernetes_version
,image_id
,boot_volume_size
,node_metadata
,ssh_public_key
,volume_kms_key_id
) can be changed withboot_volume
cycling. - The cycling operation will attempt to bring all nodes in the NodePool in sync with the NodePool specification. If
boot_volume
cycling mode is chosen, and the node needs changes to fields that can not be updated via aboot_volume
cycle, the cycle attempt for that node will fail. The cycle_mode has to be changed toinstance
and the node-cycle operation needs to be retried.
When cycling nodes, the OKE cordons, drains, and terminates nodes according to the node pool's cordon and drain options.
Notes:
- It's strongly recommended to use readiness probes and PodDisruptionBudgets to reduce the impact of the node replacement operation.
- This operation is supported only with the
enhanced
OKE clusters. - New nodes will be created within the same AD/FD as the ones they replace.
- Node cycle requests can be canceled but can't be reverted.
- When setting a high
node_cycling_max_surge
value, check your tenancy compute limits to confirm availability of resources for the new worker nodes. - Compatible with the cluster_autoscaler. During node-cycling execution, the request to reduce node_pool size is rejected, and all the worker nodes within the cycled node_pool are annotated with
"cluster-autoscaler.kubernetes.io/scale-down-disabled": "true"
to prevent the termination of the newly created nodes. node_cycling_enabled = true
is incompatible with changes to the node_poolplacement_config
(subnet_id, availability_domains, placement_fds, etc.)- If the
kubernetes_version
attribute is changed whenimage_type = custom
, ensure a compatibleimage_id
with the new Kubernetes version is provided.
Usage
# Example worker pool node-cycle configuration.
worker_pools = {
cycled-node-pool = {
description = "Cycling nodes in a node_pool.",
size = 4,
node_cycling_enabled = true
node_cycling_max_surge = "25%"
node_cycling_max_unavailable = 0
node_cycling_mode = ["instance"] # An alternative value is boot_volume. Only a single mode is supported.
}
}