Skip to content

feat: add node public IP support to AKSNodeClass (AKS Machine API) - #1982

Merged
Alex Leites (tallaxes) merged 6 commits into
mainfrom
tallaxes/node-public-ip-api
Oct 11, 2026
Merged

Alex Leites (tallaxes) merged 6 commits into
mainfrom
tallaxes/node-public-ip-api

Conversation

@tallaxes

@tallaxes Alex Leites (tallaxes) commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

fixes: #656

Adds an optional spec.nodePublicIP block to the AKSNodeClass, and supports it in the AKS Machine API provisioning modes (aksmachineapi and aksmachineapiheaderbatch). It lets a NodeClass give every node its own instance-level public IP address, optionally allocated from a public IP prefix or carrying IP tags such as routing preference. This is the Karpenter equivalent of an agent pool's node public IPs.

AKS creates each node's public IP with the node and deletes it with the node, so Karpenter has no cleanup of its own to do. In the VM-based modes (aksscriptless, bootstrappingclient), enabling the field sets ValidationSucceeded=False with reason NodePublicIPUnsupportedProvisionMode, so the NodeClass isn't ready and no nodes are created from it.

API (v1beta1, mirrored in v1alpha2)

  • enabled (optional bool).
  • prefixIDs (optional list, 1 item). Each item must be a Microsoft.Network/publicIPPrefixes ARM ID.
  • ipTags (optional list, 1 to 4 items). Each item has a required ipTagType and tag, 1 to 256 characters each, and entries must be unique.
  • CEL requires enabled: true when prefixIDs or ipTags is set, and rejects setting both prefixIDs and ipTags. Empty lists are rejected rather than treated as omitted.
  • The nodeclass hash ignores the block when it's omitted or disabled, so existing nodes don't drift on upgrade. Omitted, {} and {enabled: false} all hash the same. Prefix IDs are compared case-insensitively, since ARM IDs are case-insensitive. Reordering IP tags doesn't change the hash; changing a tag, including its casing, does.

AKS Machine API

  • Machine template. It sets network.enableNodePublicIP, plus network.nodePublicIPPrefixID or network.nodePublicIPTags, only when the feature is enabled. A disabled NodeClass produces the same request and batch key as before. In header-batch mode, the fields are in the shared template, so Machines from different NodeClass settings are never batched together.
  • Reusing a leftover Machine. Before Karpenter adopts an existing Machine, it now checks the Machine's node public IP settings against the NodeClass. A missing value counts as disabled, prefix IDs are compared case-insensitively, and IP tags are compared as a set (order doesn't matter, casing does). On a mismatch, Karpenter doesn't reuse the Machine and deletes it, so it never adopts a node with the wrong settings under the current NodeClass hash. The check reads the Machine with a plain GET (no $expand). The E2E below confirms that GET and LIST return these fields.
  • Errors. Public IP failures from AKS stay on the existing unhandled-error path. Tests lock in that they aren't treated as insufficient capacity, that they keep AKS's code and message, and that every offering stays available, for both synchronous and provisioning failures. The tests use the error codes and messages observed below. They're NodeClass problems, not VM-size or zone problems, so trying other offerings wouldn't help.

Example

apiVersion: karpenter.azure.com/v1beta1
kind: AKSNodeClass
metadata:
  name: public
spec:
  nodePublicIP:
    enabled: true
    prefixIDs:
      - /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/publicIPPrefixes/<prefix>

or, with IP tags instead of a prefix:

spec:
  nodePublicIP:
    enabled: true
    ipTags:
      - ipTagType: RoutingPreference
        tag: Internet

This becomes the Machine's network profile:

"network": {
  "vnetSubnetID": "<unchanged>",
  "enableNodePublicIP": true,
  "nodePublicIPPrefixID": "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/publicIPPrefixes/<prefix>"
}

ipTags map to nodePublicIPTags the same way.

E2E results (aksmachineapi, self-hosted; tests are in #1984)

  • Machine readback. GET and LIST, without $expand, return enableNodePublicIP: true and the configured nodePublicIPPrefixID or nodePublicIPTags. After the feature is disabled and the nodes drift, the replacement Machines report enableNodePublicIP: false.

  • Without a prefix. Each node's NIC gets a public IP, and the node reports it as ExternalIP. Inbound traffic reaches the node on its public IP, and outbound traffic leaves from it. Disabling the feature drifts the nodes, and AKS releases the old IPs.

  • From a prefix. Nodes in all three zones get IPs from a zone-redundant prefix, and AKS releases the IPs when the nodes are deleted.

  • With IP tags. Regional nodes (karpenter.azure.com/placement-scope: regional) with RoutingPreference: Internet get a zone-redundant public IP carrying the tag.

  • Error codes. Each failing identity was observed on its own:

    Case When it fails Code Where Karpenter shows it
    Karpenter's identity can't join the prefix, or the prefix doesn't exist ARM rejects the Machine PUT (HTTP 403) LinkedAuthorizationFailed, naming Karpenter's identity NodeClaim Launched=Unknown, reason CreateInstanceFailed. The NodeClaim is kept and retried.
    The cluster identity can't join the prefix During provisioning, when AKS creates the VM LinkedAuthorizationFailed, naming the cluster identity AsyncProvisioningError event on the NodeClaim, which is then deleted and replaced
    Unsupported IP tag type AKS rejects the Machine PUT (HTTP 400) UnsupportedIPTagType NodeClaim Launched=Unknown, reason CreateInstanceFailed
    RoutingPreference on a zonal node (observed in a manual run) During provisioning; a zonal node gets a zonal public IP, which can't use routing preference No code: "A zonal PublicIPAddress … cannot support for routing preference feature" AsyncProvisioningError event; the NodeClaim is deleted and replaced

    Both identities need Microsoft.Network/publicIPPrefixes/join/action on the prefix. On NAP, Karpenter calls AKS as the cluster identity, so one grant covers both.

How was this change tested?

  • CEL validation tests (v1beta1 and v1alpha2): valid and invalid prefix IDs; prefixIDs or ipTags without enabled; more than one prefix; ipTags with prefixIDs; empty, 256- and 257-character tag values; duplicate tags; more than 4 tags.
  • Hash tests (v1beta1 and v1alpha2): omitted, {} and {enabled: false} match the pre-feature hash; prefix ID casing and IP tag order don't change the hash; enabling, changing the prefix, and adding, changing or removing a tag do; hashing doesn't mutate the NodeClass.
  • Status validation tests: the NodeClass isn't ready, with NodePublicIPUnsupportedProvisionMode, in the VM-based modes, and recovers when the feature is disabled; it's accepted in both AKS Machine API modes.
  • Unit tests for the network profile, the reuse check (enabled vs. disabled, nil vs. false, prefix casing and mismatch, IP tag order and mismatch) and the batch key.
  • AKS Machine acceptance tests: the Machine PUT contents, reuse and reuse refusal on mismatch, drift, and the error handling above (synchronous and provisioning).
  • E2E as above.
  • make verify and make test.

Does this change impact docs?

Release Note

Add spec.nodePublicIP to AKSNodeClass. In the AKS Machine API provisioning modes, it gives each node an instance-level public IP address, optionally allocated from a public IP prefix or carrying IP tags such as routing preference.

@tallaxes
Alex Leites (tallaxes) force-pushed the tallaxes/node-public-ip-api branch 2 times, most recently from 230c88b to 0f71243 Compare October 10, 2026 01:38
Comment thread pkg/providers/instance/vminstance.go Resolved
Add an optional spec.nodePublicIP block to the v1beta1 AKSNodeClass, and
mirror it in v1alpha2, so a NodeClass can give each node its own
instance-level public IP:

- enabled (optional bool)
- prefixIDs (optional, 1 item): a Microsoft.Network/publicIPPrefixes ID
- ipTags (optional, 1-4 unique items): ipTagType and tag

CEL requires enabled=true when prefixIDs or ipTags is set, rejects
combining prefixIDs with ipTags, and rejects empty lists.

The nodeclass hash ignores the block when it is omitted or disabled, so
existing nodes do not drift. Prefix IDs are compared case-insensitively
and IP tag order does not matter.

No provisioning mode supports node public IPs yet: enabling the field
sets ValidationSucceeded=False with reason
NodePublicIPUnsupportedProvisionMode.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…1983)

Enable spec.nodePublicIP for the aksmachineapi and aksmachineapiheaderbatch
provisioning modes. NodeClasses using it in VM-based modes stay rejected by
status validation.

- The Machine template sets network.enableNodePublicIP, plus
  nodePublicIPPrefixID or nodePublicIPTags, only when the feature is
  enabled, so disabled machines keep their existing template and batch key.
- Reusing a leftover Machine checks its node public IP settings against the
  NodeClass. A missing value counts as disabled and prefix IDs are compared
  case-insensitively. On a mismatch the Machine is not reused and is
  deleted, so a node with the wrong settings is never adopted under the
  current NodeClass hash.
- Public IP failures from AKS stay on the existing unhandled error path:
  they are not treated as insufficient capacity, keep AKS's code and
  message, and leave every offering available.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tallaxes Alex Leites (tallaxes) changed the title feat: add nodePublicIP to AKSNodeClass (not yet supported) feat: add node public IP support to AKSNodeClass (AKS Machine API) Oct 10, 2026
@tallaxes
Alex Leites (tallaxes) marked this pull request as ready for review October 10, 2026 18:42
Copilot AI balanced review requested due to automatic review settings October 10, 2026 18:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟑 Changes recommended

RoutingPreference can repeatedly select invalid zonal offerings, and the API is incorrectly mirrored into unserved v1alpha2.

2 open findings
What changed in this PR

Adds configurable node public IPs for AKS Machine API provisioning, including validation, hashing, request construction, reuse checks, and error handling.

Changes:

  • Adds nodePublicIP API fields and CRD validation.
  • Configures and validates AKS Machine network profiles.
  • Adds extensive unit, acceptance, drift, batching, and error-path coverage.
File Description
pkg/​providers/​instance/​offerings/​errordetailhandlers_test.go Tests provisioning errors.
pkg/​providers/​instance/​offerings/​commonerrorhandlers_test.go Adds offering assertion helper.
pkg/​providers/​instance/​offerings/​aksmachinebegincreateerrorhandlers_test.go Tests synchronous errors.
pkg/​providers/​instance/​aksmachineinstancehelpers.go Builds public-IP network settings.
pkg/​providers/​instance/​aksmachineinstancehelpers_test.go Tests network configuration and reuse validation.
pkg/​providers/​instance/​aksmachineinstance.go Validates Machines before reuse.
pkg/​providers/​azclient/​aksmachinesheaderbatch/​batchkey_test.go Verifies batching separation.
pkg/​operator/​options/​options_extensions.go Defines supported provisioning modes.
pkg/​fake/​aksmachinesapi.go Adds public-IP failure fakes.
pkg/​controllers/​nodeclass/​status/​validation.go Rejects unsupported modes.
pkg/​controllers/​nodeclass/​status/​validation_test.go Tests mode validation.
pkg/​cloudprovider/​suite_aksmachineapi_offerings_test.go Tests offering error behavior.
pkg/​cloudprovider/​suite_aksmachineapi_integration_test.go Tests Machine reuse and rejection.
pkg/​cloudprovider/​suite_aksmachineapi_features_test.go Tests Machine request fields.
pkg/​cloudprovider/​suite_aksmachineapi_drift_test.go Tests public-IP drift.
pkg/​apis/​v1beta1/​zz_generated.deepcopy.go Adds generated deep copies.
pkg/​apis/​v1beta1/​crd_validation_cel_test.go Tests API validation.
pkg/​apis/​v1beta1/​aksnodeclass.go Defines and hashes the API.
pkg/​apis/​v1beta1/​aksnodeclass_hash_test.go Tests hash semantics.
pkg/​apis/​v1alpha2/​zz_generated.deepcopy.go Mirrors generated deep copies.
pkg/​apis/​v1alpha2/​crd_validation_cel_test.go Mirrors validation tests.
pkg/​apis/​v1alpha2/​aksnodeclass.go Mirrors the API into v1alpha2.
pkg/​apis/​v1alpha2/​aksnodeclass_hash_test.go Mirrors hash tests.
pkg/​apis/​crds/​karpenter.azure.com_aksnodeclasses.yaml Updates generated CRD schemas.
charts/​karpenter-crd/​templates/​karpenter.azure.com_aksnodeclasses.yaml Updates chart CRD template.
Files not reviewed (2)
  • pkg/apis/v1alpha2/zz_generated.deepcopy.go: Generated file
  • pkg/apis/v1beta1/zz_generated.deepcopy.go: Generated file

🧠 Review effort: Balanced


πŸ’‘ Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pkg/apis/v1alpha2/aksnodeclass.go Resolved
Comment thread pkg/apis/v1beta1/aksnodeclass.go Resolved
Comment thread pkg/apis/v1beta1/aksnodeclass.go Resolved
Comment thread pkg/apis/v1beta1/aksnodeclass.go Resolved
An exhausted public IP prefix doesn't fail the Machine create request. AKS accepts it, and the
Machine fails during provisioning with a detail that has no code and the message "No more IPs
available in IpPrefix ...". Drop the PublicIpPrefixOutOfIpAddressesForVMScaleSet create-request
case, replace the CreateOrUpdatePublicIPAddressError provisioning case with the observed shape,
and use the observed GetPublicIPPrefixByResourceIDError message.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…p-api

# Conflicts:
#	pkg/apis/v1alpha2/zz_generated.deepcopy.go
#	pkg/apis/v1beta1/zz_generated.deepcopy.go
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tallaxes
Alex Leites (tallaxes) merged commit 4d56249 into main Oct 11, 2026
16 checks passed
@tallaxes
Alex Leites (tallaxes) deleted the tallaxes/node-public-ip-api branch October 11, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for Instance-level Public IPs

3 participants