Skip to content

feat: support for ultrassd - #1704

Merged
Pablo Trivino (PabloTriv) merged 95 commits into
mainfrom
pablotrivino/enalbeUltraSSD
Jul 17, 2026
Merged

Pablo Trivino (PabloTriv) merged 95 commits into
mainfrom
pablotrivino/enalbeUltraSSD

Conversation

@PabloTriv

@PabloTriv Pablo Trivino (PabloTriv) commented Jun 2, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes # #119

Description
This change introduces support for ultra ssd. Included is a design doc that talks more about the implementation. UltraSSD will be enabled/disabled via a setting on the AKSNodeClass. The implementation here should be as close to AKS as possible.

How was this change tested?
New E2E: https://github.com/Azure/karpenter-provider-azure/actions/runs/27987096466/job/82831071878
*

Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: #
  • No

Release Note


Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think my main concern (reflected in a few different comments) is if we should treat the source of truth for a NAP cluster as the MachineAPI/ManagedCluster state OR the aksnodeclass fields?

Need a few things confirmed:

  1. Is this only set during cluster creation? or can it be enabled/disabled through the cluster lifecycle?
  2. Is it set at the cluster level? agentpool level? or both?
  3. Will/can/should Machine API report this as a DriftAction when the MC field for this changes?

If this is only enabled at cluster creation for AKS, I think this should instead be set via helm values -> env vars/flags -> opts.

If this can be disabled/enabled after cluster creation, I think this nodeclass field should be limited to self-hosted and machineapi should handle setting this for machines based on MC object.

Comment thread designs/0013-ultra-ssd-support.md Resolved
Comment thread designs/0012-ultra-ssd-support.md Resolved
Comment thread designs/0012-ultra-ssd-support.md Resolved
@PabloTriv

Copy link
Copy Markdown
Collaborator Author

Is this only set during cluster creation? or can it be enabled/disabled through the cluster lifecycle?

This is set during cluster creation and nodepool creation. It can't be changed once a pool has been created. Enabling at the cluster means the system pool gets the UltraSSD capabilities, not that all nodepools/nodes in the cluster will have the functionality by default.

Is it set at the cluster level? agentpool level? or both?

It's agentpool level. When you pass it during cluster create you're just enabling it on the initial create request. And other nodepools you add need to opt in. It's also immutable.

Will/can/should Machine API report this as a DriftAction when the MC field for this changes?

The field can't change once the cluster/pool is created, so I don't think we'd have to worry about this.

If this is only enabled at cluster creation for AKS, I think this should instead be set via helm values -> env vars/flags -> opts.

I think this would make sense if it was a global field that controlled all nodepools. But today each pool needs to opt into it, which is why I think it would make sense to stick it into either the NodePool or AKSNodeClass CRDs. Clusters can have a mix of ultrassd enabled/disabled pools on the same cluster.

@rakechill

Rachel Gregory (rakechill) commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

...I think this would make sense if it was a global field that controlled all nodepools. But today each pool needs to opt into it, which is why I think it would make sense to stick it into either the NodePool or AKSNodeClass CRDs. Clusters can have a mix of ultrassd enabled/disabled pools on the same cluster.

I see, didn't realize that enabling for the cluster == enabling for the system nodepool. In that case, it makes sense to me for this to be something that gets enabled from the Karpenter side. Will customers be able to change this after the fact then? That would be different behavior than what AKS currently provides.

I'd expect there to be some validation if a customer tries to switch from Enabled to Disabled.

Or, if we don't plan to mimic RP behavior, can we get clarity on why RP doesn't allow disablement after enablement?

@rakechill Rachel Gregory (rakechill) left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your comments from the previous review make sense + I better understand the agentpool<>cluster relationship wrt this feature.

A few remaining thoughts:

  1. Is there any way to make this per-nodepool? I expect many clusters will have different Nodepools for different use cases + may prefer that level of granularity. Maybe checking with Wilson (@wdarko1) based on current customer requirements could be useful.
  2. Are the other features we have on AKSNodeClass also configurable per-agentpool in AKS? Or are they cluster-wide?
  3. Should we block disablement after enablement to align with current AKS behavior?
  4. Should we only allow ultra SSD to be enabled in Karpenter if it's enabled on the cluster? Basically introducing a validation here based on cluster state.

Comment thread designs/0012-ultra-ssd-support.md Resolved
@PabloTriv

Pablo Trivino (PabloTriv) commented Jun 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Is there any way to make this per-nodepool? I expect many clusters will have different Nodepools for different use cases + may prefer that level of granularity. Maybe checking with Wilson (@wdarko1) based on current customer requirements could be useful.

I feel like using the AKSNodeClass is the pattern that's already established. NodePools seem to be more about generic Karpenter scheduling specs and NodeClass about provider specific stuff. The NodePool also references the NodeClass so cx still has some level of granularity by swapping around the NodeClass that their NodePool points to. Will ask Wilson (@wdarko1) for his thoughts.

Are the other features we have on AKSNodeClass also configurable per-agentpool in AKS? Or are they cluster-wide?

FIPS, LocalDNS, and Artifact Streaming all operate at nodepool level in AKS. In particular, FIPS seems the most similar, since you can enable it at cluster creation time for the system pool but requires opt-in for subsequent nodepools.

Should we block disablement after enablement to align with current AKS behavior?

AKS RP does reject attempts to change the UltraSSD settings on an existing pool today. In theory, if someone tries to edit the field on the NodeClass CR, we should consider the node as drifted and replace it.

Should we only allow ultra SSD to be enabled in Karpenter if it's enabled on the cluster? Basically introducing a validation here based on cluster state.

I think No. Today in AKS you can make a cluster without enabling ultra ssd at creation and add an ultra-ssd enabled nodepool to it.

@PabloTriv

Copy link
Copy Markdown
Collaborator Author

I think that the proposed implementation would be pretty consistent with AKS. If you disable the field on the NodeClass, all the NodeClaims belonging to that NodePool/Class would be considered drifted and replaced, the same way you'd have to replace a nodepool in AKS.

Interestingly, while AKS nodepools do not let you update the ultra ssd field, standalone VMs can actually enable/disable it, but require being stopped and deallocated first.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

design lgtm, let's just add a customer experience section to the design doc that makes the enablement/disablement conditions clear + adds support/reasoning from the docs + current AKS behavior.

Comment thread designs/0012-ultra-ssd-support.md Resolved
Comment thread designs/0012-ultra-ssd-support.md Resolved
Comment thread designs/0012-ultra-ssd-support.md Resolved
Comment thread designs/0012-ultra-ssd-support.md Resolved
@wdarko1

Copy link
Copy Markdown
Contributor

Is there any way to make this per-nodepool? I expect many clusters will have different Nodepools for different use cases + may prefer that level of granularity. Maybe checking with Wilson (@wdarko1) based on current customer requirements could be useful.

I feel like using the AKSNodeClass is the pattern that's already established. NodePools seem to be more about generic Karpenter scheduling specs and NodeClass about provider specific stuff. The NodePool also references the NodeClass so cx still has some level of granularity by swapping around the NodeClass that their NodePool points to. Will ask Wilson (@wdarko1) for his thoughts.

Are the other features we have on AKSNodeClass also configurable per-agentpool in AKS? Or are they cluster-wide?

FIPS, LocalDNS, and Artifact Streaming all operate at nodepool level in AKS. In particular, FIPS seems the most similar, since you can enable it at cluster creation time for the system pool but requires opt-in for subsequent nodepools.

Should we block disablement after enablement to align with current AKS behavior?

AKS RP does reject attempts to change the UltraSSD settings on an existing pool today. In theory, if someone tries to edit the field on the NodeClass CR, we should consider the node as drifted and replace it.

Should we only allow ultra SSD to be enabled in Karpenter if it's enabled on the cluster? Basically introducing a validation here based on cluster state.

I think No. Today in AKS you can make a cluster without enabling ultra ssd at creation and add an ultra-ssd enabled nodepool to it.

Agree that customers should be able to select UltraSSD at the nodepool level. Customers can use multiple AKSNodeClasss IIRC so customers can just point their nodepools to a specific AKSNodeClass when needed and still achieve a node/nodepool level UltraSSD enablement. Pablo Trivino (@PabloTriv) Rachel Gregory (@rakechill)

@tallaxes Alex Leites (tallaxes) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left some questions and ideas re a bit more E2E test coverage, but nothing blocking.

Comment thread designs/0013-ultra-ssd-support.md Resolved
Comment thread designs/0013-ultra-ssd-support.md Resolved
Comment thread designs/0012-ultra-ssd-support.md Resolved
Comment thread go.sum Resolved

@tallaxes Alex Leites (tallaxes) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional E2E tests in #1797 tests passed, can decide to include them now or in followup

Comment thread designs/0013-ultra-ssd-support.md Resolved
Comment thread pkg/providers/instance/vminstance.go Resolved
Comment thread designs/README.md Resolved
@PabloTriv
Pablo Trivino (PabloTriv) merged commit 731b1f7 into main Jul 17, 2026
14 checks passed
@PabloTriv
Pablo Trivino (PabloTriv) deleted the pablotrivino/enalbeUltraSSD branch July 17, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants