Skip to content

conformance: make timeout and other fields configurable via yaml - #4773

Merged
k8s-ci-robot merged 26 commits into
kubernetes-sigs:mainfrom
ericdbishop:yaml-conformance-options
May 21, 2026
Merged

k8s-ci-robot merged 26 commits into
kubernetes-sigs:mainfrom
ericdbishop:yaml-conformance-options

Conversation

@ericdbishop

Copy link
Copy Markdown
Contributor

What type of PR is this?

/area conformance-machinery

What this PR does / why we need it:

See #4346. Makes timeouts and other conformance option fields configurable via a yaml file, whose path is passed in as a flag. This is ultimately a step towards making conformance a separate binary to distribute to implementations.

When a conformance option is configured via both yaml and a flag, the flag takes precedence. Flag values only override yaml configuration if the flag was explicitly passed in via the command line.

WIP: Deciding additional fields of the configuration structure that should be accepted in the yaml configuration.

AI disclosure: Used copilot to review my code and make some idiomatic code optimizations.

Which issue(s) this PR fixes:

Fixes #4346

Does this PR introduce a user-facing change?:

NONE

Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. area/conformance-machinery Issues or PRs related to the machinery and the suite used to run conformance tests. labels Apr 20, 2026
@k8s-ci-robot
k8s-ci-robot requested review from kl52752 and snorwin April 20, 2026 21:36
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 20, 2026
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Hi @ericdbishop. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 20, 2026
@rikatz

rikatz commented Apr 20, 2026

Copy link
Copy Markdown
Member

/cc

@k8s-ci-robot
k8s-ci-robot requested a review from rikatz April 20, 2026 22:36
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
@ericdbishop
ericdbishop marked this pull request as ready for review April 21, 2026 13:53
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 21, 2026
Comment thread conformance/utils/config/timeout.go Resolved
Comment thread conformance/utils/suite/suite.go Resolved
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Comment thread conformance/conformance_options_test.go Resolved
Comment thread conformance/conformance.go Resolved
Comment thread conformance/conformance.go Resolved
Comment thread conformance/conformance.go Resolved
@linux-foundation-easycla

linux-foundation-easycla Bot commented Apr 22, 2026 •

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 22, 2026
@ericdbishop
ericdbishop force-pushed the yaml-conformance-options branch from fd6ed57 to f381a14 Compare April 22, 2026 17:14
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 22, 2026
@ericdbishop
ericdbishop requested review from rikatz and snorwin April 24, 2026 21:31
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
return err
}

v := reflect.ValueOf(tc).Elem()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's just confirm this is the best approach.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this is the best approach to avoid changing the type of the duration fields or wrapping Duration. I had tried changing the type to metav1.Duration and it resulted in a lot of code changes: f9ba683#diff-072d1356145c7619d3b242a24c0b5178dd155e84cca0880894a10cc4f2a50302

Also, it might help to check out this other comment thread: #4773 (comment).

All that being said, I will make a change to avoid the panic if a non-duration json tag is added to the timeout struct.

Comment thread conformance/utils/suite/conformance.go Resolved
Comment thread conformance/utils/suite/conformance.go Resolved
Comment thread conformance/utils/suite/suite.go Resolved
Comment thread conformance/utils/suite/suite.go Resolved
Comment thread conformance/utils/suite/suite.go Outdated
ClientOptions client.Options
Clientset clientset.Interface
RestConfig *rest.Config
AddressType string

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on our sync review, we are not sure this is being used, so please take a look what happens if we just drop this field

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be removed since it has no affect on the testing. Making that change.

Comment thread conformance/utils/flags/flags.go Resolved
Comment thread conformance/utils/flags/flags.go Resolved
@rikatz

rikatz commented May 20, 2026

Copy link
Copy Markdown
Member

As we also have discussed, here are AI findings :)

https://gist.github.com/rikatz/a10ecf57342aac61e5dff50039f1721d

Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
Signed-off-by: Eric Bishop <ericbish.dev@gmail.com>
@rikatz

rikatz commented May 21, 2026

Copy link
Copy Markdown
Member

/lgtm
/approve
thanks @ericdbishop this is a great start to move Conformance to its own binary/container

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2026
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ericdbishop, rikatz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2026
@k8s-ci-robot
k8s-ci-robot merged commit a392c6d into kubernetes-sigs:main May 21, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/conformance-machinery Issues or PRs related to the machinery and the suite used to run conformance tests. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conformance - Make timeout and other fields configurable

4 participants