[kj4ezj]

Change Description

@vzqzhang introduced a new long-running test, the nodeos_contrl_c_lr_test, in pull request 9494. @nksanthosh asked me to perform stability testing on this new test. This means setting up Buildkite to execute the same test over and over hundreds or thousands of times in order to either detect non-deterministic results, or to build confidence that this test has a deterministic result.

The pipeline almost supported doing this using environment variables alone...

ROUNDS='100'
SKIP_DOCKER='true'
SKIP_INSTALL='true'
SKIP_LONG_RUNNING_TESTS='false'
SKIP_MULTIVERSION_TEST='true'
SKIP_PACKAGE_BUILDER='true'
SKIP_SERIAL_TESTS='true'
SKIP_SYNC_TESTS='true'
SKIP_UNIT_TESTS='true'
SKIP_WASM_SPEC_TESTS='true'

...but the pipeline code currently stops short of providing a way to isolate a single test. Worse more, the existing pipeline code uploads jobs as placeholders for skipped steps instead of not creating skipped jobs at all. This was done intentionally, so it is both documented and more obvious to engineers steps were skipped in the Buildkite UI. However, in this case, all the skipped steps count against a global limit on the number of jobs per build enforced by Buildkite, meaning we'd only be able to perform a handful of runs anyways.

Instead, this pull request simply deletes all unused steps and hard-codes the desired test.

I also considered adding support to the pipeline to do this without any code changes but, due to unnecessary complexity, I don't think this would be worth the time investment compared to just deleting code like this. Instead, I recommend we rewrite the pipeline code in a real programming language (JavaScript) that will enable us to support this level of pipeline complexity without having such obfuscated code.

Change Type

Select ONE

• Documentation
• Stability bug fix
• Other
• Other - special case
  CI.

Consensus Changes

• Consensus Changes
  None.

API Changes

• API Changes
  None.

Documentation Additions

• Documentation Additions
  None.

[kj4ezj]

Tested in build 24759, which successfully completed 21 runs in batches of 14.

Edit: All 21 runs passed!

[kj4ezj]

I kicked off build 24760 to perform 2100 runs in 30 rounds of 70 runs each (ten for each operating system).

Edit: All 2100 runs passed!

[kj4ezj]

Closing, as pull request 9494 has been merged.