Skip to content

Add retry logic for gem install to handle QEMU arm64 segfaults - #1648

Merged
suyadav1 merged 1 commit into
ci_prodfrom
sky/gem-install-retry
Apr 16, 2026
Merged

suyadav1 merged 1 commit into
ci_prodfrom
sky/gem-install-retry

Conversation

@suyadav1

Copy link
Copy Markdown
Contributor

Problem

The build_linux job in the multi-arch build pipeline fails intermittently due to sporadic GCC segfaults when compiling Ruby native gem extensions under QEMU emulation for arm64. Each failure hits a different gem dependency of fluentd (http_parser.rb, msgpack, yajl-ruby). Because setup.sh has no error handling on gem install, the failure is silently ignored, and the build later fails at the Dockerfile COPY --from=builder /usr/bin/fluentd step with a confusing "not found" error.

Build #116473 required 5 manual retries before build_linux succeeded.

Fix

Added a gem_install_with_retry() wrapper function in kubernetes/linux/setup.sh with:

  • 3 retry attempts per gem install command
  • exit 1 after exhausting retries (fail fast with a clear error)
  • Applied to all gem install calls, not just fluentd

Key Result — Retry logic validated in build #116513

The build succeeded on the first pipeline attempt. Logs show the retry logic was exercised:

  • amd64 — all gems installed on attempt 1/3, no retries needed
  • arm64 — gem install fluentd failed on attempt 1/3 (QEMU segfault), then succeeded on attempt 2/3
#47 535.6 gem install attempt 1/3: gem install fluentd -v 1.16.3 --no-document
#47 627.8 WARNING: gem install failed (attempt 1/3)
#47 629.9 gem install attempt 2/3: gem install fluentd -v 1.16.3 --no-document
#47 951.4 Successfully installed fluentd-1.16.3

Without this fix, build #116513 would have failed and required another manual re-run.

Native extension builds under QEMU emulation for arm64 hit sporadic
segfaults in GCC/make. Added gem_install_with_retry() wrapper with
3 retries and explicit exit 1 on exhaustion. Applied to all gem
install calls in setup.sh.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@suyadav1
suyadav1 requested a review from a team as a code owner April 16, 2026 23:25
@suyadav1
suyadav1 enabled auto-merge (squash) April 16, 2026 23:27
@suyadav1
suyadav1 merged commit c4f0e57 into ci_prod Apr 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants