[pelikhan]

Organization administrators need to roll out workflow updates and upgrades across many repositories without creating duplicate PRs or exhausting GitHub API budgets.

Approach

• Add --enterprise, --interactive, --dry-run, --concurrency, and --replace-existing to both update and upgrade.
• Provide a searchable repository-selection TUI with live concurrent worker statuses and resulting PR URLs. Use isolated subprocesses because the existing apply pipeline changes the process-wide working directory.
• Detect existing rollout PRs in batches, including older versions and drafts. Skip them by default; when replacement is requested, keep old PRs open until a replacement is successfully created. Workers recheck before applying changes.
• Reduce API usage with batched GraphQL discovery, a temporary shared dependency-read cache, paced worker starts, and reused quota checks. Stop admitting work on quota exhaustion or throttling. Fall back to paginated repository inventory instead of silently truncating GitHub code-search results at the 1,000-file ceiling.
• Preserve repository glob filtering against both bare names and owner/repository names, before checkout or worker execution. Document unattended usage, cancellation, replacement semantics, and rollout limitations.

Examples

# Match repository names within an organization
gh aw update --org githubnext --repos 'gh-aw-*' --interactive --pr

# Match organization-qualified names across an enterprise
gh aw upgrade --enterprise my-enterprise --repos 'githubnext/gh-aw-*' --dry-run

# Unattended rollout with explicit replacement
gh aw upgrade --org my-org --pr --yes --replace-existing --concurrency 3

Quote wildcard patterns so the shell does not expand them locally. Dry-run performs discovery and local previews without creating or closing PRs or issues, committing, pushing, or upgrading the CLI extension. Rollout workers use the currently running compiler consistently. Separate simultaneous CLI invocations are not an atomic distributed lock.

Architecture decision

ADR: ADR-72418: Isolated Subprocess Workers and Skip-by-Default Deduplication for Concurrent Org and Enterprise Rollouts

Validation

• make agent-report-progress passed, including impacted Go tests, standard and custom lint gates, and workflow lock-file drift checks.
• Focused race-detector coverage passed for worker concurrency, cancellation, throttling, argument forwarding, PR discovery/replacement, enterprise pagination, search-ceiling fallback, and shared dependency caching.
• An offline smoke test exercised the real CLI and subprocess dry-run across a mocked enterprise, skipped a repository with an existing PR, and rejected API mutation attempts. No live organization PRs were created.

[github-actions]

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · ~21 AIC · ⌖ ~6 AIC · ⊞ 14.8K · ◷

Generated by Ponytail Reviewer for #72418

[github-actions]

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

[github-actions]

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

[github-actions]

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · sonnet50 · ~14 AIC · ⌖ ~16 AIC · ◇ ~8 AIC · ⊞ 12.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

[github-actions]

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Commit pushed: 2ad71a6

🏗️ ADR gate enforced by Design Decision Gate 🏗️

[github-actions]

Review summary

No actionable blocking issues found in the changed lines. The specialist pass was unavailable because its diff-read invocation was denied, so this assessment is based on the independent static review of the supplied patch; no tests or builds were run per review instructions.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~4 AIC · ⌖ ~1 AIC · ⊞ 1.6K
_{Comment /review to run again}

[github-actions]

🏗️ Design Decision Gate — ADR Required

This PR makes significant changes to core business logic (1,774 new lines in pkg/) but does not have a linked Architecture Decision Record (ADR).

📄 Draft ADR committed: docs/adr/72418-subprocess-workers-for-concurrent-org-rollouts.md. Review and complete it before merging.

🔒 This PR cannot merge until an ADR is linked in the PR body.

Decision captured in the draft: each repository rollout runs as an isolated gh aw subprocess worker. A scheduler bounds and paces those workers and stops admitting work when the API budget runs out. Existing rollout PRs are skipped by default. With --replace-existing, old PRs are closed only after the replacement PR is created. Repository discovery falls back to the full repository inventory when code search hits its 1,000-file ceiling.

⚠️ This changes the close-before-create behavior from ADR-41580 for the rollout path. Please confirm the draft reflects that intent.

📋 What to do next

1. Review the draft ADR committed to your branch. It was generated from the PR diff.
2. Complete the missing sections. Add context the AI couldn't infer, refine the decision rationale, and list the real alternatives you considered.
3. Commit the finalized ADR to docs/adr/ on your branch.
4. Reference the ADR in this PR body by adding a line such as:

   ADR: ADR-72418: Isolated Subprocess Workers and Skip-by-Default Deduplication for Concurrent Org and Enterprise Rollouts

Once an ADR is linked in the PR body, this gate will re-run and check that the implementation matches the decision.

❓ Why ADRs Matter

"AI made me procrastinate on key design decisions. Because refactoring was cheap, I could always say 'I'll deal with this later.' Deferring decisions corroded my ability to think clearly."

An ADR is a permanent, searchable record of why the codebase looks the way it does, so future contributors don't have to guess.

📋 Michael Nygard ADR Format Reference

An ADR must contain these four sections to be considered complete:

• Context: What is the problem? What forces are at play?
• Decision: What did you decide? Why?
• Alternatives Considered: What else could have been done?
• Consequences: What are the trade-offs (positive and negative)?

All ADRs are stored in docs/adr/ as Markdown files numbered by PR number.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · opus55 · ~31 AIC · ⌖ ~42 AIC · ⊞ 1.9K · ◷
_{Comment /review to run again}

[github-actions]

Skills-Based Review 🧠

Applied /tdd and /grill-with-docs (per pr-triage, change_type: new_feature). One actionable gap found; otherwise this is a well-tested, carefully designed feature.

📋 Key Themes & Highlights

Key Themes

• Confirmation prompt shows no preview data: runOrgRolloutPlain (org_rollout.go:182) calls confirmOrgAction with an empty orgRepoPreview{Repo: row.repo} — no workflow/version diff is shown before the user approves PR creation in plain --interactive mode, unlike the legacy org_runner.go path which populates TotalWorkflows/Workflows. No test covers this call site. Flagged inline with a concrete fix suggestion.
• Docs: otherwise excellent coverage of replacement/skip semantics, flags, and examples; added a note to call out the preview gap once addressed.

Positive Highlights

• ✅ Strong race-detector test coverage for concurrency, cancellation, pacing, and budget admission (TestOrgRolloutConcurrentWorkers, TestOrgRolloutBudgetFailureStopsAdmission, TestOrgRolloutThrottlingStopsPendingWorkers, TestOrgRolloutWorkerStartsArePaced).
• ✅ Replacement lifecycle is safe-by-construction: old PRs are never closed before a new one exists (finishOrgRolloutWorker), and this is explicitly tested (TestOrgRolloutReplacementCloseError, TestOrgRolloutExistingPRPolicy).
• ✅ The rate-limit "hard-fail on budget-check error" change in update_org_search.go initially looked like a regression vs. the legacy soft-warning pattern in org_runner.go, but it's intentional ("Stop admitting work on quota exhaustion or throttling" per PR description) and is directly tested (TestOrgRolloutSearchBudgetErrorDoesNotSearch) — no action needed.
• ✅ Search-ceiling fallback to paginated listing instead of silently truncating at 1,000 results is a solid correctness improvement, with dedicated test coverage.
• ✅ Worker-subprocess isolation is well-justified and documented (avoids os.Chdir/global-compiler-state races from goroutines).

Nothing blocking; recommend addressing the confirmation-preview gap before merge or as a fast follow-up.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · ~153 AIC · ⌖ ~16 AIC · ⊞ 10.2K
_{Comment /matt to run again}

[Copilot]

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

4 open issues

• sync.WaitGroup does not have a Go method, so this code will not compile. Either switch… · New
• The unix build tag is not a standard GOOS constraint and is typically not set unless your build… · New
• This condition suppresses the compiler-version summary when a repo has no known versions and no… · New
• This PR expands rollout targeting to include --enterprise, but the RunE validation error for… · New

What changed in this PR

This PR introduces an org/enterprise “rollout” execution mode for gh aw update and gh aw upgrade, enabling bounded concurrency via subprocess workers, optional interactive TUI selection, and improved upgrade scanning/reporting of compiler versions.

Changes:

• Add org/enterprise rollout orchestration (discovery, scheduling, subprocess workers, and interactive TUI), including skip-by-default behavior for existing rollout PRs and optional “replace existing” flow.
• Enhance upgrade --org/--enterprise to scan all workflow lock headers for distinct compiler versions, report mixed/unknown/uncompiled states, and skip upgrades when versions already match.
• Add native GitHub API client caching for rollout workers and improve org search handling when code search is incomplete or at the 1,000-result ceiling.

File	Description
pkg/​cli/​upgrade_org_versions_test.go	Adds tests for lock-header compiler version parsing and upgrade preview/version reporting behaviors.
pkg/​cli/​upgrade_org_versions.go	Implements lock-header scanning and formatting for compiler version sets.
pkg/​cli/​upgrade_org_test.go	Updates org-upgrade dry-run expectations to the new version-set reporting format.
pkg/​cli/​upgrade_org_skip_test.go	Adds tests for version-match skip logic, markdown/lock counting, and worker-side “skip before mutations” behavior.
pkg/​cli/​upgrade_org.go	Refactors upgrade scanning to collect version sets, adds skip logic, and integrates rollout-worker lifecycle handling.
pkg/​cli/​upgrade_command.go	Wires org/enterprise rollout configuration into upgrade and updates help text/flag docs.
pkg/​cli/​update_org_search.go	Adds code-search completeness detection and fallback to full org repo listing; routes search via host-aware GH runner.
pkg/​cli/​update_org.go	Extends orgRepoPreview to carry compiler version sets and refactors checkout preview into a helper.
pkg/​cli/​update_github_api_test.go	Adds a test ensuring rollout workers share cached dependency reads across processes.
pkg/​cli/​update_github_api.go	Enables optional REST client caching (TTL + cache dir) for rollout workers and improves response-body close error handling.
pkg/​cli/​update_command.go	Wires org/enterprise rollout configuration into update and updates help text/flag docs.
pkg/​cli/​org_runner.go	Routes org commands through the rollout orchestrator when enabled and updates compiler version summary output.
pkg/​cli/​org_rollout_tui.go	Adds Bubble Tea-based interactive selection + live progress UI for rollouts.
pkg/​cli/​org_rollout_test.go	Adds tests for rollout flags, worker args, pacing/concurrency, cancellation, and TUI selection logic.
pkg/​cli/​org_rollout_process_unix.go	Adds Unix process-group cancellation so terminating a worker kills descendants.
pkg/​cli/​org_rollout_process_other.go	Adds non-Unix fallback process configuration.
pkg/​cli/​org_rollout_flags.go	Introduces org/enterprise rollout flags, validation, and worker arg forwarding.
pkg/​cli/​org_rollout_discovery_test.go	Adds tests for PR discovery batching/pagination, enterprise org discovery, and search ceiling fallback.
pkg/​cli/​org_rollout_discovery.go	Implements rollout PR discovery, replacement semantics, and org repo listing fallback.
pkg/​cli/​org_rollout.go	Implements rollout orchestration, IPC result protocol, worker process execution, and result aggregation/reporting.
docs/​src/​content/​docs/​setup/​cli.md	Documents org/enterprise rollouts, interactive mode, concurrency, replacement behavior, and upgrade version scanning/skip rules.
docs/​adr/​72418-subprocess-workers-for-concurrent-org-rollouts.md	Adds ADR describing the subprocess-worker rollout approach and tradeoffs.

_{Review effort: Lite}

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (pkg/cli/org_rollout.go:344): Prompt injection risk: rollout worker trusts a sentinel substring in combined stdout. Require the prefix to appear only as a dedicated line (start-of-line match). - Add interactive org and enterprise update rollouts #72418 (comment)
3. Review (pkg/cli/org_rollout.go:354): Sensitive information disclosure risk: on worker failures, stderr output is appended into result.Error; redact common credential patterns before surfacing it. - Add interactive org and enterprise update rollouts #72418 (comment)
4. Review (pkg/cli/org_rollout.go:182): confirmOrgAction is called with an empty preview, so the plain-mode confirmation prompt shows no workflow/version diff before the user approves PR creation. - Add interactive org and enterprise update rollouts #72418 (comment)
5. Review (docs/src/content/docs/setup/cli.md:1129): Document the preview gap limitation or fix org_rollout.go:182 as noted. - Add interactive org and enterprise update rollouts #72418 (comment)
6. Review (pkg/cli/org_rollout.go:337): Sensitive information disclosure risk: org rollout workers set GH_HOST from untrusted git remote URL. Restrict/validate the host or require explicit user-provided host. - Add interactive org and enterprise update rollouts #72418 (comment)
7. Review (pkg/cli/org_rollout.go:344): Prompt injection risk: the rollout parent parses worker stdout by searching for prefix. Authenticate the IPC or require result to be the final exact line. - Add interactive org and enterprise update rollouts #72418 (comment)
8. Review (pkg/cli/org_rollout.go:283): sync.WaitGroup does not have a Go method. Switch to errgroup.Group or use WaitGroup.Add(1)/Done(). - Add interactive org and enterprise update rollouts #72418 (comment)
9. Review (pkg/cli/org_rollout_process_unix.go:1): Use OS constraints that Go sets by default (e.g., (go/redacted):build !windows) instead of the non-standard unix tag. - Add interactive org and enterprise update rollouts #72418 (comment)
10. Review (pkg/cli/org_runner.go:99): Include preview.MissingWorkflowLocks > 0 in condition so uncompiled-only repos still display compiler status. - Add interactive org and enterprise update rollouts #72418 (comment)
11. Review (pkg/cli/upgrade_command.go:233): Update validation error message to mention --enterprise alongside --org. - Add interactive org and enterprise update rollouts #72418 (comment)
12. Review (pkg/cli/org_rollout.go:337): Sanitize high-risk debug/trace env vars (GH_DEBUG, GH_CURL_VERBOSE, etc.) before launching workers and redact patterns from stderr. - Add interactive org and enterprise update rollouts #72418 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef state: eb944ccdf8d26242697fa4874278e040ea1c93c1c6d23c086426d37db34955b9

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · ~14 AIC · ⌖ ~9 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[github-actions]

🎉 This pull request is included in a new release.

Release: v1.0.1