Repository navigation
ci: run the tests CI skipped, and fail on test files with no CI job - #8203
Closed
AlemTuzlak wants to merge 7 commits into
Closed
AlemTuzlak wants to merge 7 commits into
AlemTuzlak wants to merge 7 commits into
Conversation
PE-1113 found these test files with no CI entry point: seven scripts/__tests__ vitest files, scripts/deprecations, the doc-tests extractor, the release shell test, codemods/__tests__, the v2-migration codemod cases, the custom oxlint rules and the learned-skill workspace-artifact check. A new `repo-scripts:test` target lists the scripts files, and test_unit-repo-tooling.yml runs it with the three existing project targets. The live-tree case in validate-integration-pins.test.ts fails on its first run (the adk starter pins 1.79.0 while the repo is at 1.84.0, and the validator compares @copilotkit/channels against the react-core version), so it is skipped with that reason until the pins and the validator are fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
packages/vue's vitest include is `**/__tests__/**/*.test.ts`, so CopilotChatMessageView.testid.spec.ts never ran, locally or in CI (PE-1113). It passes under the new name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The showcase pytest loop only entered packages with tests/python/, so PE-1113 found Python tests no job ran: tests/test_*.py in agno, langgraph-fastapi, langgraph-python and pydantic-ai, langgraph-python's src/agents/test_*.py, and showcase/integrations/_shared/tests. The loop now also runs those modules, in a second pytest process so tests/conftest.py never reaches tests/python/. Some import an agent module that builds an OpenAI client at import time, so that process gets a placeholder OPENAI_API_KEY. langgraph-python and langgraph-fastapi join the 3.10 skip list: their pinned langgraph-api requires Python 3.11. A new step runs _shared/tests. The showcase shell's vitest suite (10 files) also had no CI job; it now runs before the shell build. Delete langgraph-python's src/agents/test_agent_config_agent.py: it tests read_properties, build_system_prompt and DEFAULT_* constants that agent_config_agent.py no longer has (the agent now reads its settings through useAgentContext), so it fails at import and guards nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PE-1113 found three showcase suites no job ran: - showcase/harness test/integration (browser mechanism tests against fake pages). test_unit-showcase.yml now runs `test:integration`. The four bubble-race repro files drive `bin/showcase test --d5` against a locally running stack, so vitest.integration.config.ts excludes them unless BUBBLE_RACE_LIVE=1, and the one such case in bubble-race-mechanisms is skipped. probe-contract's multi-turn ctx case fails on its first run (turn 2 gets turn 1's bubble text) and is skipped with that reason until someone investigates. - showcase/angular and showcase/vue unit suites, in a new `hosts` job. The Angular a2ui catalog test failed to resolve @copilotkit/a2ui-renderer/engine, which @copilotkit/angular/a2ui imports; the host now declares that workspace dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PE-1113 found these integration test files with no CI job: - vitest suites in built-in-agent, claude-sdk-typescript, langgraph-typescript and mastra (test_unit-mastra.yml ran one mastra file). A matrix job installs each integration's own lockfile and runs its vitest config; langgraph-typescript also installs src/agent so @langchain/* resolves to the agent's versions, and runs the agent's node:test file. - the shared-tools/__tests__ copies in claude-sdk-typescript and langgraph-typescript (real files where showcase/AGENTS.md expects symlinks). They pass, and now run through each integration's include. - _shared/ts and the strands-typescript proxy null-guard script, in the existing job. - the three xUnit projects next to the .NET agents, in a dotnet matrix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PE-1113 found these example tests with no CI job. The new test_unit-examples.yml runs them: - examples/slack, examples/teams and examples/v2/react/demo, through their nx `test` targets. The React demo's `test` script ran vitest in watch mode; it now runs `vitest run`. - standalone npm examples: agentcore infra-cdk (jest), the strands-typescript starter agent, the claude-managed-agents showcase server, and strands-crm agent and frontend. - source-only node tests in chatkit-studio and mcp-apps. - uv-locked Python agents: a2a-a2ui's a2ui_extension, llamaindex, ms-agent-framework-python and antigravity-repo-doctor. test_reskinnable-demo.yml now also runs the Playwright specs that use aimock (27 pass, 2 are `test.fixme`). Empty Intelligence variables keep the config from pointing at the local Docker memory stack. Delete the React Native demo's __tests__/App.test.tsx, jest.config.js and `test` script: it is the default React Native template test, which renders <App /> with no assertion and cannot run under pnpm (the jest preset's setup file fails to parse), so it guarded nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test that CI never runs looks like coverage but guards nothing, and mutation testing only scores tests that run (PE-1113). scripts/test-ci-coverage-map.mjs maps every tracked test file (TS/JS, Python, Go, Ruby, Java, .NET, Dart, Swift, shell) to the workflow step that runs it, with globs as narrow as each runner's own include, or to an allowlist entry that says why nothing runs it. scripts/validate-test-ci-coverage.mjs fails on a test file with neither, on a rule whose workflow no longer contains the command it names, and on a rule or allowlist entry that matches no file. It is pure Node over `git ls-files` and the workflow files. static_test-ci-coverage.yml runs it through `repo-scripts:validate-test-ci-coverage` on every pull request, with no paths filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review: 1
Please check the settings in the CodeRabbit UI or the
 |
Contributor
Production Digest-Pinning AuditAll 43 services digest-pinned. Run 2026-11-04 05:19:07 PST — 0 finding(s). |
Contributor
|
Size Change: 0 B Total Size: 2.39 MB ℹ️ View Unchanged
|
@copilotkit/a2ui-renderer
@copilotkit/agentcore-runner
@copilotkit/angular
@copilotkit/channels
@copilotkit/channels-core
@copilotkit/channels-discord
@copilotkit/channels-intelligence
@copilotkit/channels-slack
@copilotkit/channels-teams
@copilotkit/channels-ui
@copilotkit/channels-whatsapp
@copilotkit/core
@copilotkit/intelligence-langgraph
@copilotkit/intelligence-mastra
@copilotkit/learning
@copilotkit/markdown
@copilotkit/mcp-apps-renderer
@copilotkit/react-core
@copilotkit/react-native
@copilotkit/runtime
@copilotkit/sdk-js
@copilotkit/shared
@copilotkit/sqlite-runner
@copilotkit/voice
@copilotkit/vue
@copilotkit/web-components
@copilotkit/web-inspector
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Next action for a maintainer: make the job
static / test ci coverage→Every test file has a CI joba required check on2.0.0(and onmainafter the next sync). It runs on every pull request, with nopathsfilter, in a few seconds. This PR does not change branch protection.About 280 test files (73 of them codemod cases) had no CI job, so they guarded nothing. Mutation testing (PE-1076) scores only tests that run. This PR adds those tests to CI, deletes 2 dead tests, and adds a check that fails when a test file has no CI job. Now all 3400 tracked test files map to a CI job or to an allowlist entry with a reason.
What does this PR do?
scripts/validate-test-ci-coverage.mjsreadsgit ls-filesand matches each test file againstscripts/test-ci-coverage-map.mjs(123 rules, 13 allowlist entries). It covers TS/JS, Python, Go, Ruby, Java, .NET, Dart, Swift and shell tests.static_test-ci-coverage.yml(the check),test_unit-repo-tooling.yml,test_unit-examples.yml.showcase_validate.yml,test_unit-showcase.yml,test_unit-showcase-integrations.yml,test_reskinnable-demo.yml.pnpm nx run repo-scripts:validate-test-ci-coverage. Add--reportto the script to print the full map.Test files that had no CI job, and what this PR does
packages/vue/.../CopilotChatMessageView.testid.spec.ts(1)*.test.ts).test.ts. Runs intest_unit.yml.scripts/__tests__: agentcore ×2, lint-baseline, lint-gate, runtime-entitlement-recovery, telemetry-fragment, validate-integration-pins (7)repo-scripts:testintest_unit-repo-tooling.yml. 1 case skipped (see below).scripts/deprecations/v1-dist-notices,scripts/doc-tests/__tests__/extract,scripts/release/__tests__/detect-py-version-changes.test.sh,codemods/__tests__(4)repo-scripts:testcodemods/v2-migration/tests/*/*jssg cases (73)@copilotkit/v2-migration:testtools/oxlint-rules(3),tools/learned-skill-conformance/workspace-artifacts(1)oxlint-rules:test,test-workspace-artifactsshowcase/angular(22),showcase/vue(6)hostsjob intest_unit-showcase.yml. The Angular host now declares@copilotkit/a2ui-renderer. Without it, the a2ui catalog test could not resolve@copilotkit/a2ui-renderer/engine.showcase/shell/src(10)npm teststep inshowcase_validate.ymlshowcase/harness/test/integration(9)test:integrationintest_unit-showcase.yml. 4 bubble-race repros are allowlisted, and 2 cases are skipped.tests/python/: agnotests/(5), langgraph-fastapi (1), langgraph-pythontests/(2) +src/agents/(3), pydantic-ai (1)tests/python/showcase/integrations/_shared/tests(6)showcase_validate.ymltest_unit-mastra.ymlran 1 mastra file)test_unit-showcase-integrations.ymlthat installs each integration's lockfile_shared/ts(1), strands-typescript proxy null-guard (1)test_unit-showcase-integrations.yml_shared/dotnet, ms-agent-dotnet, ms-agent-harness-dotnet (3 projects, 97 tests)dotnet testmatrix intest_unit-showcase-integrations.ymlexamples/slack(16),examples/teams(1),examples/v2/react/demo(3)test_unit-examples.yml(nx). The React demotestscript ran vitest in watch mode. It now runsvitest run.test_unit-examples.ymlnode --testintest_unit-examples.ymltest_unit-examples.ymlexamples/showcases/reskinnable-demo/e2e: 5 of 7 specstest_reskinnable-demo.yml, run against aimock (27 pass, 2 aretest.fixme)examples/v2/react-native/demo/__tests__/App.test.tsx(1)langgraph-python/src/agents/test_agent_config_agent.py(1)Allowlisted: no CI job, on purpose
Each entry in
ALLOWLISThas a reason in the code.test/e2e(1);showcase/scripts/__tests__/e2e(3);showcase/testsintegration-smoke and env-routing;showcase/pocketbase/tests(4, need a PocketBase container); reskinnablememory-learning.spec.ts; oracle-agent-memory (3, need an Oracle DB).test_multi_turn.pyandevals/test_agent.pyare scripts that call OpenAI and LangSmith.ogui-routing.spec.ts(describe.fixme); shell-dashboard spike and visual specs (excluded already, with reasons intest_unit-showcase.yml); FlutterRunnerTests.swift(unmodifiedflutter createscaffold).Skipped tests (they failed on their first CI run)
scripts/__tests__/validate-integration-pins.test.ts› live tree. The adk starter pins 1.79.0 while the repo is at 1.84.0. The validator also compares@copilotkit/channels(0.x) against the react-core version.showcase/harness/test/integration/probe-contract.test.ts› multi-turn ctx. Turn 2 gets the bubble text of turn 1. This needs investigation.showcase/harness/test/integration/bubble-race-mechanisms.test.ts› s1. It needs a running showcase stack, the same as the allowlisted repros.Findings to know about
/test-aimock <slug>comment,test_e2e-showcase-on-demand.yml). The map records them that way.test_unit.ymlexcludes 8 non-TS projects. Each one has its own workflow (intelligence-runtimes.yml,intelligence-skill-adapters.yml), and the map checks that each workflow still runs it.claude-sdk-typescriptandlanggraph-typescriptkeep real copies ofshared-tools/instead of symlinks (seeshowcase/AGENTS.md). Their tests pass and now run. Restoring the symlinks is out of scope.oxlint-rules:testnow runs here. PE-1069 (lint rules) can move it to its own job if it wants.Related PRs and Issues
Checklist
!(for examplefeat(react-core)!: ...) and the description has aBREAKING CHANGE:line with migration steps. See VERSIONING.mdNo user-facing docs change: CI and test files only.
Testing
Commands run locally (macOS, Node 22). All passed, except where the table above lists a skip.
pnpm nx run repo-scripts:validate-test-ci-coverage: 9 checker tests pass.OK: 3400 test files map to a CI job.pnpm nx run repo-scripts:test,@copilotkit/v2-migration:test(55 cases pass; jssg reportstsandtsxseparately),oxlint-rules:test,learned-skill-conformance:test-workspace-artifacts.pnpm nx run-many -t test -p slack-example,teams-example,demo,@copilotkit/showcase-angular-host:test(20 files),@copilotkit/showcase-vue-host:test(6 files), harnesstest:integration.dotnet teston the 3 .NET projects (.NET 9).Not run locally: bats, the Ruby spec and the Flutter steps. These steps did not change. The CI runs on this PR are the first ubuntu runs of the new jobs.
Manual test
packages/foo/x.test.ts. No rule covers this path.git add packages/foo/x.test.ts. The check reads the git index.node scripts/validate-test-ci-coverage.mjs. The command fails and names the file.git rm -f packages/foo/x.test.ts. Then run the check again. It printsOK.How this PR makes testing easy: the checker has its own tests (
scripts/__tests__/validate-test-ci-coverage.test.mjs). Every new CI step is an nx target or a short command that you can copy from the workflow.Risk / rollback
🤖 Generated with Claude Code