⚽ Live Match: aws cloudformation/cfn lint/pull/

kddejong · 2026-10-23T21:37:35Z

Problem

The agent context files under .kiro/ rank evidence for a constraint as
"real templates that deploy, then a live API probe, then AWS docs, then the
base CFN schema". The phrase "a live API probe" was never defined, and an agent
reading it treated aws cloudformation validate-template as satisfying tier 2 —
running it 18 times and reporting the result as the strongest evidence in a
review.

validate-template parses a template and checks template-level structure. It
never invokes a resource provider, so for the overwhelming majority of cfn-lint
rules it proves nothing, and where it does respond it may be stricter or looser
than CreateStack. Treating it as proof risks encoding a constraint
CloudFormation does not actually enforce, which is exactly the false-positive
class .kiro/context/false-positives.md exists to prevent.

What changed

Docs only — nothing under src/ or test/. There is no code change here, so
there are no unit tests to add.

File	Change
`.kiro/context/false-positives.md`	Evidence list reranked to five items with a real deployment first; added the named `validate-template` exclusion; review checklist item 1 now asks for a deployment or the owning service's API
`.kiro/context/verification.md`	Same reranking in "Verifying against reality"; added the `validate-template` exclusion, an AWS-docs fetching note, and five verified environment traps (see below)
`.kiro/steering/rule-development.md`	"Validation Approach" step 1 and the "Documentation" evidence bullet reworded to match

The corrected hierarchy, strongest first: a real deployment; real templates
known to deploy; a service-level API probe; AWS documentation; the base CFN
resource schema. Plus, stated as its own item: aws cloudformation validate-template is not evidence for a property-level constraint.

Wording was kept in each file's existing voice and length — the ranked lists
stay ranked lists.

Secondary: environment facts added to `verification.md`

Each was re-run in this clone before being written. Five of six went in; one was
corrected by testing and one claim in it was dropped.

Verified and added

The -m data trap. pyproject.toml sets addopts = ["-m not data"], so
test/integration/test_data_file_format.py is deselected locally, while
.github/workflows/ci-pr-data.yaml runs tox -e py -- -m "data" on any
change under src/cfnlint/data/**. Confirmed by replicating the test's
assertion on a tracked data file: the on-disk content passes, the same
content with reversed top-level keys fails. format_json_string is
indent=1, sort_keys=True, separators=(",", ": ") plus a trailing newline.
Also documented a trap within the trap: running -m data locally fails on a
gitignored downloaded schema under src/cfnlint/data/schemas/resources/ that
arrives without a trailing newline, so the red result is often about a file
you never touched.
strict_types makes negative unit tests lie. test/unit/conftest.py
defaults strict_types=True; rules/resources/Metadata.py:71,
rules/outputs/Value.py:55 and
rules/resources/properties/Properties.py:87 evolve to False — line
references confirmed. Demonstrated directly: "123" against
{"type": "integer"} yields '123' is not of type 'integer' under
strict_types=True and zero errors under False.
Testing a branch's code. python -m cfnlint fails with
No module named cfnlint.__main__ (there is no __main__.py; the entry
point is cfnlint.runner:main). Confirmed the editable-install behaviour by
adding a marker line to a throwaway worktree's src/cfnlint/__init__.py:
run from the worktree, the venv interpreter resolved the main checkout and
the marker was absent; with PYTHONPATH=<worktree>/src it resolved the
worktree and the marker was present. The .pth holds the main checkout's
absolute src path. The throwaway worktree was reverted and removed.
Scratch templates need alphanumeric names. Confirmed: a template with
my_bucket-1 and Output my_out-1 reports E3001 and E6001
(does not match any of the regexes: '^[a-zA-Z0-9]+$'); the identical
template with MyBucket1/MyOut1 exits 0.
Decoder entry points differ — added, with a correction. cfn_yaml.load
does return the template dict directly, but the stated failure signature is
only half the story: unpacking it raises ValueError: too many values to unpack only when the template has a number of top-level keys other than
two. With exactly two keys it silently binds the two top-level key names,
which is the more dangerous outcome. Both confirmed, and the text says so.

Corrected before writing

Reading AWS docs. The claim as given — that web_fetch truncates
docs.aws.amazon.com at ~8000 characters and that appending .md returns
the full source — did not reproduce. Fetching the HTML page in full-content
mode returned the complete page including every property's
Minimum/Maximum/Allowed values. The ~8000-character cut is the
default extraction mode, and it truncated the .md URL as well, mid-way
through the property list; the .md rendering additionally mangles inline
URLs. So the note that went in says: use full-content mode, and treat .md
as an alternative source subject to the same default truncation. The original
framing was not written.

Commands run

# sweep for the ambiguous wording, including .kiro/prompts/
grep -rn -i 'probe\|validate-template' .kiro/

# item 1
.venv/bin/python -m pytest test/integration/test_data_file_format.py          # 1 deselected
.venv/bin/python -m pytest test/integration/test_data_file_format.py -m data  # selected
# plus a script replicating the assertion against format_json_string on a
# tracked data file, sorted vs reversed keys

# item 2
.venv/bin/python -c "<CfnTemplateValidator with strict_types True vs False on '123'>"

# items 3, 4, 5
.venv/bin/python -m cfnlint --version
.venv/bin/cfn-lint -t /tmp/kiro_names.yaml      # E3001 + E6001, exit 2
.venv/bin/cfn-lint -t /tmp/kiro_names2.yaml     # exit 0
# worktree + PYTHONPATH marker test; cfn_yaml.load unpacking at 2 and 3 keys

# gate
.venv/bin/pre-commit run --files .kiro/context/false-positives.md \
  .kiro/context/verification.md .kiro/steering/rule-development.md

pre-commit passes on the changed files: check vcs permalinks,
fix end of files, mixed line ending and trim trailing whitespace all pass;
the Python-only hooks are skipped because nothing Python changed.

A bare local ruff format --check .kiro/ (ruff 0.16.3) reports one file it
would reformat, .kiro/steering/rule-development.md. This is pre-existing
and not what CI runs: extracting .kiro/ from upstream/main into a
scratch directory produces the identical single finding, and the pinned hook is
ruff v0.12.4, which does not format markdown at all. pre-commit run --all-files is clean. It is a fenced Python block this change does not touch,
and it was deliberately left alone.

Proposed agent-prompt changes (NOT in this PR)

.kiro/agents/*.json carry the same flawed wording, but all three files have
uncommitted local edits, so nothing under .kiro/agents/ is staged, changed
or reverted by this PR. The replacements below are for applying by hand.

.kiro/agents/pr-reviewer.json needs no change — its prompt contains no
evidence hierarchy and never mentions a probe.

`.kiro/agents/issue-triage.json`

Before, inside the FALSE POSITIVE paragraph:

Evidence ranked strongest first: it deploys, or an API probe accepts it, or AWS documentation says it is legal. The base CFN schema is the weakest evidence and is frequently wrong.

After:

Evidence ranked strongest first: a real deployment succeeds, or real templates known to deploy use the value, or the service that owns the constraint accepts it through its own API, or AWS documentation says it is legal. The base CFN schema is the weakest evidence and is frequently wrong. aws cloudformation validate-template is NOT evidence: it checks template-level structure and never invokes a resource provider, so it cannot establish a property-level constraint.

`.kiro/agents/issue-worker.json`

Before, step 2:

PROVE THE CONSTRAINT. If the change encodes a limit, a pattern, or an enum, establish the real value. Evidence ranked strongest first: real templates that deploy, then a live API probe, then AWS documentation, then the base CFN schema (weakest, and often silent or wrong). Never derive a constraint from the base schema alone. Record the evidence for the PR description, including account, region, and date for any live probe.

After:

PROVE THE CONSTRAINT. If the change encodes a limit, a pattern, or an enum, establish the real value. Evidence ranked strongest first: a real deployment (CreateStack/UpdateStack), then real templates known to deploy, then a service-level API probe against the service that owns the constraint, then AWS documentation, then the base CFN schema (weakest, and often silent or wrong). Never derive a constraint from the base schema alone, and never from aws cloudformation validate-template — it checks template-level structure without invoking a resource provider, so it cannot establish a property-level constraint. Record the evidence for the PR description: account, region, date and the verbatim error for a deployment; account, region and date for a service API probe.

Follow-up commit `6f3a809` — a free deployment probe for parameter constraints

The reranked hierarchy puts a real deployment first and names
validate-template as not evidence, but it offered no technique for obtaining
a deployment. That gap is what pushed a reviewing agent to validate-template
in the first place: the file demanded deploy-grade evidence and gave no
affordable route to it.

.kiro/context/verification.md gains a section immediately after the ranked
list and the validate-template paragraph, in the file's existing voice:

A parameter's Default is validated in the CloudFormation control plane
before any resource exists, so a create-stack probe is free and leaves
nothing behind.
The shape: the parameter under test plus one
AWS::CloudFormation::WaitConditionHandle. A rejection returns synchronously
and creates no stack at all; an acceptance creates one free resource, deleted
immediately.
It covers the whole parameter-constraint family, E2015, E2003 and their
neighbours.
The measured result: CloudFormation renders a boolean Default lowercase.
Default: true deploys against AllowedValues: ["true", "false"]; against
["True", "False"] it fails Parameter 'P' must be one of AllowedValues;
against AllowedPattern: ^(True|False)$ it fails Parameter 'P' must match pattern ^(True|False)$. A CommaDelimitedList fails Each value of parameter 'P' must be one of AllowedValues.

Seven cases, recorded in us-east-1 on 2026-10-23. That coercion model is
what three claims in the review of PR #4797 turned on. Per the ranked list's own
instruction the account id belongs in a PR description rather than a checked-in
file, since .kiro/ ships in this public repo, so the committed text carries
region and date only.

Pointer, not duplication

.kiro/context/false-positives.md gains one line pointing at the technique,
because its §4 makes the strongest demand for deployment evidence and previously
offered no route to it.

.kiro/steering/rule-development.md gets nothing: its "Before Submitting"
section already routes command-level detail to verification.md, so a second
pointer would be redundant. One home plus one pointer.

Still docs-only — nothing under src/ or test/ — so there are no unit tests
to add.

The `unitlint (ubuntu-latest, 3.14)` failure is pre-existing

Verdict: not this PR's doing, and already fixed on upstream/main, so the
merge below resolves it.

The failing job's log names exactly one test:

FAILED test/integration/test_integration_templates.py::TestQuickStartTemplates::test_templates

on test/fixtures/templates/integration/aws-ec2-subnet.yaml, with
Second list contains 4 additional elements — four E3020 matches
('CidrBlock' should not be included with 'Ipv4NetmaskLength' and its converse,
on Subnet1 and Subnet5) that a newer schema set produces and the recorded
snapshot did not list. Upstream commit 8f7e4857e ("Update tests for schema
changes", #4872) adds precisely those four entries to
test/fixtures/results/integration/aws-ec2-subnet_yaml.json.

Reproduced on both trees against one identical freshly downloaded schema set
(schema_date 2026-10-23T22:24:42Z) in an isolated XDG_CACHE_HOME:

Tree	Result
this branch before the merge (`c9be2d04d`)	FAILED, the same four extra `E3020` entries as CI
clean `upstream/main` (`a06f5b206`)	passed
this branch after merging `upstream/main`	passed

The inverse holds too, which confirms the coupling: with an older local schema
set (2026-10-22) the outcome flips, this branch passing and upstream/main
failing. The snapshot is pinned to schema behaviour, not to any code here.

The alternative explanation — pre-commit's ruff-format reaching fenced code
blocks in markdown — is ruled out four ways:

tox-gh maps Python 3.14 to py314 only. The style env that runs
pre-commit run --all-files is mapped to 3.13, so the failing job never ran
it. The log agrees: the one tox env is py314, and neither ruff nor
pre-commit appears anywhere in it.
unitlint (macos-latest, 3.13) and (windows-latest, 3.13) both passed, so
the style env itself passes on this branch.
The pinned hook is ruff v0.12.4, which does not format markdown at all.
Running the hooks against the changed files reports
ruff format ... (no files to check).
pre-commit run --all-files is clean on this branch, every hook passing.

Nothing was changed to address the failure beyond the merge, which was required
anyway.

Bringing the branch current: merged, not rebased

The branch was BEHIND. A rebase would require a force-push, which this
environment refuses, so upstream/main was merged into the branch instead
(1634243). No history was rewritten and nothing was force-pushed. This PR is
squash-merged, so the merge commit costs nothing in the final history; if a
linear history is preferred, a maintainer can rebase and force-push directly.

Commands run for this follow-up

gh run view --job <failing job id> --log-failed       # named the single failing test

# both-trees reproduction against one identical fresh schema set
XDG_CACHE_HOME=/tmp/iso <venv>/python -c "from cfnlint.runner.cli import main; main()" --update-specs
XDG_CACHE_HOME=/tmp/iso PYTHONPATH=<tree>/src <venv>/python -m pytest \
  test/integration/test_integration_templates.py::TestQuickStartTemplates::test_templates -q

# full gate on the merged branch
XDG_CACHE_HOME=/tmp/iso PYTHONPATH=<tree>/src <venv>/python -m pytest test -q
XDG_CACHE_HOME=/tmp/iso PYTHONPATH=<tree>/src <venv>/python -m pytest test -q -m data
<venv>/pre-commit run --files .kiro/context/verification.md .kiro/context/false-positives.md
<venv>/pre-commit run --all-files

pre-commit run --all-files: clean, all fourteen hooks pass.

The full suite reports 19 failed, 3127 passed — identical on clean
upstream/main, caused by pydot, junit_xml, sarif_om and
jschema_to_python being absent from this local venv while CI installs
.[full]. -m data passes (4 tests) on both trees. No file under
src/cfnlint/data/** changed, so ci-pr-data.yaml does not trigger.

.kiro/agents/ remains untouched: git diff upstream/main...HEAD -- .kiro/agents/
is empty, and the branch diff contains no credential block and no account id.

The evidence hierarchy in .kiro said 'a live API probe' without defining it, and an agent read 'aws cloudformation validate-template' as satisfying it. validate-template parses a template and checks template-level structure; it never invokes a resource provider, so it cannot establish a property-level constraint, and treating it as proof risks encoding a limit CloudFormation does not enforce. Rank a real deployment first, then real templates known to deploy, then a service-level API probe, and name validate-template explicitly as not evidence. Also record environment facts that produced wrong conclusions: the -m data sorted-keys requirement that only CI enforces, strict_types diverging between conftest and real rule paths, editable-install resolution from a worktree, E3001/E6001 on non-alphanumeric scratch names, and the two decoder entry-point shapes.

The evidence hierarchy ranks a real deployment first and names validate-template as not evidence, but it gave no technique for obtaining a deployment. That gap is what pushed a reviewing agent to validate-template: the file demanded deploy-grade evidence and offered no affordable route to it. A parameter's Default is validated in the CloudFormation control plane before any resource exists, so a create-stack probe is free. The template carries the parameter under test plus one AWS::CloudFormation::WaitConditionHandle. A rejection returns synchronously and creates no stack at all; an acceptance creates one free resource, deleted immediately. This covers the whole parameter-constraint family, E2015, E2003 and their neighbours. Record the measured coercion model: CloudFormation renders a boolean Default lowercase. Default: true deploys against AllowedValues ["true", "false"], and fails against ["True", "False"]. Seven cases, us-east-1, 2026-10-23. The account id stays in the PR description, since .kiro ships in the public repo. Point false-positives.md at the technique instead of repeating it. rule-development.md already routes command detail to verification.md, so it needs no pointer.

…hierarchy

codecov · 2026-10-24T00:10:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.68%. Comparing base (a06f5b2) to head (1634243).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4867   +/-   ##
=======================================
  Coverage   94.68%   94.68%           
=======================================
  Files         439      439           
  Lines       16004    16004           
  Branches     3108     3108           
=======================================
  Hits        15153    15153           
  Misses        468      468           
  Partials      383      383

Flag	Coverage Δ
unittests	`94.67% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…d constraints (#4932) validate-template invokes no resource provider, so it cannot establish a property value, limit, enum or cross-property rule. That reasoning does not extend to template-level structure, which the control plane itself parses and which validate-template is therefore testing directly. Recorded us-east-1, 2026-10-26: a parameter of Type: List<AWS::SSM::Parameter::Value<String>> draws the identical "Template format error: List of AWS::SSM::Parameter::Value is not supported." from validate-template and from create-stack, and no stack is created. Raised by issue #4854, where the reporter cited validate-template for a parameter type and our own context said that was not evidence. The deployment-first evidence ranking from #4867 is unchanged.

kddejong added 3 commits October 23, 2026 21:36

Merge remote-tracking branch 'upstream/main' into docs/kiro-evidence-…

1634243

…hierarchy

kddejong marked this pull request as ready for review October 24, 2026 00:17

kddejong merged commit 8d888d2 into aws-cloudformation:main Oct 24, 2026
23 of 24 checks passed

kddejong deleted the docs/kiro-evidence-hierarchy branch October 24, 2026 00:17

This was referenced Oct 24, 2026

docs: check quota adjustability before encoding a limit #4895

Merged

docs(kiro): narrow the validate-template prohibition to provider-owned constraints #4932

Merged

Navigation Menu

Repository navigation

docs: rank deployment above probes in agent evidence hierarchy - #4867

docs: rank deployment above probes in agent evidence hierarchy#4867
kddejong merged 3 commits into
aws-cloudformation:mainfrom
kddejong:docs/kiro-evidence-hierarchy

kddejong commented Oct 23, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

kddejong commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

What changed

Secondary: environment facts added to verification.md

Commands run

Proposed agent-prompt changes (NOT in this PR)

.kiro/agents/issue-triage.json

.kiro/agents/issue-worker.json

Follow-up commit 6f3a809 — a free deployment probe for parameter constraints

Pointer, not duplication

The unitlint (ubuntu-latest, 3.14) failure is pre-existing

Bringing the branch current: merged, not rebased

Commands run for this follow-up

Uh oh!

codecov Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kddejong commented Oct 23, 2026 •

edited

Loading

Secondary: environment facts added to `verification.md`

`.kiro/agents/issue-triage.json`

`.kiro/agents/issue-worker.json`

Follow-up commit `6f3a809` — a free deployment probe for parameter constraints

The `unitlint (ubuntu-latest, 3.14)` failure is pre-existing

codecov Bot commented Oct 24, 2026 •

edited

Loading