All issues

Search results

Response cache (--use_cache) is not keyed by model, so one cache silently serves model A's outputs while reporting them as model B's results

Status: Open.

#4063 In EleutherAI/lm-evaluation-harness;

· AUTHENSOR opened on Aug 29, 2026

Unscorable samples silently leave the metric denominator, inflating headline scores

Status: Open.

#4062 In EleutherAI/lm-evaluation-harness;

· AUTHENSOR opened on Aug 29, 2026

OpenAI-compat: local-chat-completions with full /v1/chat/completions URL

Status: Open.

#4058 In EleutherAI/lm-evaluation-harness;

· cursor opened on Aug 28, 2026

Feature request: EvalPort import/export for per-document samples and results

Status: Open.

#4022 In EleutherAI/lm-evaluation-harness;

· adhabnr-ux opened on Aug 20, 2026

Proposal: Wilson CIs + content-hash integrity field on results

Status: Open.

#4019 In EleutherAI/lm-evaluation-harness;

· CSOAI-ORG opened on Aug 19, 2026

Proposal: Wilson CI bounds, MDE, and optional BH-FDR flags on results output

Status: Open.

#4017 In EleutherAI/lm-evaluation-harness;

· CSOAI-ORG opened on Aug 19, 2026

Generative-task scorers cannot distinguish an unparseable response from a wrong answer (census: 2,936/4,524 tasks affected)

Status: Open.

#4007 In EleutherAI/lm-evaluation-harness;

· Rajveer-code opened on Aug 16, 2026

--use_cache crashes before inference for multimodal image requests

Status: Open.

#4005 In EleutherAI/lm-evaluation-harness;

· sdivyanshu90 opened on Aug 16, 2026

Fix broken Discord link in contributing guide

Status: Open.

#3989 In EleutherAI/lm-evaluation-harness;

· sdivyanshu90 opened on Aug 10, 2026

mmlu_flan_cot_zeroshot group-level aggregate never populates (metric name mismatch: acc vs exact_match)

Status: Open.

#3986 In EleutherAI/lm-evaluation-harness;

· zeeshanhaque21 opened on Aug 7, 2026

All 31 tmlu tasks fail to load: miulab/tmlu returns HTTP 401 and has no mirror

Status: Open.

#3985 In EleutherAI/lm-evaluation-harness;

· ayaangazali opened on Aug 7, 2026

Proposal: opt-in pre-registration for eval claims (lock threshold + task-config hash before the run)

Status: Open.

#3967 In EleutherAI/lm-evaluation-harness;

· sk8ordie84 opened on Jul 30, 2026