intellisu-coder-pro

Typed Decisions evaluation

Model display name: intellisu-coder-pro
Organization display name: 江苏银行 (Bank of Jiangsu)

This repository records evaluation of a user-supplied API service under this display name. No model weights are included and no architecture, parameter count, upstream training provenance or independently verified ownership of an upstream model is asserted. The HF repository namespace identifies the hosting account, separately from the declared display organization.

Benchmark	Coverage	Metric	Result
Typed Decisions	400/400 test cases, 2,000 decisions	Accuracy (agreement with teacher labels)	0.7735
Same full test		KL from gold (lower is better)	0.3025644435
Same full test		Brier (lower is better)	0.1164229198
Same full test		ECE (lower is better)	0.011837

Evaluated 2026-10-31; evaluation files published 2026-11-03. All 400 public test cases were retained, with one generated response per case, low reasoning and a 1,024 completion-token cap including reasoning. API-reported usage was 342,613 tokens. There were zero truncated or invalid outputs. No answer-dependent resampling or paid model judge was used.

Protocol: a zero-shot chat JSON adapter produces self-reported probability distributions, rather than native System One classifier logits. Locked official scoring functions were used. This is a self-reported custom-adapter evaluation, not organizer certification or an exact reproduction of native classifier inference. The dataset's model-generated soft labels reflect a teacher's judgments; agreement does not establish real financial, compliance or security correctness. No fitting on this benchmark's training split was performed by this runner; upstream training exposure is unknown.

Only state and questions were sent to the service. Gold distributions, factors, panel labels and rewards were not sent as additional inputs. Raw outputs, metric details and dataset revision/hash are in evaluation/typed-decisions/. The dataset revision was 1c9479de422cc0117abcddfadc3afe2cd8383e40.

Attribution: LocalLLaMA / Typed Decisions, Apache-2.0 as declared by the dataset. This source-dataset license is not a license grant for the evaluated API model or its weights. No API model license is asserted here. Original reference-answer data and API credentials are not included.

The scores in .eval_results/typed-decisions.yaml are intended for HF Community Evals automatic aggregation. No independently verified badge is claimed.

SafeAgents Judge Bench evaluation

Model display name: intellisu-coder-pro
Organization display name: 江苏银行 (Bank of Jiangsu)

Benchmark	Coverage	Metric	Result
SafeAgents Judge Bench	562/562 public test tool calls	Balanced accuracy	0.830
Same full test		AUROC	0.991
Same full test		False block rate (lower is better)	0.333
Same full test		Category macro F1	0.535

Evaluated 2026-10-31; evaluation files published 2026-11-03. One generated response per case, low reasoning, maximum 768 completion tokens including reasoning. API-reported usage was 652,034 tokens. Three invalid JSON/probability outputs and zero truncated responses were retained in scoring. Invalid verdicts count as allow under the official fail-open policy; missing categories count as wrong. No label-dependent retries or paid model judges were used in this run.

Protocol and limitations: a zero-shot chat JSON adapter returns self-reported probabilities, not native System One classifier logits. Locked official state, verdict and scoring functions were used, including the official task/argument truncation. This is a self-reported custom-adapter evaluation, not independently verified or organizer-certified. Labels are model-generated and the dataset includes only 39 allowed calls versus 523 blocked calls, so the false-block estimate has limited negative-sample support. The result does not establish production security or compliance correctness.

The model received only the official state built from task text, tool name/arguments and scanner findings, plus the two official questions. Reference labels, categories, reward and panel rationales were not provided as extra model inputs. No training or calibration on this benchmark's train split was performed by the runner; upstream training exposure is unknown.

Full raw responses, binary/category verdicts, metrics and dataset revision/hash are in evaluation/safeagents-judge-bench/. Dataset revision: 5671a11198785784d72822b8df1bce2238f726a4. The metrics JSON retains its generation-time local-file status; this model card and the published result files record the current publication.

Attribution: harness-hub / SafeAgents Judge Bench, Apache-2.0 as declared by the dataset. This source license does not grant rights to the evaluated API model or its weights. No model weights, original reference-answer data or API credentials are uploaded. Result YAML.

AIME 2026 evaluation

Model display name: intellisu-coder-pro
Organization display name: 江苏银行 (Bank of Jiangsu)

Benchmark	Coverage	Protocol	Local self-reported result
AIME 2026	All 30 public problems	Single generated response per problem; exact final-integer comparison	30/30 (100.0%)

Evaluated 2026-10-31; files published 2026-11-03. Fixed low reasoning and maximum 512 completion tokens including reasoning. All 30 raw responses were bare integers; there were zero truncated or unparsed outputs. API-reported usage was 12,636 tokens across 30 model calls. No resampling, answer-dependent retries or model judge was used. This publication reused saved outputs and did not rerun inference.

This is a custom single-pass, budget-limited baseline, not a reproduced or certified official MathArena leaderboard run. It does not estimate a multi-sample average, establish unseen-test performance or support an independently verified first-place claim. The benchmark is public, and upstream training exposure is unknown. No training on this benchmark was performed by this runner.

Only the problem text and a fixed final-integer instruction were sent to the API. The reference answer field was used only for local comparison after generation. Original reference-answer data are not uploaded. Raw outputs, exact protocol, metrics and source revision/hash are in evaluation/aime2026/. The dataset revision is d61f486805306e6e6b5913504c07763a99e1b112.

Attribution: MathArena / AIME 2026, license declared by the source dataset: CC-BY-NC-SA-4.0 (dataset). This attribution does not assign an open-source license to the evaluated API model or its weights. No model weights or API credentials are included. The metrics JSON retains generation-time local-file status; this model card records the later publication.

Result YAML. Self-reported; no independent-verification badge is claimed.