Evaluation Science

Ranking Leaderboard Evaluatology Analysis

This document uses the Evaluatology methodology[1] to design the PerfMatrix test process, determine EO, AO, and COs combinations across real-world ranking comparison dimensions, and provide scientific guidance for the ranking website.

1. Evaluatology Concept to Ranking Entity Mapping

Based on the five-step Evaluatology methodology, abstract concepts are mapped to the ranking website data model:

Evaluatology Concept	Ranking Entity	Fixed By	Description
EO (Evaluated Object)	Compiler + CPU + Memory + Operating System	Ranking result information	Compiler (cross / native compilation) + CPU (real silicon \| RTL simulation; incl. architecture, frequency, cache, cores) + memory (physical DRAM \| DRAMsim3 under RTL; capacity, channels, frequency) + operating system (compile toolchain, runtime libraries)
AO (Affected Object)	SPEC CPU Benchmark Suite configuration	Benchmark metadata + config file	SPEC version, input size, speed/rate mode, base/peak optimization level, pinned by the config file
COs (Confounding Objects)	PerfMatrix test framework	PerfMatrix test framework + CI (Continuous Integration) configuration	Standardized SPEC test framework and toolset: it orchestrates configuration, build/packaging, execution, validation, and scoring, thereby fixing the common measurement process and rules across ranking rows.

EO: Evaluated Object

Compiler

cross / native

name/version/type

CPU

real silicon | RTL

frequency / cache / cores

RTL ver (when simulated)

Memory

physical DRAM | DRAMsim3

capacity / channels

freq / latency

Operating System

kernel / distro

compile toolchain

runtime libraries

AO: Affected Object (SPEC Benchmark)

Suite

spec06/17/26

Config file

Input set

ref/train/test

Run mode

speed/rate

Tuning mode

base/peak

COs: Confounding Objects

PerfMatrix test framework

standardized toolset

Build / package / run

workflow orchestration

Validation / scoring

fixed rules

SCS: Self-Contained System

TestResult

score, score/GHz

Hardware realization modes are paired: real silicon ↔ physical DRAM; CPU RTL ↔ DRAMsim3.

SCS: Self-Contained System

EO + AO + COs together form a self-contained system (SCS). Every TestResult on the leaderboard is the output of one SCS. Changing any element creates a different SCS; such rows can still be ranked descriptively, but causal interpretation requires the changed variables to be stated.

2. EO Enumeration: Compiler + CPU + Memory + Operating System

Compiler, CPU, OS, and EO — Current Snapshot

2.1 Compiler List (12 total)

No.	Name	Version	Type	Target Arch
1	LLVM	19.1.0	llvm	x86_64
2	LLVM	22.1.0	llvm	riscv64
3	XSCC	1.1	xscc	riscv64
4	LLVM	21.1.0	llvm	riscv64
5	GCC	15.1.0	gcc	riscv64
6	LLVM	21.1.0	llvm	x86_64
7	GCC	15.1.0	gcc	x86_64
8	LLVM	22.1.0	llvm	x86_64
9	LLVM	21.1.0	llvm	aarch64
10	GCC	15.1.0	gcc	aarch64
11	XSCC	1.1	xscc	aarch64
12	GCC	12.1.0	gcc	riscv64

2.2 Platform List (17 total)

No.	Name	Type	CPU	MHz	Memory	Cores	OS	Arch
1	Intel-11900K-CypressCove	x86_native	11th Gen Intel Core i9-11900K	5300	126 GiB	8C/16T	Ubuntu 22.04.1 LTS	X86
2	Zhihe-A210-C920	riscv_board	A210(Xuantie-c920)	1900	15 GiB	8C/8T	RuyiOS 2.0	RISC-V
3	OpenXiangShan-KMHv3-4bfb226bf	riscv_emu	Xiangshan KunminghuV3	3000	32 GB (2 x 16 GB DDR4-3200)	1C/1T	Linux (RISC-V)	RISC-V
4	SiFive-EIC7700X-P550	riscv_board	ESWIN EIC7700X (SiFive P550)	1400	25 GiB	4C/4T	Ubuntu 24.04.3 LTS	RISC-V
5	Sophgo-SG2042-C920	riscv_board	C920 (T-Head)	2000	126 GiB	64C/64T	Fedora Linux 38 (Workstation Edition)	RISC-V
6	SpacemiT-K3-X100	riscv_board	SpacemiT X100	2400	32 GiB	8C/8T	Bianbu 4.0rc2	RISC-V
7	SpacemiT-K1-X60	riscv_board	Spacemit(R) X60	1600	16 GiB	8C/8T	Bianbu 2.1	RISC-V
8	Intel-14900K-RaptorCove	x86_native	Intel Core i9-14900K	6000	125 GiB	24C/48T	Ubuntu 22.04.4 LTS	X86
9	AMD-9950X-Zen5	x86_native	AMD Ryzen 9 9950X 16-Core	5700	123 GiB	16C/32T	Ubuntu 22.04.5 LTS	X86
10	OpenXiangShan-KMHv3-9443d04bd	riscv_emu	Xiangshan KunminghuV3	3000	32 GB (2 x 16 GB DDR4-3200)	1C/1T	Linux (RISC-V)	RISC-V
11	T-Head-Yitian710-NeoverseN2	arm	neoverse-n2	3000	31 GiB	4C/4T	CentOS Stream 8	Arm
12	OpenXiangShan-KMHv3-64e7bff7f	riscv_emu	Xiangshan KunminghuV3	3000	32 GB (2 x 16 GB DDR4-3200)	1C/1T	Linux (RISC-V)	RISC-V
13	OpenXiangShan-KMHv3-1a9a2f52c	riscv_emu	Xiangshan KunminghuV3	3000	32 GB (2 x 16 GB DDR4-3200)	1C/1T	Linux (RISC-V)	RISC-V
14	OpenXiangShan-KMHv3-a20df0c36	riscv_emu	Xiangshan KunminghuV3	3000	32 GB (2 x 16 GB DDR4-3200)	1C/1T	Linux (RISC-V)	RISC-V
15	AWS-Graviton4-NeoverseV2	arm	Neoverse-V2	2800	768 GiB DDR5-5600	96C/96T	—	Arm
16	Sophgo-SG2044-C920v2	riscv_board	SOPHON SG2044 (XuanTie C920v2)	2600	126 GiB	64C/64T	openEuler 24.03 (LTS-SP2)	RISC-V
17	AWS-Graviton5-NeoverseV3	arm	ARM Neoverse V3	3300	371 GB DDR5	192C/192T	Ubuntu 26.04 LTS (kernel 7.0.0-1008-aws)	Arm

2.3 Effective EO Combinations (completed/partial TestResults)

The final EO is defined by the compiler, CPU (real silicon or RTL simulation), memory, and operating system (compile toolchain + runtime libraries).

Simulation note: RTL-simulation runs use SimPoint to sample the program into representative slices and build checkpoint images in a custom checkpoint format; the selected SimPoint clusters carry weights that sum to 100% (the whole program is represented by weighted slices, not executed in full). The scores above are estimates derived from these program slices — not a full SPEC CPU evaluation — and may deviate from the actual performance of real silicon.

No.	EO	Compiler	Platform
1	EO-GCC12.1.0@OpenXiangShan-KMHv3-1a9a2f52c	GCC 12.1.0 riscv64	OpenXiangShan-KMHv3-1a9a2f52c
2	EO-GCC12.1.0@OpenXiangShan-KMHv3-64e7bff7f	GCC 12.1.0 riscv64	OpenXiangShan-KMHv3-64e7bff7f
3	EO-GCC15.1.0@AMD-9950X-Zen5	GCC 15.1.0 x86_64	AMD-9950X-Zen5
4	EO-GCC15.1.0@AWS-Graviton4-NeoverseV2	GCC 15.1.0 aarch64	AWS-Graviton4-NeoverseV2
5	EO-GCC15.1.0@AWS-Graviton5-NeoverseV3	GCC 15.1.0 aarch64	AWS-Graviton5-NeoverseV3
6	EO-GCC15.1.0@Intel-14900K-RaptorCove	GCC 15.1.0 x86_64	Intel-14900K-RaptorCove
7	EO-GCC15.1.0@OpenXiangShan-KMHv3-4bfb226bf	GCC 15.1.0 riscv64	OpenXiangShan-KMHv3-4bfb226bf
8	EO-GCC15.1.0@SpacemiT-K3-X100	GCC 15.1.0 riscv64	SpacemiT-K3-X100
9	EO-GCC15.1.0@T-Head-Yitian710-NeoverseN2	GCC 15.1.0 aarch64	T-Head-Yitian710-NeoverseN2
10	EO-LLVM19.1.0@Intel-11900K-CypressCove	LLVM 19.1.0 x86_64	Intel-11900K-CypressCove
11	EO-LLVM19.1.0@Intel-14900K-RaptorCove	LLVM 19.1.0 x86_64	Intel-14900K-RaptorCove
12	EO-LLVM21.1.0@AMD-9950X-Zen5	LLVM 21.1.0 x86_64	AMD-9950X-Zen5
13	EO-LLVM21.1.0@AWS-Graviton4-NeoverseV2	LLVM 21.1.0 aarch64	AWS-Graviton4-NeoverseV2
14	EO-LLVM21.1.0@Intel-14900K-RaptorCove	LLVM 21.1.0 x86_64	Intel-14900K-RaptorCove
15	EO-LLVM21.1.0@OpenXiangShan-KMHv3-4bfb226bf	LLVM 21.1.0 riscv64	OpenXiangShan-KMHv3-4bfb226bf
16	EO-LLVM21.1.0@T-Head-Yitian710-NeoverseN2	LLVM 21.1.0 aarch64	T-Head-Yitian710-NeoverseN2
17	EO-LLVM22.1.0@AMD-9950X-Zen5	LLVM 22.1.0 x86_64	AMD-9950X-Zen5
18	EO-LLVM22.1.0@SiFive-EIC7700X-P550	LLVM 22.1.0 riscv64	SiFive-EIC7700X-P550
19	EO-LLVM22.1.0@Sophgo-SG2042-C920	LLVM 22.1.0 riscv64	Sophgo-SG2042-C920
20	EO-LLVM22.1.0@SpacemiT-K1-X60	LLVM 22.1.0 riscv64	SpacemiT-K1-X60
21	EO-LLVM22.1.0@SpacemiT-K3-X100	LLVM 22.1.0 riscv64	SpacemiT-K3-X100
22	EO-LLVM22.1.0@Zhihe-A210-C920	LLVM 22.1.0 riscv64	Zhihe-A210-C920
23	EO-XSCC1.1@AWS-Graviton4-NeoverseV2	XSCC 1.1 aarch64	AWS-Graviton4-NeoverseV2
24	EO-XSCC1.1@AWS-Graviton5-NeoverseV3	XSCC 1.1 aarch64	AWS-Graviton5-NeoverseV3
25	EO-XSCC1.1@OpenXiangShan-KMHv3-4bfb226bf	XSCC 1.1 riscv64	OpenXiangShan-KMHv3-4bfb226bf
26	EO-XSCC1.1@OpenXiangShan-KMHv3-9443d04bd	XSCC 1.1 riscv64	OpenXiangShan-KMHv3-9443d04bd
27	EO-XSCC1.1@OpenXiangShan-KMHv3-a20df0c36	XSCC 1.1 riscv64	OpenXiangShan-KMHv3-a20df0c36
28	EO-XSCC1.1@Sophgo-SG2042-C920	XSCC 1.1 riscv64	Sophgo-SG2042-C920
29	EO-XSCC1.1@Sophgo-SG2044-C920v2	XSCC 1.1 riscv64	Sophgo-SG2044-C920v2
30	EO-XSCC1.1@T-Head-Yitian710-NeoverseN2	XSCC 1.1 aarch64	T-Head-Yitian710-NeoverseN2

2.4 Compiler Optimization Flags (via config file)

Each EO's compiler flags are defined by a config file (the base/peak tuning policy itself is an AO property — see §3.2 AO Configuration Matrix). They include:

Optimization level: e.g. -O2 / -O3 / -Ofast
Target architecture flags: e.g. -march=rv64gc / -march=native
Link-time optimization: e.g. -flto / -static / -ljemalloc
Portability flags: e.g. -DSPEC_CPU_LP64
Base options: e.g. -O3 -ffast-math -flto -ljemalloc

2.5 EO Difference Analysis: Key Differences in Cross-Platform Comparison

When performing cross-platform comparisons, the following EO factors (CPU, compilation mode, memory, OS) change and must be noted:

Platform	Compilation Mode	Memory	OS
Intel-11900K-CypressCove (x86)	Native compilation	126 GiB	Ubuntu 22.04
Intel-14900K-RaptorCove (x86)	Native compilation	125 GiB	Ubuntu 22.04
AMD-9950X-Zen5 (x86)	Native compilation	123 GiB	Ubuntu 22.04
Zhihe-A210-C920	Cross compilation (riscv64 GNU toolchain)	15 GiB	RuyiOS 2.0
KMHv3-emu	Cross compilation (riscv64 GNU toolchain)	2 GiB	Linux (RISC-V) [RTL simulation environment]
SiFive-EIC7700X-P550	Cross compilation (riscv64 GNU toolchain)	25 GiB	Ubuntu 24.04
Sophgo-SG2042-C920	Cross compilation (riscv64 GNU toolchain)	126 GiB	Fedora 38
SpacemiT-K3-X100	Cross compilation (riscv64 GNU toolchain)	32 GiB	Bianbu 4.0
SpacemiT-K1-X60	Cross compilation (riscv64 GNU toolchain)	16 GiB	Bianbu 2.1
T-Head-Yitian710-NeoverseN2	Native compilation	31 GiB	CentOS Stream 8
AWS-Graviton4-NeoverseV2	Native compilation	768 GiB DDR5-5600	—
AWS-Graviton5-NeoverseV3	Native compilation	371 GB DDR5	Ubuntu 26.04 LTS

Note: The EO is ultimately defined by the combination of the compiler + CPU + memory + operating system (section 2.3).

3. AO Enumeration: SPEC CPU Benchmark Suite Configuration

3.1 SPEC Suite Dimensions

Suite	Version	INT Items	FP Items	Mode	Description
spec06	SPEC CPU 2006	12	17	speed / rate defined by SPEC; this site currently exposes speed for spec06 data and pages	Classic integer / floating-point benchmarks
spec17	SPEC CPU 2017	10 (_s) + 10 (_r)	10 (_s) + 13 (_r)	speed / rate	speed = single-copy; rate = throughput, but this site runs rate with a single copy (copy=1), so rate scores here reflect single-copy performance, not full-load throughput
spec26	SPEC CPU 2026	13 (_s) + 14 (_r)	13 (_s) + 12 (_r)	speed / rate	Officially released May 2026 (52 benchmarks across 4 suites)

Official SPEC CPU index pages are cited for suite information [2]-[5].

3.2 AO Configuration Matrix

Dimension	Values	Impact
Suite	spec06 / spec17 / spec26	Completely different benchmark sets and reference times
Input set	ref / train / test	Input dataset size (ref = largest, test = smallest), with different reference times
Run mode	speed / rate	SPEC CPU2006 defines both speed and rate metrics; this site currently exposes speed for spec06 data and pages. spec17/spec26 distinguish single-copy vs throughput; this site runs rate with a single copy (copy=1), so its rate scores here reflect single-copy performance, not full-load throughput
Tuning mode	base / peak	base = uniform compiler flags across all benchmarks; peak = per-benchmark tuned flags

3.3 AOs Actually Used on the Ranking Website

All current results use the ref input set. The ref workload is the reference input set used for formal/reportable SPEC runs. Whether a result is a publicly quotable compliant SPEC result still depends on the full run rules, configuration, disclosure, and review requirements. The SPEC version, input size, speed/rate mode, and base/peak level are part of the AO and are represented by the leaderboard data plus the config file.

AO	Suite	Mode	Base/Peak	Count
AO-spec06-base-speed	spec06	speed	base	21 completed / 24 total
AO-spec06-peak-speed	spec06	speed	peak	1 completed / 1 total
AO-spec17-base-speed	spec17	speed	base	13 completed / 19 total
AO-spec17-base-rate	spec17	rate	base	6 completed / 7 total
AO-spec26-base-rate	spec26	rate	base	3 completed / 4 total

3.4 Reference Time (Reftime) — Core COs Constant

Reference times are hardcoded by SPEC. See the appendix for the complete listing.

4. COs Enumeration: PerfMatrix Test Framework

COs (PerfMatrix Test Framework) — Current Snapshot

Actual measurements on the ranking website are performed by the standardized PerfMatrix test framework based on the evaluatology diagram specification. It explicitly pins the SCS elements (EO/AO/COs) through three configuration layers:

4.1 Three-Layer Configuration System (PerfMatrix Design)

The three-layer configuration explicitly pins the EO/AO/COs elements:

Layer	Configuration Layer	Content
Layer 1	Project setup	SPEC version + target platform/compiler combination
Layer 2	Run parameters	compiler/toolchain paths, workload input, run count
Layer 3	Config file	compiler flags and portability options

4.2 PerfMatrix Test Framework (COs) Elements

COs Class	Content	Fixed By	Fixed / Variable
Scoring logic	scoring component: reference time / median runtime	PerfMatrix test framework	Globally fixed
Validation tool	SPEC-compatible output validation	PerfMatrix test framework	Globally fixed
Test runner	run orchestration component	PerfMatrix test framework	Globally fixed
Build server	x86 build server	PerfMatrix CI (Continuous Integration) configuration	Globally fixed

Reference times are fixed constants defined by SPEC and used in the scoring environment. The Reference Time Tables appendix is generated from benchmark metadata.

5. Ranking Leaderboard Comparison Scenarios and EO/AO/COs Combination Matrix

Scenario Details

5.1 Scenario Overview

The Ranking website forms multiple comparison dimensions through URL routes and user filters. Each dimension corresponds to a different evaluatology analysis framework:

URL Routes

/cross-arch x86+arm+KMHv3

/riscv RISC-V zone

Suite

spec06

spec17

spec26

Filters

Metric: int/fp/@1GHz

Base / Peak

speed / rate

Ranking Modules

Top leaderboard

By platform type

By compiler type

By benchmark item

Recent tests

5.2 Scenario A: Cross-Compiler Comparison (same platform, same Suite, same Mode)

Evaluatology meaning: with AO and COs fixed, only the compiler part of EO changes, so the compiler performance impact can be inferred.

Evaluatology Dimension	Content	Fixed / Variable
EO-compiler type	GCC / LLVM / XSCC	Variable
EO-compiler version	19.1.0 / 21.1.0 / 22.1.0 / 15.1.0 / 1.1	Variable
EO-CPU platform	Same platform, such as AMD-9950X-Zen5	Fixed
AO-Suite	Same suite	Fixed
AO-Mode	Same speed/rate + base/peak	Fixed
EO-OS	Same OS	Fixed
EO-toolchain	Same toolchain	Fixed
COs-PerfMatrix test framework	Timing / scoring fixed	Fixed

Ranking website implementation: the compiler grouping view groups and displays results by compiler category.

Actual comparable pairs (using spec17 base speed as an example):

#	EO-A	EO-B	AO	Difference Attribution
1	GCC 15.1.0 @ AMD-9950X-Zen5	LLVM 21.1.0 @ AMD-9950X-Zen5	AO-spec17-base-speed	Compiler difference
2	GCC 15.1.0 @ Intel-14900K-RaptorCove	LLVM 19.1.0 @ Intel-14900K-RaptorCove	AO-spec17-base-speed	Compiler difference
3	GCC 15.1.0 @ AMD-9950X-Zen5	LLVM 21.1.0 @ AMD-9950X-Zen5	AO-spec06-base-speed	Compiler difference
4	XSCC 1.1 @ OpenXiangShan-KMHv3	LLVM 21.1.0 @ OpenXiangShan-KMHv3	AO-spec06-base-speed	Compiler difference
5	LLVM 22.1.0 @ SpacemiT-K3-X100	GCC 15.1.0 @ SpacemiT-K3-X100	AO-spec17-base-speed	Compiler difference

Evaluatology analysis: AO and COs are fixed, and EO changes only in the compiler dimension. According to evaluatology, when the compared EOs differ only in compiler or compiler options while the hardware is the same, the difference can be attributed to compiler performance impact.

5.3 Scenario B: Cross-Platform Comparison (same compiler, same Suite, same Mode)

Evaluatology meaning: the CPU, memory, and operating-system parts of EO change while the compiler is fixed.

Evaluatology Dimension	Content	Fixed / Variable
EO-compiler	Same compiler	Fixed
EO-CPU platform	Different CPU	Variable
AO	Same suite + mode	Fixed
EO-OS	May differ	Variable
EO-toolchain	Same toolchain within one ISA, or different toolchains across ISAs	Partially variable
COs-PerfMatrix test framework	Timing / scoring fixed	Fixed

Ranking website implementation: the platform grouping view groups and displays results by platform category.

Actual comparable pairs:

#	EO-A	EO-B	AO	EO Difference
1	LLVM 22.1.0 @ Zhihe-A210-C920	LLVM 22.1.0 @ SiFive-EIC7700X-P550	AO-spec06-base-speed	Different CPU + Memory + OS
2	LLVM 22.1.0 @ Zhihe-A210-C920	LLVM 22.1.0 @ Sophgo-SG2042-C920	AO-spec17-base-speed	Different CPU + Memory + OS
3	LLVM 22.1.0 @ Zhihe-A210-C920	LLVM 22.1.0 @ SpacemiT-K3-X100	AO-spec17-base-rate	Different CPU + Memory + OS

Evaluatology analysis: the compiler part of EO is fixed. The EO differs in CPU, memory, and operating system, so the difference reflects the combined effect of the EO's CPU, memory, and operating system — not the CPU alone.

5.4 Scenario C: 1GHz-Normalized Cross-Platform Comparison

Evaluatology meaning: frequency normalization converts the SPEC score into a score/GHz metric, reducing the direct effect of nominal clock frequency.

Evaluatology Dimension	Content	Fixed / Variable
EO-CPU platform	Different CPU	Variable
EO-compiler	May be the same or different	Variable
EO-OS	May differ	Variable
Normalization formula	score/GHz = score / (mhz / 1000)	Fixed rule
AO	Same suite + mode	Fixed
COs-PerfMatrix test framework	Timing / scoring fixed	Fixed

Ranking website implementation: the score/GHz options in user filters are available across the leaderboard, grouping, and per-benchmark views.

Important note:

score/GHz is computed consistently from each platform frequency.
For KMHv3-emu, the normalization uses its 3GHz simulation frequency.

Evaluatology validity: score/GHz is a frequency-normalized SPEC score. It assumes approximate linear scaling with frequency and cannot remove differences in instruction count, compiler quality, memory hierarchy, OS scheduling, runtime libraries, or simulator behavior. These factors are now EO sub-dimensions.

5.5 Scenario D: Base vs Peak Comparison

Evaluatology meaning: the compiler-option part of EO changes, so the boundary of compiler optimization impact can be inferred.

Evaluatology Dimension	Content	Fixed / Variable
EO-compiler	Same compiler	Fixed
EO-CPU platform	Same platform	Fixed
EO-compiler options	base (uniform flags) vs peak (per-benchmark tuning)	Variable
AO	Same suite + mode	Fixed
COs-PerfMatrix test framework	Exactly the same	Fixed

Ranking website implementation: the base/peak switch in user filters.

Actual comparable pairs:

#	Base EO	Peak EO	AO
1	GCC 15.1.0 @ Intel-14900K-RaptorCove base	GCC 15.1.0 @ Intel-14900K-RaptorCove peak	AO-spec06-speed
2	LLVM 21.1.0 @ AMD-9950X-Zen5 base	LLVM 21.1.0 @ AMD-9950X-Zen5 peak	AO-spec06-speed

Evaluatology analysis: EO changes only in the compiler-option dimension, while AO and COs remain unchanged. The score difference between peak and base reflects the observed benefit of per-benchmark tuning under the current SPEC rules and configuration; it is not a theoretical optimization limit.

5.6 Scenario E: Cross-Suite Comparison

Evaluatology meaning: the entire AO changes because different SPEC versions are used, while EO and COs remain unchanged.

Evaluatology Dimension	Content	Fixed / Variable
EO	Same compiler + CPU + Memory + OS	Fixed
AO-Suite	spec06 / spec17 / spec26	Variable
COs-PerfMatrix test framework	Same test framework	Fixed

Ranking website implementation: the suite switcher.

Actual comparable pairs:

#	EO	AO-A	AO-B
1	LLVM 22.1.0 @ Zhihe-A210-C920	AO-spec06-base-speed	AO-spec17-base-speed
2	LLVM 22.1.0 @ SiFive-EIC7700X-P550	AO-spec06-base-speed	AO-spec17-base-speed

Evaluatology analysis: scores cannot be directly compared. Different SPEC versions have completely different benchmark items, reference times, and scoring rules. Cross-Suite comparison can only observe trend direction, such as whether the same compiler remains leading across multiple suites, and cannot support numerical comparison. EOs can be analyzed under their corresponding AOs.

5.7 Scenario F: Cross-Architecture Comparison

Evaluatology meaning: this is the most macro-level comparison dimension on the ranking website. The compiler, CPU, and memory of EO change simultaneously across different ISA families.

Evaluatology Dimension	Content	Fixed / Variable
EO-compiler	Different compilers targeting different ISAs	Variable
EO-CPU platform	x86 / Arm / RISC-V hardware	Variable
AO	Same suite + mode	Fixed
EO-toolchain	Different compile toolchains	Variable
EO-OS	Different OS	May vary
EO-CPU (RTL sim)	KMHv3 has an RTL simulator; x86 does not	Variable
COs-PerfMatrix test framework	Timing / scoring fixed	Fixed

Ranking website implementation: the cross-architecture suite page, scoped to x86 + Arm + KMHv3.

Special EO (CPU / RTL) notes for KMHv3:

KMHv3-emu runs on an RTL simulator and is not real silicon.
The RTL simulator introduces additional EO (CPU) factors: RTL version (kmhv3-cr260430-4bfb226bf) and DRAMsim3 configuration.
Its score/GHz values are normalized using the 3GHz simulation frequency.

Cross-Architecture Comparison

x86_nativeAMD-9950X-Zen5 / Intel-14900K-RaptorCove / Intel-11900K-CypressCoveGCC15.1.0 / LLVM19.1.0~22.1.0Native compile · real silicon · Ubuntu

Arm / T-Head-Yitian710-NeoverseN2 / AWS-Graviton4-NeoverseV2 / AWS-Graviton5-NeoverseV3LLVM21.1.0 / GCC15.1.0 / XSCC1.1Native compile · real silicon · CentOS Stream 8 / Ubuntu 26.04 LTS

riscv_emu / OpenXiangShan-KMHv3XSCC1.1 / LLVM21.1.0 / GCC15.1.0Cross compile · RTL sim + DRAMsim3 · Linux(RISC-V)

score/GHz ranking

Actual cross-architecture comparable pairs (spec06 base speed unless otherwise noted):

EO-x86	EO-RISC-V	1GHz Normalization?
LLVM 21.1.0 @ AMD-9950X-Zen5	XSCC 1.1 @ OpenXiangShan-KMHv3	Required
GCC 15.1.0 @ Intel-14900K-RaptorCove	LLVM 21.1.0 @ OpenXiangShan-KMHv3	Required
LLVM 22.1.0 @ AMD-9950X-Zen5 (spec26 rate)	LLVM 22.1.0 @ SpacemiT-K3-X100 (spec26 rate)	Required

Evaluatology analysis: use caution. In cross-architecture comparison, multiple EO sub-dimensions (compiler, CPU/ISA, OS, compile toolchain, RTL simulation) change at the same time while the COs PerfMatrix test framework stays fixed, so performance differences cannot be simply attributed to a single factor. score/GHz reduces nominal-frequency effects, but the result still reflects ISA design, compiler quality, memory hierarchy, OS/runtime differences, and RTL simulator behavior. This dimension is suitable for descriptive trends and relative position observation. Each comparison should explicitly state which EO sub-dimensions differ.

5.8 Scenario G: Top Leaderboard

Evaluatology meaning: all EOs are mixed into one ranking under the same AO.

Evaluatology Dimension	Content
EO	All compiler + CPU + Memory + OS combinations
AO	Same suite + tuning mode + run mode
COs-PerfMatrix test framework	Same test framework across rows

Ranking website implementation: the Overall Ranking area.

Evaluatology analysis: the top leaderboard mixes many different EOs (compiler + CPU + memory + OS) under one fixed COs test framework and one AO. It is suitable for a descriptive overview, not strict causal inference. Differences between rows may come from compiler, hardware, OS, runtime libraries, or RTL simulator behavior. Each comparison should explicitly state which EO sub-dimensions differ.

6. Reliability Checks

The ranking data is directly produced by the PerfMatrix test framework. Its reliability and evaluation checks are as follows:

Reliability Checks and Dataset — Current Snapshot

6.1 Correctness Verification

Check	Mechanism	Evaluatology Requirement
Output correctness	SPEC official specdiff output validation	"If any sub-test output is incorrect, the overall EO evaluation result is invalidated"
Run count	Three runs by default, median selected	Each EO is measured over three runs by default.
Anomaly handling	Retry all 3 if any run is abnormal	"If one of the 3 runs is significantly disturbed, retry all"
Test status	run status: completed / partial / failed	failed results are excluded; partial results enter a ranking only when the selected metric value exists

6.2 Measurement & Verification Decoupling

Step	Tool	Runs On
Score calculation	PerfMatrix scoring component	Target hardware
Output verification	SPEC-compatible output validation	x86 server (independent)

6.3 Data Integrity Rules

The ranking website's data filtering rules improve consistency and make the ranking inputs explicit:

Records with status=failed: excluded from all ranking sections.
Records with status=partial: enter a ranking only when the selected metric value exists.
Records missing the selected metric value are excluded from that ranking section.

7. Comparison Reliability — Scientific Assessment Summary

Only scenarios A and D isolate a single EO dimension and support causal attribution; B and C constrain most variables and support a relative-performance reading; F and G leave many EO sub-dimensions variable and support a descriptive overview only. The Suitable Conclusion column states the strongest claim each scenario can bear.

Scenario	EO	AO	COs	Suitable Conclusion
A: Compiler Comparison	Compiler only	Fixed	Fixed	Compiler performance difference
D: Base vs Peak	Compiler flags only	Fixed	Fixed	Compiler optimization potential
B: Same-ISA Platform Comparison	CPU + Memory + OS	Fixed	Fixed	Relative hardware performance
C: 1GHz Normalization	Hardware + compiler + OS	Fixed	Fixed	Relative score/GHz performance
G: Top Leaderboard	All	Fixed	Fixed	Descriptive overview
F: Cross-Architecture Comparison	Compiler + CPU + Memory + OS + RTL	Fixed	Fixed	Architecture-level trend observation

Appendix: Dataset Statistics

Compiler total: 12 (gcc=4, llvm=6, xscc=2)
Platform total: 17 (x86_native=3, riscv_board=6, riscv_emu=5, arm=3)
TestResult total: 55 (completed=44, partial=11)
Completed TestResult by suite: spec06=22, spec17=19, spec26=3
Effective EO combinations with completed/partial TestResults: 30
Effective AO combinations: 5
Supported suites: spec06 / spec17 / spec26

Complete Reference Time Tables

Reference times are fixed constants defined by SPEC and used in the scoring environment. The Reference Time Tables appendix is generated from benchmark metadata.

SPEC CPU 2006

Benchmark	Suite	Reference time (seconds)
400.perlbench	int	9770
401.bzip2	int	9650
403.gcc	int	8050
429.mcf	int	9120
445.gobmk	int	10490
456.hmmer	int	9330
458.sjeng	int	12100
462.libquantum	int	20720
464.h264ref	int	22130
471.omnetpp	int	6250
473.astar	int	7020
483.xalancbmk	int	6900
998.specrand	int	10
410.bwaves	fp	13590
416.gamess	fp	19580
433.milc	fp	9180
434.zeusmp	fp	9100
435.gromacs	fp	7140
436.cactusadm	fp	11950
437.leslie3d	fp	9400
444.namd	fp	8020
447.dealii	fp	11440
450.soplex	fp	8340
453.povray	fp	5320
454.calculix	fp	8250
459.gemsfdtd	fp	10610
465.tonto	fp	9840
470.lbm	fp	13740
481.wrf	fp	11170
482.sphinx3	fp	19490
999.specrand	fp	10

SPEC CPU 2017 — speed

Benchmark	Suite	Reference time (seconds)
600.perlbench_s	int	1775
602.gcc_s	int	3982
605.mcf_s	int	4721
620.omnetpp_s	int	1631
623.xalancbmk_s	int	1417
625.x264_s	int	1764
631.deepsjeng_s	int	1433
641.leela_s	int	1706
648.exchange2_s	int	2940
657.xz_s	int	6182
603.bwaves_s	fp	58998
607.cactubssn_s	fp	16670
619.lbm_s	fp	5238
621.wrf_s	fp	13226
627.cam4_s	fp	8863
628.pop2_s	fp	11873
638.imagick_s	fp	14426
644.nab_s	fp	17472
649.fotonik3d_s	fp	9116
654.roms_s	fp	15745

SPEC CPU 2017 — rate

Benchmark	Suite	Reference time (seconds)
500.perlbench_r	int	1592
502.gcc_r	int	1416
505.mcf_r	int	1616
520.omnetpp_r	int	1312
523.xalancbmk_r	int	1056
525.x264_r	int	1751
531.deepsjeng_r	int	1146
541.leela_r	int	1656
548.exchange2_r	int	2620
557.xz_r	int	1080
503.bwaves_r	fp	10028
507.cactubssn_r	fp	1266
508.namd_r	fp	950
510.parest_r	fp	2616
511.povray_r	fp	2335
519.lbm_r	fp	1054
521.wrf_r	fp	2240
526.blender_r	fp	1523
527.cam4_r	fp	1749
538.imagick_r	fp	2487
544.nab_r	fp	1683
549.fotonik3d_r	fp	3897
554.roms_r	fp	1589

SPEC CPU 2026 — speed

Benchmark	Suite	Reference time (seconds)
801.xz_s	int	591
807.ntest_s	int	1140
817.flac_s	int	1737
821.gcc_s	int	2070
823.llvm_s	int	1411
827.cppcheck_s	int	1119
829.abc_s	int	831
834.vpr_s	int	954
835.gem5_s	int	1139
838.diamond_s	int	1001
846.minizinc_s	int	670
853.ns3_s	int	1153
854.graph500_s	int	611
800.pot3d_s	fp	673
803.sph_exa_s	fp	1238
809.cactus_s	fp	1122
811.tealeaf_s	fp	557
816.nab_s	fp	1126
820.cloverleaf_s	fp	857
822.palm_s	fp	1228
849.fotonik3d_s	fp	660
857.namd_s	fp	1452
865.roms_s	fp	1090
867.nest_s	fp	2160
872.marian_s	fp	1082
881.neutron_s	fp	815

SPEC CPU 2026 — rate

Benchmark	Suite	Reference time (seconds)
706.stockfish_r	int	1260
707.ntest_r	int	592
708.sqlite_r	int	528
710.omnetpp_r	int	486
714.cpython_r	int	479
721.gcc_r	int	686
723.llvm_r	int	507
727.cppcheck_r	int	359
729.abc_r	int	459
734.vpr_r	int	461
735.gem5_r	int	487
750.sealcrypto_r	int	536
753.ns3_r	int	613
777.zstd_r	int	644
709.cactus_r	fp	858
722.palm_r	fp	1320
731.astcenc_r	fp	840
736.ocio_r	fp	875
737.gmsh_r	fp	459
748.flightdm_r	fp	716
749.fotonik3d_r	fp	1156
765.roms_r	fp	1575
766.femflow_r	fp	1467
767.nest_r	fp	793
772.marian_r	fp	1579
782.lbm_r	fp	573

References

Results shown on this site are for research and internal comparison, and are not equivalent to SPEC-published results. Public use of SPEC-related numbers should follow the SPEC Fair Use Rule and the relevant SPEC CPU Run and Reporting Rules, and should clearly distinguish official compliant results, non-official results, and estimates.

BenchCouncil organization information is cited from its official website [6].

[1] Zhan J, Wang L, Gao W, et al. Evaluatology: The Science of Uncovering the Effects[M]. Hong Kong: BenchCouncil Press, 2025. ISBN 978-988-71596-8-1. https://press.benchcouncil.org/book.html
[2] Standard Performance Evaluation Corporation. SPEC CPU Benchmarks[EB/OL]. https://www.spec.org/cpu/
[3] Standard Performance Evaluation Corporation. SPEC CPU 2006[EB/OL]. https://www.spec.org/cpu2006/
[4] Standard Performance Evaluation Corporation. SPEC CPU 2017[EB/OL]. https://www.spec.org/cpu2017/
[5] Standard Performance Evaluation Corporation. SPEC CPU 2026[EB/OL]. https://www.spec.org/cpu2026/
[6] BenchCouncil. International Open Benchmark Council[EB/OL]. https://www.benchcouncil.org/cn/index.html