SPEC 公平使用规则

06/30 15:04
阅读数 83

SPEC Fair Use Rules
Updated 9 February 2023
Change history

Consistency and fairness are guiding principles for SPEC. To help assure that these principles are met, the following requirements must be met by any organization or individual who makes public use of SPEC benchmark results.

Section I lists general requirements that apply to public use of all SPEC benchmarks. Section II lists additional specific requirements for individual benchmarks.

It is intended that this document provides the information needed for compliance with Fair Use, and in the event of any inconsistencies, this document takes precedence over individual benchmark run rules fair use requirements.


I. General Requirements For Public Use
   of All SPEC Benchmark Results

A. Requirements List

1. Compliance

2. Data Sources

3. Clear and correct, as of a specific date

4. Trademarks

5. Required Metrics

6. Comparisons

B. Generic Example

C. Compliance Exceptions

1. Academic/Research Usage

2. Estimates

D. Derived Values

E. Non-SPEC Information

F. Retired Benchmarks

1. Disclosure

2. Benchmarks that require review

3. Normalized historical comparisons

I. General Requirements For Public Use of All SPEC Benchmark Results
I.A. Requirements List
Compliance. Claimed results must be compliant with that benchmark's rules. See definition: compliant result. (Certain Exceptions may apply.)

Data Sources

Source(s) must be stated for quoted SPEC results.

Such sources must be publicly available, from SPEC or elsewhere.

The licensee (the entity responsible for the result) must be clearly identifiable from the source.

The date that the data was retrieved must be stated.

The SPEC web site (http://www.spec.org) or a suitable sub page must be noted as a resource for additional information about the benchmark.

Clear and correct, as of a specific date

Statements regarding SPEC, its benchmarks, and results published by SPEC, must be clear and correct.

A claim must state a date as of which data was retrieved.

A claim may compare newly announced compliant results vs. data retrieved earlier.

There is no requirement to update a claim when later results are published.

For example, an Acme web page dated 28 January 2011 announces performance results for the Model A and claims "the best SPECweb2009 performance when compared vs. results published at www.spec.org as of 26 January 2011". If SPEC publishes better results on 1 February, there is no requirement to update the page.


Reference must be made to the SPEC trademark. Such reference may be included in a notes section with other trademark references (SPEC trademarks are listed at http://www.spec.org/spec/trademarks.html).

SPEC's trademarks may not be used to mislabel something that is not a SPEC metric.

For example, suppose that a Gaming Society compares performance using a composite of a weighted subset of SPEC CPU 2006 plus a weighted subset of SPECviewperf 11, and calls its composite "GamePerfMark". The composite, weighting, and subsetting are done by the Society, not by SPEC. The composite may be useful and interesting, but it may not be represented as a SPEC metric. It would be a Fair Use violation to reference it as "SPECgame".

Required Metrics. In the tables below, some benchmarks have Required Metrics. Public statements must include these.

Comparisons. It is fair to compare compliant results to other compliant results. Enabling such comparisons is a core reason why the SPEC benchmarks exist. Each benchmark product has workloads, software tools, run rules, and review processes that are intended to improve the technical credibility and relevance of such comparisons.

When comparisons are made,

SPEC metrics may be compared only to SPEC metrics.

The basis for comparison must be stated.

Results of one benchmark are not allowed to be compared to a different benchmark
(e.g. SPECjAppServer2004 to TPC-C; or SPECvirt_sc2010 to SPECweb2005).

Results of a benchmark may not be compared to a different major release of the same benchmark
(e.g. SPECweb2005 to SPECweb2009). Exception: normalized historical comparisons may be made as described under Retired Benchmarks.

Comparisons of non-compliant numbers. The comparison of non-compliant numbers to compliant results is restricted to certain exceptional cases described later in this Fair Use Rule (Academic/Research usage; Estimates, for those benchmarks that allow estimates; Normalized Historical Comparisons). Where allowed, comparisons that include non-compliant numbers must not be misleading or deceptive as to compliance. It must be clear from the context of the comparison which numbers are compliant and which are not.

Back to Contents

I.B. Generic Example
This example for a generic SPEC benchmark illustrates the points above. See also the examples for specific benchmarks below, for additional requirements that may apply.

Example: New York, NY, January 28, 2011: Acme Corporation announces that the Model A achieves 100 for SPECgeneric2011, a new record among systems running Linux [1].
[1] Comparison based on best performing systems using the Linux operating system published at www.spec.org as of 26 January 2011. SPEC® and the benchmark name SPECgeneric® are registered trademarks of the Standard Performance Evaluation Corporation. For more information about SPECgeneric2011, see www.spec.org/generic2011/.
Back to Contents

I.C. Compliance Exceptions
Exceptions regarding the compliance requirement are described in this section.

Academic/research usage. SPEC encourages use of its benchmarks in research and academic contexts, on the grounds that SPEC benchmarks represent important characteristics of real world applications and therefore research innovations measured with SPEC benchmarks may benefit real users. SPEC understands that academic use of the SPEC benchmarks may be seen as enhancing the credibility of both the researcher and SPEC.

Research use of SPEC benchmarks may not be able to meet the compliance requirement.

Examples: (1) Testing is done with a simulator rather than real hardware. (2) The software innovation is not generally available or is not of product quality. (3) The SPEC test harness is modified without approval of SPEC.

SPEC has an interest in protecting the integrity of the SPEC metrics, including consistency of methods of measurement and the meaning of the units of measure that are defined by SPEC benchmarks. It would be unfair to those who do meet the compliance requirements if non-compliant numbers were misrepresented as compliant results. Therefore, SPEC recommends that researchers consider using the SPEC workload, but do not call the measurements by the SPEC metric name.

The requirements for Fair Use in academic/research contexts are:

It is a Fair Use violation to imply, to the reasonable reader, that a non-compliant number is a compliant result.
Non-compliance must be clearly disclosed. If the SPEC metric name is used, it is recommended that (nc), for non-compliant, be added after each mention of the metric name. It is understood that there may be other ways to accomplish this in context, for example adding words such as "experimental" or "simulated" or "estimated" or "non-compliant".
Diagrams, Tables, and Abstracts (which, often, are excerpted and used separately) must have sufficient context on their own so that they are not misleading as to compliance.
If non-compliant numbers are compared to compliant results it must be clear from the context which is which.
Example: The Acme Corporation Model A achieves SPECint2006  100 in testing published at www.spec.org. Our Research Compiler improves the same hardware to SPECint2006  125(nc). The notation (nc), for non-compliant, is used because our compiler does not meet SPEC's requirements for general availability.
Other Fair Use Requirements Still Apply. This section discusses an exception to only the compliance requirement from the Requirements List. Fair Use in academic/research context must still meet the other requirements, including but not limited to making correct use of SPEC results with dated citations of sources.

Estimates. Some SPEC benchmarks allow estimates, as shown in the tables below. Only for those benchmarks, it is acceptable to compare estimates to compliant results provided that:

Estimates must be clearly identified as such.

Each use of a SPEC metric as an estimate must be clearly marked as an estimate.

If estimates are used in graphs, the word "estimated" or "est." must be plainly visible within the graph, for example in the title, the scale, the legend, or next to each individual number that is estimated.

Licensees are encouraged to give a rationale or methodology for any estimates, together with other information that may help the reader assess the accuracy of the estimate.

Example 1: The Acme Corporation Model A achieves SPECint2006  100 in testing published at www.spec.org. The Bugle Corporation Model B will nearly double that performance to SPECint2006  198(est). The notation (est), for estimated, is used because SPECint2006 was run on pre-production hardware. Customer systems, planned for Q4, are expected to be similar.

Example 2: Performance estimates are modeled using the cycle simulator GrokSim Mark IV. It is likely that actual hardware, if built, would include significant differences.

Back to Contents

I.D. Derived Values
It is sometimes useful to define a numeric unit that includes a SPEC metric plus other information, and then use the new number to compare systems. This is called a Derived Value.

Examples:    SPECint_rate2006 per chip
SPECvirt_sc2010 per gigabyte
Note: the examples above are not intended to imply that all derived values use ratios of the form above. The definition is intentionally broad, and includes additional examples

Derived values are acceptable, provided that they follow this Fair Use rule, including but not limited to using compliant results, listing sources for SPEC result data, and including any required metrics.

A derived value must not be represented as a SPEC metric. The context must not give the appearance that SPEC has created or endorsed the derived value. In particular, it is a Fair Use violation, and may be a Trademark violation, to form a new word that looks like a SPEC metric name when there is no such metric.

Not Acceptable:    SPECint_chiprate2006
If a derived value is used as the basis of an estimate, the estimate must be correctly labeled. A derived value may introduce seeming opportunities to extrapolate beyond measured data. For example, if 4 different systems all have the same ratio of SPECwhatever per chip, it can be tempting to estimate that another, unmeasured, system will have the same ratio. This may be a very good estimate; but it is still an estimate, and must be correctly labeled. If used in public, it must be for a benchmark that allows estimates.

Back to Contents

I.E. Non-SPEC Information
A basis of comparison or a derived value may use information from both SPEC and non-SPEC sources.

SPEC values truthfulness and clarity at all times:

When information from SPEC sources is used in public, SPEC requires that such information be reported correctly (per section I.A.3).
SPEC recommends that non-SPEC material should be accurate, relevant, and not misleading. Data and methods should be explained and substantiated.
Disclaimer.    SPEC is not responsible for non-SPEC information. The SPEC Fair Use rule is limited to the information derived from SPEC sources. (Other rules may apply to the non-SPEC information, such as industry business standards, ethics, or Truth in Advertising law.)

SPEC may point out non-SPEC content.    SPEC reserves the right to publicly comment to distinguish SPEC information from non-SPEC information.

Integrity of results and trademarks.    The non-SPEC information must not be presented in a manner that may reasonably lead the reader to untrue conclusions about SPEC, its results, or its trademarks.


Example 1 (basis): ACME Corporation claims the best SPECjEnterprise2010 performance for systems available as (example 1a) rack mount, or (1b) with more than 8 disk device slots, or (1c) with Art Deco paint. Bugle Corporation asserts that the basis of comparison is irrelevant or confusing or silly. Bugle may be correct. Nevertheless, such irrelevance, confusion, or silliness would not alone be enough to constitute a SPEC Fair Use violation.

Example 2 (derived value): ACME claims that its model A has better SPECint_rate2006 per unit of cooling requirement than does the Bugle Model B. SPEC is not responsible for judging thermal characteristics.

Example 3: ACME claims the "best SPECmpiM_2007 performance among industry-leading servers". This claim violates the requirement that the basis must be clear.

Example 4: ACME computes SPECint_rate2006 per unit of cooling, but inexplicably selects SPECint_rate_base2006 for some systems and SPECint_rate2006 for others. The computation violates the requirement that the SPEC information must be accurate, and may also violate the requirement that a claim should not lead the reasonable reader to untrue conclusions about SPEC's results.

Back to Contents

I.F. Retired Benchmarks
Disclosure. If public claims are made using a retired benchmark, with compliant results that have not been previously reviewed and accepted by SPEC, then the fact that the benchmark has been retired and new results are no longer being accepted for review and publication by SPEC must be plainly disclosed.

Example: The Acme Corporation Model A achieves a score of 527 SPECjvm98. Note: SPECjvm98 has been retired and SPEC is no longer reviewing or publishing results with that benchmark. We are providing this result as a comparison to older hardware that may still be in use at some customer sites.
Benchmarks that require review. Some benchmarks require that SPEC review and accept results prior to public use. For such benchmarks, the review process is not available after benchmark retirement, and therefore no new results may be published.

Normalized historical comparisons. When SPEC releases a new major version of a benchmark, the SPEC metrics are generally not comparable to the previous version, and there is no formula for converting from one to the other. Nevertheless, SPEC recognizes that there is value in historical comparisons, which are typically done by normalizing performance across current and one or more generations of retired benchmarks, using systems that have been measured with both the older and newer benchmarks as the bridges for the normalization. Historical comparisons are inherently approximate because picking differing 'bridge' systems may yield differing ratios and because an older workload exercises different system capabilities than a more modern workload.

Normalized historical comparisons are acceptable only if their inherently approximate nature is not misrepresented. At minimum:

It must not be claimed that SPEC metrics for one benchmark generation are precisely comparable to metrics from another generation.
The approximate nature must be apparent from the context.
For example, a graph shown briefly in a presentation is labelled "Normalized Historic Trends for SPEC<benchmark>". As another example, in a white paper (where the expectation is for greater detail than presentations), the author explicitly calls out that workloads have differed over time, and explains how numbers are calculated.







I. 公众使用的一般要求

A. 要求清单

1. 遵守规定

2. 数据来源

3. 清晰和正确,截至具体日期

4. 商标

5. 必要的指标

6. 比较

B. 一般性的例子

C. 合规的例外情况

1. 学术/研究用途

2. 估算

D. 衍生的价值

E. 非SPEC信息

F. 退役基准

1. 披露

2. 需要审查的基准

3. 正常化的历史比较

I. 公开使用所有SPEC基准测试结果的一般要求
I.A. 要求列表












例如,一个Acme网页在2011年1月28日宣布了A型车的性能结果,并声称 "与2011年1月26日在www.spec.org 公布的结果相比,SPECweb2009的性能最好"。如果SPEC在2月1日公布了更好的结果,就不需要更新该网页。




例如,假设一个游戏协会使用SPEC CPU 2006的加权子集加上SPECviewperf 11的加权子集来比较性能,并将其复合称为 "GamePerfMark"。复合、加权和子集是由该协会完成的,而不是由SPEC完成的。这种合成可能是有用的和有趣的,但它可能不会被表示为SPEC的指标。将其作为 "SPECgame "来引用是违反公平使用原则的。










I.B. 通用实例

例子: 2011年1月28日,纽约,纽约: Acme公司宣布A型机在SPECgeneric2011中取得了100分的成绩,这是运行Linux的系统中的一个新纪录[1]。
[1] 比较基于截至2011年1月26日在www.spec.org 公布的使用Linux操作系统的最佳性能系统。SPEC®和基准名称SPECgeneric®是标准性能评估公司的注册商标。关于SPECgeneric2011的更多信息,见www.spec.org/generic2011/。

I.C. 合规性的例外情况



举例来说: (1)用模拟器而不是真实的硬件进行测试。(2) 软件创新不是普遍可用的,或者不是产品质量的。(3)SPEC测试线束未经SPEC的批准而被修改。



必须明确披露不符合规定的情况。如果使用SPEC公制名称,建议在每次提到公制名称后加上(nc),表示不符合规定。可以理解的是,在上下文中可能有其他方法来实现这一点,例如,添加诸如 "实验 "或 "模拟 "或 "估计 "或 "不符合要求 "等词语。
例子: Acme公司的A型机在发表于www.spec.org 的测试中达到了SPECint2006 100。我们的研究编译器将相同的硬件提高到SPECint2006 125(nc)。符号(nc),表示不符合要求,因为我们的编译器不符合SPEC的要求,不能普遍使用。




如果在图表中使用估计值,"估计 "或 "EST "一词必须在图表中明显可见,例如,在标题、比例、图例中,或在每个被估计的单独数字旁边。


例1:Acme公司的A型机在发表于www.spec.org 的测试中达到了SPECint2006 100。Bugle公司的B型机的性能几乎翻了一番,达到了SPECint2006的198(est)。由于SPECint2006是在试生产的硬件上运行的,所以使用了估计的符号(est)。计划在第四季度使用的客户系统,预计也会类似。

例2:性能估计是使用周期模拟器GrokSim Mark IV来模拟的。实际的硬件,如果建立的话,很可能会有很大的差异。


I.D. 衍生值

举例来说:    SPECint_rate2006每芯片



不接受:    SPECint_chiprate2006


I.E. 非SPEC信息


免责声明。   SPEC对非SPEC的信息不负责任。SPEC公平使用规则只限于从SPEC来源获得的信息。(其他规则可能适用于非SPEC信息,如行业商业标准、道德或广告法中的真实性)。

SPEC可以指出非SPEC的内容。   SPEC保留公开评论的权利,以区分SPEC信息和非SPEC信息。

结果和商标的完整性。   非SPEC信息不能以一种可能合理地导致读者对SPEC、其结果或其商标得出不真实的结论的方式呈现。


例子1(基础): ACME公司声称,对于可作为(例1a)机架安装、或(1b)具有8个以上磁盘设备插槽、或(1c)具有装饰艺术涂料的系统来说,SPECjEnterprise2010性能最佳。Bugle公司声称,比较的基础是不相关的,或令人困惑的,或愚蠢的。Bugle公司可能是正确的。然而,这种不相关、混乱或愚蠢并不足以单独构成SPEC合理使用的违反。

例2(衍生价值): ACME声称它的模型A比Bugle模型B有更好的SPECint_rate2006每单位的冷却需求,而SPEC不负责判断热特性。

例3:ACME声称 "在业界领先的服务器中具有最好的SPECmpiM_2007性能"。这种说法违反了依据必须明确的要求。



I.F. 退役基准

例子: Acme 公司的 A 型机获得了 527 分的 SPECjvm98。注意:SPECjvm98 已经退役,SPEC 不再审查或公布该基准的结果。我们提供这个结果是为了与一些客户现场可能仍在使用的旧硬件作比较。
需要审查的基准。一些基准要求 SPEC 在公开使用之前审查和接受结果。对于这样的基准,在基准退役后,审查过程是不可用的,因此可能没有新的结果被公布。

规范化的历史比较。当 SPEC 发布一个新的基准的主要版本时,SPEC 的指标通常不能与以前的版本相比较,而且也没有从一个转换到另一个的公式。然而,SPEC 认识到,历史比较是有价值的,这通常是通过对当前和一代或多代退役基准的性能进行归一化来完成的,使用已经用较早和较新的基准测量的系统作为归一化的桥梁。历史比较本质上是近似的,因为挑选不同的 "桥梁 "系统可能会产生不同的比率,也因为较早的工作负载与较现代的工作负载所行使的系统能力不同。


例如,在演示文稿中简要显示的图表被标记为 "SPEC<benchmark>的归一化历史趋势"。另一个例子是,在白皮书中(期望比演讲更详细),作者明确指出工作负载随着时间的推移而不同,并解释数字是如何计算的。

点击引领话题📣 发布并加入讨论🔥
0 评论
0 收藏