Internet Engineering Task Force (IETF) A. Morton
Request for Comments: 8172 AT&T Labs
Category: Informational July 2017
ISSN: 2070-1721
Internet Engineering Task Force (IETF) A. Morton
Request for Comments: 8172 AT&T Labs
Category: Informational July 2017
ISSN: 2070-1721
Considerations for Benchmarking Virtual Network Functions and Their Infrastructure
对虚拟网络功能及其基础设施进行基准测试的注意事项
Abstract
摘要
The Benchmarking Methodology Working Group has traditionally conducted laboratory characterization of dedicated physical implementations of internetworking functions. This memo investigates additional considerations when network functions are virtualized and performed in general-purpose hardware.
基准方法工作组传统上对网络互连功能的专用物理实现进行实验室表征。本备忘录调查了在通用硬件中虚拟化和执行网络功能时的其他注意事项。
Status of This Memo
关于下段备忘
This document is not an Internet Standards Track specification; it is published for informational purposes.
本文件不是互联网标准跟踪规范；它是为了提供信息而发布的。
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 7841.
本文件是互联网工程任务组（IETF）的产品。它代表了IETF社区的共识。它已经接受了公众审查，并已被互联网工程指导小组（IESG）批准出版。并非IESG批准的所有文件都适用于任何级别的互联网标准；见RFC 7841第2节。
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc8172.
有关本文件当前状态、任何勘误表以及如何提供反馈的信息，请访问http://www.rfc-editor.org/info/rfc8172.
Copyright Notice
版权公告
Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.
版权所有（c）2017 IETF信托基金和确定为文件作者的人员。版权所有。
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
本文件受BCP 78和IETF信托有关IETF文件的法律规定的约束(http://trustee.ietf.org/license-info)自本文件出版之日起生效。请仔细阅读这些文件，因为它们描述了您对本文件的权利和限制。从本文件中提取的代码组件必须包括信托法律条款第4.e节中所述的简化BSD许可证文本，并提供简化BSD许可证中所述的无担保。
Table of Contents
目录
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Considerations for Hardware and Testing . . . . . . . . . . . 4
3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4
3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5
3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6
3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7
4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 8
4.1. Comparison with Physical Network Functions . . . . . . . 8
4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8
4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 9
4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 10
4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12
5. Security Considerations . . . . . . . . . . . . . . . . . . . 12
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 13
7.1. Normative References . . . . . . . . . . . . . . . . . . 13
7.2. Informative References . . . . . . . . . . . . . . . . . 14
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 15
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Considerations for Hardware and Testing . . . . . . . . . . . 4
3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4
3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5
3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6
3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7
4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 8
4.1. Comparison with Physical Network Functions . . . . . . . 8
4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8
4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 9
4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 10
4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12
5. Security Considerations . . . . . . . . . . . . . . . . . . . 12
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 13
7.1. Normative References . . . . . . . . . . . . . . . . . . 13
7.2. Informative References . . . . . . . . . . . . . . . . . 14
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 15
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15
The Benchmarking Methodology Working Group (BMWG) has traditionally conducted laboratory characterization of dedicated physical implementations of internetworking functions (or physical network functions (PNFs)). The black-box benchmarks of throughput, latency, forwarding rates, and others have served our industry for many years. [RFC1242] and [RFC2544] are the cornerstones of the work.
基准测试方法工作组（BMWG）传统上对互联网功能（或物理网络功能（PNF））的专用物理实现进行实验室表征。吞吐量、延迟、转发速率等黑匣子基准多年来一直服务于我们的行业。[RFC1242]和[RFC2544]是这项工作的基石。
A set of service provider and vendor development goals has emerged: reduce costs while increasing flexibility of network devices and drastically reduce deployment time. Network Function Virtualization (NFV) has the promise to achieve these goals and therefore has garnered much attention. It now seems certain that some network functions will be virtualized following the success of cloud computing and virtual desktops supported by sufficient network path capacity, performance, and widespread deployment; many of the same techniques will help achieve NFV.
出现了一系列服务提供商和供应商的发展目标：降低成本，同时提高网络设备的灵活性，并大幅缩短部署时间。网络功能虚拟化（NFV）有望实现这些目标，因此受到了广泛关注。现在看来，随着云计算和虚拟桌面在足够的网络路径容量、性能和广泛部署的支持下取得成功，一些网络功能将被虚拟化是肯定的；许多相同的技术将有助于实现NFV。
In the context of Virtual Network Functions (VNFs), the supporting Infrastructure requires general-purpose computing systems, storage systems, networking systems, virtualization support systems (such as hypervisors), and management systems for the virtual and physical resources. There will be many potential suppliers of Infrastructure
在虚拟网络功能（VNF）方面，支持基础设施需要通用计算系统、存储系统、网络系统、虚拟化支持系统（如虚拟机监控程序）以及虚拟和物理资源的管理系统。将有许多潜在的基础设施供应商
systems and significant flexibility in configuring the systems for best performance. There are also many potential suppliers of VNFs, adding to the combinations possible in this environment. The separation of hardware and software suppliers has a profound implication on benchmarking activities: much more of the internal configuration of the black-box Device Under Test (DUT) must now be specified and reported with the results, to foster both repeatability and comparison testing at a later time.
在配置系统以获得最佳性能方面具有极大的灵活性。也有许多潜在的VNF供应商，在这种环境中增加了可能的组合。硬件和软件供应商的分离对基准测试活动有着深远的影响：现在必须指定更多被测黑匣子设备（DUT）的内部配置，并随结果报告，以促进以后的重复性和比较测试。
Consider the following user story as further background and motivation:
考虑下面的用户故事作为进一步的背景和动机：
I'm designing and building my NFV Infrastructure platform. The first steps were easy because I had a small number of categories of VNFs to support and the VNF vendor gave hardware recommendations that I followed. Now I need to deploy more VNFs from new vendors, and there are different hardware recommendations. How well will the new VNFs perform on my existing hardware? Which among several new VNFs in a given category are most efficient in terms of capacity they deliver? And, when I operate multiple categories of VNFs (and PNFs) *concurrently* on a hardware platform such that they share resources, what are the new performance limits, and what are the software design choices I can make to optimize my chosen hardware platform? Conversely, what hardware platform upgrades should I pursue to increase the capacity of these concurrently operating VNFs?
我正在设计和构建我的NFV基础设施平台。第一步很简单，因为我需要支持少量的VNF类别，并且VNF供应商给出了我遵循的硬件建议。现在我需要从新的供应商那里部署更多的VNF，并且有不同的硬件建议。新的VNF在我现有的硬件上的性能如何？在给定类别的几个新VNF中，哪一个在提供容量方面最有效？而且，当我在硬件平台上同时操作多个类别的VNF（和PNF）*以共享资源时，新的性能限制是什么，我可以做出什么样的软件设计选择来优化我选择的硬件平台？相反，我应该进行哪些硬件平台升级以增加这些并发运行的VNF的容量？
See <http://www.etsi.org/technologies-clusters/technologies/nfv> for more background; the white papers there may be a useful starting place. The "NFV Performance & Portability Best Practices" document [NFV.PER001] is particularly relevant to BMWG. There are also documents available among the Approved ETSI NFV Specifications [Approved_ETSI_NFV], including documents describing Infrastructure performance aspects and service quality metrics, and drafts in the ETSI NFV Open Area [Draft_ETSI_NFV], which may also have relevance to benchmarking.
看<http://www.etsi.org/technologies-clusters/technologies/nfv>了解更多背景；那里的白皮书可能是一个有用的起点。“NFV性能和可移植性最佳实践”文件[NFV.PER001]与BMWG特别相关。在已批准的ETSI NFV规范[已批准的ETSI NFV]中也有可用的文件，包括描述基础设施性能方面和服务质量指标的文件，以及ETSI NFV开放区域的草案[草案]，这些文件也可能与基准相关。
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
本文件中的关键词“必须”、“不得”、“必需”、“应”、“不应”、“建议”、“不建议”、“可”和“可选”在所有大写字母出现时（如图所示）应按照BCP 14[RFC2119][RFC8174]所述进行解释。
At the time of this writing, BMWG is considering the new topic of Virtual Network Functions and related Infrastructure to ensure that common issues are recognized from the start; background materials from respective standards development organizations and Open Source development projects (e.g., IETF, ETSI NFV, and the Open Platform for Network Function Virtualization (OPNFV)) are being used.
在撰写本文时，BMWG正在考虑虚拟网络功能和相关基础设施的新主题，以确保从一开始就认识到常见问题；正在使用来自各个标准开发组织和开源开发项目（如IETF、ETSI NFV和网络功能虚拟化开放平台（OPNFV））的背景材料。
This memo investigates additional methodological considerations necessary when benchmarking VNFs instantiated and hosted in general-purpose hardware, using bare metal hypervisors [BareMetal] or other isolation environments such as Linux containers. An essential consideration is benchmarking physical and Virtual Network Functions in the same way when possible, thereby allowing direct comparison. Benchmarking combinations of physical and virtual devices and functions in a System Under Test (SUT) is another topic of keen interest.
本备忘录调查了在使用裸机管理程序[裸机]或其他隔离环境（如Linux容器）对在通用硬件中实例化和托管的VNF进行基准测试时需要考虑的其他方法。一个重要的考虑因素是尽可能以相同的方式对物理和虚拟网络功能进行基准测试，从而允许直接比较。测试系统（SUT）中物理和虚拟设备及功能的基准组合是另一个非常感兴趣的话题。
A clearly related goal is investigating benchmarks for the capacity of a general-purpose platform to host a plurality of VNF instances. Existing networking technology benchmarks will also be considered for adaptation to NFV and closely associated technologies.
一个明确相关的目标是研究通用平台承载多个VNF实例的能力基准。还将考虑现有的网络技术基准，以适应NFV和密切相关的技术。
A non-goal is any overlap with traditional computer benchmark development and their specific metrics (e.g., SPECmark suites such as SPEC CPU).
非目标是与传统计算机基准测试开发及其特定指标（例如SPECmark套件，如SPEC CPU）的任何重叠。
A continued non-goal is any form of architecture development related to NFV and associated technologies in BMWG, consistent with all chartered work since BMWG began in 1989.
一个持续的非目标是与BMWG中NFV和相关技术相关的任何形式的架构开发，与BMWG自1989年开始以来的所有特许工作一致。
This section lists the new considerations that must be addressed to benchmark VNF(s) and their supporting Infrastructure. The SUT is composed of the hardware platform components, the VNFs installed, and many other supporting systems. It is critical to document all aspects of the SUT to foster repeatability.
本节列出了基准VNF及其支持基础架构必须考虑的新问题。SUT由硬件平台组件、安装的VNFs和许多其他支持系统组成。记录SUT的所有方面以培养可重复性至关重要。
The following new hardware components will become part of the test setup:
以下新硬件组件将成为测试设置的一部分：
1. High-volume server platforms (general-purpose, possibly with virtual technology enhancements)
1. 高容量服务器平台（通用，可能具有虚拟技术增强功能）
2. Storage systems with large capacity, high speed, and high reliability
2. 大容量、高速、高可靠性的存储系统
3. Network interface ports specially designed for efficient service of many virtual Network Interface Cards (NICs)
3. 网络接口端口专门为许多虚拟网络接口卡（NIC）的高效服务而设计
4. High-capacity Ethernet switches
4. 高容量以太网交换机
The components above are subjects for development of specialized benchmarks that focus on the special demands of network function deployment.
上述组件是开发专门基准的主题，重点关注网络功能部署的特殊需求。
Labs conducting comparisons of different VNFs may be able to use the same hardware platform over many studies, until the steady march of innovations overtakes their capabilities (as happens with the lab's traffic generation and testing devices today).
对不同的VNF进行比较的实验室可能能够在许多研究中使用相同的硬件平台，直到稳步推进的创新超过了它们的能力（就像今天实验室的流量生成和测试设备一样）。
It will be necessary to configure and document the settings for the entire general-purpose platform to ensure repeatability and foster future comparisons, including, but clearly not limited to, the following:
有必要配置和记录整个通用平台的设置，以确保可重复性并促进未来的比较，包括但不限于以下内容：
o number of server blades (shelf occupation)
o 刀片服务器数量（机架占用）
o CPUs
o 中央处理器
o caches
o 贮藏
o memory
o 记忆力
o storage system
o 存储系统
o I/O
o 输入输出
as well as configurations that support the devices that host the VNF itself:
以及支持承载VNF本身的设备的配置：
o Hypervisor (or other forms of virtual function hosting)
o 虚拟机监控程序（或其他形式的虚拟函数托管）
o Virtual Machine (VM)
o 虚拟机（VM）
o Infrastructure virtual network (which interconnects virtual machines with physical network interfaces or with each other through virtual switches, for example)
o 基础设施虚拟网络（例如，通过虚拟交换机将虚拟机与物理网络接口或彼此互连）
and finally, the VNF itself, with items such as:
最后是VNF本身，包括以下项目：
o specific function being implemented in VNF
o 正在VNF中实现的特定功能
o reserved resources for each function (e.g., CPU pinning and Non-Uniform Memory Access (NUMA) node assignment)
o 每个功能的保留资源（例如，CPU固定和非统一内存访问（NUMA）节点分配）
o number of VNFs (or sub-VNF components, each with its own VM) in the service function chain (see Section 1.1 of [RFC7498] for a definition of service function chain)
o 服务功能链中的VNF数量（或子VNF组件，每个组件都有自己的VM）（服务功能链的定义见[RFC7498]第1.1节）
o number of physical interfaces and links transited in the service function chain
o 服务功能链中传输的物理接口和链接的数量
In the physical device benchmarking context, most of the corresponding Infrastructure configuration choices were determined by the vendor. Although the platform itself is now one of the configuration variables, it is important to maintain emphasis on the networking benchmarks and capture the platform variables as input factors.
在物理设备基准测试环境中，大多数相应的基础设施配置选择由供应商确定。尽管平台本身现在是配置变量之一，但重要的是保持对网络基准的重视，并将平台变量捕获为输入因素。
The concept of characterizing performance at capacity limits may change. For example:
在容量限制下表征性能的概念可能会改变。例如：
1. It may be more representative of system capacity to characterize the case where the VMs hosting the VNFs are operating at 50% utilization and therefore sharing the "real" processing power across many VMs.
1. 在承载VNF的虚拟机以50%的利用率运行，因此在多个虚拟机之间共享“真实”处理能力的情况下，这可能更能代表系统容量。
2. Another important test case stems from the need to partition (or isolate) network functions. A noisy neighbor (VM hosting a VNF in an infinite loop) would ideally be isolated; the performance of other VMs would continue according to their specifications, and tests would evaluate the degree of isolation.
2. 另一个重要的测试用例源于划分（或隔离）网络功能的需要。一个嘈杂的邻居（在无限循环中承载一个VNF的VM）在理想情况下会被隔离；其他虚拟机的性能将根据其规格继续进行，测试将评估隔离程度。
3. System errors will likely occur as transients, implying a distribution of performance characteristics with a long tail (like latency) and leading to the need for longer-term tests of each set of configuration and test parameters.
3. 系统错误可能会以瞬态的形式出现，这意味着性能特征的分布具有长尾（如延迟），并导致需要对每组配置和测试参数进行长期测试。
4. The desire for elasticity and flexibility among network functions will include tests where there is constant flux in the number of VM instances, the resources the VMs require, and the setup/ teardown of network paths that support VM connectivity. Requests for and instantiation of new VMs, along with releases for VMs hosting VNFs that are no longer needed, would be a normal
4. 对网络功能的弹性和灵活性的需求将包括测试，其中虚拟机实例的数量、虚拟机所需的资源以及支持虚拟机连接的网络路径的设置/拆除是恒定的。对新虚拟机的请求和实例化，以及对承载不再需要的VNF的虚拟机的发布，将是一个正常的过程
operational condition. In other words, benchmarking should include scenarios with production life-cycle management of VMs and their VNFs and network connectivity in progress, including VNF scaling up/down operations, as well as static configurations.
操作条件。换句话说，基准测试应包括虚拟机及其VNF的生产生命周期管理和正在进行的网络连接（包括VNF放大/缩小操作）以及静态配置的场景。
5. All physical things can fail, and benchmarking efforts can also examine recovery aided by the virtual architecture with different approaches to resiliency.
5. 所有物理事物都可能失败，基准测试工作还可以通过不同的恢复方法检查虚拟体系结构辅助的恢复。
6. The sheer number of test conditions and configuration combinations encourage increased efficiency, including automated testing arrangements, combination sub-sampling through an understanding of inter-relationships, and machine-readable test results.
6. 测试条件和配置组合的绝对数量鼓励提高效率，包括自动测试安排、通过理解相互关系进行组合子采样以及机器可读的测试结果。
Since many components of the new NFV Infrastructure are virtual, test setup design must have prior knowledge of interactions/dependencies within the various resource domains in the SUT. For example, a virtual machine performing the role of a traditional tester function, such as generating and/or receiving traffic, should avoid sharing any SUT resources with the DUT. Otherwise, the results will have unexpected dependencies not encountered in physical device benchmarking.
由于新NFV基础设施的许多组件都是虚拟的，因此测试设置设计必须事先了解SUT中各种资源域内的交互/依赖关系。例如，执行传统测试仪功能（如生成和/或接收流量）的虚拟机应避免与DUT共享任何SUT资源。否则，结果将具有物理设备基准测试中未遇到的意外依赖项。
Note that the term "tester" has traditionally referred to devices dedicated to testing in BMWG literature. In this new context, "tester" additionally refers to functions dedicated to testing, which may be either virtual or physical. "Tester" has never referred to the individuals performing the tests.
注意，术语“测试仪”传统上指BMWG文献中专用于测试的设备。在这个新的上下文中，“tester”还指专用于测试的功能，可以是虚拟的，也可以是物理的。“测试人员”从未指执行测试的个人。
The possibility to use shared resources in test design while producing useful results remains one of the critical challenges to overcome. Benchmarking setups may designate isolated resources for the DUT and other critical support components (such as the host/ kernel) as the first baseline step and add other loading processes. The added complexity of each setup leads to shared-resource testing scenarios, where the characteristics of the competing load (in terms of memory, storage, and CPU utilization) will directly affect the benchmarking results (and variability of the results), but the results should reconcile with the baseline.
在测试设计中使用共享资源同时产生有用结果的可能性仍然是需要克服的关键挑战之一。基准测试设置可以指定DUT和其他关键支持组件（如主机/内核）的独立资源作为第一个基准步骤，并添加其他加载过程。每个设置增加的复杂性导致共享资源测试场景，其中竞争负载的特征（内存、存储和CPU利用率方面）将直接影响基准测试结果（以及结果的可变性），但结果应与基线一致。
The physical test device remains a solid foundation to compare with results using combinations of physical and virtual test functions or results using only virtual testers when necessary to assess virtual interfaces and other virtual functions.
物理测试设备仍然是一个坚实的基础，与物理和虚拟测试功能的组合或使用虚拟测试器的结果在必要时评估虚拟接口和其他虚拟功能的结果进行比较。
This section discusses considerations related to benchmarks applicable to VNFs and their associated technologies.
本节讨论与适用于VNF及其相关技术的基准相关的注意事项。
In order to compare the performance of VNFs and system implementations with their physical counterparts, identical benchmarks must be used. Since BMWG has already developed specifications for many network functions, there will be re-use of existing benchmarks through references, while allowing for the possibility of benchmark curation during development of new methodologies. Consideration should be given to quantifying the number of parallel VNFs required to achieve comparable scale/capacity with a given physical device or whether some limit of scale was reached before the VNFs could achieve the comparable level. Again, implementation based on different hypervisors or other virtual function hosting remain as critical factors in performance assessment.
为了将VNF和系统实现的性能与其物理实现进行比较，必须使用相同的基准测试。由于BMWG已经为许多网络功能制定了规范，因此将通过引用重新使用现有基准，同时允许在开发新方法时进行基准管理。应考虑量化使用给定物理设备实现可比规模/容量所需的并行VNF数量，或者在VNF达到可比水平之前是否达到了一定的规模限制。同样，基于不同虚拟机监控程序或其他虚拟函数宿主的实现仍然是性能评估的关键因素。
When the network functions under test are based on open-source code, there may be a tendency to rely on internal measurements to some extent, especially when the externally observable phenomena only support an inference of internal events (such as routing protocol convergence observed in the data plane). Examples include CPU/Core utilization, network utilization, storage utilization, and memory committed/used. These "white-box" metrics provide one view of the resource footprint of a VNF. Note that the resource utilization metrics do not easily match the 3x4 Matrix, described in Section 4.4.
当测试中的网络功能基于开放源代码时，可能会在某种程度上依赖于内部测量，特别是当外部可观察现象仅支持内部事件的推断（例如在数据平面中观察到的路由协议收敛）时。示例包括CPU/核心利用率、网络利用率、存储利用率和提交/使用的内存。这些“白盒”指标提供了VNF资源占用的一个视图。请注意，资源利用率指标不容易匹配第4.4节中描述的3x4矩阵。
However, external observations remain essential as the basis for benchmarks. Internal observations with fixed specification and interpretation may be provided in parallel (as auxiliary metrics), to assist the development of operations procedures when the technology is deployed, for example. Internal metrics and measurements from open-source implementations may be the only direct source of performance results in a desired dimension, but corroborating external observations are still required to assure the integrity of measurement discipline was maintained for all reported results.
然而，外部观察仍然是基准的基础。例如，在部署技术时，可以并行提供具有固定规格和解释的内部观察（作为辅助指标），以协助制定操作程序。开源实现的内部度量和测量可能是期望维度中性能结果的唯一直接来源，但仍然需要确凿的外部观察，以确保所有报告结果的测量规程的完整性。
A related aspect of benchmark development is where the scope includes multiple approaches to a common function under the same benchmark. For example, there are many ways to arrange for activation of a network path between interface points, and the activation times can be compared if the start-to-stop activation interval has a generic
基准开发的一个相关方面是，范围包括对同一基准下的公共功能的多种方法。例如，有许多方法可以安排在接口点之间激活网络路径，并且如果从开始到停止的激活间隔具有通用性，则可以比较激活时间
and unambiguous definition. Thus, generic benchmark definitions are preferred over technology/protocol-specific definitions where possible.
明确的定义。因此，在可能的情况下，通用基准定义优于特定于技术/协议的定义。
There will be new classes of benchmarks needed for network design and assistance when developing operational practices (possibly automated management and orchestration of deployment scale). Examples follow in the paragraphs below, many of which are prompted by the goals of increased elasticity and flexibility of the network functions, along with reduced deployment times.
在开发操作实践（可能是部署规模的自动化管理和协调）时，网络设计和协助将需要新的基准类。下面的段落中有一些例子，其中许多例子都是为了提高网络功能的弹性和灵活性，以及减少部署时间。
o Time to deploy VNFs: In cases where the general-purpose hardware is already deployed and ready for service, it is valuable to know the response time when a management system is tasked with "standing up" 100s of virtual machines and the VNFs they will host.
o 部署VNF的时间：在通用硬件已经部署并准备好服务的情况下，了解管理系统被指派“站立”100个虚拟机以及它们将承载的VNF的响应时间是很有价值的。
o Time to migrate VNFs: In cases where a rack or shelf of hardware must be removed from active service, it is valuable to know the response time when a management system is tasked with "migrating" some number of virtual machines and the VNFs they currently host to alternate hardware that will remain in service.
o 迁移VNF的时间：在必须从活动服务中移除硬件机架或机架的情况下，了解管理系统执行“迁移”某些数量的虚拟机及其当前承载的VNF到将继续服务的备用硬件的任务时的响应时间是很有价值的。
o Time to create a virtual network in the general-purpose Infrastructure: This is a somewhat simplified version of existing benchmarks for convergence time, in that the process is initiated by a request from (centralized or distributed) control, rather than inferred from network events (link failure). The successful response time would remain dependent on data-plane observations to confirm that the network is ready to perform.
o 在通用基础设施中创建虚拟网络的时间：这是现有收敛时间基准的简化版本，因为该过程由来自（集中式或分布式）控制的请求启动，而不是从网络事件（链路故障）推断。成功的响应时间仍取决于数据面观察，以确认网络已准备好执行。
o Effect of verification measurements on performance: A complete VNF, or something as simple as a new policy to implement in a VNF, is implemented. The action to verify instantiation of the VNF or policy could affect performance during normal operation.
o 验证度量对性能的影响：实现了一个完整的VNF，或者像要在VNF中实现的新策略这样简单的东西。验证VNF或策略实例化的操作可能会影响正常操作期间的性能。
Also, it appears to be valuable to measure traditional packet transfer performance metrics during the assessment of traditional and new benchmarks, including metrics that may be used to support service engineering such as the spatial composition metrics found in [RFC6049]. Examples include mean one-way delay in Section 4.1 of [RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet Reordering [RFC4737] [RFC4689].
此外，在评估传统基准和新基准期间，测量传统分组传输性能指标似乎很有价值，包括可用于支持服务工程的指标，如[RFC6049]中的空间组合指标。示例包括[RFC6049]第4.1节中的平均单向延迟、[RFC5481]中的数据包延迟变化（PDV）以及数据包重新排序[RFC4737][RFC4689]。
It can be useful to organize benchmarks according to their applicable life-cycle stage and the performance criteria they were designed to assess. The table below (derived from [X3.102]) provides a way to organize benchmarks such that there is a clear indication of coverage for the intersection of life-cycle stages and performance criteria.
根据基准的适用生命周期阶段和设计用于评估的性能标准来组织基准是有用的。下表（来源于[X3.102]）提供了一种组织基准的方法，以明确指示生命周期阶段和性能标准交叉点的覆盖范围。
|----------------------------------------------------------|
| | | | |
| | SPEED | ACCURACY | RELIABILITY |
| | | | |
|----------------------------------------------------------|
| | | | |
| Activation | | | |
| | | | |
|----------------------------------------------------------|
| | | | |
| Operation | | | |
| | | | |
|----------------------------------------------------------|
| | | | |
| De-activation | | | |
| | | | |
|----------------------------------------------------------|
|----------------------------------------------------------|
| | | | |
| | SPEED | ACCURACY | RELIABILITY |
| | | | |
|----------------------------------------------------------|
| | | | |
| Activation | | | |
| | | | |
|----------------------------------------------------------|
| | | | |
| Operation | | | |
| | | | |
|----------------------------------------------------------|
| | | | |
| De-activation | | | |
| | | | |
|----------------------------------------------------------|
For example, the "Time to deploy VNFs" benchmark described above would be placed in the intersection of Activation and Speed, making it clear that there are other potential performance criteria to benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups" in a set of 100 attempts. This example emphasizes that the Activation and De-activation life-cycle stages are key areas for NFV and related Infrastructure and encourages expansion beyond traditional benchmarks for normal operation. Thus, reviewing the benchmark coverage using this table (sometimes called the 3x3 Matrix) can be a worthwhile exercise in BMWG.
例如，上面描述的“部署VNF的时间”基准将放在激活和速度的交叉点上，这表明还有其他潜在的性能标准需要基准测试，例如100次尝试中的“失败VM/VNF站立的百分比”。该示例强调，激活和去激活生命周期阶段是NFV和相关基础设施的关键领域，并鼓励超出正常运行的传统基准的扩展。因此，在BMWG中，使用此表（有时称为3x3矩阵）审查基准覆盖率是一项值得做的工作。
In one of the first applications of the 3x3 Matrix in BMWG [SDN-BENCHMARK], we discovered that metrics on measured size, capacity, or scale do not easily match one of the three columns above. Following discussion, this was resolved in two ways:
在BMWG[SDN-BENCHMARK]中3x3矩阵的第一个应用程序中，我们发现度量大小、容量或规模的指标不容易匹配上述三列中的一列。经过讨论，通过两种方式解决了这一问题：
o Add a column, Scale, for use when categorizing and assessing the coverage of benchmarks (without measured results). An example of this use is found in [OPNFV-BENCHMARK] (and a variation may be found in [SDN-BENCHMARK]). This is the 3x4 Matrix.
o 添加一列量表，用于分类和评估基准覆盖率（无测量结果）。[OPNFV-BENCHMARK]中有此用法的一个示例，（在[SDN-BENCHMARK]中可以找到一个变体）。这是3x4矩阵。
o If using the matrix to report results in an organized way, keep size, capacity, and scale metrics separate from the 3x3 Matrix and incorporate them in the report with other qualifications of the results.
o 如果使用矩阵以有组织的方式报告结果，请将大小、容量和规模指标与3x3矩阵分开，并将其与结果的其他限定条件合并到报告中。
Note that the resource utilization (e.g., CPU) metrics do not fit in the matrix. They are not benchmarks, and omitting them confirms their status as auxiliary metrics. Resource assignments are configuration parameters, and these are reported separately.
请注意，资源利用率（例如CPU）指标不适合该矩阵。它们不是基准，省略它们将确认它们作为辅助度量的状态。资源分配是配置参数，它们是单独报告的。
This approach encourages use of the 3x3 Matrix to organize reports of results, where the capacity at which the various metrics were measured could be included in the title of the matrix (and results for multiple capacities would result in separate 3x3 Matrices, if there were sufficient measurements/results to organize in that way).
这种方法鼓励使用3x3矩阵来组织结果报告，其中测量各种指标的能力可以包括在矩阵标题中（如果有足够的测量/结果以这种方式组织，多个能力的结果将产生单独的3x3矩阵）。
For example, results for each VM and VNF could appear in the 3x3 Matrix, organized to illustrate resource occupation (CPU Cores) in a particular physical computing system, as shown below.
例如，每个VM和VNF的结果可能出现在3x3矩阵中，组织起来以说明特定物理计算系统中的资源占用（CPU核心），如下所示。
VNF#1
.-----------.
|__|__|__|__|
Core 1 |__|__|__|__|
|__|__|__|__|
| | | | |
'-----------'
VNF#2
.-----------.
|__|__|__|__|
Cores 2-5 |__|__|__|__|
|__|__|__|__|
| | | | |
'-----------'
VNF#3 VNF#4 VNF#5
.-----------. .-----------. .-----------.
|__|__|__|__| |__|__|__|__| |__|__|__|__|
Core 6 |__|__|__|__| |__|__|__|__| |__|__|__|__|
|__|__|__|__| |__|__|__|__| |__|__|__|__|
| | | | | | | | | | | | | | |
'-----------' '-----------' '-----------'
VNF#6
.-----------.
|__|__|__|__|
Core 7 |__|__|__|__|
|__|__|__|__|
| | | | |
'-----------'
VNF#1
.-----------.
|__|__|__|__|
Core 1 |__|__|__|__|
|__|__|__|__|
| | | | |
'-----------'
VNF#2
.-----------.
|__|__|__|__|
Cores 2-5 |__|__|__|__|
|__|__|__|__|
| | | | |
'-----------'
VNF#3 VNF#4 VNF#5
.-----------. .-----------. .-----------.
|__|__|__|__| |__|__|__|__| |__|__|__|__|
Core 6 |__|__|__|__| |__|__|__|__| |__|__|__|__|
|__|__|__|__| |__|__|__|__| |__|__|__|__|
| | | | | | | | | | | | | | |
'-----------' '-----------' '-----------'
VNF#6
.-----------.
|__|__|__|__|
Core 7 |__|__|__|__|
|__|__|__|__|
| | | | |
'-----------'
The combination of tables above could be built incrementally, beginning with VNF#1 and one Core, then adding VNFs according to their supporting Core assignments. X-Y plots of critical benchmarks would also provide insight to the effect of increased hardware utilization. All VNFs might be of the same type, or to match a production environment, there could be VNFs of multiple types and categories. In this figure, VNFs #3-#5 are assumed to require small CPU resources, while VNF#2 requires four Cores to perform its function.
上述表的组合可以增量构建，从VNF#1和一个核心开始，然后根据支持的核心分配添加VNF。关键基准的X-Y图也将提供对提高硬件利用率的影响的洞察。所有VNF可能是同一类型的，或者为了匹配生产环境，可能有多种类型和类别的VNF。在此图中，假设VNF#3-#5需要较小的CPU资源，而VNF#2需要四个内核来执行其功能。
Although there is incomplete work to benchmark physical network function power consumption in a meaningful way, the desire to measure the physical Infrastructure supporting the virtual functions only adds to the need. Both maximum power consumption and dynamic power consumption (with varying load) would be useful. The Intelligent Platform Management Interface (IPMI) standard [IPMI2.0] has been implemented by many manufacturers and supports measurement of instantaneous energy consumption.
尽管以一种有意义的方式对物理网络功能功耗进行基准测试的工作还不完善，但测量支持虚拟功能的物理基础设施的愿望只会增加需求。最大功耗和动态功耗（负载变化时）都很有用。许多制造商已经实施了智能平台管理接口（IPMI）标准[IPMI2.0]，并支持瞬时能耗测量。
To assess the instantaneous energy consumption of virtual resources, it may be possible to estimate the value using an overall metric based on utilization readings, according to [NFVIaas-FRAMEWORK].
为了评估虚拟资源的瞬时能量消耗，根据[NFVIaas FRAMEWORK]，可以使用基于利用率读数的总体指标来估计该值。
Benchmarking activities as described in this memo are limited to technology characterization of a DUT/SUT using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above.
本备忘录中所述的基准测试活动仅限于在实验室环境中使用受控刺激对DUT/SUT进行技术表征，并具有专用地址空间和上述章节中规定的约束条件。
The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network.
基准网络拓扑将是一个独立的测试设置，不得连接到可能将测试流量转发到生产网络或将流量错误路由到测试管理网络的设备。
Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT/SUT.
此外，基准测试是在“黑盒”的基础上进行的，仅依赖于DUT/SUT外部可观察到的测量。
Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks.
DUT/SUT中不应存在专门用于基准测试的特殊能力。DUT/SUT对网络安全的任何影响应在实验室和生产网络中相同。
This document does not require any IANA actions.
本文件不要求IANA采取任何行动。
[NFV.PER001] ETSI, "Network Function Virtualization: Performance & Portability Best Practices", ETSI GS NFV-PER 001, V1.1.2, December 2014.
[NFV.PER001]ETSI，“网络功能虚拟化：性能和可移植性最佳实践”，ETSI GS NFV-PER001，V1.1.22014年12月。
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <http://www.rfc-editor.org/info/rfc2119>.
[RFC2119]Bradner，S.，“RFC中用于表示需求水平的关键词”，BCP 14，RFC 2119，DOI 10.17487/RFC2119，1997年3月<http://www.rfc-editor.org/info/rfc2119>.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, <http://www.rfc-editor.org/info/rfc2544>.
[RFC2544]Bradner，S.和J.McQuaid，“网络互连设备的基准测试方法”，RFC 2544，DOI 10.17487/RFC2544，1999年3月<http://www.rfc-editor.org/info/rfc2544>.
[RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, "Terminology for Benchmarking Network-layer Traffic Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689, October 2006, <http://www.rfc-editor.org/info/rfc4689>.
[RFC4689]Poretsky，S.，Perser，J.，Erramilli，S.，和S.Khurana，“基准网络层流量控制机制的术语”，RFC 4689，DOI 10.17487/RFC4689，2006年10月<http://www.rfc-editor.org/info/rfc4689>.
[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S., and J. Perser, "Packet Reordering Metrics", RFC 4737, DOI 10.17487/RFC4737, November 2006, <http://www.rfc-editor.org/info/rfc4737>.
[RFC4737]Morton，A.，Ciavattone，L.，Ramachandran，G.，Shalunov，S.，和J.Perser，“数据包重新排序度量”，RFC 4737，DOI 10.17487/RFC4737，2006年11月<http://www.rfc-editor.org/info/rfc4737>.
[RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for Service Function Chaining", RFC 7498, DOI 10.17487/RFC7498, April 2015, <http://www.rfc-editor.org/info/rfc7498>.
[RFC7498]Quinn，P.，Ed.和T.Nadeau，Ed.，“服务功能链接的问题陈述”，RFC 7498，DOI 10.17487/RFC7498，2015年4月<http://www.rfc-editor.org/info/rfc7498>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <http://www.rfc-editor.org/info/rfc8174>.
[RFC8174]Leiba，B.，“RFC 2119关键词中大写与小写的歧义”，BCP 14，RFC 8174，DOI 10.17487/RFC8174，2017年5月<http://www.rfc-editor.org/info/rfc8174>.
[Approved_ETSI_NFV] ETSI, Network Functions Virtualisation Technical Committee, "ETSI NFV", <http://www.etsi.org/standards-search>.
[Approved_ETSI_NFV]ETSI，网络功能虚拟化技术委员会，“ETSI NFV”<http://www.etsi.org/standards-search>.
[BareMetal] Popek, G. and R. Goldberg, "Formal requirements for virtualizable third generation architectures", Communications of the ACM, Volume 17, Issue 7, Pages 412-421, DOI 10.1145/361011.361073, July 1974.
[BareMetal]Popek，G.和R.Goldberg，“可虚拟化第三代架构的正式要求”，《ACM通讯》，第17卷，第7期，第412-421页，DOI 10.1145/361011.361073，1974年7月。
[Draft_ETSI_NFV] ETSI, "Network Functions Virtualisation: Specifications", <http://www.etsi.org/technologies-clusters/technologies/ nfv>.
[Draft_ETSI_NFV]ETSI，“网络功能虚拟化：规范”<http://www.etsi.org/technologies-clusters/technologies/ nfv>。
[IPMI2.0] Intel Corporation, Hewlett-Packard Company, NEC Corporation, and Dell Inc., "Intelligent Platform Management Interface Specification v2.0 (with latest errata)", April 2015, <http://www.intel.com/content/dam/www/public/us/en/ documents/specification-updates/ipmi-intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec-update.pdf>.
[IPMI2.0]英特尔公司、惠普公司、NEC公司和戴尔公司，“智能平台管理接口规范v2.0（最新勘误表）”，2015年4月<http://www.intel.com/content/dam/www/public/us/en/ 文档/规范更新/ipmi-intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec-update.pdf>。
[NFVIaas-FRAMEWORK] Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D., Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra, "NFVIaaS Architectural Framework for Policy Based Resource Placement and Scheduling", Work in Progress, draft-krishnan-nfvrg-policy-based-rm-nfviaas-06, March 2016.
[NFVIaas框架]Krishnan，R.，Figueira，N.，Krishnaswamy，D.，Lopez，D.，Wright，S.，Hinrichs，T.，Krishnaswamy，R.，和A.Yerra，“NFVIaas基于政策的资源配置和调度架构框架”，在建工程，草稿-Krishnan-nfvrg-Policy-Based-rm-NFVIaas-062016年3月。
[OPNFV-BENCHMARK] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking Virtual Switches in OPNFV", Work in Progress, draft-ietf-bmwg-vswitch-opnfv-04, June 2017.
[OPNFV-BENCHMARK]Tahan，M.，O'Mahony，B.，和A.Morton，“OPNFV中虚拟交换机的基准测试”，正在进行的工作，草稿-ietf-bmwg-vswitch-OPNFV-042017年6月。
[RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <http://www.rfc-editor.org/info/rfc1242>.
[RFC1242]Bradner，S.，“网络互连设备的基准术语”，RFC 1242，DOI 10.17487/RFC1242，1991年7月<http://www.rfc-editor.org/info/rfc1242>.
[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, March 2009, <http://www.rfc-editor.org/info/rfc5481>.
[RFC5481]Morton，A.和B.Claise，“数据包延迟变化适用性声明”，RFC 5481，DOI 10.17487/RFC5481，2009年3月<http://www.rfc-editor.org/info/rfc5481>.
[RFC6049] Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, <http://www.rfc-editor.org/info/rfc6049>.
[RFC6049]Morton，A.和E.Stephan，“度量的空间构成”，RFC 6049，DOI 10.17487/RFC6049，2011年1月<http://www.rfc-editor.org/info/rfc6049>.
[SDN-BENCHMARK] Vengainathan, B., Basil, A., Tassinari, M., Manral, V., and S. Banks, "Terminology for Benchmarking SDN Controller Performance", Work in Progress, draft-ietf-bmwg-sdn-controller-benchmark-term-04, June 2017.
[SDN-BENCHMARK]Vengianthan，B.，Basil，A.，Tassinari，M.，Manral，V.，和S.Banks，“SDN控制器性能基准术语”，正在进行的工作，草案-ietf-bmwg-SDN-Controller-BENCHMARK-term-042017年6月。
[X3.102] ANSI, "Information Systems - Data Communication Systems and Services - User-Oriented Performance Parameters Communications Framework", ANSI X3.102, 1983.
[X3.102]ANSI，“信息系统-数据通信系统和服务-面向用户的性能参数通信框架”，ANSI X3.1021983。
Acknowledgements
致谢
The author acknowledges an encouraging conversation on this topic with Mukhtiar Shaikh and Ramki Krishnan in November 2013. Bhavani Parise and Ilya Varlashkin have provided useful suggestions to expand these considerations. Bhuvaneswaran Vengainathan has already tried the 3x3 Matrix with the SDN controller document and contributed to many discussions. Scott Bradner quickly pointed out shared resource dependencies in an early vSwitch measurement proposal, and the topic was included here as a key consideration. Further development was encouraged by Barry Constantine's comments following the BMWG session at IETF 92: the session itself was an affirmation for this memo. There have been many interesting contributions from Maryam Tahhan, Marius Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others.
作者承认2013年11月与Mukhtiar Shaikh和Ramki Krishnan就此话题进行了令人鼓舞的对话。Bhavani Parise和Ilya Varlashkin提供了有用的建议来扩展这些考虑。Bhuvaneswaran Vengianthan已经在SDN控制器文档中尝试了3x3矩阵，并参与了许多讨论。Scott Bradner在早期的vSwitch度量提案中很快指出了共享资源的依赖性，该主题作为一个关键考虑事项包含在这里。Barry Constantine在IETF 92 BMWG会议后的评论鼓励了进一步的发展：会议本身就是对本备忘录的肯定。玛亚姆·塔汉、马吕斯·乔治斯库、雅各布·拉普、索拉布·查托帕迪亚和其他人做出了许多有趣的贡献。
Author's Address
作者地址
Al Morton AT&T Labs 200 Laurel Avenue South Middletown, NJ 07748 United States of America
美国新泽西州劳雷尔大道南米德尔顿200号阿尔莫顿AT&T实验室，邮编：07748
Phone: +1 732 420 1571
Fax: +1 732 368 1192
Email: acmorton@att.com
Phone: +1 732 420 1571
Fax: +1 732 368 1192
Email: acmorton@att.com