Internet Engineering Task Force (IETF) T. Haynes
Request for Comments: 8434 Hammerspace
Updates: 5661 August 2018
Category: Standards Track
ISSN: 2070-1721
Internet Engineering Task Force (IETF) T. Haynes
Request for Comments: 8434 Hammerspace
Updates: 5661 August 2018
Category: Standards Track
ISSN: 2070-1721
Requirements for Parallel NFS (pNFS) Layout Types
对并行NFS（pNFS）布局类型的要求
Abstract
摘要
This document defines the requirements that individual Parallel NFS (pNFS) layout types need to meet in order to work within the pNFS framework as defined in RFC 5661. In so doing, this document aims to clearly distinguish between requirements for pNFS as a whole and those specifically directed to the pNFS file layout. The lack of a clear separation between the two sets of requirements has been troublesome for those specifying and evaluating new layout types. In this regard, this document updates RFC 5661.
本文档定义了各个并行NFS（pNFS）布局类型需要满足的要求，以便在RFC 5661中定义的pNFS框架内工作。因此，本文件旨在明确区分pNFS整体要求和专门针对pNFS文件布局的要求。对于那些指定和评估新布局类型的人来说，这两组需求之间缺乏明确的分离一直是个麻烦。在这方面，本文件更新了RFC 5661。
Status of This Memo
关于下段备忘
This is an Internet Standards Track document.
这是一份互联网标准跟踪文件。
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841.
本文件是互联网工程任务组（IETF）的产品。它代表了IETF社区的共识。它已经接受了公众审查，并已被互联网工程指导小组（IESG）批准出版。有关互联网标准的更多信息，请参见RFC 7841第2节。
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8434.
有关本文件当前状态、任何勘误表以及如何提供反馈的信息，请访问https://www.rfc-editor.org/info/rfc8434.
Copyright Notice
版权公告
Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.
版权所有（c）2018 IETF信托基金和确定为文件作者的人员。版权所有。
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
本文件受BCP 78和IETF信托有关IETF文件的法律规定的约束(https://trustee.ietf.org/license-info)自本文件出版之日起生效。请仔细阅读这些文件，因为它们描述了您对本文件的权利和限制。从本文件中提取的代码组件必须包括信托法律条款第4.e节中所述的简化BSD许可证文本，并提供简化BSD许可证中所述的无担保。
Table of Contents
目录
1. Introduction ....................................................2
2. Definitions .....................................................3
2.1. Use of the Terms "Data Server" and "Storage Device" ........5
2.2. Requirements Language ......................................6
3. The Control Protocol ............................................6
3.1. Control Protocol Requirements ..............................8
3.2. Previously Undocumented Protocol Requirements ..............9
3.3. Editorial Requirements ....................................10
4. Specifications of Original Layout Types ........................11
4.1. File Layout Type ..........................................11
4.2. Block Layout Type .........................................12
4.3. Object Layout Type ........................................13
5. Summary ........................................................14
6. Security Considerations ........................................15
7. IANA Considerations ............................................15
8. References .....................................................16
8.1. Normative References ......................................16
8.2. Informative References ....................................16
Acknowledgments ...................................................17
Author's Address ..................................................17
1. Introduction ....................................................2
2. Definitions .....................................................3
2.1. Use of the Terms "Data Server" and "Storage Device" ........5
2.2. Requirements Language ......................................6
3. The Control Protocol ............................................6
3.1. Control Protocol Requirements ..............................8
3.2. Previously Undocumented Protocol Requirements ..............9
3.3. Editorial Requirements ....................................10
4. Specifications of Original Layout Types ........................11
4.1. File Layout Type ..........................................11
4.2. Block Layout Type .........................................12
4.3. Object Layout Type ........................................13
5. Summary ........................................................14
6. Security Considerations ........................................15
7. IANA Considerations ............................................15
8. References .....................................................16
8.1. Normative References ......................................16
8.2. Informative References ....................................16
Acknowledgments ...................................................17
Author's Address ..................................................17
The concept of "layout type" has a central role in the definition and implementation of Parallel NFS (pNFS) (see [RFC5661]). Clients and servers implementing different layout types behave differently in many ways while conforming to the overall pNFS framework defined in [RFC5661] and this document. Layout types may differ as to:
“布局类型”的概念在并行NFS（pNFS）的定义和实现中起着核心作用（参见[RFC5661]）。实现不同布局类型的客户端和服务器在许多方面表现不同，同时符合[RFC5661]和本文档中定义的总体pNFS框架。布局类型可能在以下方面有所不同：
o The method used to do I/O operations directed to data storage devices.
o 用于执行指向数据存储设备的I/O操作的方法。
o The requirements for communication between the metadata server (MDS) and the storage devices.
o 元数据服务器（MDS）和存储设备之间的通信要求。
o The means used to ensure that I/O requests are only processed when the client holds an appropriate layout.
o 用于确保仅当客户端拥有适当布局时才处理I/O请求的方法。
o The format and interpretation of nominally opaque data fields in pNFS-related NFSv4.x data structures.
o pNFS相关NFSv4.x数据结构中名义上不透明数据字段的格式和解释。
Each layout type will define the needed details for its usage in the specification for that layout type; layout type specifications are always Standards Track RFCs. Except for the file layout type defined
每个布局类型将在该布局类型的规范中定义其使用所需的详细信息；布局类型规范始终是标准轨道RFC。定义的文件布局类型除外
in Section 13 of [RFC5661], existing layout types are defined in their own Standards Track documents, and it is anticipated that new layout types will be defined in similar documents.
在[RFC5661]第13节中，现有布局类型在其自身的标准跟踪文件中定义，预计新布局类型将在类似文件中定义。
The file layout type was defined in the Network File System (NFS) version 4.1 protocol specification [RFC5661]. The block layout type was defined in [RFC5663], and the object layout type was defined in [RFC5664]. Subsequently, the Small Computer System Interface (SCSI) layout type was defined in [RFC8154].
文件布局类型在网络文件系统（NFS）版本4.1协议规范[RFC5661]中定义。块布局类型在[RFC5663]中定义，对象布局类型在[RFC5664]中定义。随后，在[RFC8154]中定义了小型计算机系统接口（SCSI）布局类型。
Some implementers have interpreted the text in Sections 12 ("Parallel NFS (pNFS)") and 13 ("NFSv4.1 as a Storage Protocol in pNFS: the File Layout Type") of [RFC5661] as applying only to the file layout type. Because Section 13 was not covered in a separate Standards Track document such as those for both the block and object layout types, there was some confusion as to the responsibilities of both the metadata server and the data servers (DSs) that were laid out in Section 12.
一些实施者将[RFC5661]第12节（“并行NFS（pNFS）”）和第13节（“NFSv4.1作为pNFS中的存储协议：文件布局类型”）中的文本解释为仅适用于文件布局类型。由于第13节未包含在单独的标准跟踪文件中，例如块和对象布局类型的标准跟踪文件，因此第12节中列出的元数据服务器和数据服务器（DSs）的责任存在一些混淆。
As a consequence, authors of new specifications (see [RFC8435] and [Lustre]) may struggle to meet the requirements to be a pNFS layout type. This document gathers the requirements from all of the original Standards Track documents regarding layout type and then specifies the requirements placed on all layout types independent of the particular type chosen.
因此，新规范（见[RFC8435]和[Lustre]）的作者可能难以满足pNFS布局类型的要求。本文档从所有原始标准跟踪文档中收集有关布局类型的要求，然后指定独立于所选特定类型的所有布局类型上的要求。
control communication requirement: the specification for information on layouts, stateids, file metadata, and file data that must be communicated between the metadata server and the storage devices. There is a separate set of requirements for each layout type.
控制通信要求：有关布局、状态ID、文件元数据和必须在元数据服务器和存储设备之间通信的文件数据的信息的规范。每种布局类型都有一套单独的要求。
control protocol: the particular mechanism that an implementation of a layout type would use to meet the control communication requirement for that layout type. This need not be a protocol as normally understood. In some cases, the same protocol may be used as both a control protocol and storage protocol.
控制协议：布局类型的实现用于满足该布局类型的控制通信要求的特定机制。这不需要是通常理解的协议。在某些情况下，同一协议可同时用作控制协议和存储协议。
storage protocol: the protocol used by clients to do I/O operations to the storage device. Each layout type specifies the set of storage protocols.
存储协议：客户端用于对存储设备执行I/O操作的协议。每种布局类型指定一组存储协议。
loose coupling: when the control protocol is a storage protocol.
松耦合：当控制协议是存储协议时。
tight coupling: an arrangement in which the control protocol is one designed specifically for control communication. It may be either a proprietary protocol adapted specifically to a particular metadata server or a protocol based on a Standards Track document.
紧密耦合：一种安排，其中控制协议是专门为控制通信设计的协议。它可以是专门适用于特定元数据服务器的专有协议，也可以是基于标准跟踪文档的协议。
(file) data: that part of the file system object that contains the data to be read or written. It is the contents of the object rather than the attributes of the object.
（文件）数据：文件系统对象中包含要读取或写入的数据的部分。它是对象的内容，而不是对象的属性。
data server (DS): a pNFS server that provides the file's data when the file system object is accessed over a file-based protocol. Note that this usage differs from that in [RFC5661], which applies the term in some cases even when other sorts of protocols are being used. Depending on the layout, there might be one or more data servers over which the data is striped. While the metadata server is strictly accessed over the NFSv4.1 protocol, the data server could be accessed via any file access protocol that meets the pNFS requirements.
数据服务器（DS）：通过基于文件的协议访问文件系统对象时提供文件数据的pNFS服务器。请注意，此用法与[RFC5661]中的用法不同，后者在某些情况下甚至在使用其他种类的协议时也应用此术语。根据布局的不同，可能有一个或多个数据服务器将数据分条。虽然元数据服务器严格通过NFSv4.1协议访问，但数据服务器可以通过满足pNFS要求的任何文件访问协议访问。
See Section 2.1 for a comparison of this term and "storage device".
有关该术语与“存储设备”的比较，请参见第2.1节。
storage device: the target to which clients may direct I/O requests when they hold an appropriate layout. Note that each data server is a storage device but that some storage device are not data servers. See Section 2.1 for further discussion.
存储设备：当客户端持有适当的布局时，可以将I/O请求定向到的目标。请注意，每个数据服务器都是一个存储设备，但有些存储设备不是数据服务器。进一步讨论见第2.1节。
fencing: the process by which the metadata server prevents the storage devices from processing I/O from a specific client to a specific file.
防护：元数据服务器阻止存储设备处理从特定客户端到特定文件的I/O的过程。
layout: the information a client uses to access file data on a storage device. This information includes specification of the protocol (layout type) and the identity of the storage devices to be used.
布局：客户端用于访问存储设备上的文件数据的信息。此信息包括协议规范（布局类型）和要使用的存储设备的标识。
The bulk of the contents of the layout are defined in [RFC5661] as nominally opaque, but individual layout types are responsible for specifying the format of the layout data.
在[RFC5661]中，版面的大部分内容被定义为名义上不透明的，但个别版面类型负责指定版面数据的格式。
layout iomode: a grant of either read-only or read/write I/O to the client.
布局iomode：向客户端授予只读或读/写I/O。
layout stateid: a 128-bit quantity returned by a server that uniquely defines the layout state provided by the server for a specific layout that describes a layout type and file (see
layout stateid：服务器返回的128位数量，它唯一地定义了服务器为描述布局类型和文件的特定布局提供的布局状态（请参见
Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 of [RFC5661] describes differences in handling between layout stateids and other stateid types.
[RFC5661]第12.5.2节。此外，[RFC5661]第12.5.3节描述了布局stateid和其他stateid类型之间的处理差异。
layout type: a specification of both the storage protocol used to access the data and the aggregation scheme used to lay out the file data on the underlying storage devices.
布局类型：用于访问数据的存储协议和用于在底层存储设备上布局文件数据的聚合方案的规范。
recalling a layout: a graceful recall, via a callback, of a specific layout by the metadata server to the client. Graceful here means that the client would have the opportunity to flush any WRITEs, etc., before returning the layout to the metadata server.
调用布局：通过回调，元数据服务器向客户端优雅地调用特定布局。这里的优雅意味着，在将布局返回到元数据服务器之前，客户端将有机会刷新任何写入操作等。
revoking a layout: an invalidation of a specific layout by the metadata server. Once revocation occurs, the metadata server will not accept as valid any reference to the revoked layout, and a storage device will not accept any client access based on the layout.
撤销布局：元数据服务器对特定布局的失效。一旦发生撤销，元数据服务器将不接受对已撤销布局的任何引用作为有效引用，并且存储设备将不接受基于该布局的任何客户端访问。
(file) metadata: the part of the file system object that contains various descriptive data relevant to the file object, as opposed to the file data itself. This could include the time of last modification, access time, EOF position, etc.
（文件）元数据：文件系统对象的一部分，它包含与文件对象相关的各种描述性数据，而不是文件数据本身。这可能包括上次修改的时间、访问时间、EOF位置等。
metadata server (MDS): the pNFS server that provides metadata information for a file system object. It is also responsible for generating, recalling, and revoking layouts for file system objects, for performing directory operations, and for performing I/O operations to regular files when the clients direct these to the metadata server itself.
元数据服务器（MDS）：为文件系统对象提供元数据信息的pNFS服务器。它还负责生成、调用和撤销文件系统对象的布局，执行目录操作，以及在客户端将常规文件定向到元数据服务器本身时对常规文件执行I/O操作。
stateid: a 128-bit quantity returned by a server that uniquely defines the set of locking-related state provided by the server. Stateids may designate state related to open files, byte-range locks, delegations, or layouts.
stateid：服务器返回的128位数量，它唯一地定义了服务器提供的锁定相关状态集。StateID可以指定与打开的文件、字节范围锁、委托或布局相关的状态。
In [RFC5661], the terms "data server" and "storage device" are used somewhat inconsistently:
在[RFC5661]中，术语“数据服务器”和“存储设备”的使用有些不一致：
o In Section 12, where pNFS in general is discussed, the term "storage device" is used.
o 在第12节中，一般讨论PNF时，使用术语“存储设备”。
o In Section 13, where the file layout type is discussed, the term "data server" is used.
o 在讨论文件布局类型的第13节中，使用了术语“数据服务器”。
o In other sections, the term "data server" is used, even in contexts where the storage access type is not NFSv4.1 or any other file access protocol.
o 在其他章节中，使用术语“数据服务器”，即使在存储访问类型不是NFSv4.1或任何其他文件访问协议的上下文中也是如此。
As this document deals with pNFS in general, it uses the more generic term "storage device" in preference to "data server". The term "data server" is used only in contexts in which a file server is used as a storage device. Note that every data server is a storage device, but storage devices that use protocols that are not file access protocols (such as NFS) are not data servers.
由于本文档一般涉及PNF，因此使用了更通用的术语“存储设备”，而不是“数据服务器”。术语“数据服务器”仅在文件服务器用作存储设备的上下文中使用。请注意，每个数据服务器都是存储设备，但使用非文件访问协议（如NFS）的存储设备不是数据服务器。
Since a given storage device may support multiple layout types, a given device can potentially act as a data server for some set of storage protocols while simultaneously acting as a storage device for others.
由于给定的存储设备可能支持多种布局类型，因此给定的设备可能充当某些存储协议集的数据服务器，同时充当其他存储协议集的存储设备。
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
本文件中的关键词“必须”、“不得”、“必需”、“应”、“不应”、“建议”、“不建议”、“可”和“可选”在所有大写字母出现时（如图所示）应按照BCP 14[RFC2119][RFC8174]所述进行解释。
This document differs from most Standards Track documents in that it specifies requirements for those defining future layout types rather than defining the requirements for implementations directly. This document makes clear whether:
本文档与大多数标准跟踪文档的不同之处在于，它为那些定义未来布局类型的文档指定了需求，而不是直接定义实现的需求。本文件明确说明：
(1) any particular requirement applies to implementations.
(1) 任何特定的要求都适用于实现。
(2) any particular requirement applies to those defining layout types.
(2) 任何特殊要求都适用于定义布局类型的要求。
(3) the requirement is a general requirement that implementations need to conform to, with the specific means left to layout type definitions type to specify.
(3) 该需求是实现需要遵循的一般需求，具体方法留给布局类型定义来指定。
A layout type has to meet the requirements that apply to the interaction between the metadata server and the storage device such that they present to the client a consistent view of stored data and locking state (Section 12.2.6 of [RFC5661]). Particular implementations may satisfy these requirements in any manner they choose, and the mechanism chosen need not be described as a protocol. Specifications defining layout types need to clearly show how implementations can meet the requirements discussed below, especially
布局类型必须满足适用于元数据服务器和存储设备之间交互的要求，以便向客户提供存储数据和锁定状态的一致视图（RFC5661第12.2.6节）。特定实现可以以它们选择的任何方式满足这些需求，并且所选择的机制不需要描述为协议。定义布局类型的规范需要清楚地显示实现如何满足下面讨论的需求，特别是
with respect to those that have security implications. In addition, such specifications may find it necessary to impose requirements on implementations of the layout type to ensure appropriate interoperability.
对于那些有安全影响的。此外，此类规范可能会发现有必要对布局类型的实现施加要求，以确保适当的互操作性。
In some cases, there may be no control protocol other than the storage protocol. This is often described as using a "loosely coupled" model. In such cases, the assumption is that the metadata server, storage devices, and client may be changed independently and that the implementation requirements in the layout type specification need to ensure this degree of interoperability. This model is used in the block and object layout type specification.
在某些情况下，除了存储协议之外，可能没有其他控制协议。这通常被描述为使用“松散耦合”模型。在这种情况下，假设元数据服务器、存储设备和客户端可以独立更改，布局类型规范中的实现要求需要确保这种程度的互操作性。此模型用于块和对象布局类型规范。
In other cases, it is assumed that there will be a purpose-built control protocol that may be different for different implementations of the metadata server and data server. The assumption here is that the metadata server and data servers are designed and implemented as a unit and interoperability needs to be assured between clients and metadata-data server pairs, developed independently. This is the model used for the file layout.
在其他情况下，假设将有一个专门构建的控制协议，该协议对于元数据服务器和数据服务器的不同实现可能不同。这里的假设是，元数据服务器和数据服务器是作为一个单元设计和实现的，并且需要确保独立开发的客户端和元数据数据服务器对之间的互操作性。这是用于文件布局的模型。
Another possibility is for the definition of a control protocol to be specified in a Standards Track document. There are two subcases to consider:
另一种可能性是在标准跟踪文件中规定控制协议的定义。有两个子类别需要考虑：
o A new layout type includes a definition of a particular control protocol whose use is obligatory for metadata servers and storage devices implementing the layout type. In this case, the interoperability model is similar to the first case above, and the defining document should assure interoperability among metadata servers, storage devices, and clients developed independently.
o 新布局类型包括特定控制协议的定义，元数据服务器和实现该布局类型的存储设备必须使用该协议。在这种情况下，互操作性模型类似于上面的第一种情况，定义文档应确保独立开发的元数据服务器、存储设备和客户端之间的互操作性。
o A control protocol is defined in a Standards Track document that meets the control protocol requirements for one of the existing layout types. In this case, the new document's job is to assure interoperability between metadata servers and storage devices developed separately. The existing definition document for the selected layout type retains the function of assuring interoperability between clients and a given collection of metadata servers and storage devices. In this context, implementations that implement the new protocol are treated in the same way as those that use an internal control protocol or a functional equivalent.
o 控制协议在标准跟踪文档中定义，该文档满足现有布局类型之一的控制协议要求。在这种情况下，新文档的任务是确保元数据服务器和单独开发的存储设备之间的互操作性。选定布局类型的现有定义文档保留了确保客户端与给定元数据服务器和存储设备集合之间互操作性的功能。在这种情况下，实现新协议的实现与使用内部控制协议或等效功能的实现的处理方式相同。
An example of this last case is the SCSI layout type [RFC8154], which extends the block layout type. The block layout type had a requirement for fencing of clients but did not present a way for the
最后一种情况的示例是SCSI布局类型[RFC8154]，它扩展了块布局类型。区块布局类型对客户的围栏有要求，但没有为客户提供一种方式
control protocol (in this case, the SCSI storage protocol) to fence the client. The SCSI layout type remedies that in [RFC8154] and, in effect, has a tightly coupled model.
用于隔离客户端的控制协议（在本例中为SCSI存储协议）。SCSI布局类型弥补了[RFC8154]中的缺陷，实际上具有紧密耦合的模型。
The requirements of interactions between the metadata server and the storage devices are:
元数据服务器和存储设备之间的交互要求如下：
(1) The metadata server MUST be able to service the client's I/O requests if the client decides to make such requests to the metadata server instead of to the storage device. The metadata server must be able to retrieve the data from the constituent storage devices and present it back to the client. A corollary to this is that even though the metadata server has successfully given the client a layout, the client MAY still send I/O requests to the metadata server.
(1) 如果客户机决定向元数据服务器而不是向存储设备发出请求，则元数据服务器必须能够为客户机的I/O请求提供服务。元数据服务器必须能够从组成存储设备检索数据，并将其呈现回客户端。由此推论，即使元数据服务器已成功地为客户机提供了布局，客户机仍可以向元数据服务器发送I/O请求。
(2) The metadata server MUST be able to restrict access to a file on the storage devices when it revokes a layout. The metadata server typically would revoke a layout whenever a client fails to respond to a recall or a client's lease is expired due to non-renewal. It might also revoke the layout as a means of enforcing a change in locking state or access permissions that the storage device cannot directly enforce.
(2) 元数据服务器必须能够在撤销布局时限制对存储设备上文件的访问。元数据服务器通常会在客户端无法响应回调或客户端的租约因未续订而过期时撤销布局。它还可以撤销布局，以强制更改存储设备无法直接强制执行的锁定状态或访问权限。
Effective revocation may require client cooperation in using a particular stateid (file layout) or principal (e.g., flexible file layout) when performing I/O.
有效的撤销可能需要客户端在执行I/O时配合使用特定的stateid（文件布局）或主体（例如，灵活的文件布局）。
In contrast, there is no requirement to restrict access to a file on the storage devices when a layout is recalled. It is only after the metadata server determines that the client is not gracefully returning the layout and starts the revocation that this requirement is enforced.
相反，当调用布局时，不需要限制对存储设备上文件的访问。只有在元数据服务器确定客户机没有正常返回布局并开始撤销之后，此要求才会强制执行。
(3) A pNFS implementation MUST NOT allow the violation of NFSv4.1's access controls: Access Control Lists (ACLs) and file open modes. Section 12.9 of [RFC5661] specifically lays this burden on the combination of clients, storage devices, and the metadata server. However, the specification of the individual layout type might create requirements as to how this is to be done. This may include a possible requirement for the metadata server to update the storage device so that it can enforce security.
(3) pNFS实现不得违反NFSv4.1的访问控制：访问控制列表（ACL）和文件打开模式。[RFC5661]的第12.9节明确规定了客户机、存储设备和元数据服务器的组合。但是，单个布局类型的规范可能会对如何实现这一点产生要求。这可能包括元数据服务器可能需要更新存储设备，以便它能够实施安全性。
The file layout requires the storage device to enforce access whereas the flexible file layout requires both the storage device and the client to enforce security.
文件布局要求存储设备强制访问，而灵活的文件布局要求存储设备和客户端强制安全。
(4) Interactions between locking and I/O operations MUST obey existing semantic restrictions. In particular, if an I/O operation would be invalid when directed at the metadata server, it is not to be allowed when performed on the storage device.
(4) 锁定和I/O操作之间的交互必须遵守现有的语义限制。特别是，如果I/O操作在指向元数据服务器时无效，则在存储设备上执行时不允许该操作。
For the block and SCSI layouts, as the storage device is not able to reject the I/O operation, the client is responsible for enforcing this requirement.
对于块和SCSI布局，由于存储设备无法拒绝I/O操作，因此客户机负责强制执行此要求。
(5) Any disagreement between the metadata server and the data server as to the value of attributes such as modify time, the change attribute, and the EOF position MUST be of limited duration with clear means of resolution of any discrepancies being provided. Note the following:
(5) 元数据服务器和数据服务器之间关于属性值（如修改时间、更改属性和EOF位置）的任何分歧必须持续时间有限，并提供明确的解决方法。注意以下几点：
(a) Discrepancies need not be resolved unless any client has accessed the file in question via the metadata server, typically by performing a GETATTR.
(a) 除非任何客户机通过元数据服务器访问了相关文件（通常通过执行GETATTR），否则不需要解决差异。
(b) A particular storage device might be striped, and as such, its local view of the EOF position does not match the global EOF position.
(b) 特定存储设备可能是条带化的，因此，其EOF位置的本地视图与全局EOF位置不匹配。
(c) Both clock skew and network delay can lead to the metadata server and the storage device having different values of the time attributes. As long as those differences can be accounted for in what is presented to the client in a GETATTR, then no violation results.
(c) 时钟偏移和网络延迟都可能导致元数据服务器和存储设备具有不同的时间属性值。只要这些差异可以在GETATTR中呈现给客户机的内容中得到解释，那么就不会产生冲突结果。
(d) A LAYOUTCOMMIT requires that changes in attributes resulting from operations on the storage device need to be reflected in the metadata server by the completion of the operation.
(d) LAYOUTCOMMIT要求存储设备上的操作导致的属性更改需要在操作完成时反映在元数据服务器中。
These requirements may be satisfied in different ways by different layout types. As an example, while the file layout type uses the stateid to fence off the client, there is no requirement that other layout types use this stateid approach.
不同的布局类型可能以不同的方式满足这些要求。例如，虽然文件布局类型使用stateid隔离客户端，但不要求其他布局类型使用此stateid方法。
Each new Standards Track document for a layout type MUST address how the client, metadata server, and storage devices are to interact to meet these requirements.
布局类型的每个新标准跟踪文档都必须说明客户端、元数据服务器和存储设备如何交互以满足这些要求。
While not explicitly stated as requirements in Section 12 of [RFC5661], the existing layout types do have more requirements that they need to enforce.
虽然在[RFC5661]第12节中没有明确规定为要求，但现有布局类型确实有更多需要强制执行的要求。
The client has these obligations when making I/O requests to the storage devices:
客户机在向存储设备发出I/O请求时有以下义务：
(1) Clients MUST NOT perform I/O to the storage device if they do not have layouts for the files in question.
(1) 如果客户机没有相关文件的布局，则不得对存储设备执行I/O。
(2) Clients MUST NOT perform I/O operations outside of the specified ranges in the layout segment.
(2) 客户端不得在布局段的指定范围之外执行I/O操作。
(3) Clients MUST NOT perform I/O operations that would be inconsistent with the iomode specified in the layout segments it holds.
(3) 客户机不得执行与其所持有的布局段中指定的iomode不一致的I/O操作。
Under the file layout type, the storage devices are able to reject any request made not conforming to these requirements. This may not be possible for other known layout types, which puts the burden of enforcing such violations solely on the client. For these layout types:
在文件布局类型下，存储设备能够拒绝任何不符合这些要求的请求。对于其他已知的布局类型，这可能是不可能的，这将强制执行此类冲突的负担完全放在客户机上。对于这些布局类型：
(1) The metadata server MAY use fencing operations to the storage devices to enforce layout revocation against the client.
(1) 元数据服务器可以使用对存储设备的防护操作来对客户端强制布局撤销。
(2) The metadata server MUST allow the clients to perform data I/O against it, even if it has already granted the client a layout. A layout type might discourage such I/O, but it cannot forbid it.
(2) 元数据服务器必须允许客户端对其执行数据I/O，即使它已经授予客户端布局。布局类型可能不鼓励这样的I/O，但不能禁止。
(3) The metadata server MUST be able to do storage allocation, whether that is to create, delete, extend, or truncate files.
(3) 元数据服务器必须能够进行存储分配，无论是创建、删除、扩展还是截断文件。
The means to address these requirements will vary with the layout type. A control protocol will be used to effect these; the control protocol could be a purpose-built one, one identical to the storage protocol, or a new Standards Track control protocol.
满足这些要求的方法将因布局类型而异。将使用控制协议来实现这些功能；控制协议可以是专门构建的、与存储协议相同的协议，也可以是新的标准跟踪控制协议。
This section discusses how the protocol requirements discussed above need to be addressed in documents specifying a new layout type. Depending on the interoperability model for the layout type in question, this may involve the imposition of layout-type-specific requirements that ensure appropriate interoperability of pNFS components that are developed separately.
本节讨论如何在指定新布局类型的文档中解决上述协议要求。根据所讨论布局类型的互操作性模型，这可能涉及实施特定于布局类型的要求，以确保单独开发的pNFS组件的适当互操作性。
The specification of the layout type needs to make clear how the client, metadata server, and storage device act together to meet the protocol requirements discussed previously. If the document does not
布局类型的规范需要明确客户机、元数据服务器和存储设备如何协同工作以满足前面讨论的协议要求。如果文件没有
impose implementation requirements sufficient to ensure that these semantic requirements are met, it is not appropriate for publication as an RFC from the IETF stream.
如果实施要求足以确保满足这些语义要求，则不适合从IETF流发布为RFC。
Some examples include:
一些例子包括：
o If the metadata server does not have a means to invalidate a stateid issued to the storage device to keep a particular client from accessing a specific file, then the layout type specification has to document how the metadata server is going to fence the client from access to the file on that storage device.
o 如果元数据服务器无法使发给存储设备的stateid无效，从而阻止特定客户端访问特定文件，则布局类型规范必须记录元数据服务器将如何阻止客户端访问该存储设备上的文件。
o If the metadata server implements mandatory byte-range locking when accessed directly by the client, then the layout type specification must require that this also be done when data is read or written using the designated storage protocol.
o 如果元数据服务器在客户端直接访问时实现强制字节范围锁定，则布局类型规范必须要求在使用指定的存储协议读取或写入数据时也执行此操作。
This section discusses how the original layout types interact with Section 12 of [RFC5661], which enumerates the requirements of pNFS layout type specifications. It is not normative with regards to the file layout type presented in Section 13 of [RFC5661], the block layout type [RFC5663], and the object layout type [RFC5664]. These are discussed here only to illuminate the updates Section 3 of this document makes to Section 12 of [RFC5661].
本节讨论原始布局类型如何与[RFC5661]第12节交互，该节列举了pNFS布局类型规范的要求。关于[RFC5661]第13节中介绍的文件布局类型、块布局类型[RFC5663]和对象布局类型[RFC5664]，这是不规范的。此处讨论这些内容只是为了说明本文件第3节对[RFC5661]第12节的更新。
Because the storage protocol is a subset of NFSv4.1, the semantics of the file layout type comes closest to the semantics of NFSv4.1 in the absence of pNFS. In particular, the stateid and principal used for I/O MUST have the same effect and be subject to the same validation on a data server as it would have if the I/O were being performed on the metadata server itself. The same set of validations are applied whether or not pNFS is in effect.
因为存储协议是NFSv4.1的子集，所以在没有pNFS的情况下，文件布局类型的语义最接近NFSv4.1的语义。特别是，用于I/O的stateid和principal必须具有与在元数据服务器本身上执行I/O时相同的效果，并在数据服务器上接受相同的验证。无论pNFS是否有效，都会应用相同的验证集。
While for most implementations, the storage devices can do the following validations that are each presented as a "SHOULD" and not a "MUST" in [RFC5661]:
对于大多数实现，存储设备可以执行以下验证，每个验证在[RFC5661]中表示为“应该”，而不是“必须”：
(1) client holds a valid layout,
(1) 客户端拥有有效的布局，
(2) client I/O matches the layout iomode, and
(2) 客户端I/O与布局iomode匹配，并且
(3) client does not go out of the byte ranges,
(3) 客户端未超出字节范围，
Actually, the first point is presented in [RFC5661] as both:
实际上，第一点在[RFC5661]中表示为：
"MUST": in Section 13.6
“必须”：在第13.6节中
As described in Section 12.5.1, a client MUST NOT send an I/O to a data server for which it does not hold a valid layout; the data server MUST reject such an I/O.
如第12.5.1节所述，客户机不得向其未持有有效布局的数据服务器发送I/O；数据服务器必须拒绝此类I/O。
"SHOULD": in Section 13.8
“应该”：在第13.8节中
The iomode need not be checked by the data servers when clients perform I/O. However, the data servers SHOULD still validate that the client holds a valid layout and return an error if the client does not.
当客户端执行I/O时，数据服务器无需检查iomode。但是，数据服务器仍应验证客户端是否拥有有效布局，如果客户端没有，则返回错误。
It should be noted that it is just these layout-specific checks that are optional, not the normal file access semantics. The storage devices MUST make all of the required access checks on each READ or WRITE I/O as determined by the NFSv4.1 protocol. If the metadata server would deny a READ or WRITE operation on a file due to its ACL, mode attribute, open access mode, open deny mode, mandatory byte-range locking state, or any other attributes and state, the storage device MUST also deny the READ or WRITE operation. Also, while the NFSv4.1 protocol does not mandate export access checks based on the client's IP address, if the metadata server implements such a policy, then that counts as such state as outlined above.
应该注意的是，只是这些特定于布局的检查是可选的，而不是正常的文件访问语义。存储设备必须根据NFSv4.1协议对每个读或写I/O进行所有必需的访问检查。如果元数据服务器由于其ACL、模式属性、开放访问模式、开放拒绝模式、强制字节范围锁定状态或任何其他属性和状态而拒绝对文件执行读或写操作，则存储设备还必须拒绝读或写操作。此外，尽管NFSv4.1协议不强制基于客户端的IP地址进行导出访问检查，但如果元数据服务器实现了这样的策略，则该策略将被视为如上所述的状态。
The data filehandle provided by the PUTFH operation to the data server provides sufficient context to enable the data server to ensure that the client has a valid layout for the I/O being performed for the subsequent READ or WRITE operation in the compound.
PUTFH操作向数据服务器提供的数据文件句柄提供了足够的上下文，使数据服务器能够确保客户端具有有效的I/O布局，以便为化合物中的后续读取或写入操作执行I/O。
Finally, the data server can check the stateid presented in the READ or WRITE operation to see if that stateid has been rejected by the metadata server; if so, the data server will cause the I/O to be fenced. Whilst it might just be the open owner or lock owner on that client being fenced, the client should take the NFS4ERR_BAD_STATEID error code to mean it has been fenced from the file and contact the metadata server.
最后，数据服务器可以检查读或写操作中显示的stateid，以查看该stateid是否已被元数据服务器拒绝；如果是这样，数据服务器将导致I/O被隔离。虽然它可能只是被隔离的客户机上的开放所有者或锁所有者，但客户机应使用NFS4ERR_BAD_STATEID错误代码表示它已从文件中隔离，并与元数据服务器联系。
With the block layout type, the storage devices are generally not able to enforce file-based security. Typically, storage area network (SAN) disk arrays and SAN protocols provide coarse-grained access control mechanisms (e.g., Logical Unit Number (LUN) mapping and/or masking), with a target granularity of disks rather than individual blocks and a source granularity of individual hosts rather than of
对于块布局类型，存储设备通常无法实施基于文件的安全性。通常，存储区域网络（SAN）磁盘阵列和SAN协议提供粗粒度的访问控制机制（例如，逻辑单元号（LUN）映射和/或掩蔽），目标粒度为磁盘，而不是单个块，源粒度为单个主机，而不是主机
users or owners. Access to block storage is logically at a lower layer of the I/O stack than NFSv4. Since NFSv4 security is not directly applicable to protocols that access such storage directly, Section 2.1 of [RFC5663] specifies that:
用户或所有者。与NFSv4相比，对块存储的访问逻辑上位于I/O堆栈的较低层。由于NFSv4安全性不直接适用于直接访问此类存储的协议，[RFC5663]第2.1节规定：
in environments where pNFS clients cannot be trusted to enforce such policies, pNFS block layout types SHOULD NOT be used.
在无法信任pNFS客户端执行此类策略的环境中，不应使用pNFS块布局类型。
Due to these granularity issues, the security burden has been shifted from the storage devices to the client. Those deploying implementations of this layout type need to be sure that the client implementation can be trusted. This is not a new sort of requirement in the context of SAN protocols. In such environments, the client is expected to provide block-based protection.
由于这些粒度问题，安全负担已从存储设备转移到客户端。部署这种布局类型的实现的人需要确保客户端实现是可信的。在SAN协议的上下文中，这不是一种新的需求。在这种环境中，客户机需要提供基于块的保护。
This shift of the burden also extends to locks and layouts. The storage devices are not able to enforce any of these, and the burden is pushed to the client to make the appropriate checks before sending I/O to the storage devices. For example, the server may use a layout iomode only allowing reading to enforce a mandatory read-only lock. In such cases, the client has to support that use by not sending WRITEs to the storage devices. The fundamental issue here is that the storage device is treated by this layout type in the same fashion as a local disk device. Once the client has access to the storage device, it is able to perform both READ and WRITE I/O to the entire storage device. The byte ranges in the layout, any locks, the layout iomode, etc., can only be enforced by the client. Therefore, the client is required to provide that enforcement.
这种负担的转移也延伸到锁和布局上。存储设备无法强制执行其中任何一项，在向存储设备发送I/O之前，将负担推给客户端进行适当的检查。例如，服务器可能使用仅允许读取的布局iomode来强制执行强制只读锁定。在这种情况下，客户端必须通过不向存储设备发送写操作来支持这种使用。这里的基本问题是，这种布局类型以与本地磁盘设备相同的方式处理存储设备。一旦客户机能够访问存储设备，它就能够对整个存储设备执行读写I/O操作。布局中的字节范围、任何锁、布局iomode等只能由客户端强制执行。因此，要求客户提供强制执行。
In the context of fencing off of the client upon revocation of a layout, these limitations come into play again, i.e., the granularity of the fencing can only be at the level of the host and logical unit. Thus, if one of a client's layouts is revoked by the server, it will effectively revoke all of the client's layouts for files located on the storage units comprising the logical volume. This may extend to the client's layouts for files in other file systems. Clients need to be prepared for such revocations and reacquire layouts as needed.
在撤销布局后将客户端隔离的情况下，这些限制再次发挥作用，即隔离的粒度只能在主机和逻辑单元级别。因此，如果服务器撤销了客户机的一个布局，它将有效地撤销位于构成逻辑卷的存储单元上的文件的所有客户机布局。这可能会扩展到其他文件系统中的文件的客户端布局。客户需要为这种撤销做好准备，并根据需要重新获取布局。
With the object layout type, security checks occur during the allocation of the layout. The client will typically ask for layouts covering all of the file and may do so for either READ or READ/WRITE. This enables it to do subsequent I/O operations without the need to obtain layouts for specific byte ranges. At that time, the metadata server should verify permissions against the layout iomode, the file mode bits or ACLs, etc. As the client may be acting for multiple
对于对象布局类型，在布局分配期间进行安全检查。客户机通常会要求提供覆盖所有文件的布局，也可能要求提供读或读/写布局。这使它能够执行后续的I/O操作，而无需获取特定字节范围的布局。此时，元数据服务器应根据布局iomode、文件模式位或ACL等验证权限，因为客户端可能会执行多个操作
local users, it MUST authenticate and authorize the user by issuing respective OPEN and ACCESS calls to the metadata server, similar to having NFSv4 data delegations.
对于本地用户，它必须通过向元数据服务器发出相应的打开和访问调用来对用户进行身份验证和授权，类似于NFSv4数据委托。
Upon successful authorization, the client receives within the layout a set of object capabilities allowing it I/O access to the specified objects corresponding to the requested iomode. These capabilities are used to enforce access control and locking semantics at the storage devices. Whenever one of the following occurs on the metadata server, then the metadata server MUST change the capability version attribute on all objects comprising the file in order to invalidate any outstanding capabilities before committing to one of these changes:
成功授权后，客户端将在布局内接收一组对象功能，允许其对与请求的iomode对应的指定对象进行I/O访问。这些功能用于在存储设备上实施访问控制和锁定语义。每当元数据服务器上发生以下情况之一时，元数据服务器必须更改构成文件的所有对象的“功能版本”属性，以便在提交这些更改之前使任何未完成的功能无效：
o the permissions on the object change,
o 对象上的权限已更改，
o a conflicting mandatory byte-range lock is granted, or
o 授予冲突的强制字节范围锁，或
o a layout is revoked and reassigned to another client.
o 布局被撤销并重新分配给另一个客户端。
When the metadata server wishes to fence off a client to a particular object, then it can use the above approach to invalidate the capability attribute on the given object. The client can be informed via the storage device that the capability has been rejected and is allowed to fetch a refreshed set of capabilities, i.e., reacquire the layout.
当元数据服务器希望将客户机隔离到特定对象时，它可以使用上述方法使给定对象上的capability属性无效。可以通过存储设备通知客户机该功能已被拒绝，并允许客户机获取一组刷新的功能，即重新获取布局。
In the three original layout types, the burden of enforcing the security of NFSv4.1 can fall to either the storage devices (files), the client (blocks), or the metadata server (objects). Such choices are conditioned by the native capabilities of the storage devices -- if a control protocol can be implemented, then the burden can be shifted primarily to the storage devices.
在三种原始布局类型中，强制NFSv4.1安全性的负担可能落在存储设备（文件）、客户机（块）或元数据服务器（对象）上。这些选择取决于存储设备的本机功能——如果可以实现控制协议，那么负担可以主要转移到存储设备上。
In the context of this document, we treat the control protocol as a set of requirements. As new layout types are published, the defining documents MUST address:
在本文档中，我们将控制协议视为一组需求。随着新布局类型的发布，定义文档必须解决：
(1) The fencing of clients after a layout is revoked.
(1) 撤销布局后客户端的防护。
(2) The security implications of the native capabilities of the storage devices with respect to the requirements of the NFSv4.1 security model.
(2) 存储设备的本机功能对NFSv4.1安全模型要求的安全影响。
In addition, these defining documents need to make clear how other semantic requirements of NFSv4.1 (e.g., locking) are met in the context of the proposed layout type.
此外，这些定义文档需要明确在拟定布局类型的上下文中如何满足NFSv4.1的其他语义要求（例如锁定）。
This section does not deal directly with security considerations for existing or new layout types. Instead, it provides a general framework for understating security-related issues within the pNFS framework. Specific security considerations will be addressed in the Security Considerations sections of documents specifying layout types. For example, in Section 3 of [RFC5663], the lack of finer-than-physical disk access control necessitates that the client is delegated the responsibility to enforce the access provided to them in the layout extent that they were granted by the metadata server.
本节不直接讨论现有或新布局类型的安全注意事项。相反，它为在pNFS框架内低估安全相关问题提供了一个通用框架。具体的安全注意事项将在指定布局类型的文件的安全注意事项部分中说明。例如，在[RFC5663]的第3节中，由于缺少比物理磁盘更精细的访问控制，客户机必须被授权负责在元数据服务器授予的布局范围内强制执行向其提供的访问。
The layout type specification must ensure that only data access consistent with the NFSV4.1 security model is allowed. It may do this directly, by providing that appropriate checks be performed at the time each access is performed. It may do it indirectly by allowing the client or the storage device to be responsible for making the appropriate checks. In the latter case, I/O access rights are reflected in layouts, and the layout type must provide a way to prevent inappropriate access due to permissions changes between the time a layout is granted and the time the access is performed.
布局类型规范必须确保仅允许符合NFSV4.1安全模型的数据访问。它可以通过在执行每次访问时执行适当的检查来直接执行此操作。它可以通过允许客户机或存储设备负责进行适当的检查来间接实现这一点。在后一种情况下，I/O访问权限反映在布局中，布局类型必须提供一种方法，以防止由于授予布局和执行访问之间的权限更改而导致不适当的访问。
The metadata server MUST be able to fence off a client's access to the data file on a storage device. When it revokes the layout, the client's access MUST be terminated at the storage devices. The client has a subsequent opportunity to reacquire the layout and perform the security check in the context of the newly current access permissions.
元数据服务器必须能够阻止客户端访问存储设备上的数据文件。当它撤销布局时，客户端的访问必须在存储设备上终止。客户端随后有机会重新获取布局，并在新的当前访问权限上下文中执行安全检查。
This document has no IANA actions.
本文档没有IANA操作。
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.
[RFC2119]Bradner，S.，“RFC中用于表示需求水平的关键词”，BCP 14，RFC 2119，DOI 10.17487/RFC2119，1997年3月<https://www.rfc-editor.org/info/rfc2119>.
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, <https://www.rfc-editor.org/info/rfc5661>.
[RFC5661]Shepler，S.，Ed.，Eisler，M.，Ed.，和D.Noveck，Ed.，“网络文件系统（NFS）版本4次要版本1协议”，RFC 5661，DOI 10.17487/RFC5661，2010年1月<https://www.rfc-editor.org/info/rfc5661>.
[RFC5663] Black, D., Fridella, S., and J. Glasgow, "Parallel NFS (pNFS) Block/Volume Layout", RFC 5663, DOI 10.17487/RFC5663, January 2010, <https://www.rfc-editor.org/info/rfc5663>.
[RFC5663]Black，D.，Fridella，S.，和J.Glasgow，“并行NFS（pNFS）块/卷布局”，RFC 5663，DOI 10.17487/RFC5663，2010年1月<https://www.rfc-editor.org/info/rfc5663>.
[RFC5664] Halevy, B., Welch, B., and J. Zelenka, "Object-Based Parallel NFS (pNFS) Operations", RFC 5664, DOI 10.17487/RFC5664, January 2010, <https://www.rfc-editor.org/info/rfc5664>.
[RFC5664]Halevy，B.，Welch，B.，和J.Zelenka，“基于对象的并行NFS（pNFS）操作”，RFC 5664，DOI 10.17487/RFC5664，2010年1月<https://www.rfc-editor.org/info/rfc5664>.
[RFC8154] Hellwig, C., "Parallel NFS (pNFS) Small Computer System Interface (SCSI) Layout", RFC 8154, DOI 10.17487/RFC8154, May 2017, <https://www.rfc-editor.org/info/rfc8154>.
[RFC8154]Hellwig，C.，“并行NFS（pNFS）小型计算机系统接口（SCSI）布局”，RFC8154，DOI 10.17487/RFC8154，2017年5月<https://www.rfc-editor.org/info/rfc8154>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8174]Leiba，B.，“RFC 2119关键词中大写与小写的歧义”，BCP 14，RFC 8174，DOI 10.17487/RFC8174，2017年5月<https://www.rfc-editor.org/info/rfc8174>.
[Lustre] Faibish, S., Cote, D., and P. Tao, "Parallel NFS (pNFS) Lustre Layout Operations", Work in Progress, draft-faibish-nfsv4-pnfs-lustre-layout-07, May 2014.
[Lustre]Faibish，S.，Cote，D.，和P.Tao，“并行NFS（pNFS）Lustre布局操作”，正在进行的工作，草稿-Faibish-nfsv4-pNFS-Lustre-Layout-072014年5月。
[RFC8435] Halevy, B. and T. Haynes, "Parallel NFS (pNFS) Flexible File Layout", RFC 8435, DOI 10.17487/RFC8435, August 2018, <https://www.rfc-editor.org/info/rfc8435>.
[RFC8435]Halevy，B.和T.Haynes，“并行NFS（pNFS）灵活文件布局”，RFC 8435，DOI 10.17487/RFC8435，2018年8月<https://www.rfc-editor.org/info/rfc8435>.
Acknowledgments
致谢
Dave Noveck provided an early review that sharpened the clarity of the definitions. He also provided a more comprehensive review of the document.
戴夫·诺维克（Dave Noveck）提供了一份早期评论，使定义更加清晰。他还对该文件进行了更全面的审查。
Both Chuck Lever and Christoph Helwig provided insightful comments during the working group last call.
Chuck Lever和Christoph Helwig在工作组最后一次电话会议上都发表了富有洞察力的评论。
Author's Address
作者地址
Thomas Haynes Hammerspace 4300 El Camino Real Ste 105 Los Altos, CA 94022 United States of America
Thomas Haynes Hammerspace 4300 El Camino Real Ste 105 Los Altos，加利福尼亚州，美利坚合众国94022
Email: loghyr@gmail.com
Email: loghyr@gmail.com