Internet Engineering Task Force (IETF)                    O. Bonaventure
Request for Comments: 8041                                     UCLouvain
Category: Informational                                        C. Paasch
ISSN: 2070-1721                                              Apple, Inc.
                                                                G. Detal
                                                            January 2017
Internet Engineering Task Force (IETF)                    O. Bonaventure
Request for Comments: 8041                                     UCLouvain
Category: Informational                                        C. Paasch
ISSN: 2070-1721                                              Apple, Inc.
                                                                G. Detal
                                                            January 2017

Use Cases and Operational Experience with Multipath TCP




This document discusses both use cases and operational experience with Multipath TCP (MPTCP) in real networks. It lists several prominent use cases where Multipath TCP has been considered and is being used. It also gives insight to some heuristics and decisions that have helped to realize these use cases and suggests possible improvements.


Status of This Memo


This document is not an Internet Standards Track specification; it is published for informational purposes.


This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 7841.

本文件是互联网工程任务组(IETF)的产品。它代表了IETF社区的共识。它已经接受了公众审查,并已被互联网工程指导小组(IESG)批准出版。并非IESG批准的所有文件都适用于任何级别的互联网标准;见RFC 7841第2节。

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at


Copyright Notice


Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.

版权所有(c)2017 IETF信托基金和确定为文件作者的人员。版权所有。

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents ( in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

本文件受BCP 78和IETF信托有关IETF文件的法律规定的约束(自本文件出版之日起生效。请仔细阅读这些文件,因为它们描述了您对本文件的权利和限制。从本文件中提取的代码组件必须包括信托法律条款第4.e节中所述的简化BSD许可证文本,并提供简化BSD许可证中所述的无担保。

Table of Contents


   1. Introduction ....................................................3
   2. Use Cases .......................................................4
      2.1. Datacenters ................................................4
      2.2. Cellular/WiFi Offload ......................................5
      2.3. Multipath TCP Proxies ......................................8
   3. Operational Experience ..........................................9
      3.1. Middlebox Interference .....................................9
      3.2. Congestion Control ........................................11
      3.3. Subflow Management ........................................12
      3.4. Implemented Subflow Managers ..............................13
      3.5. Subflow Destination Port ..................................15
      3.6. Closing Subflows ..........................................16
      3.7. Packet Schedulers .........................................17
      3.8. Segment Size Selection ....................................18
      3.9. Interactions with the Domain Name System ..................19
      3.10. Captive Portals ..........................................20
      3.11. Stateless Webservers .....................................20
      3.12. Load-Balanced Server Farms ...............................21
   4. Security Considerations ........................................21
   5. References .....................................................23
      5.1. Normative References ......................................23
      5.2. Informative References ....................................23
   Acknowledgements ..................................................30
   Authors' Addresses ................................................30
   1. Introduction ....................................................3
   2. Use Cases .......................................................4
      2.1. Datacenters ................................................4
      2.2. Cellular/WiFi Offload ......................................5
      2.3. Multipath TCP Proxies ......................................8
   3. Operational Experience ..........................................9
      3.1. Middlebox Interference .....................................9
      3.2. Congestion Control ........................................11
      3.3. Subflow Management ........................................12
      3.4. Implemented Subflow Managers ..............................13
      3.5. Subflow Destination Port ..................................15
      3.6. Closing Subflows ..........................................16
      3.7. Packet Schedulers .........................................17
      3.8. Segment Size Selection ....................................18
      3.9. Interactions with the Domain Name System ..................19
      3.10. Captive Portals ..........................................20
      3.11. Stateless Webservers .....................................20
      3.12. Load-Balanced Server Farms ...............................21
   4. Security Considerations ........................................21
   5. References .....................................................23
      5.1. Normative References ......................................23
      5.2. Informative References ....................................23
   Acknowledgements ..................................................30
   Authors' Addresses ................................................30
1. Introduction
1. 介绍

Multipath TCP was specified in [RFC6824] and five independent implementations have been developed. As of November 2016, Multipath TCP has been or is being implemented on the following platforms:


o Linux kernel [MultipathTCP-Linux]

o Linux内核[多路径TCP Linux]

o Apple iOS and macOS

o 苹果iOS和macOS

o Citrix load balancers

o Citrix负载平衡器



o Oracle Solaris

o Oracle Solaris

The first three implementations are known to interoperate. Three of these implementations are open source (Linux kernel, FreeBSD and Apple's iOS and macOS). Apple's implementation is widely deployed.


Since the publication of [RFC6824] as an Experimental RFC, experience has been gathered by various network researchers and users about the operational issues that arise when Multipath TCP is used in today's Internet.


When the MPTCP working group was created, several use cases for Multipath TCP were identified [RFC6182]. Since then, other use cases have been proposed and some have been tested and even deployed. We describe these use cases in Section 2.


Section 3 focuses on the operational experience with Multipath TCP. Most of this experience comes from the utilization of the Multipath TCP implementation in the Linux kernel [MultipathTCP-Linux]. This open-source implementation has been downloaded and implemented by thousands of users all over the world. Many of these users have provided direct or indirect feedback by writing documents (scientific articles or blog messages) or posting to the mptcp-dev mailing list (see This Multipath TCP implementation is actively maintained and continuously improved. It is used on various types of hosts, ranging from smartphones or embedded routers to high-end servers.

第3节重点介绍多路径TCP的操作经验。这些经验大多来自Linux内核[MultipathTCP Linux]中多路径TCP实现的使用。这个开源实现已经被全世界成千上万的用户下载和实现。其中许多用户通过撰写文档(科学文章或博客消息)或发布到mptcp开发人员邮件列表(参见 这种多路径TCP实现得到积极维护和不断改进。它用于各种类型的主机,从智能手机或嵌入式路由器到高端服务器。

The Multipath TCP implementation in the Linux kernel is not, by far, the most widespread deployment of Multipath TCP. Since September 2013, Multipath TCP is also supported on smartphones and tablets beginning with iOS7 [IETFJ]. There are likely hundreds of millions of MPTCP-enabled devices. This Multipath TCP implementation is


currently only used to support the Siri voice recognition/control application. Some lessons learned from this deployment are described in [IETFJ].


Section 3 is organized as follows. Supporting the middleboxes was one of the difficult issues in designing the Multipath TCP protocol. We explain in Section 3.1 which types of middleboxes the Linux Kernel implementation of Multipath TCP supports and how it reacts upon encountering these. Section 3.2 summarizes the MPTCP-specific congestion controls that have been implemented. Sections 3.3 to 3.7 discuss heuristics and issues with respect to subflow management as well as the scheduling across the subflows. Section 3.8 explains some problems that occurred with subflows having different Maximum Segment Size (MSS) values. Section 3.9 presents issues with respect to content delivery networks and suggests a solution to this issue. Finally, Section 3.10 documents an issue with captive portals where MPTCP will behave suboptimally.


2. Use Cases
2. 用例

Multipath TCP has been tested in several use cases. There is already an abundant amount of scientific literature on Multipath TCP [MPTCPBIB]. Several of the papers published in the scientific literature have identified possible improvements that are worth being discussed here.


2.1. Datacenters
2.1. 数据中心

A first, although initially unexpected, documented use case for Multipath TCP has been in datacenters [HotNets][SIGCOMM11]. Today's datacenters are designed to provide several paths between single-homed servers. The multiplicity of these paths comes from the utilization of Equal-Cost Multipath (ECMP) and other load-balancing techniques inside the datacenter. Most of the deployed load-balancing techniques in datacenters rely on hashes computed over the five tuple. Thus, all packets from the same TCP connection follow the same path: so they are not reordered. The results in [HotNets] demonstrate by simulations that Multipath TCP can achieve a better utilization of the available network by using multiple subflows for each Multipath TCP session. Although [RFC6182] assumes that at least one of the communicating hosts has several IP addresses, [HotNets] demonstrates that Multipath TCP is beneficial when both hosts are single-homed. This idea is analyzed in more details in [SIGCOMM11], where the Multipath TCP implementation in the Linux kernel is modified to be able to use several subflows from the same IP address. Measurements in a public datacenter show the quantitative benefits of Multipath TCP [SIGCOMM11] in this environment.


Although ECMP is widely used inside datacenters, this is not the only environment where there are different paths between a pair of hosts. ECMP and other load-balancing techniques such as Link Aggregation Groups (LAGs) are widely used in today's networks; having multiple paths between a pair of single-homed hosts is becoming the norm instead of the exception. Although these multiple paths often have the same cost (from an IGP metrics viewpoint), they do not necessarily have the same performance. For example, [IMC13c] reports the results of a long measurement study showing that load-balanced Internet paths between that same pair of hosts can have huge delay differences.


2.2. Cellular/WiFi Offload
2.2. 蜂窝/WiFi卸载

A second use case that has been explored by several network researchers is the cellular/WiFi offload use case. Smartphones or other mobile devices equipped with two wireless interfaces are a very common use case for Multipath TCP. In September 2015, this is also the largest deployment of MPTCP-enabled devices [IETFJ]. It has been briefly discussed during IETF 88 [IETF88], but there is no published paper or report that analyses this deployment. For this reason, we only discuss published papers that have mainly used the Multipath TCP implementation in the Linux kernel for their experiments.

几位网络研究人员探讨的第二个用例是蜂窝/WiFi卸载用例。配备两个无线接口的智能手机或其他移动设备是多路径TCP的一个非常常见的用例。2015年9月,这也是最大规模的MPTCP设备部署[IETFJ]。在IETF 88[IETF88]中对其进行了简要讨论,但没有发表过分析此部署的论文或报告。因此,我们只讨论在Linux内核中主要使用多路径TCP实现进行实验的已发表论文。

The performance of Multipath TCP in wireless networks was briefly evaluated in [NSDI12]. One experiment analyzes the performance of Multipath TCP on a client with two wireless interfaces. This evaluation shows that when the receive window is large, Multipath TCP can efficiently use the two available links. However, if the window becomes smaller, then packets sent on a slow path can block the transmission of packets on a faster path. In some cases, the performance of Multipath TCP over two paths can become lower than the performance of regular TCP over the best performing path. Two heuristics, reinjection and penalization, are proposed in [NSDI12] to solve this identified performance problem. These two heuristics have since been used in the Multipath TCP implementation in the Linux kernel. [CONEXT13] explored the problem in more detail and revealed some other scenarios where Multipath TCP can have difficulties in efficiently pooling the available paths. Improvements to the Multipath TCP implementation in the Linux kernel are proposed in [CONEXT13] to cope with some of these problems.


The first experimental analysis of Multipath TCP in a public wireless environment was presented in [Cellnet12]. These measurements explore the ability of Multipath TCP to use two wireless networks (real WiFi and 3G networks). Three modes of operation are compared. The first mode of operation is the simultaneous use of the two wireless networks. In this mode, Multipath TCP pools the available resources


and uses both wireless interfaces. This mode provides fast handover from WiFi to cellular or the opposite when the user moves. Measurements presented in [CACM14] show that the handover from one wireless network to another is not an abrupt process. When a host moves, there are regions where the quality of one of the wireless networks is weaker than the other, but the host considers this wireless network to still be up. When a mobile host enters such regions, its ability to send packets over another wireless network is important to ensure a smooth handover. This is clearly illustrated from the packet trace discussed in [CACM14].


Many cellular networks use volume-based pricing; users often prefer to use unmetered WiFi networks when available instead of metered cellular networks. [Cellnet12] implements support for the MP_PRIO option to explore two other modes of operation.


In the backup mode, Multipath TCP opens a TCP subflow over each interface, but the cellular interface is configured in backup mode. This implies that data flows only over the WiFi interface when both interfaces are considered to be active. If the WiFi interface fails, then the traffic switches quickly to the cellular interface, ensuring a smooth handover from the user's viewpoint [Cellnet12]. The cost of this approach is that the WiFi and cellular interfaces are likely to remain active all the time since all subflows are established over the two interfaces.


The single-path mode is slightly different. This mode benefits from the break-before-make capability of Multipath TCP. When an MPTCP session is established, a subflow is created over the WiFi interface. No packet is sent over the cellular interface as long as the WiFi interface remains up [Cellnet12]. This implies that the cellular interface can remain idle and battery capacity is preserved. When the WiFi interface fails, a new subflow is established over the cellular interface in order to preserve the established Multipath TCP sessions. Compared to the backup mode described earlier, measurements reported in [Cellnet12] indicate that this mode of operation is characterized by a throughput drop while the cellular interface is brought up and the subflows are reestablished.


From a protocol viewpoint, [Cellnet12] discusses the problem posed by the unreliability of the REMOVE_ADDR option and proposes a small protocol extension to allow hosts to reliably exchange this option. It would be useful to analyze packet traces to understand whether the unreliability of the REMOVE_ADDR option poses an operational problem in real deployments.


Another study of the performance of Multipath TCP in wireless networks was reported in [IMC13b]. This study uses laptops connected to various cellular ISPs and WiFi hotspots. It compares various file transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms 2-path MPTCP, especially for larger files. However, for three congestion-control algorithms (LIA, OLIA, and Reno -- see Section 3.2), there is no significant performance difference for file sizes smaller than 4 MB.

[IMC13b]中报告了无线网络中多径TCP性能的另一项研究。这项研究使用笔记本电脑连接到各种手机ISP和WiFi热点。它比较了各种文件传输场景。[IMC13b]观察到4路径MPTCP优于2路径MPTCP,尤其是对于较大的文件。但是,对于三种拥塞控制算法(LIA、OLIA和Reno——请参见第3.2节),对于小于4 MB的文件大小,没有明显的性能差异。

A different study of the performance of Multipath TCP with two wireless networks is presented in [INFOCOM14]. In this study the two networks had different qualities: a good network and a lossy network. When using two paths with different packet-loss ratios, the Multipath TCP congestion-control scheme moves traffic away from the lossy link that is considered to be congested. However, [INFOCOM14] documents an interesting scenario that is summarized hereafter.


   client ----------- path1 -------- server
     |                                  |
     +--------------- path2 ------------+
   client ----------- path1 -------- server
     |                                  |
     +--------------- path2 ------------+

Figure 1: Simple network topology


Initially, the two paths in Figure 1 have the same quality and Multipath TCP distributes the load over both of them. During the transfer, the path2 becomes lossy, e.g., because the client moves. Multipath TCP detects the packet losses and they are retransmitted over path1. This enables the data transfer to continue over this path. However, the subflow over path2 is still up and transmits one packet from time to time. Although the N packets have been acknowledged over the first subflow (at the MPTCP level), they have not been acknowledged at the TCP level over the second subflow. To preserve the continuity of the sequence numbers over the second subflow, TCP will continue to retransmit these segments until either they are acknowledged or the maximum number of retransmissions is reached. This behavior is clearly inefficient and may lead to blocking since the second subflow will consume window space to be able to retransmit these packets. [INFOCOM14] proposes a new Multipath TCP option to solve this problem. In practice, a new TCP option is probably not required. When the client detects that the data transmitted over the second subflow has been acknowledged over the first subflow, it could decide to terminate the second subflow by sending a RST segment. If the interface associated to this subflow is still up, a new subflow could be immediately reestablished. It would then be immediately usable to send new data and would not be forced to first retransmit the previously transmitted data. As of this writing, this dynamic management of the subflows is not yet implemented in the Multipath TCP implementation in the Linux kernel.


Some studies have started to analyze the performance of Multipath TCP on smartphones with real applications. In contrast with the bulk transfers that are used by many publications, many deployed applications do not exchange huge amounts of data and mainly use small connections. [COMMAG2016] proposes a software testing framework that allows to automate Android applications to study their interactions with Multipath TCP. [PAM2016] analyses a one-month packet trace of all the packets exchanged by a dozen of smartphones utilized by regular users. This analysis reveals that short connections are important on smartphones and that the main benefit of using Multipath TCP on smartphones is the ability to perform seamless handovers between different wireless networks. Long connections benefit from these handovers.


2.3. Multipath TCP Proxies
2.3. 多路径TCP代理

As Multipath TCP is not yet widely deployed on both clients and servers, several deployments have used various forms of proxies. Two families of solutions are currently being used or tested.


A first use case is when an MPTCP-enabled client wants to use several interfaces to reach a regular TCP server. A typical use case is a smartphone that needs to use both its WiFi and its cellular interface to transfer data. Several types of proxies are possible for this use case. An HTTP proxy deployed on a MPTCP-capable server would enable the smartphone to use Multipath TCP to access regular web servers. Obviously, this solution only works for applications that rely on HTTP. Another possibility is to use a proxy that can convert any Multipath TCP connection into a regular TCP connection. MPTCP-specific proxies have been proposed [HotMiddlebox13b] [HAMPEL].


Another possibility leverages the SOCKS protocol [RFC1928]. SOCKS is often used in enterprise networks to allow clients to reach external servers. For this, the client opens a TCP connection to the SOCKS server that relays it to the final destination. If both the client and the SOCKS server use Multipath TCP, but not the final destination, then Multipath TCP can still be used on the path between the clients and the SOCKS server. At IETF 93, Korea Telecom announced that they have deployed (in June 2015) a commercial service that uses Multipath TCP on smartphones. These smartphones access regular TCP servers through a SOCKS proxy. This enables them to achieve throughputs of up to 850 Mbps [KT].

另一种可能性是利用SOCKS协议[RFC1928]。SOCKS通常用于企业网络,以允许客户端访问外部服务器。为此,客户机打开一个到SOCKS服务器的TCP连接,该服务器将其中继到最终目的地。如果客户端和SOCKS服务器都使用多路径TCP,但不是最终目的地,则多路径TCP仍可在客户端和SOCKS服务器之间的路径上使用。在IETF 93上,韩国电信宣布(2015年6月)在智能手机上部署了一项使用多路径TCP的商业服务。这些智能手机通过SOCKS代理访问常规TCP服务器。这使它们能够实现高达850 Mbps[KT]的吞吐量。

Measurements performed with Android smartphones [Mobicom15] show that popular applications work correctly through a SOCKS proxy and MPTCP-enabled smartphones. Thanks to Multipath TCP, long-lived connections can be spread over the two available interfaces. However, for short-lived connections, most of the data is sent over the initial subflow that is created over the interface corresponding to the default route and the second subflow is almost not used [PAM2016].


A second use case is when Multipath TCP is used by middleboxes, typically inside access networks. Various network operators are discussing and evaluating solutions for hybrid access networks [TR-348]. Such networks arise when a network operator controls two different access network technologies, e.g., wired and cellular, and wants to combine them to improve the bandwidth offered to the end users [HYA-ARCH]. Several solutions are currently investigated for such networks [TR-348]. Figure 2 shows the organization of such a network. When a client creates a normal TCP connection, it is intercepted by the Hybrid CPE (HPCE) that converts it in a Multipath TCP connection so that it can use the available access networks (DSL and LTE in the example). The Hybrid Access Gateway (HAG) does the opposite to ensure that the regular server sees a normal TCP connection. Some of the solutions currently discussed for hybrid networks use Multipath TCP on the HCPE and the HAG. Other solutions rely on tunnels between the HCPE and the HAG [GRE-NOTIFY].


   client --- HCPE ------ DSL ------- HAG --- internet --- server
               |                       |
               +------- LTE -----------+
   client --- HCPE ------ DSL ------- HAG --- internet --- server
               |                       |
               +------- LTE -----------+

Figure 2: Hybrid Access Network


3. Operational Experience
3. 操作经验
3.1. Middlebox Interference
3.1. 中间箱干扰

The interference caused by various types of middleboxes has been an important concern during the design of the Multipath TCP protocol. Three studies on the interactions between Multipath TCP and middleboxes are worth discussing.


The first analysis appears in [IMC11]. This paper was the main motivation for Multipath TCP incorporating various techniques to cope with middlebox interference. More specifically, Multipath TCP has been designed to cope with middleboxes that:


o change source or destination addresses

o 更改源地址或目标地址

o change source or destination port numbers

o 更改源或目标端口号

o change TCP sequence numbers

o 更改TCP序列号

o split or coalesce segments

o 分割或合并段

o remove TCP options

o 删除TCP选项

o modify the payload of TCP segments

o 修改TCP段的有效负载

These middlebox interferences have all been included in the MBtest suite [MBTest]. This test suite is used in [HotMiddlebox13] to verify the reaction of the Multipath TCP implementation in the Linux kernel [MultipathTCP-Linux] when faced with middlebox interference. The test environment used for this evaluation is a dual-homed client connected to a single-homed server. The middlebox behavior can be activated on any of the paths. The main results of this analysis are:

这些中间箱干扰都包含在MBtest套件[MBtest]中。此测试套件在[HotMiddlebox13]中用于验证Linux内核[MultipathTCP Linux]中的多路径TCP实现在遇到中间盒干扰时的反应。用于此评估的测试环境是连接到单主机服务器的双主机客户端。可以在任何路径上激活中间盒行为。该分析的主要结果如下:

o the Multipath TCP implementation in the Linux kernel is not affected by a middlebox that performs NAT or modifies TCP sequence numbers

o Linux内核中的多路径TCP实现不受执行NAT或修改TCP序列号的中间盒的影响

o when a middlebox removes the MP_CAPABLE option from the initial SYN segment, the Multipath TCP implementation in the Linux kernel falls back correctly to regular TCP

o 当一个中间盒从初始SYN段中删除支持MP_的选项时,Linux内核中的多路径TCP实现会正确地返回到常规TCP

o when a middlebox removes the DSS option from all data segments, the Multipath TCP implementation in the Linux kernel falls back correctly to regular TCP

o 当中间盒从所有数据段中删除DSS选项时,Linux内核中的多路径TCP实现会正确地返回到常规TCP

o when a middlebox performs segment coalescing, the Multipath TCP implementation in the Linux kernel is still able to accurately extract the data corresponding to the indicated mapping

o 当中间盒执行段合并时,Linux内核中的多路径TCP实现仍然能够准确地提取与指定映射对应的数据

o when a middlebox performs segment splitting, the Multipath TCP implementation in the Linux kernel correctly reassembles the data corresponding to the indicated mapping. [HotMiddlebox13] shows, in Figure 4 in Section 3.3, a corner case with segment splitting that may lead to a desynchronization between the two hosts.

o 当中间盒执行段拆分时,Linux内核中的多路径TCP实现会正确地重新组装与指定映射对应的数据。[HotMiddlebox13]在第3.3节的图4中显示了一种可能导致两台主机之间失步的段分裂的情况。

The interactions between Multipath TCP and real deployed middleboxes are also analyzed in [HotMiddlebox13]; a particular scenario with the FTP Application Level Gateway running on a NAT is described.


Middlebox interference can also be detected by analyzing packet traces on MPTCP-enabled servers. A closer look at the packets received on the server [TMA2015] shows that among the 184,000 Multipath TCP connections, only 125 of them were falling back to regular TCP. These connections originated from 28 different client IP addresses. These include 91 HTTP connections and 34 FTP connections. The FTP interference is expected since Application Level Gateways used for FTP modify the TCP payload and the DSS Checksum detects these modifications. The HTTP interference appeared only on the direction from server to client and could have been caused by transparent proxies deployed in cellular or enterprise networks. A longer trace is discussed in [COMCOM2016] and similar conclusions about the middlebox interference are provided.


From an operational viewpoint, knowing that Multipath TCP can cope with various types of middlebox interference is important. However, there are situations where the network operators need to gather information about where a particular middlebox interference occurs. The tracebox software [tracebox] described in [IMC13a] is an extension of the popular traceroute software that enables network operators to check at which hop a particular field of the TCP header (including options) is modified. It has been used by several network operators to debug various middlebox interference problems. Experience with tracebox indicates that supporting the ICMP extension defined in [RFC1812] makes it easier to debug middlebox problems in IPv4 networks.


Users of the Multipath TCP implementation have reported some experience with middlebox interference. The strangest scenario has been a middlebox that accepts the Multipath TCP options in the SYN segment but later replaces Multipath TCP options with a TCP EOL option [StrangeMbox]. This causes Multipath TCP to perform a fallback to regular TCP without any impact on the application.

多路径TCP实现的用户报告了一些关于中间盒干扰的经验。最奇怪的场景是一个中间盒,它接受SYN段中的多路径TCP选项,但后来用TCP EOL选项[SquendMBOX]替换多路径TCP选项。这会导致多路径TCP在不影响应用程序的情况下执行常规TCP的回退。

3.2. Congestion Control
3.2. 拥塞控制

Congestion control has been an important challenge for Multipath TCP. The coupled congestion-control scheme defined in [RFC6356] in an adaptation of the NewReno algorithm. A detailed description of this coupled algorithm is provided in [NSDI11]. It is the default scheme in the Linux implementation of Multipath TCP, but Linux supports other schemes.


The second congestion-control scheme is OLIA [CONEXT12]. It is also an adaptation of the NewReno single path congestion-control scheme to support multiple paths. Simulations [CONEXT12] and measurements [CONEXT13] have shown that it provides some performance benefits compared to the default coupled congestion-control scheme.


The delay-based scheme proposed in [ICNP12] has also been ported to the Multipath TCP implementation in the Linux kernel. It has been evaluated by using simulations [ICNP12] and measurements [PaaschPhD].


BALIA, defined in [BALIA], provides a better balance between TCP friendliness, responsiveness, and window oscillation.


These different congestion-control schemes have been compared in several articles. [CONEXT13] and [PaaschPhD] compare these algorithms in an emulated environment. The evaluation showed that the delay-based congestion-control scheme is less able to efficiently use the available links than the three other schemes.


3.3. Subflow Management
3.3. 亚流管理

The multipath capability of Multipath TCP comes from the utilization of one subflow per path. The Multipath TCP architecture [RFC6182] and the protocol specification [RFC6824] define the basic usage of the subflows and the protocol mechanisms that are required to create and terminate them. However, there are no guidelines on how subflows are used during the lifetime of a Multipath TCP session. Most of the published experiments with Multipath TCP have been performed in controlled environments. Still, based on the experience running them and discussions on the mptcp-dev mailing list, interesting lessons have been learned about the management of these subflows.


From a subflow viewpoint, the Multipath TCP protocol is completely symmetrical. Both the clients and the server have the capability to create subflows. However, in practice, the existing Multipath TCP implementations have opted for a strategy where only the client creates new subflows. The main motivation for this strategy is that often the client resides behind a NAT or a firewall, preventing passive subflow openings on the client. Although there are environments such as datacenters where this problem does not occur, as of this writing, no precise requirement has emerged for allowing the server to create new subflows.


3.4. Implemented Subflow Managers
3.4. 实现的子流管理器

The Multipath TCP implementation in the Linux kernel includes several strategies to manage the subflows that compose a Multipath TCP session. The basic subflow manager is the full-mesh. As the name implies, it creates a full-mesh of subflows between the communicating hosts.


The most frequent use case for this subflow manager is a multihomed client connected to a single-homed server. In this case, one subflow is created for each interface on the client. The current implementation of the full-mesh subflow manager is static. The subflows are created immediately after the creation of the initial subflow. If one subflow fails during the lifetime of the Multipath TCP session (e.g., due to excessive retransmissions or the loss of the corresponding interface), it is not always reestablished. There is ongoing work to enhance the full-mesh path manager to deal with such events.


When the server is multihomed, using the full-mesh subflow manager may lead to a large number of subflows being established. For example, consider a dual-homed client connected to a server with three interfaces. In this case, even if the subflows are only created by the client, six subflows will be established. This may be excessive in some environments, in particular when the client and/or the server have a large number of interfaces. Implementations should limit the number of subflows that are used.


Creating subflows between multihomed clients and servers may sometimes lead to operational issues as observed by discussions on the mptcp-dev mailing list. In some cases, the network operators would like to have a better control on how the subflows are created by Multipath TCP [MPTCP-MAX-SUB]. This might require the definition of policy rules to control the operation of the subflow manager. The two scenarios below illustrate some of these requirements.


                host1 ----------  switch1 ----- host2
                  |                   |            |
                  +--------------  switch2 --------+
                host1 ----------  switch1 ----- host2
                  |                   |            |
                  +--------------  switch2 --------+

Figure 3: Simple Switched Network Topology


Consider the simple network topology shown in Figure 3. From an operational viewpoint, a network operator could want to create two subflows between the communicating hosts. From a bandwidth utilization viewpoint, the most natural paths are host1-switch1-host2 and host1-switch2-host2. However, a Multipath TCP implementation running on these two hosts may sometimes have difficulties to obtain this result.


To understand the difficulty, let us consider different allocation strategies for the IP addresses. A first strategy is to assign two subnets: subnetA (resp. subnetB) contains the IP addresses of host1's interface to switch1 (resp. switch2) and host2's interface to switch1 (resp. switch2). In this case, a Multipath TCP subflow manager should only create one subflow per subnet. To enforce the utilization of these paths, the network operator would have to specify a policy that prefers the subflows in the same subnet over subflows between addresses in different subnets. It should be noted that the policy should probably also specify how the subflow manager should react when an interface or subflow fails.


A second strategy is to use a single subnet for all IP addresses. In this case, it becomes more difficult to specify a policy that indicates which subflows should be established.


The second subflow manager that is currently supported by the Multipath TCP implementation in the Linux kernel is the ndiffport subflow manager. This manager was initially created to exploit the path diversity that exists between single-homed hosts due to the utilization of flow-based load-balancing techniques [SIGCOMM11]. This subflow manager creates N subflows between the same pair of IP addresses. The N subflows are created by the client and differ only in the source port selected by the client. It was not designed to be used on multihomed hosts.


A more flexible subflow manager has been proposed, implemented and evaluated in [CONEXT15]. This subflow manager exposes various kernel events to a user space daemon that decides when subflows need to be created and terminated based on various policies.


3.5. Subflow Destination Port
3.5. 子流目的端口

The Multipath TCP protocol relies on the token contained in the MP_JOIN option to associate a subflow to an existing Multipath TCP session. This implies that there is no restriction on the source address, destination address and source or destination ports used for the new subflow. The ability to use different source and destination addresses is key to support multihomed servers and clients. The ability to use different destination port numbers is worth discussing because it has operational implications.


For illustration, consider a dual-homed client that creates a second subflow to reach a single-homed server as illustrated in Figure 4.


           client ------- r1 --- internet --- server
               |                   |
           client ------- r1 --- internet --- server
               |                   |

Figure 4: Multihomed-Client Connected to Single-Homed Server


When the Multipath TCP implementation in the Linux kernel creates the second subflow, it uses the same destination port as the initial subflow. This choice is motivated by the fact that the server might be protected by a firewall and only accept TCP connections (including subflows) on the official port number. Using the same destination port for all subflows is also useful for operators that rely on the port numbers to track application usage in their network.


There have been suggestions from Multipath TCP users to modify the implementation to allow the client to use different destination ports to reach the server. This suggestion seems mainly motivated by traffic-shaping middleboxes that are used in some wireless networks. In networks where different shaping rates are associated with different destination port numbers, this could allow Multipath TCP to reach a higher performance. This behavior is valid according to the Multipath TCP specification [RFC6824]. An application could use an enhanced socket API [SOCKET] to behave in this way.


However, from an implementation point-of-view supporting different destination ports for the same Multipath TCP connection can cause some issues. A legacy implementation of a TCP stack creates a listening socket to react upon incoming SYN segments. The listening socket is handling the SYN segments that are sent on a specific port number. Demultiplexing incoming segments can thus be done solely by looking at the IP addresses and the port numbers. With Multipath TCP however, incoming SYN segments may have an MP_JOIN option with a different destination port. This means that all incoming segments


that did not match on an existing listening-socket or an already established socket must be parsed for an eventual MP_JOIN option. This imposes an additional cost on servers, previously not existent on legacy TCP implementations.


3.6. Closing Subflows
3.6. 关闭子流
                    client                       server
                       |                           |
   MPTCP: ESTABLISHED  |                           | MPTCP: ESTABLISHED
   Sub: ESTABLISHED    |                           | Sub: ESTABLISHED
                       |                           |
                       |         DATA_FIN          |
   MPTCP: CLOSE-WAIT   | <------------------------ | close()   (step 1)
   Sub: ESTABLISHED    |         DATA_ACK          |
                       | ------------------------> | MPTCP: FIN-WAIT-2
                       |                           | Sub: ESTABLISHED
                       |                           |
                       |  DATA_FIN + subflow-FIN   |
   close()/shutdown()  | ------------------------> | MPTCP: TIME-WAIT
   (step 2)            |        DATA_ACK           | Sub: CLOSE-WAIT
   MPTCP: CLOSED       | <------------------------ |
   Sub: FIN-WAIT-2     |                           |
                       |                           |
                       |        subflow-FIN        |
   MPTCP: CLOSED       | <------------------------ | subflow-close()
   Sub: TIME-WAIT      |        subflow-ACK        |
   (step 3)            | ------------------------> | MPTCP: TIME-WAIT
                       |                           | Sub: CLOSED
                       |                           |
                    client                       server
                       |                           |
   MPTCP: ESTABLISHED  |                           | MPTCP: ESTABLISHED
   Sub: ESTABLISHED    |                           | Sub: ESTABLISHED
                       |                           |
                       |         DATA_FIN          |
   MPTCP: CLOSE-WAIT   | <------------------------ | close()   (step 1)
   Sub: ESTABLISHED    |         DATA_ACK          |
                       | ------------------------> | MPTCP: FIN-WAIT-2
                       |                           | Sub: ESTABLISHED
                       |                           |
                       |  DATA_FIN + subflow-FIN   |
   close()/shutdown()  | ------------------------> | MPTCP: TIME-WAIT
   (step 2)            |        DATA_ACK           | Sub: CLOSE-WAIT
   MPTCP: CLOSED       | <------------------------ |
   Sub: FIN-WAIT-2     |                           |
                       |                           |
                       |        subflow-FIN        |
   MPTCP: CLOSED       | <------------------------ | subflow-close()
   Sub: TIME-WAIT      |        subflow-ACK        |
   (step 3)            | ------------------------> | MPTCP: TIME-WAIT
                       |                           | Sub: CLOSED
                       |                           |

Figure 5: Multipath TCP may not be able to avoid time-wait state on the subflow (indicated as Sub in the drawing), even if enforced by the application on the client-side.


Figure 5 shows a very particular issue within Multipath TCP. Many high-performance applications try to avoid TIME-WAIT state by deferring the closure of the connection until the peer has sent a FIN. That way, the client on the left of Figure 5 does a passive closure of the connection, transitioning from CLOSE-WAIT to Last-ACK and finally freeing the resources after reception of the ACK of the FIN. An application running on top of an MPTCP-enabled Linux kernel might also use this approach. The difference here is that the close() of the connection (step 1 in Figure 5) only triggers the

图5显示了多路径TCP中的一个非常特殊的问题。许多高性能应用程序试图通过延迟连接的关闭直到对等方发送FIN来避免时间等待状态。这样,图5左侧的客户端将被动关闭连接,从CLOSE-WAIT转换到Last ACK,并在接收到FIN的ACK后释放资源。在支持MPTCP的Linux内核上运行的应用程序也可能使用这种方法。这里的区别是连接的close()只触发

sending of a DATA_FIN. Nothing guarantees that the kernel is ready to combine the DATA_FIN with a subflow-FIN. The reception of the DATA_FIN will make the application trigger the closure of the connection (step 2), trying to avoid TIME-WAIT state with this late closure. This time, the kernel might decide to combine the DATA_FIN with a subflow-FIN. This decision will be fatal, as the subflow's state machine will not transition from CLOSE_WAIT to Last-ACK, but rather go through FIN_WAIT-2 into TIME-WAIT state. The TIME-WAIT state will consume resources on the host for at least 2 MSL (Maximum Segment Lifetime). Thus, a smart application that tries to avoid TIME-WAIT state by doing late closure of the connection actually ends up with one of its subflows in TIME-WAIT state. A high-performance Multipath TCP kernel implementation should honor the desire of the application to do passive closure of the connection and successfully avoid TIME-WAIT state -- even on the subflows.

发送数据。没有任何东西可以保证内核准备好将数据_FIN与子流FIN组合在一起。接收数据_FIN将使应用程序触发连接关闭(步骤2),试图避免延迟关闭的等待状态。这一次,内核可能决定将数据_FIN与子流FIN组合起来。此决定将是致命的,因为子流的状态机不会从CLOSE_WAIT转换到Last ACK,而是通过FIN_WAIT-2进入TIME-WAIT状态。等待时间状态将消耗主机上的资源至少2 MSL(最大段生存期)。因此,试图通过延迟关闭连接来避免时间等待状态的智能应用程序实际上会以其一个子流处于时间等待状态而告终。一个高性能的多路径TCP内核实现应该满足应用程序被动关闭连接的愿望,并成功避免时间等待状态——即使在子流上也是如此。

The solution to this problem lies in an optimistic assumption that a host doing active-closure of a Multipath TCP connection by sending a DATA_FIN will soon also send a FIN on all its subflows. Thus, the passive closer of the connection can simply wait for the peer to send exactly this FIN -- enforcing passive closure even on the subflows. Of course, to avoid consuming resources indefinitely, a timer must limit the time our implementation waits for the FIN.


3.7. Packet Schedulers
3.7. 数据包调度器

In a Multipath TCP implementation, the packet scheduler is the algorithm that is executed when transmitting each packet to decide on which subflow it needs to be transmitted. The packet scheduler itself does not have any impact on the interoperability of Multipath TCP implementations. However, it may clearly impact the performance of Multipath TCP sessions. The Multipath TCP implementation in the Linux kernel supports a pluggable architecture for the packet scheduler [PaaschPhD]. As of this writing, two schedulers have been implemented: round-robin and lowest-rtt-first. The second scheduler relies on the round-trip time (rtt) measured on each TCP subflow and sends first segments over the subflow having the lowest round-trip time. They are compared in [CSWS14]. The experiments and measurements described in [CSWS14] show that the lowest-rtt-first scheduler appears to be the best compromise from a performance viewpoint. Another study of the packet schedulers is presented in [PAMS2014]. This study relies on simulations with the Multipath TCP implementation in the Linux kernel. They compare the lowest-rtt-first with the round-robin and a random scheduler. They show some situations where the lowest-rtt-first scheduler does not perform as well as the other schedulers, but there are many scenarios where the

在多路径TCP实现中,数据包调度器是在传输每个数据包时执行的算法,用于决定需要传输哪个子流。数据包调度器本身对多路径TCP实现的互操作性没有任何影响。但是,它可能会明显影响多路径TCP会话的性能。Linux内核中的多路径TCP实现支持数据包调度器[PaaschPhD]的可插拔体系结构。在撰写本文时,已经实现了两个调度器:循环调度和最低rtt优先。第二个调度器依赖于在每个TCP子流上测量的往返时间(rtt),并通过具有最低往返时间的子流发送第一段。它们在[CSWS14]中进行了比较。[CSWS14]中描述的实验和测量表明,从性能角度来看,最低的rtt优先调度程序似乎是最好的折衷方案。关于数据包调度器的另一项研究见[PAMS2014]。本研究依赖于Linux内核中多路径TCP实现的模拟。他们首先将最低的rtt与循环调度和随机调度进行比较。它们显示了一些情况,其中最低的rtt first调度程序的性能不如其他调度程序,但在许多情况下

opposite is true. [PAMS2014] notes that "it is highly likely that the optimal scheduling strategy depends on the characteristics of the paths being used."


3.8. Segment Size Selection
3.8. 段大小选择

When an application performs a write/send system call, the kernel allocates a packet buffer (sk_buff in Linux) to store the data the application wants to send. The kernel will store at most one MSS (Maximum Segment Size) of data per buffer. As the MSS can differ amongst subflows, an MPTCP implementation must select carefully the MSS used to generate application data. The Linux kernel implementation had various ways of selecting the MSS: minimum or maximum amongst the different subflows. However, these heuristics of MSS selection can cause significant performance issues in some environments. Consider the following example. An MPTCP connection has two established subflows that respectively use an MSS of 1420 and 1428 bytes. If MPTCP selects the maximum, then the application will generate segments of 1428 bytes of data. An MPTCP implementation will have to split the segment in two (1420-byte and 8-byte) segments when pushing on the subflow with the smallest MSS. The latter segment will introduce a large overhead as this single data segment will use 2 slots in the congestion window (in packets) therefore reducing by roughly twice the potential throughput (in bytes/s) of this subflow. Taking the smallest MSS does not solve the issue as there might be a case where the subflow with the smallest MSS only sends a few packets, therefore reducing the potential throughput of the other subflows.


The Linux implementation recently took another approach [DetalMSS]. Instead of selecting the minimum and maximum values, it now dynamically adapts the MSS based on the contribution of all the subflows to the connection's throughput. For each subflow, it computes the potential throughput achieved by selecting each MSS value and by taking into account the lost space in the congestion window. It then selects the MSS that allows to achieve the highest potential throughput.


Given the prevalence of middleboxes that clamp the MSS, Multipath TCP implementations must be able to efficiently support subflows with different MSS values. The strategy described above is a possible solution to this problem.


3.9. Interactions with the Domain Name System
3.9. 与域名系统的互动

Multihomed clients such as smartphones can send DNS queries over any of their interfaces. When a single-homed client performs a DNS query, it receives from its local resolver the best answer for its request. If the client is multihomed, the answer in response to the DNS query may vary with the interface over which it has been sent.


           client -- cellular -- internet -- cdn3
              |                   |
              +----- wifi --------+
           client -- cellular -- internet -- cdn3
              |                   |
              +----- wifi --------+

Figure 6: Simple Network Topology


If the client sends a DNS query over the WiFi interface, the answer will point to the cdn2 server while the same request sent over the cellular interface will point to the cdn1 server. This might cause problems for CDN providers that locate their servers inside ISP networks and have contracts that specify that the CDN server will only be accessed from within this particular ISP. Assume now that both the client and the CDN servers support Multipath TCP. In this case, a Multipath TCP session from cdn1 or cdn2 would potentially use both the cellular network and the WiFi network. Serving the client from cdn2 over the cellular interface could violate the contract between the CDN provider and the network operators. A similar problem occurs with regular TCP if the client caches DNS replies. For example, the client obtains a DNS answer over the cellular interface and then stops this interface and starts to use its WiFi interface. If the client retrieves data from cdn1 over its WiFi interface, this may also violate the contract between the CDN and the network operators.


A possible solution to prevent this problem would be to modify the DNS resolution on the client. The client subnet Extension Mechanisms for DNS (EDNS) defined in [RFC7871] could be used for this purpose. When the client sends a DNS query from its WiFi interface, it should also send the client subnet corresponding to the cellular interface in this request. This would indicate to the resolver that the answer should be valid for both the WiFi and the cellular interfaces (e.g., the cdn3 server).


3.10. Captive Portals
3.10. 俘虏门户

Multipath TCP enables a host to use different interfaces to reach a server. In theory, this should ensure connectivity when at least one of the interfaces is active. However, in practice, there are some particular scenarios with captive portals that may cause operational problems. The reference environment is shown in Figure 7.


           client -----  network1
                +------- internet ------------- server
           client -----  network1
                +------- internet ------------- server

Figure 7: Issue with Captive Portal


The client is attached to two networks: network1 that provides limited connectivity and the entire Internet through the second network interface. In practice, this scenario corresponds to an open WiFi network with a captive portal for network1 and a cellular service for the second interface. On many smartphones, the WiFi interface is preferred over the cellular interface. If the smartphone learns a default route via both interfaces, it will typically prefer to use the WiFi interface to send its DNS request and create the first subflow. This is not optimal with Multipath TCP. A better approach would probably be to try a few attempts on the WiFi interface and then, upon failure of these attempts, try to use the second interface for the initial subflow as well.


3.11. Stateless Webservers
3.11. 无状态Web服务器

MPTCP has been designed to interoperate with webservers that benefit from SYN-cookies to protect against SYN-flooding attacks [RFC4987]. MPTCP achieves this by echoing the keys negotiated during the MP_CAPABLE handshake in the third ACK of the three-way handshake. Reception of this third ACK then allows the server to reconstruct the state specific to MPTCP.

MPTCP设计用于与受益于SYN Cookie的Web服务器进行互操作,以防止SYN洪泛攻击[RFC4987]。MPTCP通过在三方握手的第三个ACK中回显支持MP_的握手过程中协商的密钥来实现这一点。然后,接收到第三个ACK允许服务器重构特定于MPTCP的状态。

However, one caveat to this mechanism is the unreliable nature of the third ACK. Indeed, when the third ACK gets lost, the server will not be able to reconstruct the MPTCP state. MPTCP will fall back to regular TCP in this case. This is in contrast to regular TCP. When the client starts sending data, the first data segment also includes the SYN-cookie, which allows the server to reconstruct the TCP-state. Further, this data segment will be retransmitted by the client in case it gets lost and thus is resilient against loss. MPTCP does not include the keys in this data segment and thus the server cannot reconstruct the MPTCP state.

然而,该机制的一个警告是第三个ACK的不可靠性质。实际上,当第三个ACK丢失时,服务器将无法重建MPTCP状态。在这种情况下,MPTCP将退回到常规TCP。这与常规TCP不同。当客户端开始发送数据时,第一个数据段还包括SYN cookie,它允许服务器重建TCP状态。此外,该数据段将由客户端重新传输,以防丢失,因此具有抗丢失的能力。MPTCP不包括此数据段中的密钥,因此服务器无法重建MPTCP状态。

This issue might be considered as a minor one for MPTCP. Losing the third ACK should only happen when packet loss is high; in this case, MPTCP provides a lot of benefits as it can move traffic away from the lossy link. It is undesirable that MPTCP has a higher chance to fall back to regular TCP in those lossy environments.


[MPTCP-DEPLOY] discusses this issue and suggests a modified handshake mechanism that ensures reliable delivery of the MP_CAPABLE, following the three-way handshake. This modification will make MPTCP reliable, even in lossy environments when servers need to use SYN-cookies to protect against SYN-flooding attacks.

[MPTCP-DEPLOY]讨论了这个问题,并提出了一种改进的握手机制,以确保在三方握手后可靠地提供MP_功能。此修改将使MPTCP可靠,即使在服务器需要使用SYN Cookie来防止SYN洪泛攻击的有损环境中也是如此。

3.12. Load-Balanced Server Farms
3.12. 负载平衡服务器场

Large-scale server farms typically deploy thousands of servers behind a single virtual IP (VIP). Steering traffic to these servers is done through Layer 4 load-balancers that ensure that a TCP-flow will always be routed to the same server [Presto08].


As Multipath TCP uses multiple different TCP subflows to steer the traffic across the different paths, load-balancers need to ensure that all these subflows are routed to the same server. This implies that the load-balancers need to track the MPTCP-related state, allowing them to parse the token in the MP_JOIN and assign those subflows to the appropriate server. However, server farms typically deploy several load-balancers for reliability and capacity reasons. As a TCP subflow might get routed to any of these load-balancers, they would need to synchronize the MPTCP-related state -- a solution that is not feasible on a large scale.


The token (carried in the MP_JOIN) contains the information indicating to which MPTCP-session the subflow belongs. As the token is a hash of the key, servers are not able to generate the token in such a way that the token can provide the necessary information to the load-balancers, which would allow them to route TCP subflows to the appropriate server. [MPTCP-LOAD] discusses this issue in detail and suggests two alternative MP_CAPABLE handshakes to overcome these.


4. Security Considerations
4. 安全考虑

This informational document discusses use cases and operational experience with Multipath TCP. An extensive analysis of the remaining security issues in the Multipath TCP specification has been published in [RFC7430], together with suggestions for possible solutions.


From a security viewpoint, it is important to note that Multipath TCP, like other multipath solutions such as SCTP, has the ability to send packets belonging to a single connection over different paths. This design feature of Multipath TCP implies that middleboxes that have been deployed on-path assuming that they would observe all the packets exchanged for a given connection in both directions may not function correctly anymore. A typical example are firewalls, Intrusion Detection System (IDS) or deep packet inspections (DPIs) deployed in enterprise networks. Those devices expect to observe all the packets from all TCP connections. With Multipath TCP, those middleboxes may not observe anymore all packets since some of them may follow a different path. The two examples below illustrate typical deployments of such middleboxes. The first example, Figure 8, shows an MPTCP-enabled smartphone attached to both an enterprise and a cellular network. If a Multipath TCP connection is established by the smartphone towards a server, some of the packets sent by the smartphone or the server may be transmitted over the cellular network and thus be invisible for the enterprise middlebox.


     smartphone +----- enterprise net --- MBox----+------ server
                |                                 |
                +----- cellular net  -------------+
     smartphone +----- enterprise net --- MBox----+------ server
                |                                 |
                +----- cellular net  -------------+

Figure 8: Enterprise Middlebox May Not Observe All Packets from Multihomed Host


The second example, Figure 9, shows a possible issue when multiple middleboxes are deployed inside a network. For simplicity, we assume that network1 is the default IPv4 path while network2 is the default IPv6 path. A similar issue could occur with per-flow load-balancing such as ECMP [RFC2992]. With regular TCP, all packets from each connection would either pass through Mbox1 or Mbox2. With Multipath TCP, the client can easily establish a subflow over network1 and another over network2 and each middlebox would only observe a part of the traffic of the end-to-end Multipath TCP connection.


     client ----R-- network1  --- MBox1 -----R------------- server
                |                            |
                +-- network2  --- MBox2 -----+
     client ----R-- network1  --- MBox1 -----R------------- server
                |                            |
                +-- network2  --- MBox2 -----+

Figure 9: Interactions between Load-Balancing and Security Middleboxes


In these two cases, it is possible for an attacker to evade some security measures operating on the TCP byte stream and implemented on the middleboxes by controlling the bytes that are actually sent over each subflow and there are tools that ease those kinds of evasion [PZ15] [PT14]. This is not a security issue for Multipath TCP itself


since Multipath TCP behaves correctly. However, this demonstrates the difficulty of enforcing security policies by relying only on on-path middleboxes instead of enforcing them directly on the endpoints.


5. References
5. 工具书类
5.1. Normative References
5.1. 规范性引用文件

[RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. Iyengar, "Architectural Guidelines for Multipath TCP Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, <>.

[RFC6182]福特,A.,雷丘,C.,汉德利,M.,巴尔,S.,和J.艾扬格,“多路径TCP开发的体系结构指南”,RFC 6182,DOI 10.17487/RFC6182,2011年3月<>.

[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, <>.

[RFC6824]Ford,A.,Raiciu,C.,Handley,M.,和O.Bonaventure,“多地址多路径操作的TCP扩展”,RFC 6824DOI 10.17487/RFC68242013年1月<>.

5.2. Informative References
5.2. 资料性引用

[BALIA] Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath TCP: analysis, design, and implementation", IEEE/ACM Trans. on Networking (TON), Volume 24, Issue 1, February 2016.

[BALIA]Peng,Q.,Walid,A.,Hwang,J.,和S.Low,“多路径TCP:分析、设计和实现”,IEEE/ACM Trans。关于网络(TON),第24卷,第1期,2016年2月。

[CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", Communications of the ACM, 57(4):51-57, April 2014, <>.


[Cellnet12] Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O. Bonaventure, "Exploring Mobile/WiFi Handover with Multipath TCP", ACM SIGCOMM workshop on Cellular Networks (Cellnet12), August 2012, < exploring-mobilewifi-handover-multipath-tcp>.

[Cellnet12]Paasch,C.,Detal,G.,Duchene,F.,Raiciu,C.,和O.Bonaventure,“利用多路径TCP探索移动/WiFi切换”,ACM SIGCOMM蜂窝网络研讨会(Cellnet12),2012年8月< 探索mobilewifi切换多路径tcp>。

[COMCOM2016] Tran, V., De Coninck, Q., Hesmans, B., Sadre, R., and O. Bonaventure, "Observing real Multipath TCP traffic", Computer Communications, DOI 10.1016/j.comcom.2016.01.014, April 2016, < observing-real-multipath-tcp-traffic>.

[COMCOM2016]Tran,V.,De Coninck,Q.,Hesmans,B.,Sadre,R.,和O.Bonaventure,“观察真实的多路径TCP流量”,计算机通信,DOI 10.1016/j.comcom.2016.01.014,2016年4月< 观察真实的多路径tcp流量>。

[COMMAG2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. Bonaventure, "Observing Real Smartphone Applications over Multipath TCP", IEEE Communications Magazine Network Testing Series, 54(3), March 2016, <>.

[COMMAG2016]De Coninck,Q.,Baerts,M.,Hesmans,B.,和O.Bonaventure,“通过多路径TCP观察真实的智能手机应用”,IEEE通信杂志网络测试系列,54(3),2016年3月<>.

[CONEXT12] Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. Leboudec, "MPTCP is not Pareto-Optimal: Performance Issues and a Possible Solution", CoNEXT '12: Proceedings of the 8th international conference on Emerging networking experiments and technologies, DOI 10.1145/2413176.2413178, December 2012.

[CONEXT12]Khalili,R.,Gast,N.,Popovic,M.,Upadhyay,U.,和J.Leboudec,“MPTCP不是帕累托最优的:性能问题和可能的解决方案”,CoNEXT'12:第八届新兴网络实验和技术国际会议记录,DOI 10.1145/2413176.2413178,2012年12月。

[CONEXT13] Paasch, C., Khalili, R., and O. Bonaventure, "On the Benefits of Applying Experimental Design to Improve Multipath TCP", Conference on emerging Networking EXperiments and Technologies (CoNEXT), DOI 10.1145/2535372.2535403, December 2013, <>.

[CONEXT13]Paasch,C.,Khalili,R.,和O.Bonaventure,“应用实验设计改善多路径TCP的益处”,新兴网络实验和技术会议(CoNEXT),DOI 10.1145/2535372.2535403,2013年12月, <>.

[CONEXT15] Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O. Bonaventure, "SMAPP: Towards Smart Multipath TCP-enabled APPlications", Proc. Conext 2015, Heidelberg, Germany, December 2015, < smapp-towards-smart-multipath-tcp-enabled-applications>.

[CONEXT15]Hesmans,B.,Detal,G.,Barre,S.,Bauduin,R.,和O.Bonaventure,“SMAPP:迈向智能多路径TCP支持应用”,Proc。Conext 2015,德国海德堡,2015年12月< smapp面向支持tcp的智能多路径应用>。

[CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure, "Experimental evaluation of multipath TCP schedulers", CSWS '14: Proceedings of the 2014 ACM SIGCOMM workshop on Capacity sharing workshop, DOI 10.1145/2630088.2631977, August 2014.

[CSWS14]Paasch,C.,Ferlin,S.,Alay,O.,和O.Bonaventure,“多路径TCP调度器的实验评估”,CSWS'14:2014年ACM SIGCOMM容量共享研讨会论文集,DOI 10.1145/2630088.2631977,2014年8月。

[DetalMSS] Detal, G., "dynamically adapt mss value", Post on the mptcp-dev mailing list, September 2014, < 2014-09/msg00130.html>.

[DetalMSS]Detal,G.,“动态调整mss值”,发布在mptcp开发邮件列表上,2014年9月< 2014-09/msg00130.html>。

[FreeBSD-MPTCP] Williams, N., "Multipath TCP For FreeBSD Kernel Patch v0.5", <>.

[FreeBSD MPTCP]Williams,N.,“FreeBSD内核修补程序v0.5的多路径TCP”<>.

[GRE-NOTIFY] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. Zhang, "GRE Notifications for Hybrid Access", Work in Progress, draft-lhwxz-gre-notifications-hybrid-access-01, January 2015.


[HAMPEL] Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility using lightweight MPTCP proxy", MobiWac '13: Proceedings of the 11th ACM international symposium on Mobility management and wireless access, DOI 10.1145/2508222.2508226, November 2013.

[HAMPEL]HAMPEL,G.,Rana,A.,和T.Klein,“使用轻量级MPTCP代理的无缝TCP移动性”,MobiWac'13:第11届ACM移动性管理和无线接入国际研讨会论文集,DOI 10.1145/2508222.2508226,2013年11月。

[HotMiddlebox13] Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT workshop Hot Middlebox, December 2013, < are-tcp-extensions-middlebox-proof>.

[HotMiddlebox13]Hesmans,B.,Duchene,F.,Paasch,C.,Detal,G.,和O.Bonaventure,“TCP扩展中端箱是否可靠?”,CoNEXT研讨会热中端箱,2013年12月< tcp扩展是否具有“防中间包”能力>。

[HotMiddlebox13b] Detal, G., Paasch, C., and O. Bonaventure, "Multipath in the Middle(Box)", HotMiddlebox '13, December 2013, < multipath-middlebox>.

[HotMiddlebox 13B]Detal,G.,Paasch,C.,和O.Bonaventure,“中间的多路径(框)”,HotMiddlebox'132013年12月< 多路径中间件>。

[HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., Wischik, D., and M. Handley, "Data center networking with multipath TCP", Hotnetx-IX: Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks Article No. 10, DOI 10.1145/1868447.1868457, October 2010, <>.

[HotNets]Raiciu,C.,Pluntke,C.,Barre,S.,Greenhalgh,A.,Wischik,D.,和M.Handley,“具有多路径TCP的数据中心网络”,Hotnetx IX:第九届ACM SIGCOMM网络热点研讨会论文集第10期,DOI 10.1145/1868447.1868457,2010年10月<>.

[HYA-ARCH] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. Zhang, "Hybrid Access Network Architecture", Work in Progress, draft-lhwxz-hybrid-access-network-architecture-02, January 2015.


[ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion control for multipath TCP", 20th IEEE International Conference on Network Protocols (INCP), DOI 10.1109/ICNP.2012.6459978, October 2012.

[ICNP12]Cao,Y.,Xu,M.,和X.Fu,“基于延迟的多路径TCP拥塞控制”,第20届IEEE网络协议国际会议(INCP),DOI 10.1109/ICNP.2012.6459978,2012年10月。

[IETF88] Stewart, L., "IETF 88 Meeting minutes of the MPTCP working group", November 2013, < 88/minutes/minutes-88-mptcp>.

[IETF88]Stewart,L.“IETF 88 MPTCP工作组会议记录”,2013年11月< 88/分钟/分钟-88-mptcp>。

[IETFJ] Bonaventure, O. and S. Seo, "Multipath TCP Deployments", IETF Journal, Vol. 12, Issue 2, November 2016.


[IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., Handley, M., and H. Tokuda, "Is it still possible to extend TCP?", IMC '11: Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference, DOI 10.1145/2068816.2068834, November 2011, <>.

[IMC11]本田,M.,西田,Y.,雷丘,C.,格林哈勒,A.,汉德利,M.,和H.德田,“是否仍有可能扩展TCP?”,IMC'11:2011年ACM SIGCOMM互联网测量会议记录,DOI 10.1145/2068816.2068834,2011年11月<>.

[IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and B. Donnet, "Revealing Middlebox Interference with Tracebox", Proceedings of the 2013 ACM SIGCOMM conference on Internet measurement conference, DOI 10.1145/2504730.2504757, October 2013, < revealing-middlebox-interference-tracebox>.

[IMC13a]Detal,G.,Hesmans,B.,Bonaventure,O.,Vanaubel,Y.,和B.Donnet,“揭示对Tracebox的中间盒干扰”,2013年ACM SIGCOMM互联网测量会议记录,DOI 10.1145/2504730.2504757,2013年10月< 显示中间盒干扰tracebox>。

[IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R., and D. Towsley, "A measurement-based study of MultiPath TCP performance over wireless network", ICM '13: Proceedings of the 2013 conference on Internet measurement conference, DOI 10.1145/2504730.2504751, October 2013, <>.

[IMC13b]Chen,Y.,Lim,Y.,Gibbens,R.,Nahum,E.,Khalili,R.,和D.Towsley,“基于测量的无线网络多路径TCP性能研究”,ICM'13:2013年互联网测量会议论文集,DOI 10.1145/2504730.2504751,2013年10月<>.

[IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush, "From Paris to Tokyo: on the suitability of ping to measure latency", IMC '13: Proceedings of the 2013 conference on Internet measurement Conference, DOI 10.1145/2504730.2504765, October 2013, <>.

[IMC13c]Pelsser,C.,Cittadini,L.,Vissicchio,S.,和R.Bush,“从巴黎到东京:ping测量延迟的适用性”,IMC'13:2013年互联网测量会议记录,DOI 10.1145/2504730.2504765,2013年10月<>.

[INFOCOM14] Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, "Cross-layer path management in multi-path transport protocol for mobile devices", IEEE INFOCOM'14, DOI 10.1109/INFOCOM.2014.6848120, April 2014.

[INFOCOM14]Lim,Y.,Chen,Y.,Nahum,E.,Towsley,D.,和K.Lee,“移动设备多路径传输协议中的跨层路径管理”,IEEE INFOCOM'14,DOI 10.1109/INFOCOM.2014.68481202014年4月。

[KT] Seo, S., "KT's GiGA LTE", July 2015, < slides-93-mptcp-3.pdf>.

[KT]Seo,S.,“KT的千兆LTE”,2015年7月< 幻灯片-93-mptcp-3.pdf>。

[MBTest] Hesmans, B., "MBTest", October 2013, <>.


[Mobicom15] De Coninck, Q., Baerts, M., Hesmans, B., and O. Bonaventure, "Poster - Evaluating Android Applications with Multipath TCP", Mobicom 2015 (Poster), DOI 10.1145/2789168.2795165, September 2015.

[Mobicom15]De Coninck,Q.,Baerts,M.,Hesmans,B.,和O.Bonaventure,“海报-使用多路径TCP评估Android应用程序”,Mobicom 2015(海报),DOI 10.1145/2789168.2795165,2015年9月。

[MPTCP-DEPLOY] Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP robust for stateless webservers", Work in Progress, draft-paasch-mptcp-syncookies-02, October 2015.


[MPTCP-LOAD] Paasch, C., Greenway, G., and A. Ford, "Multipath TCP behind Layer-4 loadbalancers", Work in Progress, draft-paasch-mptcp-loadbalancer-00, September 2015.


[MPTCP-MAX-SUB] Boucadair, M. and C. Jacquenet, "Negotiating the Maximum Number of Multipath TCP (MPTCP) Subflows", Work in Progress draft-boucadair-mptcp-max-subflow-02, May 2016.


[MPTCPBIB] Bonaventure, O., "Multipath TCP - Annotated bibliography", Technical report, April 2015, <>.


[MultipathTCP-Linux] Paasch, C., Barre, S., and . et al, "Multipath TCP - Linux Kernel implementation", <>.

[MultipathTCP Linux]Paasch,C.,Barre,S.,和。等,“多路径TCP-Linux内核实现”<>.

[NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley, "Design, implementation and evaluation of congestion control for multipath TCP", NSDI11: In Proceedings of the 8th USENIX conference on Networked systems design and implementation, 2011.


[NSDI12] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M., Duchene, F., Bonaventure, O., and M. Handley, "How Hard Can It Be? Designing and Implementing a Deployable Multipath TCP", NSDI '12: USENIX Symposium of Networked Systems Design and implementation, April 2012, <>.

[NSDI12]Raiciu,C.,Paasch,C.,Barre,S.,Ford,A.,Honda,M.,Duchene,F.,Bonaventure,O.,和M.Handley,“这有多难?设计和实现可部署的多路径TCP”,NSDI'12:USENIX网络系统设计和实现研讨会,2012年4月, <>.

[PaaschPhD] Paasch, C., "Improving Multipath TCP", Ph.D. Thesis , November 2014, < improving-multipath-tcp>.

[PaaschPhD]Paasch,C.,“改进多路径TCP”,博士。论文,2014年11月< 改进多路径tcp>。

[PAM2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. Bonaventure, "A First Analysis of Multipath TCP on Smartphones", 17th International Passive and Active Measurements Conference (PAM2016) volume 17, March 2016, < first-analysis-multipath-tcp-smartphones>.

[PAM2016]De Coninck,Q.,Baerts,M.,Hesmans,B.,和O.Bonaventure,“智能手机上多径TCP的首次分析”,第17届国际被动和主动测量会议(PAM2016)第17卷,2016年3月< 首先分析多路径tcp>。

[PAMS2014] Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo, "Impact of Path Selection and Scheduling Policies on MPTCP Performance", PAMS2014, DOI 10.1109/WAINA.2014.121, May 2014.

[PAMS2014]Arzani,B.,Gurney,A.,Cheng,S.,Guerin,R.,和B.Loo,“路径选择和调度策略对MPTCP性能的影响”,PAMS2014,DOI 10.1109/WAINA.2014.1212014年5月。

[Presto08] Greenberg, A., Lahiri, P., Maltz, D., Patel, P., and S. Sengupta, "Towards a next generation data center architecture: scalability and commoditization", ACM PRESTO 2008, DOI 10.1145/1397718.1397732, August 2008, <>.

[Presto08]Greenberg,A.,Lahiri,P.,Maltz,D.,Patel,P.,和S.Sengupta,“迈向下一代数据中心架构:可扩展性和商品化”,ACM PRESTO 2008,DOI 10.1145/1397718.1397732,2008年8月<>.

[PT14] Pearce, C. and P. Thomas, "Multipath TCP Breaking Today's Networks with Tomorrow's Protocols", Proc. Blackhat Briefings, 2014, < us-14/materials/us-14-Pearce-Multipath-TCP-Breaking-Todays-Networks-With-Tomorrows-Protocols-WP.pdf>.

[PT14]Pearce,C.和P.Thomas,“多路径TCP使用明天的协议打破今天的网络”,Proc。Blackhat简报会,2014年< us-14/materials/us-14-Pearce-Multipath-TCP-Breaking-Todays-Networks-With-Tomorrows-Protocols-WP.pdf>。

[PZ15] Pearce, C. and S. Zeadally, "Ancillary Impacts of Multipath TCP on Current and Future Network Security", IEEE Internet Computing, vol. 19, no. 5, pp. 58-65, DOI 10.1109/MIC.2015.70, September 2015.

[PZ15]Pearce,C.和S.Zeadally,“多路径TCP对当前和未来网络安全的辅助影响”,IEEE互联网计算,第19卷,第5期,第58-65页,DOI 10.1109/MIC.2015.702015年9月。

[RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", RFC 1812, DOI 10.17487/RFC1812, June 1995, <>.

[RFC1812]Baker,F.,Ed.,“IP版本4路由器的要求”,RFC 1812,DOI 10.17487/RFC1812,1995年6月<>.

[RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and L. Jones, "SOCKS Protocol Version 5", RFC 1928, DOI 10.17487/RFC1928, March 1996, <>.

[RFC1928]Leech,M.,Ganis,M.,Lee,Y.,Kuris,R.,Koblas,D.,和L.Jones,“SOCKS协议版本5”,RFC 1928,DOI 10.17487/RFC1928,1996年3月<>.

[RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000, <>.

[RFC2992]Hopps,C.,“等成本多路径算法的分析”,RFC 2992,DOI 10.17487/RFC2992,2000年11月<>.

[RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007, <>.

[RFC4987]Eddy,W.“TCP SYN洪泛攻击和常见缓解措施”,RFC 4987,DOI 10.17487/RFC4987,2007年8月<>.

[RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled Congestion Control for Multipath Transport Protocols", RFC 6356, DOI 10.17487/RFC6356, October 2011, <>.

[RFC6356]Raiciu,C.,Handley,M.,和D.Wischik,“多路径传输协议的耦合拥塞控制”,RFC 6356,DOI 10.17487/RFC6356,2011年10月<>.

[RFC7430] Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C. Raiciu, "Analysis of Residual Threats and Possible Fixes for Multipath TCP (MPTCP)", RFC 7430, DOI 10.17487/RFC7430, July 2015, <>.

[RFC7430]Bagnulo,M.,Paasch,C.,Gont,F.,Bonaventure,O.,和C.Raiciu,“多路径TCP(MPTCP)的残余威胁和可能修复的分析”,RFC 7430,DOI 10.17487/RFC7430,2015年7月<>.

[RFC7871] Contavalli, C., van der Gaast, W., Lawrence, D., and W. Kumari, "Client Subnet in DNS Queries", RFC 7871, DOI 10.17487/RFC7871, May 2016, <>.

[RFC7871]Contavalli,C.,van der Gaast,W.,Lawrence,D.,和W.Kumari,“DNS查询中的客户端子网”,RFC 7871,DOI 10.17487/RFC7871,2016年5月<>.

[SIGCOMM11] Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A., Wischik, D., and M. Handley, "Improving datacenter performance and robustness with multipath TCP", SIGCOMM '11: Proceedings of the ACM SIGCOMM 2011 conference, DOI 10.1145/2018436.2018467, August 2011, <>.

[SIGCOM11]Raiciu,C.,Barre,S.,Pluntke,C.,Greenhalgh,A.,Wischik,D.,和M.Handley,“利用多路径TCP改进数据中心性能和健壮性”,SIGCOM11:ACM SIGCOM2011年会议记录,DOI 10.1145/2018436.2018467,2011年8月<>.

[SOCKET] Hesmans, B. and O. Bonaventure, "An enhanced socket API for Multipath TCP", Proceedings of the 2016 Applied Networking Research Workshop, DOI 10.1145/2959424.2959433, July 2016, <>.

[SOCKET]Hesmans,B.和O.Bonaventure,“用于多路径TCP的增强SOCKET API”,2016年应用网络研究研讨会论文集,DOI 10.1145/2959424.29594332016年7月<>.

[StrangeMbox] Bonaventure, O., "Multipath TCP through a strange middlebox", Blog post, January 2015, < multipath_tcp_through_a_strange_middlebox.html>.

[StrangeMbox]Bonaventure,O.,“通过一个奇怪的中间盒的多路径TCP”,博客文章,2015年1月< 多路径\u tcp\u通过\u a\u奇怪的\u middlebox.html>。

[TMA2015] Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure, "A First Look at Real Multipath TCP Traffic", Traffic Monitoring and Analysis, 2015, < first-look-real-multipath-tcp-traffic>.

[TMA2015]Hesmans,B.,Tran Vieta,H.,Sadre,R.,和O.Bonaventure,“真实多路径TCP流量的第一眼”,流量监测和分析,2015< 首先看看真正的多路径tcp流量>。

[TR-348] Broadband Forum, ., "TR 348 - Hybrid Access Broadband Network Architecture", Issue: 1, July 2016, < TR-348.pdf>.

[TR-348]宽带论坛,“TR 348-混合接入宽带网络架构”,2016年7月1日发行< TR-348.pdf>。

[tracebox] Detal, G. and O. Tilmans, "Tracebox: A Middlebox Detection Tool", 2013, <>.




This work was partially supported by the FP7-Trilogy2 project. We would like to thank all the implementers and users of the Multipath TCP implementation in the Linux kernel. This document has benefited from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley, Jaehyun Hwang, Mirja Kuehlewind, Benoit Claise, Jari Arkko, Qin Wu, Spencer Dawkins, and Ben Campbell.


Authors' Addresses


Olivier Bonaventure UCLouvain



Christoph Paasch Apple, Inc.



Gregory Detal Tessares