CORE WG F. Zheng INTERNET-DRAFT B. Fu Intended Status: Informational Z. Cao (Ed.) Expires: Feb 3 , 2016 Huawei Tech July 4, 2016 CoAP Latency Evaluation draft-zheng-core-coap-lantency-evaluation-00 Abstract This document presents the evaluation results of CoAP in terms of various latency metrics over UDP/TCP under different network environments. We conduct experiments via both GPRS and WiFi. We also evaluate how the latency metrics are impacted by the background traffic. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Copyright and License Notice Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. Expires [Page 1] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1 General topology . . . . . . . . . . . . . . . . . . . . . . 3 2.2 CoAP Client . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3 CoAP Server . . . . . . . . . . . . . . . . . . . . . . . . 3 2.4 Bare Latency . . . . . . . . . . . . . . . . . . . . . . . . 4 2.5 Packet loss rate . . . . . . . . . . . . . . . . . . . . . . 4 2.6 Tested flows . . . . . . . . . . . . . . . . . . . . . . . . 4 3 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.1 How much CoCoA helps? . . . . . . . . . . . . . . . . . . . 5 4.2 How much impact of the competing flows under GPRS and WiFi . 6 4.4 How much difference of CoAP-TCP and CoAP-over-UDP . . . . . 7 5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 6 Security Considerations . . . . . . . . . . . . . . . . . . . . 8 7 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 8 8 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . 8 9 References . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 9 Expires [Page 2] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation 1 Introduction CoAP [RFC7252] is a light-weight application protocol for constrained networks. CoAP has been widely used in many Internet of Things scenarios and deployed by many products as well. In order to know how CoAP performs under various network environment, we conduct the experimental study and present the results in this document. This work is a complementary work of the previous evaluation work presented in [COAPCCCOMP] and [COAPCC]. In particular, (a) we intentionally measure the latency related metrics; (b) we introduce competing flows while testing, which reveals the difference of Bufferbloat impact under GPRS and WiFi; (c) we conduct study in both cellular networks (GPRS) and WiFi network. This document presents the evaluation setup and results, and intents to provide insights for further protocol design and implementation. 2 Experiment Setup This section describes the evaluation setup. 2.1 General topology The topology is depicted in Fig.1 ---GPRS--+ ________ | | ( ) CoAP Client-{ +- + Internet +----+CoAP Server | | (________) - Wi-Fi--+ Figure 1: Topology 2.2 CoAP Client We use an Android phone as the CoAP client, which has both cellular and Wi-Fi interfaces. This device is physically located at our lab. libcoap is ported to this device with the extension defined in CoCoA algorithm [CoCoA] and CoAP over TCP [COT]. 2.3 CoAP Server The CoAP server is a rented Linux server from some public cloud company in China. We also use libcoap and our extensions on the server side. Expires [Page 3] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation 2.4 Bare Latency The ICMP RTT between the client and server is as below. | | GPRS | Wi-Fi | +------+-----------+-----------+ | RTT | ~572ms | ~19ms | +------+-----------+-----------+ 2.5 Packet loss rate We conduct these tests with the native packet loss rate, as well as increased packet loss rate by using Linux TC command at the server (+2% and +5% separately). We use 0% to denote the native packet loss rate. 2.6 Tested flows The client sends 100 CoAP requests to the server, with interval of 100 ms. The size of the CoAP request is 9 Bytes, and the response payload size is 23 Bytes. We believe this reflects a certain IoT scenario, e.g., the reporting of data or status from device to server. For TCP flows, long-lived connection has been established before the test, so that TCP three-way handshake overhead has not been taken into account. We also introduce background traffic as the competing flow while measuring the latency. The background traffic is a FTP downloading flow. 3 Metrics We intend to evaluate the latency of the CoAP flow with the following metrics: 1. Flow completing time (FCT): defined as the time elapsed from the sending of first request until the receiving of the last response. 2. CoAP RTT (C-RTT): defined as the average elapsed time between the sending of the original CoAP request and receiving of the CoAP response. Note: if the original request has been lost and then retransmitted when RTO is triggered, it will certainly increase the CoAP-RTT. 3. Retransmission ratio (RTR): defined as the percentage of CoAP- layer retransmission. The maximum number of retransmission is Expires [Page 4] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation defined as FOUR in [RFC7252] for UDP. 4 Results 4.1 How much CoCoA helps? Table.1 summarizes the CoAP latency performance with and without CoCoA. The results clearly indicate that CoCoA effectively reduces the latency, especially when the packet loss rate is high, due to its dynamic calculation and update of RTO. The trade-off of CoCoA is its payment of more retransmission, alike other congestion control algorithm. GPRS | CoAP-RAW | CoAP-CoCoA | +------+---------------+----------------+ | FCT |+0%* | 10173 |+0% | 10175 | |(ms) +---------------+----------------+ | |+2% | 10868 |+2% | 10170 | | +---------------+----------------+ | |+5% | 10697 |+5% | 10176 | +------+---------------+----------------+ |C-RTT |+0% | 178 |+0% | 172 | |(ms) +---------------+----------------+ | |+2% | 243 |+2% | 182 | | +---------------+----------------+ | |+5% | 332 |+5% | 203 | +------+---------------+----------------+ | RTR |+0% | 0 |+0% | 7 | | (% ) +---------------+----------------+ | |+2% | 2.7 |+2% | 18 | | +---------------+----------------+ | |+5% | 5.3 |+5% | 9 | +++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++ WiFi | CoAP-RAW | CoAP-CoCoA | +------+---------------+----------------+ | FCT |+0% | 9988 |+0% | 10175 | |(ms) +---------------+----------------+ | |+2% | 10818 |+2% | 10170 | | +---------------+----------------+ | |+5% | 10889 |+5% | 10176 | +------+---------------+----------------+ |C-RTT |+0% | 18 |+0% | 18 | |(ms) +---------------+----------------+ | |+2% | 105 |+2% | 18 | | +---------------+----------------+ Expires [Page 5] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation | |+5% | 180 |+5% | 18 | +------+---------------+----------------+ | RTR |+0% | 0 |+0% | 3.5 | | (% ) +---------------+----------------+ | |+2% | 3.3 |+2% | 5.3 | | +---------------+----------------+ | |+5% | 6.3 |+5% | 6.8 | +------+---------------+----------------+ * 0% indicates the native packet loss rate, without any manually introduced packet loss rate. Table.1 CoAP latency performance with and without CoCoA 4.2 How much impact of the competing flows under GPRS and WiFi Table.2 summarizes the results. With competing flows under GRPS, both FCT and C-RTT deteriorate heavily, and especially C-RTT has been increased by around seven times. Comparatively, in Wi-Fi case, existence of the competing flows does not deteriorates the two latency factors much. The latency bloat given the existence of competing flows most likely stems from the famous buffer-bloat problem. The GPRS and WiFi differ from each other because of their queue management deployed in the middle. This result we present here about GPRS is consistent with what have been reported in [BufbloatLTE]. In addition, we find the WiFi suffers less from Bufferbloat than GPRS. GPRS | CoAP-RAW | CoAP-CoCoA | +------+---------------+----------------+ | FCT |wo | 10173 |wo | 10175 | |(ms) +---------------+----------------+ | |w | 11270 |w | 11288 | +------+---------------+----------------+ |C-RTT |wo | 178 |wo | 172 | |(ms) +---------------+----------------+ | |w | 1344 |w | 1315 | +------+---------------+----------------+ WiFi | CoAP-RAW | CoAP-CoCoA | +------+---------------+----------------+ | FCT |wo | 9988 |wo | 9985 | |(ms) +---------------+----------------+ | |w | 9969 |w | 9969 | +------+---------------+----------------+ |C-RTT |wo | 18 |wo | 18.3 | |(ms) +---------------+----------------+ | |w | 30.9 |w | 25.4 | Expires [Page 6] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation +------+---------------+----------------+ Table.2 CoAP latency performance under competing flows (wo=without competing flow, w=with competing flow) 4.4 How much difference of CoAP-TCP and CoAP-over-UDP We implement the CoAP-TCP per [COAP-T], which is still working-in- progress. The C-RTT comparison of the two algorithm is shown in Table.3. In the CoAP-TCP implementation, we used the default Linux congestion control algorithm -Cubic. In the test, we first establish the TCP connection and then run the CoAP flows, which means that the handshake overhead has not been included into the results. CoAP-TCP bears a longer latency possibly because of (a) TCP header overhead (20 bytes as composed to 8 bytes, and TCP/UDP header constitutes a larger portion of the packet due to the small CoAP payload; (b) GPRS low data rate, which makes this effect more obvious in GPRS than in WiFi. (d) under WiFi, retransmission contributes more to latency than packet length. GPRS | CoAP-CoCoA | CoAP-TCP | +------+---------------+----------------+ |C-RTT |+0% | 172 |+0% | 450 | |(ms) +---------------+----------------+ | |+2% | 182 |+2% | 421 | | +---------------+----------------+ | |+5% | 203 |+5% | 454 | +------+---------------+----------------+ WiFi | CoAP-CoCoA | CoAP-TCP | +------+---------------+----------------+ |C-RTT |+0% | 18 |+0% | 18 | |(ms) +---------------+----------------+ | |+2% | 18 |+2% | 31 | | +---------------+----------------+ | |+5% | 18 |+5% | 40 | +------+---------------+----------------+ Table.3 CoAP latency performance of UDP-CoCoA and CoAP-TCP 5. Summary As a summary, we find: 1. CoCoA algorithm helps shorten the CoAP message layer latency effectively for our tested cases. Expires [Page 7] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation 2. With competing flows under GRPS, both FCT and C-RTT deteriorate heavily. Comparatively, WiFi suffers less from competing flows than GPRS, because of deployment of active queue management to fight with Bufferbloat. 3. COAP-TCP (with Cubic) in general bears a longer latency than that of CoCoA in UDP. Many factors contribute to this fact as analyzed in Section.4.4. This document intents to provide insights for further protocol design and implementation. We will be glad to generate more results if the IETF audience finds this document helpful. 6 Security Considerations We have not evaluate the CoAP with DTLS/TLS yet. Security tunnels definitely add to the overall latency. 7 IANA Considerations Not applicable to this document. 8 Acknowledgement This document was inspired by related work presented in [COAPCC] [COAPCCCOMP]. We also appreciate all contributors to CoAP Congestion Control. We also thanks Dr. Fanzhao Wang for discussion and reasoning about the results. 9 References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997. [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "Constrained Application Protocol (CoAP)", RFC 7252, June 2014. [CoCoA] C. Bormann, A. Betzler, C. Gomez, I. Demirkol, "CoAP Simple Congestion Control/Advanced", draft-bormann-core-cocoa-03, Oct., 2015. Expires [Page 8] INTERNET DRAFT draft-zheng-core-coap-latency-evaluation [COAP-T] C. Bormann, S. Lemay, and H. Tschofenig, "A TCP and TLS Transport for the Constrained Application Protocol (CoAP)", draft-ietf-core-coap-tcp-tls-03, May 2016. [COAPCCCOMP] Ilpo Jaervinen, Laila Daniel, Markku Kojo, "Experimental Evaluation of Alternative Congestion Control Algorithms for Constrained Application Protocol (CoAP)", Internet of Things (WF-IoT), 2015. [COAPCC] August Betzler, Carles Gomez, Ilker Demirkol, Josep Paradells, "CoAP Congestion Control for the Internet of Things", IEEE Networks Magazine, 2016. [BufbloatLTE] Stefan Alfredsson, Giacomo Del Giudice, Johan Garcia, Anna Brunstrom, Luca De Cicco, Saverio Mascolo, "Impact of TCP congestion control on bufferbloat in cellular networks", IEEE WOWMOM 2013. Authors' Addresses Fei Zheng Huawei Tech, Shenzhen, China EMail: zhengfei10@huawei.com Baicheng Fu Huawei Tech, Shenzhen, China EMail: fubaicheng@huawei.com Zhen Cao (Ed.) Huawei Tech, Beijing, China EMail: zhencao.ietf@gmail.com Expires [Page 9]