]> Microsoft

Mountain View CA 94043 US kasubra@microsoft.com

Microsoft

Redmond WA 98052 US

General Internet Engineering Task Force template This draft identifies the set of data that needs to be collected, and analyzed to quantify router buffer sizes used in routers in the Wide Area Network (WAN). The scope of this draft is limited to WAN links that have link latencies of 40 to 150 milliseconds. Reducing router buffer sizes has many advantages, the most important being cost. However, there is not much data available today to effectively calculate this. This draft details use cases for the study, and lists data that needs to be taken into consideration to be able to quantify the size of router buffers. The details of the individual measurement metrics are beyond the scope of this document. Neither does the draft identify methods to gather the data. What it identifies is a need to be able to collect, and report this empirical data in a readable fashion thus providing the ability to study and compare data in a more standardized method.

"How much buffering do core links need?" is a question that has been under study for a while. The question boils down to quantify buffer sizes and yet achieve 100% utilization on links with maximum throughput at a feasible cost. Buffer design could substantially increase costs. While over-buffering seems intuitive it can complicate the design of high speed routers, lead to higher power consumption, more board space, and lower density. It can actually increase end-to-end delay in the presence of congestion. This can make congestion more persistent. Additionally, there is always a tradeoff between buffer sizes and the capacity of a router. On the other hand, under-buffering while doing away from the above cons of over-buffering could lead us away from our primary goal of 100 percent link utilization. This could happen in a scenario using a simple Additive Increase Multiplicative Decrease (AIMD) for TCP flows when the sender has packets to send but the window size advertised is less and as a result the receiver consumes far less that it could. The rule of thumb for router buffers has been defined as : B = RTT*C. Where B, was the buffer size, RTT the Round Trip Time, and C the capacity of the bottleneck link. also talks about the buffer size being at least one TCP window size. However later studies , show that the rule of thumb works either for a single flow or a perfectly synchronized large number of flows. Further they postulate that the buffer size is actually (2RTT * C)/sqrt(n), where n is the number of flows. This indicates a significant reduction in the buffer chip promoting lower costs. As seen, there have been proponents for large buffers and small. However, most of these studies are based on theoretical models and simulations. Today, there is no model or protocol to mine big data from a providers network to be able to answer this question efficiently. The nature of WAN traffic can be uncertain and varying. Furthermore the traffic could vastly vary between individual ISPs. This document implored the need for a model of mining empirical big data in a providers network to be able to build a network that drives down the $/GB and at the same time maximizing link utilization. This document outlines use cases for the study of router buffer sizes in the WAN and identifies the data that needs to be collected and analyzed. It could be further extended to the edge and datacenters, but it is outside the scope of this draft.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

From an operator's perspective it is imperative to monitor discards and link utilization over WAN links to be able to study the router buffer sizes. But these alone will be unable to provide an operator with enough information as to why the discards happened. The two use cases outlined here argue that more data needs to be collected, reported, and analyzed.

Trans-pacific and trans-atlantic links of latencies in the range of 150 ms and 90 ms respectively, with low link utilization of 30 percent, and small buffers have seen dropped packets. The most intuitive method has been to increase the buffer sizes for these links on noticing packet discards. While this might alleviate the issue temporarily, unless the right problem has been identified this could readily lead to buffer bloat which has many issues on its own.

Operators have also observed dropped packets on WAN links within North America with as large buffers as 125 MB per port with link utilizations of 60%. If this happens even if the router has not been specifically configured to drop certain type of packets, or there are no routing misconfigurations, then clearly the issue here is not the size of the router buffer.

This section talks about the absolute minimum requirements of the type of data that needs to be collected to be able to effectively quantify router buffer size. Data Details 1 Number of concurrent flows, N For aggregate traffic 2 Length of the flow, L [Flow start time - flow end time] 3 Packet Discards, P Per Interface 4 Reason for Packet Discards, R Buffer overflow, configuration, etc. 5 Resolution of Time Interval, T [Flow start time - flow end time] 6 5 tuple flow identity, I Src IP, Dest IP, Src port, Dest Port, Protocol. A service provider needs to take into consideration several attributes to determine the right buffer size for its WAN routers. This section enlists the details as to why the five above have been identified as the minimum essential data needed to aid the study of router buffer sizes.

Studies and show that 95% of flows in the internet today are attributed to TCP flows. The nature of these flows can vary significantly not only with various time periods, but also between providers. Flows that spend most of their time in slow-start require significantly less buffering than flows that live mostly in congestion avoidance. Due to this it is important to identify the type of concurrent flows that can live on a WAN link. Short (non-persistent) flows are those that live for less than one RTT, and large (persistent) flows are those whose lifetime is larger that one RTT with congestion overhead. Internet measurements show that while a smaller number of large flows contribute to maximum packet transfer, short flows dominate most TCP sessions and large flows are known to have a larger effect on buffer sizes. These combination flows could in turn have an effect on Round Trip Time (RTT), loss probability and flow lengths. The ability to detect large flows is necessary because while the flows can be constant in steady state, the aggregate traffic can keep changing due to various arrival and departure rates. There needs to be a way for the number of concurrent flows to be collected and analyzed with the granularity of the lifetime of short flows, as low as one millisecond.

Length of a flow can be defined as its duration: [flow stop time - flow start time], or the number of packets/bytes sent in this time duration. Identifying the length of flow in a provider's network will give information of the mix of short and large flows that are present in the WAN. This will lead to modeling implications in TCP flow control.

Number of packet discards per interface is probably the most important metric. Of this the number of outward (WAN) facing interface discards would be more intuitive to the study of buffer sizes. Interface discards can be referred to in

There can be several reasons for packet discards especially when it is observed on less utilized links. Some of them could be due to routing misconfigurations, or designed to drop certain packets due to configurations. Clearly stating a reason as insufficient buffer will help narrow down the data required. This is especially true in the case of smart buffer allocations when some ports run out of buffers but not others. We could observe that a port has been allocated only, say, 30 percent of the available total buffer space but is experiencing the highest utilization and as a result of that is seeing packet drops pointing to the fact that dynamic buffers' smart allocations scheme is not adaptive and predictive to the nature of the WAN traffic.

The time interval should be granular such that it captures not only the number of concurrent flows in steady state but also the aggregate traffic over the lifetime of a short flow. It should also be able to correlate the discards per interface to the number of concurrent flows. Today via IPFIX we can calculate the number of concurrent flows. Via Sflow counters or flows, we can calculate the discards. Using counters requires upto two times the granularity set for any changes to be visible due to Nyquist rate. Reducing the counter export interval would increase the responsiveness, but at the cost of increased overhead and reduced scalability. On the other hand, packet sampling automatically allocates monitoring resources to busy links, providing a highly scaleable way to quickly detect traffic flows wherever they occur in the network. Responsiveness is important for a more stable control.

5 tuple flows have a source IP, destination IP, source port, destination port, and protocol to identify endpoints for unidirectional flows. Having this functionality gives the network operator a way to identify the offending flows, legitimate elephant flows, and high priority flows which may happen at certain periods during the day. Being able to separate traffic using the 5 tuple, further increases the strength of the sample set of empirical data available for the study of router buffer sizes.

The below is an example information model representing the data that needs to be measured in order to make buffer size estimations easier. The methods/algorithms to determine this date is out of the scope of draft. ]]>

We see that there are numerous issues at different layers that have an effect (directly or indirectly) on the sizing of router buffers. We also notice that there is no study that takes empirical data into consideration. Ideally, what would be required is an all knowing oracle that sees the traffic flow on an end-to-end network across all layers. Due to a lack of the resource, the first step to the study of router buffer sizes is to effectively mine the big data repository of a provider for the data identified in this draft.

This memo includes no request to IANA. All drafts are required to have an IANA considerations section (see the update of RFC 2434 for a guide). If the draft does not require IANA to do anything, the section contains an explicit statement that this is the case (as above). If there are no requirements for IANA, the section will be removed during conversion into an RFC by the RFC Editor.

This document does not introduce new security issues.

&RFC2119; &I-D.narten-iana-considerations-rfc2434bis; A. Feldmann, J. Rexford, and R. Caceres K. McCloghrie, F. Kastenholz J. Postel C. Villamizar and C. Song R. Bush and D. Meyer G. Appenzeller, I. Klesassy, and N. McKeown W. R. Stevens Konstantin Avrachenkov, INRIA Sophia Antipolis

This becomes an Appendix.