Bob Briscoe's publications

Contents

Communications Architecture

• Congestion Policing
• Flow Rate Fairness: Dismantling a Religion (Jul 2007)
• Internet: Fairer is Faster (May 2009)
• A Fairer Faster Internet Protocol (Dec 2008)
• Congestion Exposure (ConEx) Concepts and Use Cases (Mar 2011)
• Congestion Exposure (ConEx) Concepts, Abstract Mechanism and Requirements (Dec 2015)
• IPv6 Destination Option for Congestion Exposure (ConEx) (Jan 2016)
  
• The Need for Congestion Exposure in the Internet (Oct 2009)
• Problem Statement: Transport Protocols Don't Have To Do Fairness (Jul 2008)
• Network Performance Isolation using Congestion Policing (Feb 2014)
• Network Performance Isolation in Data Centres using Congestion Policing (Feb 2014)
• Policing Congestion Response in an Inter-Network Using Re-Feedback (Sep 2005)
• Policing Freedom to Use the Internet Resource Pool (Dec 2008)
• Re-feedback: Freedom with Accountability for Causing Congestion in a Connectionless Internetwork (May 2009)
  
• Re-ECN: Adding Accountability for Causing Congestion to TCP/IP (Jul 2014)
• Re-ECN: A Framework for adding Congestion Accountability to TCP/IP (Mar 2014)
• Bandwidth Management; Internet Society Technology Roundtable Series (Nov 2011)
• Is There a Problem With Peer-to-peer Traffic? (May 2008)
• Solving this traffic management problem... and the next, and the next (May 2008)
• Emulating Border Flow Policing Using Re-PCN on Bulk Data (Oct 2009)
• Pre-Congestion Notification (PCN) Architecture (Jun 2009)
• A Survey of PCN-Based Admission Control and Flow Termination (Jul 2009)
• Congestion Notification
• `Data Center to the Home': Ultra-Low Latency for All (Jun 2015)
• PI²: A Linearized AQM for both Classic and Scalable TCP (Dec 2016)
  
• Ultra-Low Delay for All: Live Experience, Live Analysis (May 2016)
• Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture (Mar 2018)
  
• Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Problem Statement (Jul 2016)
  
• Identifying Modified Explicit Congestion Notification (ECN) Semantics for Ultra-Low Queuing Delay (Mar 2018)
  
• DualQ Coupled AQM for Low Latency, Low Loss and Scalable Throughput (Mar 2018)
• Implementing the `TCP Prague' Requirements for L4S (Mar 2019)
• DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM (Mar 2019)
  
• Interactions between Low Latency, Low Loss, Scalable Throughput (L4S) and Differentiated Services (Mar 2018)
  
• Using Data Center TCP (DCTCP) in the Internet (Dec 2014)
• ECN++: Adding Explicit Congestion Notification (ECN) to TCP control packets (Oct 2017)
• Measuring ECN++: Good News for ++, Bad News for ECN over Mobile (Mar 2018)
  
• Byte and Packet Congestion Notification (Feb 2014)
• Tunnelling of Explicit Congestion Notification (Nov 2010)
• Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP (Mar 2018)
• Propagating Explicit Congestion Notification Across IP Tunnel Headers Separated by a Shim (Jul 2017)
• Network Service Header (NSH) Explicit Congestion Notification (ECN) Support (Mar 2018)
  
• TRILL: ECN (Explicit Congestion Notification) Support (Feb 2018)
• Resolving Tensions between Congestion Control Scaling Requirements (Jul 2017)
  
• An Open ECN Service in the IP layer (Feb 2001)
• Congestion Control, including multipath
• Implementing the `TCP Prague' Requirements for L4S (Mar 2019)
• Paced Chirping: Rapid flow start with very low queuing delay (May 2019)
• Paced Chirping - Rethinking TCP start-up (Mar 2019)
  
• `Data Center to the Home': Ultra-Low Latency for All (Jun 2015)
• PI²: A Linearized AQM for both Classic and Scalable TCP (Dec 2016)
• Ultra-Low Delay for All: Live Experience, Live Analysis (May 2016)
• Bits-N-Bites Demonstrates Ultra-Low Delay for All (Nov 2015)
• Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture (Mar 2018)
• Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Problem Statement (Jul 2016)
• Identifying Modified Explicit Congestion Notification (ECN) Semantics for Ultra-Low Queuing Delay (Mar 2018)
• Resolving Tensions between Congestion Control Scaling Requirements (Jul 2017)
• Review: Proportional Integral controller Enhanced (PIE) Active Queue Management (AQM) (May 2015)
  
• Using Data Center TCP (DCTCP) in the Internet (Dec 2014)
• Scaling TCP's Congestion Window for Small Round Trip Times (May 2015)
• Open Research Issues in Internet Congestion Control (Feb 2011)
• Dagstuhl Perspectives Workshop on End-to-End Protocols for the Future Internet (Apr 2009)
• Review: Quick-Start for TCP and IP (Nov 2005)
• Pricing Incentives
• Architecting the Future Internet; Re-Capturing the Internet Value Chain (May 2009)
• Market Managed Multi-service Internet (M3I): Architecture (Aug 2003)
  
• The Direction of Value Flow in Open Multi-service Connectionless Networks (Nov 1999)
• Protocol Extensibility
• Tunnelling through Inner Space (Jan 2015)
  
• Inner Space for All TCP Options (Mar 2015)
• Inner Space for TCP Options (Oct 2014)
• The TCP Echo and TCP Echo Reply Options (Aug 2015)
• The Echo Cookie TCP Option (Oct 2014)
  
• Extended TCP Option Space in the Payload of an Alternative SYN (Jul 2014)
• TCP SYN Extended Option Space in the Payload of a Supplementary Segment (Jul 2014)
• Reusing the IPv4 Identification Field in Atomic Packets (Feb 2014)
• Generic UDP Tunnelling (GUT) (Jul 2010)
• Distributed Systems
• The Implications of Pervasive Computing on Network Design (Jul 2004)
• Service Discovery in Massive Scale Federations - The Web Analysed in Open Distributed Processing Terms (Mar 1996)

Myth-slaying

• Less Delay, Not Just More Bandwidth (Oct 2014)
• Internet: Fairer is Faster (May 2009)
  
• A Fairer Faster Internet Protocol (Dec 2008)
• Flow Rate Fairness: Dismantling a Religion (Jul 2007)
• Problem Statement: Transport Protocols Don't Have To Do Fairness (Jul 2008)
• Metcalfe's Law is Wrong (Jul 2006)
  (also published as When a ‘law’ isn’t a law at all (Dec 2009))

Industry roadmapping

• Network Functions Virtualisation (NFV) Network Operator Perspectives on Industry Progress (Oct 2014)
• Reducing Internet Latency: A Survey of Techniques and their Merits (Dec 2014)
• Internet Latency: Causes, Solutions and Trade-offs (May 2015)
  
• Workshop Report: Reducing Internet Latency, 2013 (Apr 2014)
• Open Research Issues in Internet Congestion Control (Feb 2011)
• Future Internet; Starting the Discussion (Jul 2009)
• Commercial Models for IP Quality of Service Interconnect (Apr 2005)
  
• The Implications of Pervasive Computing on Network Design (Jul 2004)
  
• Market Managed Multi-service Internet (M3I): ISP Business Model Report (Feb 2002)
• The Direction of Value Flow in Open Multi-service Connectionless Networks (Nov 1999)
  
• Distributed Objects on the Web (Mar 1996)
• Grand Strategy - Rationale; towards a Denial of Service Resistant Internet (Unfinished) (Nov 2005)

Internet Quality of Service 

• Congestion Policing
• Flow Rate Fairness: Dismantling a Religion (Jul 2007)
• Congestion Exposure (ConEx) Concepts and Use Cases (Mar 2011)
• Congestion Exposure (ConEx) Concepts, Abstract Mechanism and Requirements (Dec 2015)
• IPv6 Destination Option for Congestion Exposure (ConEx) (Jan 2016)
  
• The Need for Congestion Exposure in the Internet (Oct 2009)
• Problem Statement: Transport Protocols Don't Have To Do Fairness (Jul 2008)
• Problem Statement: We Don't Have To Do Fairness Ourselves (Jul 2008)
• Internet: Fairer is Faster (May 2009)
• A Fairer Faster Internet Protocol (Dec 2008)
• Commercial Models for IP Quality of Service Interconnect (Apr 2005)
• Network Performance Isolation using Congestion Policing (Feb 2014)
• Network Performance Isolation in Data Centres using Congestion Policing (Feb 2014)
• Initial Congestion Exposure (ConEx) Deployment Examples (Jul 2012)
• Policing Congestion Response in an Inter-Network Using Re-Feedback (Sep 2005)
• Re-ECN: Adding Accountability for Causing Congestion to TCP/IP (Jul 2014)
• Re-ECN: A Framework for adding Congestion Accountability to TCP/IP (Mar 2014)
• Policing Freedom to Use the Internet Resource Pool (Dec 2008)
• Quality of Service Metrics
• Byte and Packet Congestion Notification (Mar 2011)
• Pseudowire (PW) Congestion Considerations (Jun 2016)
• Latency
• Paced Chirping: Rapid flow start with very low queuing delay (May 2019)
• Paced Chirping - Rethinking TCP start-up (Mar 2019)
• Low Latency DOCSIS: Technology Overview (Feb 2019)
  
• Bits-N-Bites Demonstrates Ultra-Low Delay for All (Nov 2015)
• `Data Center to the Home': Ultra-Low Latency for All (Jun 2015)
• PI²: A Linearized AQM for both Classic and Scalable TCP (Dec 2016)
  
• Ultra-Low Delay for All: Live Experience, Live Analysis (May 2016)
• Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture (Mar 2018)
• Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Problem Statement (Jul 2016)
• Identifying Modified Explicit Congestion Notification (ECN) Semantics for Ultra-Low Queuing Delay (Mar 2018)
• DualQ Coupled AQM for Low Latency, Low Loss and Scalable Throughput (Mar 2018)
• Implementing the `TCP Prague' Requirements for L4S (Mar 2019)
• DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM (Mar 2019)
• Interactions between Low Latency, Low Loss, Scalable Throughput (L4S) and Differentiated Services (Mar 2018)
• Resolving Tensions between Congestion Control Scaling Requirements (Jul 2017)
• Review: Proportional Integral controller Enhanced (PIE) Active Queue Management (AQM) (May 2015)
• Less Delay, Not Just More Bandwidth (Oct 2014)
• Reducing Internet Latency: A Survey of Techniques and their Merits (Dec 2014)
• Internet Latency: Causes, Solutions and Trade-offs (May 2015)
• A Survey of Latency Reducing Techniques and their Merits (position paper) (Sep 2013)
• Workshop Report: Reducing Internet Latency, 2013 (Apr 2014)
• Rapid Signalling of Queue Dynamics (Sep 2017)
• The Native AQM for L4S Traffic (Sep 2017)
• Managing a Queue to a Soft Delay Target (Sep 2017)
  
• Insights from Curvy RED (Random Early Detection) (May 2015)
• Scaling TCP's Congestion Window for Small Round Trip Times (May 2015)
  
• Bandwidth Management; Internet Society Technology Roundtable Series (Nov 2011)
• Using Data Center TCP (DCTCP) in the Internet (Dec 2014)
• Network Performance Isolation using Congestion Policing (Feb 2014)
• Network Performance Isolation in Data Centres using Congestion Policing (Feb 2014)
• Policing Freedom to Use the Internet Resource Pool (Dec 2008)
• Advice on network buffering (Mar 2013)
• Problem Statement and Requirements for a More Accurate ECN Feedback (Aug 2015)
• More Accurate ECN Feedback in TCP (Mar 2018)
• Chirping for Congestion Control - Implementation Feasibility (Nov 2010)
• Review: Quick-Start for TCP and IP (Nov 2005)
• Congestion Notification
• Bits-N-Bites Demonstrates Ultra-Low Delay for All (Nov 2015)
• `Data Center to the Home': Ultra-Low Latency for All (Jun 2015)
• PI²: A Linearized AQM for both Classic and Scalable TCP (Dec 2016)
  
• Ultra-Low Delay for All: Live Experience, Live Analysis (May 2016)
• Identifying Modified Explicit Congestion Notification (ECN) Semantics for Ultra-Low Queuing Delay (May 2018)
• DualQ Coupled AQM for Low Latency, Low Loss and Scalable Throughput (Mar 2018)
• Implementing the `TCP Prague' Requirements for L4S (Mar 2019)
• DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM (Mar 2019)
• Using Data Center TCP (DCTCP) in the Internet (Dec 2014)
• ECN++: Adding Explicit Congestion Notification (ECN) to TCP control packets (Oct 2017)
• Measuring ECN++: Good News for ++, Bad News for ECN over Mobile (Mar 2018)
• Insights from Curvy RED (Random Early Detection) (May 2015)
• Problem Statement and Requirements for a More Accurate ECN Feedback (Aug 2015)
• More Accurate ECN Feedback in TCP (Mar 2018)
• Tunnelling of Explicit Congestion Notification (Nov 2010)
• Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP (Mar 2018)
  (previously know as Layered Encapsulation of Congestion Notification)
• Propagating Explicit Congestion Notification Across IP Tunnel Headers Separated by a Shim (Jul 2017)
• Network Service Header (NSH) Explicit Congestion Notification (ECN) Support (Mar 2018)
• TRILL: ECN (Explicit Congestion Notification) Support (Feb 2018)
• Explicit Congestion Marking in MPLS (Jan 2008)
• Resolving Tensions between Congestion Control Scaling Requirements (Jul 2017)
• An Open ECN Service in the IP layer (Feb 2001)
• Wireless Quality of Service
• Service Differentiation in Third Generation Mobile Networks (Oct 2002)
  
• Economic Models for Resource Control in Wireless Networks (Sep 2002)
• Managing Quality of Service using Shadow Pricing
• Practical Microeconomics and Internet Resource Sharing Protocols (Aug 2009)
• Flow Rate Fairness: Dismantling a Religion (Jul 2007)
• Internet: Fairer is Faster (May 2009)
• A Fairer Faster Internet Protocol (Dec 2008)
• Congestion Exposure (ConEx) Concepts and Use Cases (Mar 2011)
• Congestion Exposure (ConEx) Concepts, Abstract Mechanism and Requirements (Dec 2015)
• IPv6 Destination Option for Congestion Exposure (ConEx) (Jan 2016)
  
• Commercial Models for IP Quality of Service Interconnect (Apr 2005)
• Network Performance Isolation using Congestion Policing (Feb 2014)
• Network Performance Isolation in Data Centres using Congestion Policing (Feb 2014)
• Policing Congestion Response in an Inter-Network Using Re-Feedback (Sep 2005)
  
• Re-ECN: Adding Accountability for Causing Congestion to TCP/IP (Jul 2014)
• Re-ECN: A Framework for adding Congestion Accountability to TCP/IP (Mar 2014)
  (previously know as Re-ECN: The Motivation for Adding Accountability for Causing Congestion to TCP/IP)
• Re-feedback: Freedom with Accountability for Causing Congestion in a Connectionless Internetwork (May 2009)
• Policing Freedom to Use the Internet Resource Pool (Dec 2008)
• A Path-Aware Rate Policer: Design and Comparative Evaluation (Oct 2005)
• Emulating Border Flow Policing Using Re-PCN on Bulk Data (Oct 2009)
• Using Self-interest to Prevent Malice; Fixing the Denial of Service Flaw of the Internet (Aug 2006)
• Pre-Congestion Notification (PCN) Architecture (Jun 2009)
• Guaranteed QoS Synthesis for Admission Control with Shared Capacity (Feb 2006)
• Market Managed Multi-service Internet (M3I); Economics Driving Network Design (Oct 2002)
• Service Differentiation in Third Generation Mobile Networks (Oct 2002)
• Economic Models for Resource Control in Wireless Networks (Sep 2002)
  
• A Market Managed Multi-service Internet (M3I) (Feb 2003)
  
• Tariff Dissemination Protocol (Mar 2002)
  
• Market Managed Multi-service Internet (M3I): Architecture (Aug 2003)
  
• Market Managed Multi-service Internet (M3I): ISP Business Model Report; Prototype Descriptions (Feb 2002)
  
• Market Managed Multi-service Internet (M3I): ISP Business Model Report (Feb 2002)
• Market Managed Multi-service Internet (M3I): Charging and Accounting System (CAS) Design (Jul 2000)
  
• Market Managed Multi-service Internet (M3I): Pricing Mechanisms; Price Reaction Design (Jul 2000)
  
• Market Managed Multi-service Internet (M3I): Requirements Specification Reference Model (Jul 2000)
• Intelligent Client Based Charging Middleware (Jan 2000)
  
• The Direction of Value Flow in Open Multi-service Connectionless Networks (Nov 1999)
  
• Lightweight Policing and Charging for Packet Networks (Mar 2000)
  
• A Dynamic Pricing Framework to Support a Scalable, Usage-based Charging Model for Packet-switched Networks (Jun 1999)
  
• A charging model for Sessions on the Internet (Jul 1999)
  
• An End to End Price-Based QoS Control Component Using Reflective Java (Dec 1997)
• Pre-Congestion Notification (PCN)
• Pre-Congestion Notification (PCN) Architecture (Jun 2009)
  (superseded An edge-to-edge Deployment Model for Pre-Congestion Notification: Admission Control over a DiffServ Region (Oct 2006))
  (previously know as A Framework for Admission Control over DiffServ using Pre-Congestion Notification;
  previously know as An architecture for edge-to-edge controlled load service using distributed measurement-based admission control)
• Pre-Congestion Notification (PCN) – New QoS Support for Differentiated Services IP Networks (Mar 2012)
• A Survey of PCN-Based Admission Control and Flow Termination (Jul 2009)
• How to Build a Virtual Queue from Two Leaky Buckets (and why one is not enough) (Apr 2012)
• Baseline Encoding and Transport of Pre-Congestion Information (Nov 2009)
• Encoding and Transport of (Pre-)Congestion Information from Within a DiffServ Domain to the Egress (May 2008)
• Overview of Pre-Congestion Notification Encoding (Jul 2012)
  (previously known as Pre-Congestion Notification Encoding Comparison)
• Encoding Three Pre-Congestion Notification (PCN) States in the IP Header using a Single Diffserv Codepoint (DSCP) (Jul 2012)
• A PCN Encoding Using 2 DSCPs to Provide 3 or More States (Mar 2012)
  (previously known as A three state extended PCN encoding scheme)
• PCN Encoding for Packet-Specific Dual Marking (PSDM) (Mar 2012)
  
• Emulating Border Flow Policing Using Re-PCN on Bulk Data (Oct 2009)
  previously known as Emulating Border Flow Policing using Re-ECN on Bulk Data)
• Metering and Marking behaviour of PCN-nodes (Nov 2009)
  (superseded Pre-congestion Notification Marking (Oct 2006))
  (previously know as The Controlled Load per hop behaviour)
• RSVP Extensions for Admission Control over Diffserv Using Pre-Congestion Notification (Jun 2006)
• Guaranteed QoS Synthesis for Admission Control with Shared Capacity (Feb 2006)
• Guaranteed QoS Synthesis - an example of a scalable core IP quality of service solution (Apr 2005)
  
• Guaranteed QoS Synthesis (GQS): Outline (Oct 2002)

Virtualisation Security

• Network Functions Virtualisation (NFV); Security Problem Statement (Oct 2014)
• Network Performance Isolation in Data Centres using Congestion Policing (Feb 2014)

Denial of Service Resistance

• Using Self-interest to Prevent Malice; Fixing the Denial of Service Flaw of the Internet (Oct 2006)
• Policing Congestion Response in an Inter-Network Using Re-Feedback (Sep 2005)
• The TCP Echo and TCP Echo Reply Options (Aug 2015)
• The Echo Cookie TCP Option (Oct 2014)
• A TCP Test to Allow Senders to Identify Receiver Non-Compliance (Jul 2014)
• Byte and Packet Congestion Notification (Feb 2014)
• Grand Strategy - Rationale; towards a Denial of Service Resistant Internet (Unfinished) (Nov 2005)

Charging & security: Multicast streams

• Timed Efficient Stream Loss-Tolerant Authentication (TESLA): Multicast Source Authentication Transform Introduction (Jun 2005)
  
• MARKS: Zero Side-Effect Multicast Key Management using Arbitrarily Revealed Key Sequences (Nov 1999)
  
• Nark: Receiver-based Multicast Non-repudiation and Key Management (Nov 1999)
  

Loosely coupled distributed object systems & multicast

• The Implications of Pervasive Computing on Network Design (Jul 2004)
  
• GAP: The Generic Announcement Protocol for Event Messaging (Sep 2003)
• End to End Aggregation of Multicast Addresses (Nov 1997)
  
• Taxonomy of Communications Requirements for Large-scale Multicast Applications (Dec 1997)
  
• Distributed Objects on the Web (Apr 1997)
  
• Service Discovery in Massive Scale Federations - The Web Analysed in Open Distributed Processing Terms (Mar 1996)
  

Most recent papers first
(based on date of the first published version).

---

"Implementing the `TCP Prague' Requirements for Low Latency Low Loss Scalable Throughput (L4S)," Bob Briscoe (Independent), Koen De Schepper (Nokia Bell-Labs), Olga Albisser (Simula), Joakim Misund (Uni Oslo), Olivier Tilmans (Nokia Bell-Labs), Mirja Kühlewind (ETH Zurich) and Asad Sajjad Ahmed (Simula), in Proc. Netdev 0x13 (Mar 2019). (11pp, 8 Figs, 23 refs) [BibTeX]

Presentations:  [ netdev0x13 (TBA) ]

Abstract: This paper is about a new approach for ultra-low queuing delay for all Internet traffic. 'Ultra-low queuing' means a few hundred microseconds of queuing delay on average, and about 1ms at the 99th percentile. This is 10 times better than state-of-the-art AQMs (FQ-CoDel and PIE). And all traffic means not just low rate VoIP, gaming, etc. but also capacity-seeking TCP-like applications. It is achieved by addressing the root cause of the problem - the congestion controller at the source.

The solution is to use one of the family of 'scalable' congestion controls. The talk will explain what that means, but for now it will suffice to say that Data Centre TCP (DCTCP) is an example. But there are other examples, including a scalable congestion control for real-time media. So the task has been to make it possible and safe to incrementally deploy congestion controls like DCTCP over the public Internet.

The whole architecture is called Low Latency Low Loss Scalable throughput (L4S). Although this all sounds rather grandiose, it actually only involves a few incremental changes to code in hosts and network nodes. You will recognise some of these, because most are desirable in themselves, and already in progress (e.g. Accurate ECN and RACK). However, focusing on all the parts loses sight of the sum. So this paper starts by explaining the sum of the parts - the bigger motivation for all the changes together. Then the body of the paper dives into the changes to DCTCP needed to make it safely coexist with other traffic, and to improve performance when used over the Internet - to satisfy the `Prague L4S Requirements'. The resulting TCP implementation is called TCP Prague.

---

"DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM," Olga Albisser (Simula), Koen De Schepper (Nokia Bell-Labs), Bob Briscoe (Independent), Olivier Tilmans (Nokia Bell-Labs) and Henrik Steen (Simula), in Proc. Netdev 0x13 (Mar 2019). (8pp, 3 Figs, 15 refs) [BibTeX]

Presentations:  [ netdev0x13 (TBA) ]

Abstract: We implemented the DualPI2 AQM as a Linux qdisc and present the details of DUALPI2 implementation, explaining which problems it can solve.

Traditionally, classic loss-based congestion controls like Cubic or Reno need a relatively large queue to utilize the network efficiently and with reasonable loss levels. These large queues introduce unnecessary delay. Like in Data Centers, the problem can be partially solved by using scalable congestion controls that can use shallow immediate ECN marking thresholds instead of loss as congestion signal. But then, another problem is introduced - classic and scalable TCP congestion controls are not able to coexist without classic TCP starving itself. This starvation occurs due to the difference in how classic and scalable congestion controls respond to congestion signals.We explain how DUALPI2 solves this problem and can be used to mix Internet and Datacenter TCP traffic both on the Internet and in Data Centers, without compromising on neither the ultra-low latency performance of DCTCP, nor the Reno/Cubic throughput performance. DualPI2 uses a scheduler with only 2 queues, coupled for fairness, and a classifier that uses only IP header inspection, being transport protocol independent (supporting TCP, SCTP, QUIC).

The largest benefit of our solution is the ability to deploy congestion controls like DCTCP on the public Internet and allow a mix of DCTCP and Internet congestion controls in the Data Center. The standardization of the DualPI2 AQM and the way the Low Latency, Low Loss and Scalable traffic (L4S) is identified, are being finalized in the IETF.

---

"Paced Chirping - Rethinking TCP start-up," Joakim Misund (Uni Oslo) and Bob Briscoe (Independent), in Proc. Netdev 0x13 (Mar 2019). (7pp, 5 Figs, 6 refs) [BibTeX]

Presentations:  [ netdev0x13 (TBA) ]

Abstract: Paced Chirping is a replacement of slow start that continuously pulses the network with groups of packets called 'chirps'. The decreasing inter-packet gap of the chirps allows the sender to estimate the available capacity from queueing delay measurements. Each chirp creates a small measurable queue, and the queue is allowed to drain in-between subsequent chirps. Using the capacity estimates Paced Chirping transition to congestion avoidance without overshooting capacity.

Paced Chirping is implemented in Linux using the kernels pacing framework to realise chirps. We have added code to the pacing framework that allows a congestion control module to create chirps with desired characteristics. This does allow for more elaborate start-up schemes, made necessary by increasing capacity and the need for lower latency. Pacing precision is a challenge at high speed due to timing precision. With the proposed NIC API change essayed by Van Jacobson at netdev 0x12 precision can be improved significantly.

Paced Chirping achieves fast acceleration with extremely low maximum queueing delay (couple of milliseconds), and it is a promising step towards a start-up algorithm that can reach utilization fast without hurting latency-sensitive applications. In the acceleration world, Cubic (without hystart) is best and DCTCP is worst. In the delay overshoot world Cubic is worst and DCTCP is best. Paced Chirping is as good as the best in both worlds. The best acceleration of today's schemes gives the worst overshoot and the best overshoot gives the worst acceleration. With paced chirping, the acceleration and the overshoot are as good as the best of both worlds.

---

"Paced Chirping: Rapid flow start with very low queuing delay," Joakim Misund (Uni Oslo) and Bob Briscoe (Independent), in Proc. IEEE Global Internet Symposium (May 2019). (7pp, 7 Fig, 20 refs) [BibTeX]

Presentations:  [ ]

Abstract: This paper introduces a promising new direction for getting a traffic flow up to speed fast while keeping the maximum queuing delay that the new flow adds extremely low. It is therefore most interesting in environments where queue delay is already fairly low. Nonetheless, it requires no special network infrastructure, being solely delay-based, so it ought to be applicable to the general Internet.

Received wisdom from TCP slow-start is that the faster a flow accelerates, the more it will overshoot the queue before the sender will notice one round trip later. The proposed technique, called paced chirping. escapes that dilemma. The sender pulses the queue increasingly rapidly with trains of packets called `chirps' that it crafts to rapidly estimate available capacity. Critically, the sender relaxes the queue between chirps, so the queue never accumulates more than a few packets. Thus, paced chirping escapes the overshoot dilemma, but still pushes enough against any pre-existing flows, so they yield their capacity.

The algorithm has been implemented in Linux and shows great promise from initial evaluation. Work so far has set aside numerous issues that are all important, but not central to proving the concept, e.g. handling delayed ACKs, losses, ECN marking, reordering, variable rate links, etc,  The work and the code is being published at this stage to seek review and collaboration around this promising direction.
 

---

"Low Latency DOCSIS - Technology Overview",  Greg White, Karthik Sunderesan and Bob Briscoe (CableLabs) (Feb 2019). (21pp, 13 fig, 9 refs) [BibTeX]

Presentations: []

Abstract:  The evolution of the bandwidth capabilities—from kilobits per second to gigabits—across generations of DOCSIS cable broadband technology has paved the way for the applications that today form our digital lives. Along with increased bandwidth, or “speed,” the latency performance of DOCSIS technology has also improved in recent years. Although it often gets less attention, latency performance contributes as much or more to the broadband experience and the feasibility of future applications as does speed.  Now, as we are looking toward a “10G” future with a 10 gigabit per second broadband platform, we need to make a similar quantum leap in latency reduction—and that’s why we’re pioneering Low Latency DOCSIS technology.

Low Latency DOCSIS technology (LLD) is a specification developed by CableLabs in collaboration with DOCSIS vendors and cable operators that tackles the two main causes of latency in the network: queuing delay and media acquisition delay.  LLD introduces an approach wherein data traffic from applications that aren't causing latency can take a different logical path through the DOCSIS network without getting hung up behind data from applications that are causing latency, as is the case in today’s Internet architectures.  This mechanism doesn't interfere with the way applications share the total bandwidth of the connection, and it doesn't reduce one application's latency at the expense of others.  In addition, LLD improves the DOCSIS upstream media acquisition delay with a faster request-grant loop and a new proactive scheduling mechanism. LLD makes the internet experience better for latency sensitive applications without any negative impact on other applications.

The latest generation of DOCSIS that has been deployed in the field—DOCSIS 3.1—experiences typical latency performance of around 10 milliseconds (ms) on the Access Network link.  However, under heavy load, the link can experience delay spikes of 100 ms or more.  LLD systems can deliver a consistent 1 ms delay on the DOCSIS network for traffic that isn’t causing latency, imperceptible for nearly all applications. The experience will be more consistent with much smaller delay variation.

LLD can be deployed by field-upgrading DOCSIS 3.1 cable modem and cable modem termination system devices with new software.  The technology includes tools that enable automatic provisioning of these new services, and it also introduces new tools to report statistics of latency performance to the operator.

Cable operators, DOCSIS equipment manufacturers, and application providers will all have to act in order to take advantage of LLD.  This white paper explains the technology and describes the role that each of these parties plays in making LLD a reality.

---

"Low Latency DOCSIS - Technology Overview," Greg White, Karthik Sunderesan and Bob Briscoe (CableLabs), IETF Internet-Draft <draft-white-tsvwg-lld> (Mar 2019) (Work in Progress). (25pp, 0 figs, 12 refs) [BibTeX]

Presentations:  []

Abstract: NOTE: This document is a reformatted version of [LLD-white-paper] with a near-identical abstract.

---

"Interactions between Low Latency, Low Loss, Scalable Throughput (L4S) and Differentiated Services," Bob Briscoe (CableLabs), IETF Internet-Draft <draft-briscoe-tsvwg-l4s-diffserv> (Mar 2018) (Work in Progress). (17pp, 5 figs, 16 refs) [BibTeX]

Presentations:  [ IETF-101 ]

Abstract: L4S and Diffserv offer somewhat overlapping services (low latency and low loss), but bandwidth allocation is out of scope for L4S. Therefore there is scope for the two approaches to complement each other, but also to conflict.  This informational document explains how the two approaches interact, how they can be arranged to complement each other and in which cases one can stand alone without needing the other.

---

"Explicit Congestion Notification (ECN) and Congestion feedback; Using the Network Service Header (NSH) ," Donald Eastlake (Huawei), Bob Briscoe (Independent) and Andrew Malis (Huawei), IETF Internet-Draft <draft-ietf-sfc-nsh-ecn-support> (Feb 2019) (Work in Progress). (12pp, 3 figs, 7 refs) [BibTeX]

Differences between drafts: [ IETF document history ]

Presentations:  [ ]

Abstract: Explicit congestion notification (ECN) allows a forwarding element to notify downstream devices of the onset of congestion without having to drop packets. Coupled with a means to feed back information about congestion to upstream nodes, this can improve network efficiency through better congestion control, frequently without packet drops. This document specifies ECN and congestion feedback support within a Service Function Chaining (SFC) domain through use of the Network Service Header (NSH, RFC 8300) and IP Flow Information Export (IPFIX, draft-ietf-tsvwg-tunnel-congestion-feedback).

---

 
"Measuring ECN++: Good News for ++, Bad News for ECN over Mobile," Anna Mandalari, Andra Lutu (Simula), Bob Briscoe (Simula), Marcelo Bagnulo (UC3M) and Özgü Alay (Simula), IEEE Communications Magazine 56(3):180--186 (March 2018). (6pp, 1 Fig, 14 refs) [BibTeX]

[ pre-print | paper (access controlled) | IETF ECN++ Internet draft | dataset | code | ECN++ landing page ]

Presentations:  [ IETF-100 ]

Abstract: After ECN was first added to IP in 2001, it was hit by a succession of deployment problems. Studies in recent years have concluded that path traversal of ECN has become close to universal. In this article, we test whether the performance enhancement called ECN++ will face a similar deployment struggle as did base ECN. For this, we assess the feasibility of ECN++ deployment over mobile as well as fixed networks. In the process, we discover bad news for the base ECN protocol: contrary to accepted beliefs, more than half the mobile carriers we tested wipe the ECN field at the first upstream hop. All packets still get through, and congestion control still functions, just without the benefits of ECN. This throws into question whether previous studies used representative vantage points. This article also reports the good news that, wherever ECN gets through, we found no deployment problems for the “++” enhancement to ECN. The article includes the results of other in-depth tests that check whether servers that claim to support ECN actually respond correctly to explicit congestion feedback. Those interested can access the raw measurement data online.

---

Managing a Queue to a Soft Delay Target,  Bob Briscoe (Independent), bobbriscoe.net Technical Report TR-BB-2017-003; arXiv:1905.03095 [cs.NI] (Sep 2017). (3pp, 1 fig, 12 refs) [BibTeX]

Presentations: []

Abstract: This memo proposes to transplant the core idea of Curvy RED, the softened delay target, into AQMs that are better designed to deal with dynamics, such as PIE or PI2, but that suffer from the weakness of a fixed delay target.

---

The Native AQM for L4S Traffic,  Bob Briscoe (Independent), bobbriscoe.net Technical Report TR-BB-2017-002; arXiv:1904.07079 [cs.NI] (Sep 2017). (3pp, 1 fig, 12 refs) [BibTeX]

Presentations: []

Abstract: This memo focuses solely on the native AQM of Low Latency Low Loss Scalable throughput (L4S) traffic and proposes various improvements to the original step design. One motivation for DCTCP to use simple step marking was that it was possible to deploy it by merely configuring the RED implementations in existing hardware, albeit in an unexpected configuration by setting parameters to degenerate values. However, there is no longer any imperative to stick with the original DCTCP step-function design, because changes will be needed to implement the DualQ Coupled AQM anyway, and the requirements for L4S congestion controls are not yet set in stone either. This paper proposes gradient (ramp) marking and a virtual (a.k.a. phantom) queue. It provides a way to implement virtual queuing delay (a.k.a. virtual sojourn time).

---

Rapid Signalling of Queue Dynamics,  Bob Briscoe (Independent), bobbriscoe.net Technical Report TR-BB-2017-001; arXiv:1904.07044 [cs.NI] (Sep 2017). (8pp, 1 fig, 22 refs) [BibTeX]

Presentations: []

Abstract: Rather than directly considering the queuing delay of data, this memo focuses on reducing the delay that congestion signals experience within a queue management algorithm, which can be greater than the delay that the data itself experiences within the queue. Once the congestion signals are delayed, regulation of the load becomes more sloppy, and the queue tends to overshoot and undershoot more as a result, leading the data itself to experience greater peaks in queuing delay as well as intermittent under-utilization.

Where the service rate of a queue varies, it is preferable to measure the queue in units of time not bytes. However, the size of the queued backlog can be measured in bytes and signalled at the last instant as data leaves the queue, whereas measuring queuing delay introduces inherent delay. This paper proposes 'scaled sojourn time', which scales queuing delay by the ratio between the backlogs at dequeue and enqueue. This is equivalent to scaling the sojourn time by the ratio of the arrival and departure rates averaged over the time spent in the queue. The paper also proposes the removal of delays due to randomness in the signal encoding.

---

Resolving Tensions between Congestion Control Scaling Requirements,  Bob Briscoe (Simula) and Koen De Schepper (Nokia Bell Labs), Simula Technical Report TR-CS-2016-001; arXiv:1904.07605 [cs.NI] (Jul 2017). (10pp, 5 figs, 11 refs) [BibTeX]

Presentations: [ IRTF/ICCRG @IETF-98 | IRTF/ICCRG @IETF-99: (pt1 | pt2) ]

Abstract: Low Latency, Low Loss Scalable throughput (L4S) is being proposed as the new default Internet service. L4S can be considered as an `incrementally deployable clean-slate' for new Internet flow-rate control mechanisms. Because, for a brief period, researchers are free to develop host and network mechanisms in tandem, somewhat unconstrained by any pre-existing legacy.

Scaling requirements represent the main constraints on a clean-slate design space. This document confines its scope to the steady state. It aims to resolve the tensions between a number of apparently conflicting scalability requirements for L4S congestion controllers. It has been produced to inform and provide structure to the debate as researchers work towards pre-standardization consensus on this issue.

This work is important, because clean-slate opportunities like this arise only rarely and will only be available briefly---for roughly one year. The decisions we make now will tend to dirty the slate again, probably for many decades.

---

PI²: A Linearized AQM for both Classic and Scalable TCP, Koen De Schepper (Nokia Bell Labs), Olga Bondarenko (Simula), Ing-Jyh Tsang (Nokia Bell Labs), Bob Briscoe (Simula), Proc. ACM CONEXT 2016 (Dec 2016). (15pp, 20 figs, 32 refs) [BibTeX]

Presentations: [ CoNEXT'16 ]

Abstract:  This paper offers insight into why it has proved hard for a Proportional Integral (PI) controller to remain stable while controlling Classic TCP flows, such as TCP Reno and Cubic. The problem is the square root of the drop probability p in the Classic TCP throughput equation. By directly controlling p′, conceptually the square root of the Classic p, we make the PI-controlled variable proportional to the load, not the square of the load. Then, we post-process this control variable p′ into the required Classic drop probability by a simple squared drop decision mechanism (comparing p′ with two random numbers instead of one). This gives either not worse or better results than PIE AQM achieves, but without the need for all its corrective heuristics.

Additionally, with suitable packet classification, it is simple to extend our PI² AQM to support coexistence between Classic and Scalable congestion controls in the public Internet. Unlike a Classic congestion control, a Scalable congestion control ensures sufficient feedback at any flow rate, an example being Data Centre TCP (DCTCP). A Scalable congestion control has no square root, so we merely use p′ without squaring, by omitting the post-processing stage.

We implemented this PI² AQM as a Linux qdisc to extensively test our claims using Cubic (Classic) and DCTCP (Scalable).

---

ECN++: Adding Explicit Congestion Notification (ECN) to TCP control packets, Marcelo Bagnulo Braun (UC3M) and Bob Briscoe (CableLabs), IETF Internet-Draft <draft-ietf-tcpm-generalized-ecn> (Oct 2017) (Work in Progress). (33pp, 0 figs, 23 refs) [BibTeX]

Differences between drafts: [ IETF document history ]

Presentations:  [ IETF-100 (pt1 | pt2) | IETF-99 | IETF-97 | IETF-96 ]

Abstract:  This document describes an experimental modification to ECN when used with TCP.  It allows the use of ECN on the following TCP packets: SYNs, pure ACKs, Window probes, FINs, RSTs and retransmissions.

---

Propagating Explicit Congestion Notification Across IP Tunnel Headers Separated by a Shim, Bob Briscoe (Simula Research Laboratory), IETF Internet-Draft <draft-ietf-tsvwg-rfc6040update-shim> (Mar 2018) (Work in Progress). (21pp, 2 figs, 38 refs) [BibTeX]

Differences between drafts: [ IETF document history ]

Presentations: [ IETF-101 | IETF-100 | IETF-99 | IETF-96-tsvwg | IETF-96-intarea ]

Abstract:  RFC 6040 on "Tunnelling of Explicit Congestion Notification" made the rules for propagation of ECN consistent for all forms of IP in IP tunnel. This specification updates RFC 6040 to clarify that its scope includes tunnels where two IP headers are separated by at least one shim header that is not sufficient on its own for wide area packet forwarding. It surveys widely deployed IP tunnelling protocols separated by such shim header(s) and updates the specifications of those that do not mention ECN propagation (L2TPv2, L2TPv3, GRE and Teredo). This specification also updates RFC 6040 with configuration requirements needed to make any legacy tunnel ingress safe.

---

Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture, Bob Briscoe (CableLabs), Koen De Schepper (Nokia Bell Labs) and Marcelo Bagnulo (UC3M), IETF Internet-Draft <draft-ietf-tsvwg-l4s-arch> (Mar 2018) (Work in Progress). (32pp, 3 fig, 42 refs) [BibTeX]

Differences between drafts: [ IETF document history ]

Presentations: [  IETF-101 | IETF-100 | IETF-99 | Full slide-deck  | IETF-98| IETF-97 ]

Abstract: This document describes the L4S architecture for the provision of a new service that the Internet could provide to eventually replace best efforts for all traffic: Low Latency, Low Loss, Scalable throughput (L4S).  It is becoming common for all (or most) applications being run by a user at any one time to require low latency.  However, the only solution the IETF can offer for ultra-low queuing delay is Diffserv, which only favours a minority of packets at the expense of others.  In extensive testing the new L4S service keeps average queuing delay under a millisecond for all applications even under very heavy load, without sacrificing utilization; and it keeps congestion loss to zero.  It is becoming widely recognized that adding more access capacity gives diminishing returns, because latency is becoming the critical problem.  Even with a high capacity broadband access, the reduced latency of L4S remarkably and consistently improves performance under load for applications such as interactive video, conversational video, voice, Web, gaming, instant messaging, remote desktop and cloud-based apps (even when all being used at once over the same access link).  The insight is that the root cause of queuing delay is in TCP, not in the queue.  By fixing the sending TCP (and other transports) queuing latency becomes so much better than today that operators will want to deploy the network part of L4S to enable new products and services. Further, the network part is simple to deploy - incrementally with zero-config.  Both parts, sender and network, ensure coexistence with other legacy traffic.  At the same time L4S solves the long-recognized problem with the future scalability of TCP throughput.

This document describes the L4S architecture, briefly describing the different components and how the work together to provide the aforementioned enhanced Internet service.

Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Problem Statement, Bob Briscoe (Simula), Koen De Schepper (Nokia Bell Labs) and Marcelo Bagnulo (UC3M), IETF Internet-Draft <draft-briscoe-tsvwg-aqm-tcpm-rmcat-l4s-problem> (Jul 2016) (Expired). (27pp, 1 fig, 30 refs) [BibTeX]

Differences between drafts: [ IETF document history ]

Presentations: [ IETF-96 ]

Abstract: This document motivates a new service that the Internet could provide to eventually replace best efforts for all traffic: Low Latency, Low Loss, Scalable throughput (L4S).  It is becoming common for all (or most) applications being run by a user at any one time to require low latency.  However, the only solution the IETF can offer for ultra-low queuing delay is Diffserv, which only favours a minority of packets at the expense of others.  In extensive testing the new L4S service keeps average queuing delay under a millisecond for all applications even under very heavy load, without sacrificing utilization; and it keeps congestion loss to zero.  It is becoming widely recognized that adding more access capacity gives diminishing returns, because latency is becoming the critical problem.  Even with a high capacity broadband access, the reduced latency of L4S remarkably and consistently improves performance under load for applications such as interactive video, conversational video, voice, Web, gaming, instant messaging, remote desktop and cloud-based apps (even when all being used at once over the same access link).  The insight is that the root cause of queuing delay is in TCP, not in the queue.  By fixing the sending TCP (and other transports) queuing latency becomes so much better than today that operators will want to deploy the network part of L4S to enable new products and services. Further, the network part is simple to deploy - incrementally with zero-config.  Both parts, sender and network, ensure coexistence with other legacy traffic.  At the same time L4S solves the long-recognized problem with the future scalability of TCP throughput.

This document explains the underlying problems that have been preventing the Internet from enjoying such performance improvements. It then outlines the parts necessary for a solution and the steps that will be needed to standardize them.  It points out opportunities that will open up, and sets out some likely use-cases, including ultra-low latency interaction with cloud processing over the public Internet.

Abstract: Data Centre TCP (DCTCP) was designed to provide predictably low queuing latency, near-zero loss, and throughput scalability using explicit congestion notification (ECN) and an extremely simple marking behaviour on switches. However, DCTCP does not co-exist with existing TCP traffic---throughput starves. So, until now, DCTCP could only be deployed where a clean-slate environment could be arranged, such as in private data centres. This paper proposes `Coupled Active Queue Management (AQM)' to allow scalable congestion controls like DCTCP to safely co-exist with classic Internet traffic. In extensive tests within the edge gateway of a realistic broadband access testbed, the Coupled AQM ensures that a flow runs at about the same rate whether it uses DCTCP or TCP Reno/Cubic, but without inspecting transport layer flow identifiers. DCTCP achieves sub-millisecond average queuing delay and zero congestion loss under a wide range of mixes of DCTCP and `classic' broadband Internet traffic, without compromising the performance of the classic traffic. The solution also reduces network complexity and eliminates network configuration.

Abstract:  Active queue management (AQM) drops packets early in the growth of a queue, to prevent a capacity-seeking sender (e.g. TCP) from keeping the buffer full. An AQM can mark instead of dropping packets if they indicate support for explicit congestion notification (ECN). Two modern AQMs (PIE and CoDel) are designed to keep queuing delay to a target by dropping packets as load varies.

This memo uses Curvy RED and an idealised but sufficient model of TCP traffic to explain why attempting to keep delay constant is a bad idea, because it requires excessively high drop at high loads. This high drop itself takes over from queuing delay as the dominant cause of delay, particularly for short flows. At high load, a link is better able to preserve reasonable performance if the delay target is softened into a curve rather than a hard cap.

The analysis proves that the same AQM can be deployed in different parts of a network whatever the capacity with the same optimal configuration.

A surprising corollary of this analysis concerns cases with a highly aggregated number of flows through a bottleneck. Although aggregation reduces queue variation, if the target queuing delay of the AQM at that bottleneck is reduced to take advantage of this aggregation, TCP will still increase the loss level because of the reduction in round trip time. The way to resolve this dilemma is to overprovision (a formula is provided).

Nonetheless, for traffic with ECN enabled, there is no harm in an AQM holding queuing delay constant or configuring an AQM to take advantage of any reduced delay due to aggregation without over-provisioning. Recently, the requirement of the ECN standard that ECN must be treated the same as drop has been questioned. The insight that the goals of an AQM for drop and for ECN should be different proves that this doubt is justified.

Abstract:  This memo explains that deploying active queue management (AQM) to counter bufferbloat will not prevent TCP from overriding the AQM and building large queues in a range of not uncommon scenarios. This is a brief paper study to explain this effect which was observed in a number of low latency testbed experiments.
 
To keep its queue short, an AQM drops (or marks) packets to make the  TCP flow(s) traversing it reduce their packet rate. Nearly all TCP  implementations will not run at less than two packets per round trip  time (RTT). 2pkt / RTT need not imply low bit-rate if the RTT is small.  For instance, it represents 2Mb/s over a 6ms round trip. When a few TCP  flows share a link, in certain scenarios, including regular broadband  and data centres, no matter how much the AQM signals to the flows to  keep the queue short, they will not obey, because it is impossible for  them to run below this floor. The memo proposes the necessary  modification to the TCP standard.

Abstract:  The following summary of my concerns, can be used as a table of contents for the rest of the review:

1. Proposed Standard, but no normative language
1. work needed to distinguish between design intent and specific implementation
2. unclear how strongly the enhancements are recommended
2. Has PIE been separately tested with and without each enhancement, to justify each?
3. Needs to enumerate whether it satisfies each AQM Design Guideline
1. If not, say why or fix.
2. Particular concerns:
1. No spec of ECN behaviour
2. No autotuning of the two main parameters
3. Transport specific (Reno-based?) autotuning of α & β
4. Rationale for a PI controller not properly articulated
5. Technical flaws/concerns
1. Turning PIE off
2. `Autotuning' α & β parameters
3. Averaging problems
4. Burst allowance unnecessary?
5. Needs a Large Delay to Make the Delay Small
6. Derandomization: a waste of cycles
7. Bound drop probability at 100% → DoS vulnerability?
8. Avoiding Large Packet Lock-Out under Extreme Load.
6. Numerous magic numbers
1. ~20 constants, 13 of which are not in the header block.
2. About half ought to be made to depend on other constants
3. Need to state how to set the remaining constants for different environments
7. Implementation suggestion for Autotuning α & β 

Workshop on Reducing Internet Latency, 2013, with Mat Ford (Ed.) and others, in Proc. Workshop on Reducing Internet Latency, (Sep 2013). (14pp, 3 figs, 14 refs) [BibTeX]

Abstract:  Latency is increasingly becoming a performance bottleneck for Internet Protocol (IP) networks, but historically networks have been designed with aims of maximizing throughput and utilization. This article offers a broad survey of techniques aimed at tackling latency in the literature up to March 2014 and their merits. A goal of this work is to be able to quantify and compare the merits of the different Internet latency reducing techniques, contrasting their gains in delay reduction versus the pain required to implement and deploy them. We found that classifying techniques according to the sources of delay they alleviate provided the best insight into the following issues: 1) the structural arrangement of a network, such as placement of servers and suboptimal routes, can contribute significantly to latency; 2) each interaction between communicating endpoints adds a Round Trip Time (RTT) to latency, especially significant for short flows; 3) in addition to base propagation delay, several sources of delay accumulate along transmission paths, today intermittently dominated by queuing delays; 4) it takes time to sense and use available capacity, with overuse inflicting latency on other flows sharing the capacity; and 5) within end systems delay sources include operating system buffering, head-of-line blocking, and hardware interaction. No single source of delay dominates in all cases, and many of these sources are spasmodic and highly variable. Solutions addressing these sources often both reduce the overall latency and make it more predictable.

Index Terms: Data communication, networks, Internet, performance, protocols, algorithms, standards, cross-layer, comparative evaluation, taxonomy, congestion control, latency, queuing delay, bufferbloat

Internet Latency: Causes, Solutions and Trade-offs,  David A. Hayes (UiO), Ing-Jyh Tsang (Bell Labs), David Ros, Andreas Petlund (Simula) and Bob Briscoe (BT), In: Proc. EuCNC Special session on Latency (May 2015). (5pp, 5 figs, 16 refs) [BibTeX]

Abstract: This paper is a digest of [1], an extensive survey discussing the merits of over 300 techniques for reducing Internet latency. It gives a broad overview of the causes, solutions, and trade-offs involved in reducing latency in the Internet. The overview covers key sources of delays and proposed solutions: due to the structural arrangement of the network, how network end-points interact, along the end-to-end path, related to link capacities, within end-hosts, and those that address multiple sources. Trade-offs are discussed in terms of the latency reduction different techniques provide versus their deployability.

A Survey of Latency Reducing Techniques and their Merits, Bob Briscoe (BT), Anna Brunstrom (Karlstad Uni), David Ros (Institut Mines-Télécom / Télécom Bretagne), David Hayes (Uni Oslo), Andreas Petlund (Simula Research Labs), Ing-Jyh Tsang (Alcatel-Lucent Bell-Labs), Stein Gjessing (Uni Oslo), Gorry Fairhurst (Uni Aberdeen), In: Proc. ISOC/RITE Workshop on Reducing Internet Latency (Position paper) (Sep 2013). (2pp, 1 fig, 1 ref) [BibTeX]

Presentations: [ Latency Workshop Sep'13 ]

• to identify approaches that require less co-ordination between companies, between industries, between disciplines or between jurisdictions
• to identify gaps where such co-ordination is unavoidably necessary
  
• to identify approaches that are not worth pursuing. 

Pre-Congestion Notification (PCN) Architecture, Philip Eardley (BT) (Editor), IETF RFC <rfc5559.txt> (Jun 2009). (54pp, 56 refs, 4 figs) [BibTeX]

Differences between drafts: [ IETF document history ]

An edge-to-edge Deployment Model for Pre-Congestion Notification: Admission Control over a DiffServ Region (superseded by Pre-Congestion Notification Architecture), Bob Briscoe, Philip Eardley, David Songhurst (BT), Francois Le Faucheur, Anna Charny (Cisco), Jozef Babiarz, Kwok-Ho Chan, Stephen Dudley (Nortel), Georgios Karagiannis (Uni Twente), Attila Bader and Lars Westberg (Ericsson), Internet Engineering Task Force (IETF) Transport Services working group Internet Draft <draft-briscoe-tsvwg-cl-architecture-04.txt> (Oct 2006) (Expired) (63pp, 44 refs, 7 figs) [BibTeX]

Differences between drafts: [ 04-03 ]

Presentations: [ IETF-66 | IETF-65 | IETF-64 | IETF-63 ]

Pre-Congestion Notification marking (superseded by Metering and Marking behaviour of PCN-nodes), Bob Briscoe, Philip Eardley, David Songhurst (BT), Francois Le Faucheur, Anna Charny, Vassilis Liatsos (Cisco), Jozef Babiarz, Kwok-Ho Chan, Stephen Dudley (Nortel), Georgios Karagiannis (Uni Twente / Ericsson), Attila Bader and Lars Westberg (Ericsson), Internet Engineering Task Force (IETF) Transport Services working group Internet Draft <draft-briscoe-tsvwg-cl-phb-03.txt> (Expired) (Oct 2006) (48pp, 20 refs, 14 figs) [BibTeX]

Differences between drafts: [ 03-02 ]

Presentations: [ IETF-66 | IETF-65 | IETF-63 ]

Abstract:  We present and analyse an approach to service differentiation in third generation mobile networks based on Wideband CDMA, that exposes a new weighting parameter designed to reflect allocation of the congestible resource. The approach naturally takes into account the difference in resource scarcity for the uplink and downlink, because it is grounded on fundamental economic models for efficient utilization of resources in WCDMA. If required, discrete values of this weight parameter can be presented as different service classes. Finally we present numerical experiments demonstrating the effectiveness of our approach, and investigate its performance and transient behaviour under power control and signal quality estimation errors.
Keywords: resource usage, class-based, power control, congestion control.

Abstract:  We present a model based on congestion pricing for resource control in wireless CDMA networks carrying traffic streams that have fixed rate requirements, but can adapt their signal quality. Our model is based on resource usage in the uplink direction of CDMA networks, and does not differentiate users based on their distance from the base station. We compare our model with other economic models that have appeared in the literature, identifying their similarities and differences. Our investigations include the effects of a mobile's distance and the wireless network's load on the target signal quality, the transmission power and the user's benefit and charge.

Abstract:  In this paper, we describe our approach to managing quality of service (QoS) using pricing. We show how it is possible to synthesise network QoS in the end-systems along the lines of the end to end design principle, as one of many possible business models. We have: i) developed an architecture for market management; ii) invented new business models to test and demonstrate its flexibility; iii) implemented generic mechanisms that not only enable these models but also many others; iv) modelled selected features of the resulting systems & markets and v) conducted experiments on users to assess acceptability and the feasibility of the overall approach. Each of these aspects is outlined in brief overview, with numerous references to more
detailed work.

Abstract:  This draft describes a very flexible and efficient protocol for distributing price information (tariffs) inside an Internet Service Provider's management system and to its customers. It is designed to work with multiple QoS architectures, for example Intserv [2] and Diffserv [4]. It can also be used for dynamic pricing. It can use a number of different transport mechanisms, e.g. embedding tariff messages as policy objects in RSVP [9] messages. As tariffs can get very complex, it is possible but not necessary to send tariffs as code (e.g. Java).  The draft also contains clear definitions of tariff and related terms.

Abstract:  This document introduces Timed Efficient Stream Loss-tolerant Authentication (TESLA). TESLA allows all receivers to check the integrity and authenticate the source of each packet in multicast or broadcast data streams. TESLA requires no trust between receivers, uses low-cost operations per packet at both sender and receiver, can tolerate any level of loss without retransmissions, and requires no per-receiver state at the sender. TESLA can protect receivers against denial of service attacks in certain circumstances. Each receiver must be loosely time-synchronized with the source in order to verify messages, but otherwise receivers do not have to send any messages. TESLA alone cannot support non-repudiation of the data source to third parties.

This informational document is intended to assist in writing standardizable and secure specifications for protocols based on TESLA in different contexts.

Presentation

Abstract:  This document contributes to the effort to add explicit congestion notification (ECN) to IP. In the current effort to standardise ECN for TCP it is unavoidably necessary to standardise certain new aspects of IP. However, the IP aspects will not and cannot only be specific to TCP. We specify interaction with features of IP such as fragmentation, differentiated services, multicast forwarding, and a definition of the service offered to higher layer congestion control protocols. This document only concerns aspects related to the IP layer, but includes any aspects likely to be common to all higher layer protocols. Any specification of ECN support in higher layer protocols is expected to appear in a separate specification for each such protocol.

[Full report, on which the above paper is based]

(Remote M3I copy)Market Managed Multi-service Internet: Pricing Mechanisms; Price Reaction Design, Bob Briscoe, Konstantinos Damianakis and Jérôme Tassel (BT), Panayotis Antoniadis & George Stamoulis (Athens UEB), M3I Consortium Deliverable D3PtII, 10 Jul 2000 (24pp, 12 Figs, 19 refs) [BibTeX]

Presentation

Abstract:  Some applications only make sense within a very tightly bounded range of quality of service (QoS) from the network. Others are far more adaptive. For the former type of application, it is relatively easy to determine their QoS requirements. This document primarily concerns how to determine the QoS requirement of the latter type of application, given a tariff for
network quality of service. We also cover how an application might describe its policy for determining QoS with respect to price. This description can then be used as policy for another entity controlling QoS, whether a middleware function on the same host, or a remote application being communicated with through a protocol.

Abstract:  This document presents our architecture for a market managed multi-service Internet (M3I). As explained in Part I, the task is to encompass all multi-service Internet architectures simultaneously in one architecture. To avoid confusion, we therefore term these sub-architectures ‘service plans’. A service plan is a combination of a network control architecture and the business model used to offer it as a service. The architecture is open, not only encompassing current and proposed service plans, but also facilitating the creation of new ones.

This document is an ‘architecture’ not a ‘design’. We define architecture as a specification of:
• why certain functions are best provided separately;
• what service they each offer and what their interfaces are;
• information structures that will have common use across the system such as identifiers.

The M3I Architecture is delivered in two parts: Principles (Part I [8]) and Construction (Part II — this part). Part I is designed to be readable alone. It goes to the core of what a multi-service Internet is and extracts fundamental principles from this exercise. It then gives a high level overview of the building blocks described in this second part in order to describe sensible ways to put them together. In contrast, this second part cannot
be read alone — part I is an essential pre-requisite. This second part describes the construction kit of the architecture — the building blocks and their interfaces. Indeed, it will be seen that the building blocks are very rudimentary — much as the principles were very fundamental. Specification is high level, but relatively precise. Because the building blocks are so rudimentary, we also introduce various compositions of the building blocks into useful sub-systems — themselves building blocks, but molecular rather than atomic. Each building block has the types of its possible inputs and outputs
tied down, so that it is impossible to put the pieces together in illegal ways. However, what is legal is not always sensible, hence the need for the principles in Part I.

Abstract:  In this paper we describe an intelligent, client based charging middleware that can be used to enable customers' self policing over the access of network resources in a multi service network like Internet (and ultimately higher level services). By intelligence we mean the charging software, data stores and the human or agent based reaction to dynamic pricing. The middleware components described in this paper are part of the MWARE client based middleware being researched and developed at the BT Advanced Communications Technology Centre [mware].

Abstract:  In this paper we present a novel scalable accounting infrastructure to support charging for network usage and Quality of Service (QoS) in an Internet context. Measurement and accounting are two core processes of a charging mechanism that make heavy demand on resources. They reduce the resource available for the core processes of the network i.e. routing and forwarding packets. Increasing the processing power and data storage capacity of the affected network elements lead to an increment in the cost of the network. The underlying principle of the infrastructure we propose is the transfer of these processes onto the edge systems to reduce their impact on the cost of the network. The measurement, accounting, applying pricing and billing processes and related sets of data are relocated on the edge systems of the network while allowing the provider to retain control over them. This paper focuses on the measurement and accounting aspects of this infrastructure. To achieve scalability we propose not to meter all the data or QoS control packet but only samples of them. We also discuss which controls both users and network providers could desire over the relocated processes. Early implementation of this work is introduced as a practical example of the concepts we present.

MARKS: Zero Side-Effect Multicast Key Management using Arbitrarily Revealed Key Sequences, Bob Briscoe (BT), in Proc First International Workshop on Networked Group Communication (NGC'99) LNCS 1736:301--319 pub. Springer-Verlag, Pisa, Italy  (17-20 Nov 1999) (13pp, 3 figs, 20 refs) [BibTeX]

Presentation

Abstract:  The goal of this work is to separately control individual secure sessions between unlimited pairs of multicast receivers and senders. At the same time, the solution given preserves the scalability of receiver initiated Internet multicast for the data transfer itself. Unlike other multicast key management solutions, there are absolutely no side effects on other receivers when a single receiver joins or leaves a session and no smartcards are required. The cost per receiver-session is typically just one short set-up message exchange with a key manager. Key managers can be replicated without limit because they are only loosely coupled to the senders who can remainoblivious to members being added or removed. The technique is a general solution for access to an arbitrary sub-range of a sequence of information and for its revocation,as long as the end of each sub-range can be planned at the time each access is requested.

Keywords: Multicast, Group Key management, Internet.

[A full length report describing five variations on the solution to the problem and a mathematical model encompassing them all. The above NGC'99 paper is a brief extract describing one solution]

Abstract:  The goal of this work is to separately control individual secure sessions between unlimited pairs of multicast receivers and senders. At the same time, the solution given preserves the scalability of receiver initiated Internet multicast for the data transfer itself. Unlike other multicast key management solutions, there are absolutely no side effects on other receivers when a single receiver joins or leaves a session and no smartcards are required. Solutions are presented for single and for multi-sender multicast. Further, we show how each receiver's data can be subject to an individual, watermarked audit trail. The cost per receiver-session is typically just one short set-up message exchange with a key manager. Key managers can be replicated without limit because they are only loosely coupled to the senders who can remain oblivious to members being added or removed. The technique is a general solution for access to an arbitrary sub-range of a sequence of information and for its revocation, as long as each session end can be planned at the time each access is requested. It might therefore also be appropriate for virtual private networks or for information distribution on other duplicated media such as DVD.

Presentation

Abstract:  The goal of this work is to separately control individual secure sessions between unlimited pairs of multicast receivers and senders while preserving the scalability of receiver initiated Internet multicast for the data transfer itself. Unlike other secure multicast solutions, there are absolutely no side-effects on other receivers when a single receiver joins or leaves a session. Each individual receiver can also reliably prove whether any fragment of the data hasn't been delivered or wasn't delivered on time (e.g. late video frames). Further, each receiver's data can be subject to an individual, watermarked audit trail. The cost per receiver-session is typically just one set-up message exchange with a key manager. Key managers can be replicated without limit because they are only loosely coupled to the senders who can remain oblivious to members being added or removed. The solution requires a tamper-resistant processor such as a smartcard at each receiver. However, generic cards supplied by a trusted third party are used rather than cards specific to each information provider. The technique can be applied to other bulk data distribution channels instead of multicast, such as DVD.

Keywords: Multicast, Non-repudiation, Key management, Smartcard, Watermark, Audit trail, Internet

[Full report, on which the above paper is based]

The Direction of Value Flow in Open Multi-service Connectionless Networks, Bob Briscoe (BT), BT Technical Report TR-NZG12-2000-001 (20 Aug 2000) (20pp (inc 6pp appendices), 7 figs, 24 refs) [BibTeX]

Abstract:  [The union of the two papers below]

The Direction of Value Flow in Connectionless Networks, Bob Briscoe (BT & UCL), invited paper, In Proc. First International Workshop on Networked Group Communication (NGC'99) (LNCS pub. Springer-Verlag ), Pisa, Italy (17-20 Nov 1999) (17pp (inc 3pp appendices), 7 figs, 23 refs) [BibTeX]

Presentation

Abstract:  [A modified version of the ICTEC'99 paper below (with just a summary of the maths, but highlighting the multicast aspects of the work)].

The Direction of Value Flow in Multi-service Connectionless Networks, Bob Briscoe (BT & UCL), In Proc. Second International Conference on Telecommunications and Electronic Commerce (ICTEC'99), Nashville, TN, US, (6-8 Oct 1999) (19pp (inc 6pp appendices), 7 figs, 22 refs) [BibTeX]

Presentation
 

Abstract:  This paper argues that, for scalability, all network providers in a connectionless multi-service network should offer each class of their service to each neighbour for each direction at a single price. This is called 'split-edge pricing'. To preserve scalability, if sets of customers wish to reapportion their networking charges between themselves, this should be tackled end-to-end. Edge reapportionment should not be muddled with networking charges, as is the case in the telephony market. Avoiding the telephony approach is shown to offer full reapportionment flexibility, but avoids the otherwise inevitable network complexity. 'Split-edge pricing' is recursive, applying as much to relationships between providers as to edge-customers. Various scenarios are discussed, showing the advantage of the approach. These include phone to Internet gateways and even inter-domain multicast conferences with heterogeneous QoS. The business model analysis suggests a new, purely financial role of end-to-end intermediary in the Internet industry.

Keywords: Charging, pricing, end-to-end, clearing, multicast, Internet, business models.

Abstract:  The underlying objective of the work presented in this paper is to create an active multi-service network which uses pricing to manage supply and demand of resources. The paper describes a dynamic pricing framework designed to support a radical approach to usage-based charging for packet-switched networks. This approach addresses the scalability problem normally associated with usage-based charging by shifting responsibility for accounting over to customer systems, which are also assigned the task of applying tariffs to create a bill.

In this context, the role of the dynamic pricing framework is to enable a provider to establish `active tariffs' and communicate them to customer systems. These tariffs take the form of mobile code for maximum flexibility, and the framework uses an auditing process to provide a level of protection against incorrect execution of this code on customer systems. In contrast to many active networks proposals, the processing load is moved away from routers to the edge of the network.

Keywords: usage-based charging, mobile code, self-billing, pricing, resource management.

Presentation

Abstract:  This paper suggests that a multi-service packet network might be achieved by adding classification and scheduling to routers, but not policing. Instead, a lightweight, packet-granularity charging system is presented that emulates a highly open policing function and is completely separated from the data path. A high proportion of charging operations runs on customer systems to achieve this, the proportion being configurable per-customer. Functions dispersible to customers include not only metering, accounting and billing but also per-packet or per-flow policing and admission control. Lower cost is achieved through simplicity without sacrificing commercial flexibility or security. Inter-provider charging, multicast charging and open bundling of network charges with those for higher class services are all catered for within the same, simple design. The paper is primarily architectural, referring to supporting papers for reports of early implementation experience in an Internet context.

Keywords: Charging, pricing, congestion control, quality of service, policing, operational support, active networks, Internet.

Abstract:  A chargeable session on the Internet may consist of more than one underlying chargeable service.  Typically there will be two, one at the network layer and one at the session layer.  Since different applications can have different demands from the Network, a generic charging scheme has to separate the service provided by the network from the service provided by an application/service provider.
In this paper we propose a pricing model which is session based and we look at the impact of this on  real-time multimedia conferencing over the Internet. In this model, we are trying to allow for the optional integration of charging at the network layer with charging at the session layer, while keeping the underlying technologies still cleanly apart. This paper also highlights the fact that the main problem of pricing application on the Internet is not just a simple case of analyzing the most technically feasible pricing mechanism but also making the solution acceptable to users.  We take the position that session based pricing is easier for end users to accept and understand and show why this is the case in this paper.

Abstract:  [near-identical to the above ISCC'99 paper]

End to End Aggregation of Multicast Addresses, Bob Briscoe & Martin Tatham (BT), 21 Nov 1997, BT Technical Report TR-NAA12-1997-001 (source of the above Internet Draft) [BibTeX]

Abstract:  This paper presents an approach for solving the inherent problem with multicast routing scalability - by co-operation between end-systems and the network. We introduce an extremely efficient, elegant way to name arbitrary sized inter-meshed aggregations of multicast addresses. This is done in such a way that it is easy to calculate how to change the name to encompass many more related names. We describe how these aggregate names could be used anywhere in place of the set of addresses to which they refer, not by resolving them into multiple operations, but by a single bulk action throughout the routing tree, and in session descriptions potentially including those for reservations. Initial aggregation in end-systems might only reduce the problem by an order of magnitude, but it is believed that this will provide sufficient structure for routers to be able to recognise further aggregation potential.  To improve the chances of router aggregation, address set allocation schemes must fulfil certain criteria that are laid down in this paper.

Presentation

Abstract:  The main objective of the model we describe in this paper is to allow easy, flexible addition of quality of service (QoS) control to Java Internet applications. In this work the QoS is expressed in terms of network and host resources, the network QoS being controlled with RSVP. Flexibility is provided by a prototype product from the ANSA research consortium; Reflective Java which uses the Meta Object Protocol (MOP) to separate functional requirements (what the application does) from non-functional requirements (how it does it). This protocol permits the design and implementation of a generic QoS control element which can be added to an application for which QoS control is required. Alternatively, an existing application with rudimentary QoS control can be modified to use a set of QoS control classes designed by a specialist intended to reconcile competition for QoS between applications. The QoS control element we have designed also has scope for QoS adaptation, moving decisions on the introduction of QoS control from build-time to run-time when best-effort degrades below a useful point. Charging is also considered in this work.

Presentation to IETF LSMA working group, Pete Bagnall, Munich, Aug 1997, Washington, Dec 1997 [broken link - presentation lost]

Abstract:  The intention of this memo is to define a classification system for the communication requirements of any large-scale multicast application (LSMA). It is very unlikely one protocol can achieve a compromise between the diverse requirements of all the parties involved in any LSMA. It is therefore necessary to understand the worst-case scenarios in order to minimize the range of protocols needed. Dynamic protocol adaptation is likely to be necessary which will require logic to map particular combinations of requirements to particular mechanisms.  Standardizing the way that applications define their requirements is a necessary step towards this. Classification is a first step towards standardization.

Abstract:  Various distributed object technologies have traditionally been seen as necessary to protect us from the uncertainties of a world where there is a perpetual state of partial failure. The World-Wide Web is the second largest distributed system in the world, behind only the telephone network which has far simpler ambitions. This paper discusses various approaches to the task of integrating the Web with more deterministic distributed object technologies to create islands of reliability (or to add other specific capabilities) without compromising the global scale of the Web. However, it is dangerous to take the view that a globally popular system such as the Web wasn’t designed correctly. The paper goes on to explore the essence of the Web’s success and discusses whether other distributed object systems would benefit from being less obsessed with deterministic behaviour.

Abstract:  The marriage between distributed object technology (the real men) and Web technology (the earth mothers) has been announced. From the groom's point of view, preparations for the marriage will be complete once he's taught the bride how to speak IIOP, learnt a bit of HTTP to make her comfortable and finished work on some important object services in the shed at the bottom of the OMG. However, by some mysterious organic process, she has amassed knowledge of all things. Whatever question he has, he must discover the spell that will draw out the answer. He needs to get in touch with her inner feelings. Consummation will be delayed until this chauvinistic gap is breached.

It is fashionable to criticise how well the Web achieves this or that goal then invent a "proper" service to do it better. It is tempting to suppose that technology for structured applications from the traditions of the distributed object world should be inserted into the Web on the day of their marriage. This paper aims to show that, in the field of service discovery, the Web deserves a long hard look before we click on the button marked "Fixed in the Next Release". As such, no new technology is proposed, merely some optimistic thoughts leading to the insight that the Web is a "proper" system for resource discovery (well, nearly).