FileCatalyst Acceleration: Solving the problems of TCP

Starting Points: » The Problem with TCP | » Do you need Acceleration? | » Other File Transfer Issues

The FileCatalyst fast file transfer protocol offers incredible speed gains vs. traditional methods. FTP and HTTP are based on TCP/IP, the backbone of the internet. For loading a webpage, TCP is great. For sustained data transfer, it is far from ideal.

Have a look at the following chart. Even though the connection itself is 45Mbps, transfers are much slower. With fast connections, latency and packet loss bring TCP-based file transfer to a crawl.

 

 

The Problem with TCP

The time it takes to send a set of packets and receive an acknowledgment is called the Round Trip Time (RTT) or simply the “latency”. TCP is serial in nature, meaning that the next ordered set won't be delivered until the previous one is acknowledged. This creates a period of “dead air”. The higher the latency, the more dead air.

Now imagine spending more time waiting for acknowledgments than sending data. The amount of data that can be sent before waiting for acknowledgment is governed by a mechanism called the TCP Sliding Window. The window closes aggressively when congestion is detected, and opens to its maximum quite slowly. TCP's congestion detection throws many false positives, meaning that the window often closes without justification.

The resulting dead air means that with TCP, your practical transfer rate can be orders of magnitude slower than your line speed. Assuming packet loss of 0.1% and a line speed of at least 10Mbps:

  • Once you hit 50ms of latency, your transfer speed starts to take a major hit.
  • By 80 ms, your maximum speed is 2500 Kbps—even for links 1Gbps and faster.
  • By 400ms—not uncommon for satellite connections—you are crippled at 500Kbps.

If you know your own line speed and estimated latencies, you can use the “FileCatalyst vs. FTP” tool in the sidebar to see if FileCatalyst acceleration can benefit you.

How FileCatalyst Acceleration Works

Back to the top

The UDP protocol provides a mechanism by which data can be transmitted at precise rates without being impeded by network impairments such as latency and packet loss. However UDP does not have any means of recovering from lost packets. Because of this, in the past there was no way to take advantage of the UDP protocol for reliable transfer of files over a network with impairments. FileCatalyst is an application layer protocol that adds a layer of reliability and rate control while not sacrificing the other desirable properties of UDP.

Similar to TCP, the FileCatalyst implementation will break data into blocks (analogous to the TCP window) although there is no limit to the size of the blocks in FileCatalyst. These blocks are transmitted to a receiver application and are written to disk. The major difference between FileCatalyst and TCP is that there is no delay while waiting for receipt of a block of data before commencement of subsequent blocks of data. Transmission of subsequent blocks is initiated immediately even when previous blocks have not yet been acknowledged. Regardless of network latency, data transmission remains constant, thus overcoming the peak and valley effect that occurs with TCP based protocols on links with high latency.

While the subsequent block is being transmitted, the FileCatalyst implementation awaits either acknowledgment of the previous block, or a list of packets that are missing for the previous block. Missing packets are re-transmitted concurrently with new data being sent for subsequent blocks until the acknowledgment is finally received. The flow of data remains constant even when there is a large number of missing packets. This process repeats until the entire file is transferred.