Accelerating File Transfer: Quicklinks
Overview | Part 1: Compression | Part 2: Minimizing Data Sets | Part 3: Multiple Streams | Part 4: Alternatives (like FileCatalyst!) to FTP/TCP
Beginning with the introductory article
on Acceleration, and continuing with the articles shown in the list above, we have been discussing a few of the ways a software-only solution can accelerate file transfer.
In the fourth and final article on the subject, we are going to talk a bit about the technology that lies at the core of the FileCatalyst accelerated transfer solution: using an alternative to FTP or other TCP-based systems.
The problem with FTP
As discussed in earlier articles, the problem with FTP is its approach to reliability. From the FileCatalyst whitepaper, “Accelerating File Transfers”:
TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. …TCP has limits on the size of this value, so when the bandwidth‐delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”.
In layman’s terms, by trying its best to be reliable, TCP can really throttle down the speed of transfers over poor networks. Since it is a layer on top of TCP, this problem extends to FTP.
Solution: Fast File Transfer with UDP
One of the other core protocols on the internet, UDP, is by its nature NOT reliable. Packets are blasted out without concern for reliability or integrity. While TCP sends data sequentially, UDP is not concerned if the receiver gets packets in order. The benefit of this is that data being sent over UDP has the ability to “fill the pipe.” With no TCP window throttling the transmission, and with no wasted time for millions of acknowledgements, UDP can transmit data at full line speed.
“But wait,” you’re thinking, “I WANT reliability!”
Naturally, a file transfer solution is nothing if the files do not arrive intact. To overcome the shortcomings of UDP, FileCatalyst implements application layer retransmission and congestion control algorithms which means it takes care of missing packets at the same time it sends new data, and it will not overwhelm your network. All of the transfer parameters are under control by the end user, meaning that transfers may be tuned for a specific set of conditions such as high latency or high packet loss. The end result is a file transmission mechanism that can both “fill the pipe” and guarantee delivery of every packet.
The FileCatalyst product suite implements a host of additional reliability features which will resume and retry interrupted connections, resend missing data, and perform MD5 checksums. The reliability built into a FileCatalyst transfer is far more robust than FTP.
Using a protocol such as UDP opens up the possibility to optimize transfers by using all your available bandwidth. Developers also have the unique opportunity to add superior reliability features. The financial benefits of faster file transfers are a subject for another article, but suffice it to say that if you can transfer files in 20 minutes instead of 16 hours, you will quickly recover any initial cost in terms of productivity hours, turnaround, and customer satisfaction (which can result in growth of customer base).
Coming soon: a closer look at specific FileCatalyst solutions, and tips on how existing FileCatalyst deployments can be optimized.