Introduction

Beginning with the introductory article on Acceleration, we have been discussing a few of the ways a software-only solution can accelerate file transfer:

1. Minimizing Data with Compression
2. Mininmizing Data by cutting redundant data (for example, with file Deltas)
3. Optimizing data transmission with Multiple Streams

In the fourth and final article on the subject, we are going to talk a bit about the technology that lies at the core of the FileCatalyst accelerated transfer solution: using an alternative to FTP or other TCP-based systems.

The problem with FTP

As discussed in earlier articles, the problem with FTP is its approach to reliability. From the FileCatalyst whitepaper, “Accelerating File Transfers”:

TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. …TCP has limits on the size of this value, so when the bandwidth‐delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”.

In layman’s terms, by trying its best to be reliable, TCP can really throttle down the speed of transfers over poor networks. Since it is a layer on top of TCP, this problem extends to FTP.

Solution: Fast File Transfer with UDP

One of the other core protocols on the internet, UDP, is by its nature NOT reliable. Packets are blasted out without concern for reliability or integrity. While TCP sends data sequentially, UDP is not concerned if the receiver gets packets in order. The benefit of this is that data being sent over UDP has the ability to “fill the pipe.” With no TCP window throttling the transmission, and with no wasted time for millions of acknowledgements, UDP can transmit data at full line speed.

“But wait,” you’re thinking, “I WANT reliability!”

Naturally, a file transfer solution is nothing if the files do not arrive intact.  To overcome the shortcomings of UDP, FileCatalyst implements application layer retransmission and congestion control algorithms which means it takes care of missing packets at the same time it sends new data, and it will not overwhelm your network.  All of the transfer parameters are under control by the end user, meaning that transfers may be tuned for a specific set of conditions such as high latency or high packet loss.  The end result is a file transmission mechanism that can both “fill the pipe” and guarantee delivery of every packet.

The FileCatalyst product suite implements a host of additional reliability features which will resume and retry interrupted connections, resend missing data, and perform MD5 checksums. The reliability built into a FileCatalyst transfer is far more robust than FTP.

Conclusion

Using a protocol such as UDP opens up the possibility to optimize transfers by using all your available bandwidth. Developers also have the unique opportunity to add superior reliability features. The financial benefits of faster file transfers are a subject for another article, but suffice it to say that if you can transfer files in 20 minutes instead of 16 hours, you will quickly recover any initial cost in terms of productivity hours, turnaround, and customer satisfaction (which can result in growth of customer base).

Coming soon: a closer look at specific FileCatalyst solutions, and tips on how existing FileCatalyst deployments can be optimized.

Yes, it is a cliché but seriously, you need FileCatalyst and you probably don’t even know it. I am obviously biased since I wrote a lot of the back end to FileCatalyst and have been “drinking my own kool-aid” for a while now. But haven’t you ever bought something and then a few months later you think back and wonder how you ever lived without it? Now I am not saying this will be the case for everyone with FileCatalyst, but I can certainly tell you in which scenarios FileCatalyst would make you ask that very question:

First scenario: You are an IT person, and you maintain the company’s file transfer system. The company is a multi-national, with branches all around the world. You have to make sure mission critical files make it from A to B, and make sure it happens as quickly as possible. You have a lot of issues with network interruptions, and then you need to resend the files. Or even worse, the interruptions leave you with corrupt files.

Second scenario: you are once again… an IT person, but you only have to move a few files between locations. However, your files are several GB in size—maybe even hundreds of GB. Your company has spent a lot on a high speed internet link, but for some reason the transfers are still slow. Not only that, but you still experience the odd disconnect here and there, and lose several hours of time because you need to start the transfer over again.

If this sounds like you, you have to try FileCatalyst because you could save yourself and your company a lot of time and a lot of money. First of all, FileCatalyst’s speed is second to none. If you want to maximize that high-speed link your company is paying for (or considering purchasing) you need a product like FileCatalyst to get the full return. Trying to use FTP as a transfer solution will never let you use the full bandwidth the way FileCatalyst will.

When it comes to reliability, FileCatalyst also leads the way with its robust retry and checksum features. It will hammer those files across the network in the minimum possible time, even with a ridiculous number of dropped connections.

Usually, you have to realize you have a problem first before even looking for a product like FileCatalyst. Maybe you’re resigned to the mistaken idea that your current solution is the only one out there. If I’ve helped you see that you might need an alternative like FileCatalyst, why not take it for a spin? Download our entry-level offering to jump right in, or better yet contact us to start a trial of any FileCatalyst product.

Chris

When people first find out about FileCatalyst, one of the first questions asked is always, “How does your product speed up file transfer?” often followed by a best guess, “Is it some sort of compression?”

Although the deepest possible answer is going to be somewhat complicated, there are some very basic principles at play in the world of file transfer acceleration, with compression being just one. Almost all products claiming to provide fast file transfer will do one or more of the following:

1. compress data
2. minimized data sets (other than compression)
3. transmit over multiple streams
4. use an alternate to FTP

The first two are considered fairly standard for the industry. The third is the domain of specialized applications such as FileCatalyst, and the fourth is territory wide open to the creativity of developers.

Over the next several days, we will explore each of these; we will elaborate, discuss realistic expectations to have, and how they factor into the FileCatalyst family. In the meantime, here’s a brief look at each:

Compression

Using Zip, Rar, Gzip, or another compression algorithm, data sets are minimized. Compression algorithms find patterns and recreate the data using a shorter form. Just as it is quicker to write “five million zeroes in a row” than to actually type five million zeroes in a row, the number of bits needed to store the data is reduced. In order for the data to be used again, it must also be decoded (uncompressed) on the destination side.

Multiple Streams

By sending data across more than one stream at a time, there are fewer moments of “downtime”. While there is housekeeping occurring on one stream (in-between files being sent for example), the other stream(s) can still be actively sending data.

Alternate Protocols

FTP transfer occurs over TCP, a protocol with ordered delivery of packets. Since the order cannot be interrupted, latency and packet loss can cause it to slow to a crawl. Protocols such as UDP can blast out packets with disregard for order. However, UDP in and of itself does not reliably transfer data, which is why file transfer technologies based around UDP must also include their own retransmission and verification functionality.

Sending less data

Compression is one way of sending a smaller file than the original, but it is not the only way. For example, FileCatalyst offers an option to transfer file deltas, which are partial files which can be used to “patch” a file that has been sent previously. Sending a 4MB delta instead of an entire multi-GB file can save an absolutely staggering amount of time and bandwidth.

Keep your eyes on this space as we talk a bit more about these and other ways to achieve fast file transfer.

Cheers,
Greg