Introduction

Beginning with the introductory article on Acceleration, we have been discussing a few of the ways a software-only solution can accelerate file transfer:

1. Minimizing Data with Compression
2. Mininmizing Data by cutting redundant data (for example, with file Deltas)
3. Optimizing data transmission with Multiple Streams

In the fourth and final article on the subject, we are going to talk a bit about the technology that lies at the core of the FileCatalyst accelerated transfer solution: using an alternative to FTP or other TCP-based systems.

The problem with FTP

As discussed in earlier articles, the problem with FTP is its approach to reliability. From the FileCatalyst whitepaper, “Accelerating File Transfers”:

TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. …TCP has limits on the size of this value, so when the bandwidth‐delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”.

In layman’s terms, by trying its best to be reliable, TCP can really throttle down the speed of transfers over poor networks. Since it is a layer on top of TCP, this problem extends to FTP.

Solution: Fast File Transfer with UDP

One of the other core protocols on the internet, UDP, is by its nature NOT reliable. Packets are blasted out without concern for reliability or integrity. While TCP sends data sequentially, UDP is not concerned if the receiver gets packets in order. The benefit of this is that data being sent over UDP has the ability to “fill the pipe.” With no TCP window throttling the transmission, and with no wasted time for millions of acknowledgements, UDP can transmit data at full line speed.

“But wait,” you’re thinking, “I WANT reliability!”

Naturally, a file transfer solution is nothing if the files do not arrive intact.  To overcome the shortcomings of UDP, FileCatalyst implements application layer retransmission and congestion control algorithms which means it takes care of missing packets at the same time it sends new data, and it will not overwhelm your network.  All of the transfer parameters are under control by the end user, meaning that transfers may be tuned for a specific set of conditions such as high latency or high packet loss.  The end result is a file transmission mechanism that can both “fill the pipe” and guarantee delivery of every packet.

The FileCatalyst product suite implements a host of additional reliability features which will resume and retry interrupted connections, resend missing data, and perform MD5 checksums. The reliability built into a FileCatalyst transfer is far more robust than FTP.

Conclusion

Using a protocol such as UDP opens up the possibility to optimize transfers by using all your available bandwidth. Developers also have the unique opportunity to add superior reliability features. The financial benefits of faster file transfers are a subject for another article, but suffice it to say that if you can transfer files in 20 minutes instead of 16 hours, you will quickly recover any initial cost in terms of productivity hours, turnaround, and customer satisfaction (which can result in growth of customer base).

Coming soon: a closer look at specific FileCatalyst solutions, and tips on how existing FileCatalyst deployments can be optimized.

As mentioned a short time ago, we have some material from our old blog that we are slowly migrating to the new one. Being that we are in the business of fast file transfer, it made since to bring back one of our more popular articles on the subject. Although written around the time of FileCatalyst 2.0 (2.7.x is the current generation), the information is still relevant, and the article continues to be widely read by visitors to the main site. Here it is, for readers of the new blog.

Original article written by Chris Bailey on July 31, 2007 9:53 PM

Fast File Transfer: Moving beyond acceleration with data optimization

If you are in the market for a fast file transfer solution, you have a couple of options. First there is the traditional FTP client software, some of which use parallel TCP streams to speed up your transfers. Then there are the UDP based transfer applications, FileCatalyst is one of them. These applications can maximize the data across your internet connection regardless of network conditions. If you have a T3, you will get exactly T3 speeds. The other approach to fast file transfer is data optimization.

By reducing the data that needs to be transmitted, you can effectively transfer the file faster. Even if the actual data going across your line is not optimal, you may still get faster rates because of the data reduction ratio. Consider a database file, or large spreadsheet. Since this data is highly compressible, you can reduce transfer time significantly just by zipping it up.

Another way to optimize file transfers is to send only portions of a file that have changed. Consider the database mentioned above. You need to back up this database on a daily basis to a remote location over a T1. The database file is 2 GB. If you maximize your T1, it could take almost 3 hours to transfer the file each day, even with the best acceleration on the market. But what if only 100MB had changed in the file? If you could detect and transfer only the portion that has changed you would reduce the transfer time by a factor of 20. Now the transfer only takes 9 minutes!

But hold on, that database is probably compressible as well, so even with only a 2:1 compression ratio you could cut that transfer time in half again. So now it is only 4 and a half minutes, or 40 times faster than your link speed!

FileCatalyst 2.0 was released earlier this year and does acceleration, as well as differencing and compression. It does it for you in the background, so there is no wait time to compress the file prior to transferring; it is all done on the fly, from one automated tool. With FileCatalyst, file transfers are as fast as your link, i.e. T1, T3, etc… The only question is how much faster it will go beyond that speed. That depends whether you have transferred the file previously and whether the data is compressible or not.

Fast File Transfer

The table above lists some speed gain examples. Of course there are a lot of cases that do not benefit in any way from this technology, but there are just as many that do. FileCatalyst should be considered an option in either case since it offers the best of all worlds; that is, industry leading acceleration as well as data optimization. You can always be assured you are getting the fastest possible file transfer with FileCatalyst.

To read more about accelerating and optimizing file transfer with the FileCatalyst family of products visit www.filecatalyst.com

We began discussing the various ways that acceleration solutions can shorten transfer time. The first of these was the use of compression.

The main point of the article was to show how smaller data sets will (naturally!) mean less transfer time. Compression is a fairly basic way to make smaller data sets, and it is not always 100% effective, particularly when the data being sent is not compressible. But there are other ways a transfer solution can cut down on the amount of data being sent.

It boils down to: “Don’t send redundant data”:

• Don’t send redundant files: Seems obvious enough, and plenty of solutions out there (especially in the backup solution sphere) have this in place. Picture a typical copy+paste operation in your Operating System. If a duplicate file exists, you are asked, “A file with this name exists. Overwrite?” And then you can peek at the time and date, plus file size, allowing you to decide. By automating this process, the file transfer solution will just skip past the files that it doesn’t need to resend.
• Don’t send redundant bytes: This one’s a little trickier! Imagine you have a multi-GB file of some sort (say, a monolithic database or an ISO). You’ve made some small changes to it and it’s time to send it off. You sigh in despair, knowing that even though only 4MB has changed, you’ll have to transfer that behemoth all over again. Wouldn’t it be nice if only those changes could be sent? FileCatalyst offers a file Deltas function, which does just that. As the transfer initializes, the source and destination files are compared… if there’s a difference, FileCatalyst will identify the blocks that have changed and build a “patch” (the delta file). This patch gets applied to the destination file, making it identical to the source file.
  
  The benefit is obvious: a 4MB delta (for example) takes a lot less time to transfer than the whole 5GB file!

As with most features, there needs to be a realistic expectation before deciding that Deltas will enhance the efficiency of any given task. For example, adding too many source locations going to the same destination creates the potential for conflict. One source to many destinations works brilliantly, however! Also, if you have a task to transfer primarily small files, you’re often better off just sending the files. And, like compression, be aware that building deltas will use a portion of your machine’s CPU. If possible, compare the performance of a given task with or without deltas… it’ll soon be clear which one to choose for a particular job.

Deltas and compression are not mutually exclusive! Assuming a source file that’s compressible to begin with, the two will work quite well together, allowing you to send the smallest possible amount of data.

–

Accelerating transfers can broadly be broken down into two categories: minimizing data, and optimizing the link. We’ve now covered the first category by discussion compression and Deltas. Our next articles will address optimizing the link itself, via Multiple Streams, as well as ditching TCP (FTP) in favour of something better suited to acceleration.

Cheers,
Greg

Yes, it is a cliché but seriously, you need FileCatalyst and you probably don’t even know it. I am obviously biased since I wrote a lot of the back end to FileCatalyst and have been “drinking my own kool-aid” for a while now. But haven’t you ever bought something and then a few months later you think back and wonder how you ever lived without it? Now I am not saying this will be the case for everyone with FileCatalyst, but I can certainly tell you in which scenarios FileCatalyst would make you ask that very question:

First scenario: You are an IT person, and you maintain the company’s file transfer system. The company is a multi-national, with branches all around the world. You have to make sure mission critical files make it from A to B, and make sure it happens as quickly as possible. You have a lot of issues with network interruptions, and then you need to resend the files. Or even worse, the interruptions leave you with corrupt files.

Second scenario: you are once again… an IT person, but you only have to move a few files between locations. However, your files are several GB in size—maybe even hundreds of GB. Your company has spent a lot on a high speed internet link, but for some reason the transfers are still slow. Not only that, but you still experience the odd disconnect here and there, and lose several hours of time because you need to start the transfer over again.

If this sounds like you, you have to try FileCatalyst because you could save yourself and your company a lot of time and a lot of money. First of all, FileCatalyst’s speed is second to none. If you want to maximize that high-speed link your company is paying for (or considering purchasing) you need a product like FileCatalyst to get the full return. Trying to use FTP as a transfer solution will never let you use the full bandwidth the way FileCatalyst will.

When it comes to reliability, FileCatalyst also leads the way with its robust retry and checksum features. It will hammer those files across the network in the minimum possible time, even with a ridiculous number of dropped connections.

Usually, you have to realize you have a problem first before even looking for a product like FileCatalyst. Maybe you’re resigned to the mistaken idea that your current solution is the only one out there. If I’ve helped you see that you might need an alternative like FileCatalyst, why not take it for a spin? Download our entry-level offering to jump right in, or better yet contact us to start a trial of any FileCatalyst product.

Chris