Introduction

Beginning with the introductory article on Acceleration, we have been discussing a few of the ways a software-only solution can accelerate file transfer:

1. Minimizing Data with Compression
2. Mininmizing Data by cutting redundant data (for example, with file Deltas)
3. Optimizing data transmission with Multiple Streams

In the fourth and final article on the subject, we are going to talk a bit about the technology that lies at the core of the FileCatalyst accelerated transfer solution: using an alternative to FTP or other TCP-based systems.

The problem with FTP

As discussed in earlier articles, the problem with FTP is its approach to reliability. From the FileCatalyst whitepaper, “Accelerating File Transfers”:

TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. …TCP has limits on the size of this value, so when the bandwidth‐delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”.

In layman’s terms, by trying its best to be reliable, TCP can really throttle down the speed of transfers over poor networks. Since it is a layer on top of TCP, this problem extends to FTP.

Solution: Fast File Transfer with UDP

One of the other core protocols on the internet, UDP, is by its nature NOT reliable. Packets are blasted out without concern for reliability or integrity. While TCP sends data sequentially, UDP is not concerned if the receiver gets packets in order. The benefit of this is that data being sent over UDP has the ability to “fill the pipe.” With no TCP window throttling the transmission, and with no wasted time for millions of acknowledgements, UDP can transmit data at full line speed.

“But wait,” you’re thinking, “I WANT reliability!”

Naturally, a file transfer solution is nothing if the files do not arrive intact.  To overcome the shortcomings of UDP, FileCatalyst implements application layer retransmission and congestion control algorithms which means it takes care of missing packets at the same time it sends new data, and it will not overwhelm your network.  All of the transfer parameters are under control by the end user, meaning that transfers may be tuned for a specific set of conditions such as high latency or high packet loss.  The end result is a file transmission mechanism that can both “fill the pipe” and guarantee delivery of every packet.

The FileCatalyst product suite implements a host of additional reliability features which will resume and retry interrupted connections, resend missing data, and perform MD5 checksums. The reliability built into a FileCatalyst transfer is far more robust than FTP.

Conclusion

Using a protocol such as UDP opens up the possibility to optimize transfers by using all your available bandwidth. Developers also have the unique opportunity to add superior reliability features. The financial benefits of faster file transfers are a subject for another article, but suffice it to say that if you can transfer files in 20 minutes instead of 16 hours, you will quickly recover any initial cost in terms of productivity hours, turnaround, and customer satisfaction (which can result in growth of customer base).

Coming soon: a closer look at specific FileCatalyst solutions, and tips on how existing FileCatalyst deployments can be optimized.

Tags: Acceleration, ftp alternative, ftp replacement, UDP

FileCatalyst Direct and Express have been updated to version 2.7.1. As an incremental release, 2.7.1 is primarily a maintenance release, tightening up a few features and adding a few small improvements. 2.7 introduced progressive transfers on file deltas (see our recent article on minimizing data sets) but for uploads only. This release adds the functionality on downloads as well, good news for people with a 2-way workflow who are transferring large files.

Here are the release notes in brief:

• Transfer of deltas file may now be done progressively (start to transfer data while delta file is being built) for both upload and download
• Ability for administrators to see client type and version in Server Remote Admin
• Confirmation dialog before canceling transfers using the applets
• Ability in upload applet to remove browser pane
• “slow start” renamed to a more accurate “congestion control” and default control aggression lowered
• Improved: resuming large delta transfers
• Improved: compatibility of MonitoringAgent with particular SMTP servers

This is a recommended upgrade for any organizations using 2.7.0 or below. Please contact your FileCatalyst representative, or log into the download area to obtain your upgrade.

Please note that the above notes cover the FileCatalyst Server and Client applications as a whole; however, not all features appear in the entry-level FileCatalyst Express solution.

Tags: Acceleration, Direct, Express, Fast File Transfer, FileCatalyst, HotFolder

We began discussing the various ways that acceleration solutions can shorten transfer time. The first of these was the use of compression.

The main point of the article was to show how smaller data sets will (naturally!) mean less transfer time. Compression is a fairly basic way to make smaller data sets, and it is not always 100% effective, particularly when the data being sent is not compressible. But there are other ways a transfer solution can cut down on the amount of data being sent.

It boils down to: “Don’t send redundant data”:

• Don’t send redundant files: Seems obvious enough, and plenty of solutions out there (especially in the backup solution sphere) have this in place. Picture a typical copy+paste operation in your Operating System. If a duplicate file exists, you are asked, “A file with this name exists. Overwrite?” And then you can peek at the time and date, plus file size, allowing you to decide. By automating this process, the file transfer solution will just skip past the files that it doesn’t need to resend.
• Don’t send redundant bytes: This one’s a little trickier! Imagine you have a multi-GB file of some sort (say, a monolithic database or an ISO). You’ve made some small changes to it and it’s time to send it off. You sigh in despair, knowing that even though only 4MB has changed, you’ll have to transfer that behemoth all over again. Wouldn’t it be nice if only those changes could be sent? FileCatalyst offers a file Deltas function, which does just that. As the transfer initializes, the source and destination files are compared… if there’s a difference, FileCatalyst will identify the blocks that have changed and build a “patch” (the delta file). This patch gets applied to the destination file, making it identical to the source file.
  
  The benefit is obvious: a 4MB delta (for example) takes a lot less time to transfer than the whole 5GB file!

As with most features, there needs to be a realistic expectation before deciding that Deltas will enhance the efficiency of any given task. For example, adding too many source locations going to the same destination creates the potential for conflict. One source to many destinations works brilliantly, however! Also, if you have a task to transfer primarily small files, you’re often better off just sending the files. And, like compression, be aware that building deltas will use a portion of your machine’s CPU. If possible, compare the performance of a given task with or without deltas… it’ll soon be clear which one to choose for a particular job.

Deltas and compression are not mutually exclusive! Assuming a source file that’s compressible to begin with, the two will work quite well together, allowing you to send the smallest possible amount of data.

–

Accelerating transfers can broadly be broken down into two categories: minimizing data, and optimizing the link. We’ve now covered the first category by discussion compression and Deltas. Our next articles will address optimizing the link itself, via Multiple Streams, as well as ditching TCP (FTP) in favour of something better suited to acceleration.

Cheers,
Greg

Tags: Acceleration, compression, Fast File Transfer, ftp replacement

As discussed on Feb 23, one of the most common questions people ask is, “How do you accelerate file transfer? Are the files compressed or something like that?” The response is that compression is only part of our offering, and an optional part at that. But that doesn’t mean it’s not important. You just have to know exactly what compression can offer in terms of benefits, as well as be aware of what it cannot do.

In terms of acceleration, the benefit is quite simple: if there is less data to be sent, the transfer will be “virtually” faster. A 10MB compressed file will take roughly half as long to send as the original 20MB file, even though the actual line speed is the same. This “virtual” speed gain will depend on the compression ratio, ie. how compressible the files are. Text-based files such as word processor documents tend to be more compressible than binary files such as executables; compressed filetypes such as JPG and MP3 cannot be compressed further at all.

A common way to use compression in file transfer is to compress (zip, rar, stuffit, etc) each file before it is sent, and then decompress the file at the destination. Not a bad method, and it can potentially yield results. But frankly if this is all an acceleration solution is doing with compression, you’re almost as well off just using a script built in-house. Instead, an acceleration solution should build upon the concept of compression and add value to it. Here are a few of the things FileCatalyst does in terms of compression:

• on-the fly compression: Rather than compressing each separate complete file, data is compressed at the block level. A block is compressed, sent, uncompressed, and appended to the file being built on the destination. By compressing on-the-fly there is no wait for the transfer to start, and when the last byte arrives the transfer is also finished—no waiting for a final decompression operation. The only caveat to this method is that it takes up more CPU than scenarios not involving compression.
• compress as single archive: This option becomes useful if you are transferring many small files. By creating just one archive for hundreds of files, you save a huge amount of set-up and tear-down. Saving these operations saves some virtual speed. But then there’s raw speed: the nature of small files means that they often arrive at the destination before line speed is reached; by sending a single archive (ie. one larger file!), the transfer can ramp up to line speed. To maximize efficiency, FileCatalyst’s progressive transfers also allow the file to start transferring even before the compression is finished.
• algorithm options: although the default compression method will work in any circumstance, you may choose from a few different algorithm options that may work just a bit better with particular file types.

There are some creative ways to use compression, with different FileCatalyst client applications/applets offering a number of ways to capitalize on the benefits of compression. These various options mean that you do not simply select “Use THIS kind of compression” globally throughout your application, but are free to pick and choose at a granular level, depending on the needs of any particular task. These options will be explored in later articles.

“Is your file accleration some sort of compression?” No, but compression sure can help!

Greg

Tags: Acceleration, compression, Fast File Transfer, FileCatalyst, HotFolder

When people first find out about FileCatalyst, one of the first questions asked is always, “How does your product speed up file transfer?” often followed by a best guess, “Is it some sort of compression?”

Although the deepest possible answer is going to be somewhat complicated, there are some very basic principles at play in the world of file transfer acceleration, with compression being just one. Almost all products claiming to provide fast file transfer will do one or more of the following:

1. compress data
2. minimized data sets (other than compression)
3. transmit over multiple streams
4. use an alternate to FTP

The first two are considered fairly standard for the industry. The third is the domain of specialized applications such as FileCatalyst, and the fourth is territory wide open to the creativity of developers.

Over the next several days, we will explore each of these; we will elaborate, discuss realistic expectations to have, and how they factor into the FileCatalyst family. In the meantime, here’s a brief look at each:

Compression

Using Zip, Rar, Gzip, or another compression algorithm, data sets are minimized. Compression algorithms find patterns and recreate the data using a shorter form. Just as it is quicker to write “five million zeroes in a row” than to actually type five million zeroes in a row, the number of bits needed to store the data is reduced. In order for the data to be used again, it must also be decoded (uncompressed) on the destination side.

Multiple Streams

By sending data across more than one stream at a time, there are fewer moments of “downtime”. While there is housekeeping occurring on one stream (in-between files being sent for example), the other stream(s) can still be actively sending data.

Alternate Protocols

FTP transfer occurs over TCP, a protocol with ordered delivery of packets. Since the order cannot be interrupted, latency and packet loss can cause it to slow to a crawl. Protocols such as UDP can blast out packets with disregard for order. However, UDP in and of itself does not reliably transfer data, which is why file transfer technologies based around UDP must also include their own retransmission and verification functionality.

Sending less data

Compression is one way of sending a smaller file than the original, but it is not the only way. For example, FileCatalyst offers an option to transfer file deltas, which are partial files which can be used to “patch” a file that has been sent previously. Sending a 4MB delta instead of an entire multi-GB file can save an absolutely staggering amount of time and bandwidth.

Keep your eyes on this space as we talk a bit more about these and other ways to achieve fast file transfer.

Cheers,
Greg

Tags: Acceleration, Add new tag, Fast File Transfer, ftp alternative