In a few weeks, we will be in Amsterdam at one of the year’s biggest events, the IBC Conference and Exhibition. If you’re going to be in attendance, visit us during the exhibition (September 11-15) at booth 7.J41. Check out the press release here

The atmosphere is always electric at the IBC, and it’s exciting to meet face-to-face with existing European clients as well as meeting with the growing number of people interested in accelerating their file transfers or improving their workflow.

Each year, the number of attendees interested in file acceleration technology—and FileCatalyst in particular—grows significantly. Only a few years ago, most people meeting us at trade shows and conferences were unaware that there were alternatives to FTP or other TCP-based transfer protocols, and weren’t certain that they needed our kind of technology. In subsequent years, and with a bit more knowledge about TCP’s limitations under their belts, visitors were actively seeking more information or were ready to ditch FTP outright. At this point, a high percentage of our visitors are aware of accelerated file transfer software, and are asking very educated questions while looking for a solution.

At IBC, FileCatalyst will be able to answer those questions face-to-face, as well as discuss the latest products and news.

This year we are also welcoming Turning Point Integration, our integration partner and reseller in the UK, who will be on-hand to talk about the ease of integrating FileCatalyst accelerated file transfer with other 3rd party software, as well as being a point of contact for European attendees wishing to learn more about FileCatalyst after the conference.

With FileCatalyst Direct 2.7.3 released just days ago, as well as FileCatalyst Webmail and Workflow having recently moved to version 4.3, there has been quite a lot of development to discuss. Next versions are just around the corner with some much-requested new functionality. Swing by for a peek at what’s new and find out more about what’s coming up.

Enjoy the show!

As mentioned a short time ago, we have some material from our old blog that we are slowly migrating to the new one. Being that we are in the business of fast file transfer, it made since to bring back one of our more popular articles on the subject. Although written around the time of FileCatalyst 2.0 (2.7.x is the current generation), the information is still relevant, and the article continues to be widely read by visitors to the main site. Here it is, for readers of the new blog.

Original article written by Chris Bailey on July 31, 2007 9:53 PM

Fast File Transfer: Moving beyond acceleration with data optimization

If you are in the market for a fast file transfer solution, you have a couple of options. First there is the traditional FTP client software, some of which use parallel TCP streams to speed up your transfers. Then there are the UDP based transfer applications, FileCatalyst is one of them. These applications can maximize the data across your internet connection regardless of network conditions. If you have a T3, you will get exactly T3 speeds. The other approach to fast file transfer is data optimization.

By reducing the data that needs to be transmitted, you can effectively transfer the file faster. Even if the actual data going across your line is not optimal, you may still get faster rates because of the data reduction ratio. Consider a database file, or large spreadsheet. Since this data is highly compressible, you can reduce transfer time significantly just by zipping it up.

Another way to optimize file transfers is to send only portions of a file that have changed. Consider the database mentioned above. You need to back up this database on a daily basis to a remote location over a T1. The database file is 2 GB. If you maximize your T1, it could take almost 3 hours to transfer the file each day, even with the best acceleration on the market. But what if only 100MB had changed in the file? If you could detect and transfer only the portion that has changed you would reduce the transfer time by a factor of 20. Now the transfer only takes 9 minutes!

But hold on, that database is probably compressible as well, so even with only a 2:1 compression ratio you could cut that transfer time in half again. So now it is only 4 and a half minutes, or 40 times faster than your link speed!

FileCatalyst 2.0 was released earlier this year and does acceleration, as well as differencing and compression. It does it for you in the background, so there is no wait time to compress the file prior to transferring; it is all done on the fly, from one automated tool. With FileCatalyst, file transfers are as fast as your link, i.e. T1, T3, etc… The only question is how much faster it will go beyond that speed. That depends whether you have transferred the file previously and whether the data is compressible or not.

Fast File Transfer

The table above lists some speed gain examples. Of course there are a lot of cases that do not benefit in any way from this technology, but there are just as many that do. FileCatalyst should be considered an option in either case since it offers the best of all worlds; that is, industry leading acceleration as well as data optimization. You can always be assured you are getting the fastest possible file transfer with FileCatalyst.

To read more about accelerating and optimizing file transfer with the FileCatalyst family of products visit www.filecatalyst.com

As part of our ongoing discussion on fast file transfers, or “acceleration,” (which we began with the article “Accelerating File transfer“), we have already talked about how transfers can be “virtually” accelerated by minimizing the data sets. This is done with compression or sending file deltas. In short, less data = faster transfer.

An acceleration solution should also optimize your link. This takes us out of the realm of “virtual” speed boosts and into the realm of more data making it across the line in less time. One way to accomplish this is to use multiple streams.

As a TCP-based protocol, FTP is sequential in nature. For the easily distracted, the problem with this is that if there are problems with this sequence (the cycle of acknowledging packets) such as lost packets and latency, TCP transfers are throttled. You can compensate for this bottneck at least in part by sending more than one chunk of data at a time, be it separate files or separate pieces of the same file.

For the braver souls, here’s the techno-babble:

Latency and its effect on FTP

In order to provide reliable data transmission, TCP/IP requires the receiver to acknowledge each packet being sent, in sequential order. Each such communication is measured as round trip time (RTT), or the time it takes for a packet to be sent and acknowledged.

TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. The optimal amount of unacknowledged data en route should equal the end-to-end bandwidth multiplied by the RTT, also called the bandwidth-delay product. TCP continually estimates this value, setting a “TCP window” to control how much data should be sent. TCP has limits on the size of this value, so when the bandwidth-delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”. Satellite connections can be into the hundreds or even thousands of milliseconds of RTT.

Packet loss and its effect on FTP

Network congestion typically causes buffer overflows of intermediate routers, ausing packet loss. Since packets are sent sequentially, this can cause a hold-up in the cycle. Unacknowledged packets also cause the TCP window to shrink or even close completely for periods of time. Wireless and satellite transfers can have even higher packet loss due to sources of interference such as clouds or physical structures. Packet loss combined with high latency creates even worse performance for FTP transfers.

Acceleration with Multiple Streams

The idea behind multiple streams is to reduce the “dead air” created by the acknowledgment cycle. Instead of using 1 stream, multiple streams are opened, each sending a different piece of data. In theory this could be separate files, but this wouldn’t help transmission of a single large file. Instead, FileCatalyst will divide a file up into separate chunks: in a 3-stream situation, for example, stream 1 might send chunks 1, 4, 7, 10, etc; stream 2 would send 2, 5, 8, 11, etc; stream 3 would send 3, 6, 9, 12, etc. This way, there is almost always SOME data going across. No “dead air”. And even if chunks arrive out of order, they are just “slotted” into place accordingly. Sounds good, but it’s important to recognize that there ARE diminished returns on opening too many streams (and your hardware I/O needs to be able to keep up in order to make it worthwhile) but for the most part having multiple streams will produce a reasonable amount of acceleration.

When applied to TCP-based transfers such as FTP, you can see the benefit of multiple streams. Because of this, FileCatalyst acceleration technology includes multiple TCP streaming for situations in which UDP-based transfers are not possible. However, this approach still does not produce the same dramatic effect as ditching TCP in favour of alternatives like UDP. UDP-based transfers, which are at the heart of the FileCatalyst acceleration technology, will be the subject of our next article on acceleration.

Cheers,
Greg

When we launched the new blog using Wordpress, one of the long-term goals was to eventually migrate some of our articles from the old blog to this new home. One subject that comes up from time to time is the decision between a WAN appliance and a file transfer solution. Many organizations approach this as a “one or the other”, which it need not be; they can coexist and ultimately serve different purposes. Other times, the feeling is that if there is already some sort of “data acceleration” in place, that a file transfer solution would be redundant. Not necessarily so! Well, no point re-writing an already excellent article; here it is as originally written:

Over the past few years,I have encountered a lot of companies that already do WAN optimization and do not see a need to implement additional software for accelerating file transfer. The fact is, WAN optimization is great depending on what you are trying to accomplish; however, there are cases where the only requirement is file transfer. WAN optimization on its own does an OK job. But if you want real performance, combining a software solution with your WAN optimization appliance can sometimes yield staggering results.

At a recent trade show we spoke with a company called Expand Networks about areas where we could complement what they do, and vise versa. They explained that they not only do acceleration, but also advanced differencing and compression at the block level. This reduces the amount of data across the network, and does a pretty good job of accelerating it. When used in conjunction with an FTP client, this can greatly improve file transfer speeds, and even surpass line speed.

We decided it would be worth getting our hands on a couple of Expand boxes and see what happens when we run FileCatalyst in conjunction with them. What we found was that for pure file transfer we were able to get at least double the throughput of the Expand box alone, and much more when we increased the network latency and packet loss. There was a point where the TCP acceleration did not do as good a job as FileCatalyst with regards to keeping the link full of data.

Because FileCatalyst uses UDP and can increase/decrease transmission rate depending on congestion it was able to detect the reduced traffic across the WAN appliances (due to its optimization techniques) and continue to increase the speed until the link was filled. FileCatalyst also compressed large blocks of data before transmission. Expand was then able to compress each packet even further using its advanced block level compression. When the file is modified slightly, and sent a second time, FileCatalyst will send the differences at the byte level. Expand then does the same at the block level resulting in a further reduction in data. The end result is that optimizations take place at the byte level and at the block level, and FileCatalyst’s immunity to high RTT and packet loss allowed this optimized data to flow at maximum possible speed.

On a network monitoring device the actual data across the line was optimal, that is on a T1 with RTT of 350ms and 1% packet loss, roughly 1500 Kbps of data was being sent. But the actual throughput was much higher depending on the type of data, and how many times it was sent across. In some cases 10 times higher than line speed. However in all cases, the speed was faster using FileCatalyst and the WAN appliance, than just the WAN appliance on its own.

In addition to the speed gains, the FileCatalyst software suite offers a host of application level features that just aren’t there when using other 3rd party file transfer applications with a WAN appliance. Features like scheduling, auto-reconnection and resume, on the fly compression, deltas, and bandwidth scheduling. Of course no other file transfer software does a better job of maximizing actual data throughput than FileCatalyst.

For any company that already has a WAN optimization appliance, and wishes to add automation, reliability and further accelerate and optimize file transfers, they should strongly consider a software solution such as FileCatalyst.

To read more about accelerating file transfer with the FileCatalyst family of products visit www.filecatalyst.com

—
Chris Bailey is co-founder and CEO of Unlimi-Tech Software Inc., makers of software solutions that are reinventing file transfer in the enterprise.
Written by Chris Bailey on July 29, 2007 7:19 PM