As mentioned a short time ago, we have some material from our old blog that we are slowly migrating to the new one. Being that we are in the business of fast file transfer, it made since to bring back one of our more popular articles on the subject. Although written around the time of FileCatalyst 2.0 (2.7.x is the current generation), the information is still relevant, and the article continues to be widely read by visitors to the main site. Here it is, for readers of the new blog.

Original article written by Chris Bailey on July 31, 2007 9:53 PM

---

 

Fast File Transfer: Moving beyond acceleration with data optimization

If you are in the market for a fast file transfer solution, you have a couple of options. First there is the traditional FTP client software, some of which use parallel TCP streams to speed up your transfers. Then there are the UDP based transfer applications, FileCatalyst is one of them. These applications can maximize the data across your internet connection regardless of network conditions. If you have a T3, you will get exactly T3 speeds. The other approach to fast file transfer is data optimization.

By reducing the data that needs to be transmitted, you can effectively transfer the file faster. Even if the actual data going across your line is not optimal, you may still get faster rates because of the data reduction ratio. Consider a database file, or large spreadsheet. Since this data is highly compressible, you can reduce transfer time significantly just by zipping it up.

Another way to optimize file transfers is to send only portions of a file that have changed. Consider the database mentioned above. You need to back up this database on a daily basis to a remote location over a T1. The database file is 2 GB. If you maximize your T1, it could take almost 3 hours to transfer the file each day, even with the best acceleration on the market. But what if only 100MB had changed in the file? If you could detect and transfer only the portion that has changed you would reduce the transfer time by a factor of 20. Now the transfer only takes 9 minutes!

But hold on, that database is probably compressible as well, so even with only a 2:1 compression ratio you could cut that transfer time in half again. So now it is only 4 and a half minutes, or 40 times faster than your link speed!

FileCatalyst 2.0 was released earlier this year and does acceleration, as well as differencing and compression. It does it for you in the background, so there is no wait time to compress the file prior to transferring; it is all done on the fly, from one automated tool. With FileCatalyst, file transfers are as fast as your link, i.e. T1, T3, etc… The only question is how much faster it will go beyond that speed. That depends whether you have transferred the file previously and whether the data is compressible or not.

The table above lists some speed gain examples. Of course there are a lot of cases that do not benefit in any way from this technology, but there are just as many that do. FileCatalyst should be considered an option in either case since it offers the best of all worlds; that is, industry leading acceleration as well as data optimization. You can always be assured you are getting the fastest possible file transfer with FileCatalyst.

To read more about accelerating and optimizing file transfer with the FileCatalyst family of products visit www.filecatalyst.com

Tags: Fast File Transfer, FileCatalyst, ftp replacement

File-based workflow solution accelerates 100 GB of game development files per day

The following press release went out today; if you have not stumbled across it in your travels, have a read! Naturally, we are quite happy with the announcement. Several of our staff are game enthusiasts, and the gaming industry is full of interesting people (like those at SEGA). It works out well that our products are so useful for game development studios. On a related note, we’re down at the Game Developers’ Conference (GDC) in Austin as we speak. Swing by booth 6333 NH.

The press release:

San Francisco, CA March 25, 2009 — Unlimi-Tech Software, the pioneer in accelerated file transfers is pleased to announce the selection of FileCatalyst Workflow by SEGA® Corporation. The deployment of FileCatalyst Workflow allows SEGA to successfully manage and transfer game development files, including game-supporting multimedia files, raw and binary game code.

The FileCatalyst Workflow software application is a unique file-based workflow solution that offers comprehensive tracking and management of files as they are processed through a digital workflow. The uniqueness of the application is exemplified by its ability to accelerate file transfers as they move from one project node to the other. Advanced acceleration algorithms allow for improved bandwidth utilization and data reduction – leading to file transfer rates of 100x faster than traditional FTP.

“We were looking for a solution that would consolidate our disparate file transfer systems into a single file-based workflow application. Our main goal was to improve our productivity and efficiency through this application,” said Jake Salgado, Director of IT, SEGA of America, “We selected FileCatalyst technology because of the speed gains it offered – previously it would take longer than a day to transfer our project files, now we can do it in hours. The time we shave off in our transfers translate to immense productivity gains when you realize we have a global work force in different time zones.”

FileCatalyst® acceleration technology is completely software-based, and is designed to overcome network performance bottlenecks to maximize use of available bandwidth. This technology is packaged into FileCatalyst Workflow, providing file-based workflow. The process is completely customizable and offers automated and scheduling features, email notifications and LDAP/Active Directory integration.

“We are pleased to be an integral part of the applications deployed by the SEGA development group,” said Chris Bailey, CEO of Unlimi-Tech Software, “The gaming industry is experiencing huge issues relating to delivery of the massive amounts of content in today’s games. FileCatalyst allows game development companies to minimize file transfer times, and thus improve productivity across the game development workflow.”

About SEGA of America

SEGA of America, Inc. is the American arm of Tokyo, Japan-based SEGA Corporation, a worldwide leader in interactive entertainment both inside and outside the home. The company develops, publishes and distributes interactive entertainment software products for a variety of hardware platforms including PC, wireless devices, and those manufactured by Nintendo, Microsoft and Sony Computer Entertainment Inc. SEGA of America Web site is located at www.sega.com

About Unlimi-Tech Software

Located in Ottawa, Canada, Unlimi-Tech Software is the creator of FileCatalyst®, the world’s leading file transfer solution. Founded in 2000, the company has more than seven hundred and fifty commercial and government clients. FileCatalyst® technology is a software-based solution designed to accelerate and optimize file transfers across global networks. FileCatalyst® is immune to the effects that latency and packet loss have on traditional file transfer methods like FTP, HTTP or CIFS. FileCatalyst® addresses these issues for enterprise WAN, satellite and wireless communications, media and content providers, and government and military organizations.

For more information, visit the FileCatalyst website at: www.filecatalyst.com

Tags: Fast File Transfer, file transfer, FileCatalyst

As part of our ongoing discussion on fast file transfers, or “acceleration,” (which we began with the article “Accelerating File transfer“), we have already talked about how transfers can be “virtually” accelerated by minimizing the data sets. This is done with compression or sending file deltas. In short, less data = faster transfer.

An acceleration solution should also optimize your link. This takes us out of the realm of “virtual” speed boosts and into the realm of more data making it across the line in less time. One way to accomplish this is to use multiple streams.

As a TCP-based protocol, FTP is sequential in nature. For the easily distracted, the problem with this is that if there are problems with this sequence (the cycle of acknowledging packets) such as lost packets and latency, TCP transfers are throttled. You can compensate for this bottneck at least in part by sending more than one chunk of data at a time, be it separate files or separate pieces of the same file.

For the braver souls, here’s the techno-babble:

Latency and its effect on FTP

In order to provide reliable data transmission, TCP/IP requires the receiver to acknowledge each packet being sent, in sequential order. Each such communication is measured as round trip time (RTT), or the time it takes for a packet to be sent and acknowledged.

TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. The optimal amount of unacknowledged data en route should equal the end-to-end bandwidth multiplied by the RTT, also called the bandwidth-delay product. TCP continually estimates this value, setting a “TCP window” to control how much data should be sent. TCP has limits on the size of this value, so when the bandwidth-delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”. Satellite connections can be into the hundreds or even thousands of milliseconds of RTT.

Packet loss and its effect on FTP

Network congestion typically causes buffer overflows of intermediate routers, ausing packet loss. Since packets are sent sequentially, this can cause a hold-up in the cycle. Unacknowledged packets also cause the TCP window to shrink or even close completely for periods of time. Wireless and satellite transfers can have even higher packet loss due to sources of interference such as clouds or physical structures. Packet loss combined with high latency creates even worse performance for FTP transfers.

Acceleration with Multiple Streams

The idea behind multiple streams is to reduce the “dead air” created by the acknowledgment cycle. Instead of using 1 stream, multiple streams are opened, each sending a different piece of data. In theory this could be separate files, but this wouldn’t help transmission of a single large file. Instead, FileCatalyst will divide a file up into separate chunks: in a 3-stream situation, for example, stream 1 might send chunks 1, 4, 7, 10, etc; stream 2 would send 2, 5, 8, 11, etc; stream 3 would send 3, 6, 9, 12, etc. This way, there is almost always SOME data going across. No “dead air”. And even if chunks arrive out of order, they are just “slotted” into place accordingly. Sounds good, but it’s important to recognize that there ARE diminished returns on opening too many streams (and your hardware I/O needs to be able to keep up in order to make it worthwhile) but for the most part having multiple streams will produce a reasonable amount of acceleration.

When applied to TCP-based transfers such as FTP, you can see the benefit of multiple streams. Because of this, FileCatalyst acceleration technology includes multiple TCP streaming for situations in which UDP-based transfers are not possible. However, this approach still does not produce the same dramatic effect as ditching TCP in favour of alternatives like UDP. UDP-based transfers, which are at the heart of the FileCatalyst acceleration technology, will be the subject of our next article on acceleration.

Cheers,
Greg

Tags: Fast File Transfer, Multiple Streams

FileCatalyst Direct and Express have been updated to version 2.7.1. As an incremental release, 2.7.1 is primarily a maintenance release, tightening up a few features and adding a few small improvements. 2.7 introduced progressive transfers on file deltas (see our recent article on minimizing data sets) but for uploads only. This release adds the functionality on downloads as well, good news for people with a 2-way workflow who are transferring large files.

Here are the release notes in brief:

• Transfer of deltas file may now be done progressively (start to transfer data while delta file is being built) for both upload and download
• Ability for administrators to see client type and version in Server Remote Admin
• Confirmation dialog before canceling transfers using the applets
• Ability in upload applet to remove browser pane
• “slow start” renamed to a more accurate “congestion control” and default control aggression lowered
• Improved: resuming large delta transfers
• Improved: compatibility of MonitoringAgent with particular SMTP servers

This is a recommended upgrade for any organizations using 2.7.0 or below. Please contact your FileCatalyst representative, or log into the download area to obtain your upgrade.

Please note that the above notes cover the FileCatalyst Server and Client applications as a whole; however, not all features appear in the entry-level FileCatalyst Express solution.

Tags: Acceleration, Direct, Express, Fast File Transfer, FileCatalyst, HotFolder

We began discussing the various ways that acceleration solutions can shorten transfer time. The first of these was the use of compression.

The main point of the article was to show how smaller data sets will (naturally!) mean less transfer time. Compression is a fairly basic way to make smaller data sets, and it is not always 100% effective, particularly when the data being sent is not compressible. But there are other ways a transfer solution can cut down on the amount of data being sent.

It boils down to: “Don’t send redundant data”:

• Don’t send redundant files: Seems obvious enough, and plenty of solutions out there (especially in the backup solution sphere) have this in place. Picture a typical copy+paste operation in your Operating System. If a duplicate file exists, you are asked, “A file with this name exists. Overwrite?” And then you can peek at the time and date, plus file size, allowing you to decide. By automating this process, the file transfer solution will just skip past the files that it doesn’t need to resend.
• Don’t send redundant bytes: This one’s a little trickier! Imagine you have a multi-GB file of some sort (say, a monolithic database or an ISO). You’ve made some small changes to it and it’s time to send it off. You sigh in despair, knowing that even though only 4MB has changed, you’ll have to transfer that behemoth all over again. Wouldn’t it be nice if only those changes could be sent? FileCatalyst offers a file Deltas function, which does just that. As the transfer initializes, the source and destination files are compared… if there’s a difference, FileCatalyst will identify the blocks that have changed and build a “patch” (the delta file). This patch gets applied to the destination file, making it identical to the source file.
  
  The benefit is obvious: a 4MB delta (for example) takes a lot less time to transfer than the whole 5GB file!

As with most features, there needs to be a realistic expectation before deciding that Deltas will enhance the efficiency of any given task. For example, adding too many source locations going to the same destination creates the potential for conflict. One source to many destinations works brilliantly, however! Also, if you have a task to transfer primarily small files, you’re often better off just sending the files. And, like compression, be aware that building deltas will use a portion of your machine’s CPU. If possible, compare the performance of a given task with or without deltas… it’ll soon be clear which one to choose for a particular job.

Deltas and compression are not mutually exclusive! Assuming a source file that’s compressible to begin with, the two will work quite well together, allowing you to send the smallest possible amount of data.

–

Accelerating transfers can broadly be broken down into two categories: minimizing data, and optimizing the link. We’ve now covered the first category by discussion compression and Deltas. Our next articles will address optimizing the link itself, via Multiple Streams, as well as ditching TCP (FTP) in favour of something better suited to acceleration.

Cheers,
Greg

Tags: Acceleration, compression, Fast File Transfer, ftp replacement