Leading into IBC, which is primarily a European show, it seemed fitting to share this bit of information:

http://www.prweb.com/releases/fast_file_transfer/eurovision/prweb2861634.htm

In short, the article describes how Eurovision has been using FileCatalyst Direct in its pilot program for enhancing the Eurovision News Exchanges. As many of you will already know, Eurovision is a department of the European Broacasting Union, the largest such organization in the world.

We certainly look forward to working with Eurovision as they move forward with the project. Based on the results they’re already seeing in the development framework, FileCatalyst is allowing for an unprecedented usage of available bandwidth.

FileCatalyst Webmail and Workflow have been updated and released as production versions.

The latest version included some changes to internal components and libraries, as well as improvements to the way the web application interacts with the file server. Some of the changes included in 4.3 are:

• Logging improvements (memory usage, FTP logs from applets)
• Delta transfers (RSYNC-like) for HTTP uploads
• Workflow only: overwrite location of job files and transaction PDFs
• Upgraded internal libraries and components
• New session-based security as an additional security measure
• Resolved issue uploading files to 3rd party FTP server (such as IIS) using fc upload and 2-way applet
• New confirmation dialogs for certain upload functions

Version 4.3 is a recommended upgrade.

Introduction

Beginning with the introductory article on Acceleration, we have been discussing a few of the ways a software-only solution can accelerate file transfer:

1. Minimizing Data with Compression
2. Mininmizing Data by cutting redundant data (for example, with file Deltas)
3. Optimizing data transmission with Multiple Streams

In the fourth and final article on the subject, we are going to talk a bit about the technology that lies at the core of the FileCatalyst accelerated transfer solution: using an alternative to FTP or other TCP-based systems.

The problem with FTP

As discussed in earlier articles, the problem with FTP is its approach to reliability. From the FileCatalyst whitepaper, “Accelerating File Transfers”:

TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. …TCP has limits on the size of this value, so when the bandwidth‐delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”.

In layman’s terms, by trying its best to be reliable, TCP can really throttle down the speed of transfers over poor networks. Since it is a layer on top of TCP, this problem extends to FTP.

Solution: Fast File Transfer with UDP

One of the other core protocols on the internet, UDP, is by its nature NOT reliable. Packets are blasted out without concern for reliability or integrity. While TCP sends data sequentially, UDP is not concerned if the receiver gets packets in order. The benefit of this is that data being sent over UDP has the ability to “fill the pipe.” With no TCP window throttling the transmission, and with no wasted time for millions of acknowledgements, UDP can transmit data at full line speed.

“But wait,” you’re thinking, “I WANT reliability!”

Naturally, a file transfer solution is nothing if the files do not arrive intact.  To overcome the shortcomings of UDP, FileCatalyst implements application layer retransmission and congestion control algorithms which means it takes care of missing packets at the same time it sends new data, and it will not overwhelm your network.  All of the transfer parameters are under control by the end user, meaning that transfers may be tuned for a specific set of conditions such as high latency or high packet loss.  The end result is a file transmission mechanism that can both “fill the pipe” and guarantee delivery of every packet.

The FileCatalyst product suite implements a host of additional reliability features which will resume and retry interrupted connections, resend missing data, and perform MD5 checksums. The reliability built into a FileCatalyst transfer is far more robust than FTP.

Conclusion

Using a protocol such as UDP opens up the possibility to optimize transfers by using all your available bandwidth. Developers also have the unique opportunity to add superior reliability features. The financial benefits of faster file transfers are a subject for another article, but suffice it to say that if you can transfer files in 20 minutes instead of 16 hours, you will quickly recover any initial cost in terms of productivity hours, turnaround, and customer satisfaction (which can result in growth of customer base).

Coming soon: a closer look at specific FileCatalyst solutions, and tips on how existing FileCatalyst deployments can be optimized.

The following press release went out today; if you have not stumbled across it in your travels, have a read! Naturally, we are quite happy with the announcement. Several of our staff are game enthusiasts, and the gaming industry is full of interesting people (like those at SEGA). It works out well that our products are so useful for game development studios. On a related note, we’re down at the Game Developers’ Conference (GDC) in Austin as we speak. Swing by booth 6333 NH.

The press release:

San Francisco, CA March 25, 2009 — Unlimi-Tech Software, the pioneer in accelerated file transfers is pleased to announce the selection of FileCatalyst Workflow by SEGA® Corporation. The deployment of FileCatalyst Workflow allows SEGA to successfully manage and transfer game development files, including game-supporting multimedia files, raw and binary game code.

The FileCatalyst Workflow software application is a unique file-based workflow solution that offers comprehensive tracking and management of files as they are processed through a digital workflow. The uniqueness of the application is exemplified by its ability to accelerate file transfers as they move from one project node to the other. Advanced acceleration algorithms allow for improved bandwidth utilization and data reduction – leading to file transfer rates of 100x faster than traditional FTP.

“We were looking for a solution that would consolidate our disparate file transfer systems into a single file-based workflow application. Our main goal was to improve our productivity and efficiency through this application,” said Jake Salgado, Director of IT, SEGA of America, “We selected FileCatalyst technology because of the speed gains it offered – previously it would take longer than a day to transfer our project files, now we can do it in hours. The time we shave off in our transfers translate to immense productivity gains when you realize we have a global work force in different time zones.”

FileCatalyst® acceleration technology is completely software-based, and is designed to overcome network performance bottlenecks to maximize use of available bandwidth. This technology is packaged into FileCatalyst Workflow, providing file-based workflow. The process is completely customizable and offers automated and scheduling features, email notifications and LDAP/Active Directory integration.

“We are pleased to be an integral part of the applications deployed by the SEGA development group,” said Chris Bailey, CEO of Unlimi-Tech Software, “The gaming industry is experiencing huge issues relating to delivery of the massive amounts of content in today’s games. FileCatalyst allows game development companies to minimize file transfer times, and thus improve productivity across the game development workflow.”

About SEGA of America

SEGA of America, Inc. is the American arm of Tokyo, Japan-based SEGA Corporation, a worldwide leader in interactive entertainment both inside and outside the home. The company develops, publishes and distributes interactive entertainment software products for a variety of hardware platforms including PC, wireless devices, and those manufactured by Nintendo, Microsoft and Sony Computer Entertainment Inc. SEGA of America Web site is located at www.sega.com

About Unlimi-Tech Software

Located in Ottawa, Canada, Unlimi-Tech Software is the creator of FileCatalyst®, the world’s leading file transfer solution. Founded in 2000, the company has more than seven hundred and fifty commercial and government clients. FileCatalyst® technology is a software-based solution designed to accelerate and optimize file transfers across global networks. FileCatalyst® is immune to the effects that latency and packet loss have on traditional file transfer methods like FTP, HTTP or CIFS. FileCatalyst® addresses these issues for enterprise WAN, satellite and wireless communications, media and content providers, and government and military organizations.

For more information, visit the FileCatalyst website at: www.filecatalyst.com

As part of our ongoing discussion on fast file transfers, or “acceleration,” (which we began with the article “Accelerating File transfer“), we have already talked about how transfers can be “virtually” accelerated by minimizing the data sets. This is done with compression or sending file deltas. In short, less data = faster transfer.

An acceleration solution should also optimize your link. This takes us out of the realm of “virtual” speed boosts and into the realm of more data making it across the line in less time. One way to accomplish this is to use multiple streams.

As a TCP-based protocol, FTP is sequential in nature. For the easily distracted, the problem with this is that if there are problems with this sequence (the cycle of acknowledging packets) such as lost packets and latency, TCP transfers are throttled. You can compensate for this bottneck at least in part by sending more than one chunk of data at a time, be it separate files or separate pieces of the same file.

For the braver souls, here’s the techno-babble:

Latency and its effect on FTP

In order to provide reliable data transmission, TCP/IP requires the receiver to acknowledge each packet being sent, in sequential order. Each such communication is measured as round trip time (RTT), or the time it takes for a packet to be sent and acknowledged.

TCP responds to latency by adjusting the amount of unacknowledged data that can be on the link before waiting for a reply. The optimal amount of unacknowledged data en route should equal the end-to-end bandwidth multiplied by the RTT, also called the bandwidth-delay product. TCP continually estimates this value, setting a “TCP window” to control how much data should be sent. TCP has limits on the size of this value, so when the bandwidth-delay product exceeds a certain threshold, the result is a lot of waiting or “dead air”. Satellite connections can be into the hundreds or even thousands of milliseconds of RTT.

Packet loss and its effect on FTP

Network congestion typically causes buffer overflows of intermediate routers, ausing packet loss. Since packets are sent sequentially, this can cause a hold-up in the cycle. Unacknowledged packets also cause the TCP window to shrink or even close completely for periods of time. Wireless and satellite transfers can have even higher packet loss due to sources of interference such as clouds or physical structures. Packet loss combined with high latency creates even worse performance for FTP transfers.

Acceleration with Multiple Streams

The idea behind multiple streams is to reduce the “dead air” created by the acknowledgment cycle. Instead of using 1 stream, multiple streams are opened, each sending a different piece of data. In theory this could be separate files, but this wouldn’t help transmission of a single large file. Instead, FileCatalyst will divide a file up into separate chunks: in a 3-stream situation, for example, stream 1 might send chunks 1, 4, 7, 10, etc; stream 2 would send 2, 5, 8, 11, etc; stream 3 would send 3, 6, 9, 12, etc. This way, there is almost always SOME data going across. No “dead air”. And even if chunks arrive out of order, they are just “slotted” into place accordingly. Sounds good, but it’s important to recognize that there ARE diminished returns on opening too many streams (and your hardware I/O needs to be able to keep up in order to make it worthwhile) but for the most part having multiple streams will produce a reasonable amount of acceleration.

When applied to TCP-based transfers such as FTP, you can see the benefit of multiple streams. Because of this, FileCatalyst acceleration technology includes multiple TCP streaming for situations in which UDP-based transfers are not possible. However, this approach still does not produce the same dramatic effect as ditching TCP in favour of alternatives like UDP. UDP-based transfers, which are at the heart of the FileCatalyst acceleration technology, will be the subject of our next article on acceleration.

Cheers,
Greg