As mentioned a short time ago, we have some material from our old blog that we are slowly migrating to the new one. Being that we are in the business of fast file transfer, it made since to bring back one of our more popular articles on the subject. Although written around the time of FileCatalyst 2.0 (2.7.x is the current generation), the information is still relevant, and the article continues to be widely read by visitors to the main site. Here it is, for readers of the new blog.

Original article written by Chris Bailey on July 31, 2007 9:53 PM

---

 

Fast File Transfer: Moving beyond acceleration with data optimization

If you are in the market for a fast file transfer solution, you have a couple of options. First there is the traditional FTP client software, some of which use parallel TCP streams to speed up your transfers. Then there are the UDP based transfer applications, FileCatalyst is one of them. These applications can maximize the data across your internet connection regardless of network conditions. If you have a T3, you will get exactly T3 speeds. The other approach to fast file transfer is data optimization.

By reducing the data that needs to be transmitted, you can effectively transfer the file faster. Even if the actual data going across your line is not optimal, you may still get faster rates because of the data reduction ratio. Consider a database file, or large spreadsheet. Since this data is highly compressible, you can reduce transfer time significantly just by zipping it up.

Another way to optimize file transfers is to send only portions of a file that have changed. Consider the database mentioned above. You need to back up this database on a daily basis to a remote location over a T1. The database file is 2 GB. If you maximize your T1, it could take almost 3 hours to transfer the file each day, even with the best acceleration on the market. But what if only 100MB had changed in the file? If you could detect and transfer only the portion that has changed you would reduce the transfer time by a factor of 20. Now the transfer only takes 9 minutes!

But hold on, that database is probably compressible as well, so even with only a 2:1 compression ratio you could cut that transfer time in half again. So now it is only 4 and a half minutes, or 40 times faster than your link speed!

FileCatalyst 2.0 was released earlier this year and does acceleration, as well as differencing and compression. It does it for you in the background, so there is no wait time to compress the file prior to transferring; it is all done on the fly, from one automated tool. With FileCatalyst, file transfers are as fast as your link, i.e. T1, T3, etc… The only question is how much faster it will go beyond that speed. That depends whether you have transferred the file previously and whether the data is compressible or not.

The table above lists some speed gain examples. Of course there are a lot of cases that do not benefit in any way from this technology, but there are just as many that do. FileCatalyst should be considered an option in either case since it offers the best of all worlds; that is, industry leading acceleration as well as data optimization. You can always be assured you are getting the fastest possible file transfer with FileCatalyst.

To read more about accelerating and optimizing file transfer with the FileCatalyst family of products visit www.filecatalyst.com

Tags: Fast File Transfer, FileCatalyst, ftp replacement

File-based workflow solution accelerates 100 GB of game development files per day

The following press release went out today; if you have not stumbled across it in your travels, have a read! Naturally, we are quite happy with the announcement. Several of our staff are game enthusiasts, and the gaming industry is full of interesting people (like those at SEGA). It works out well that our products are so useful for game development studios. On a related note, we’re down at the Game Developers’ Conference (GDC) in Austin as we speak. Swing by booth 6333 NH.

The press release:

San Francisco, CA March 25, 2009 — Unlimi-Tech Software, the pioneer in accelerated file transfers is pleased to announce the selection of FileCatalyst Workflow by SEGA® Corporation. The deployment of FileCatalyst Workflow allows SEGA to successfully manage and transfer game development files, including game-supporting multimedia files, raw and binary game code.

The FileCatalyst Workflow software application is a unique file-based workflow solution that offers comprehensive tracking and management of files as they are processed through a digital workflow. The uniqueness of the application is exemplified by its ability to accelerate file transfers as they move from one project node to the other. Advanced acceleration algorithms allow for improved bandwidth utilization and data reduction – leading to file transfer rates of 100x faster than traditional FTP.

“We were looking for a solution that would consolidate our disparate file transfer systems into a single file-based workflow application. Our main goal was to improve our productivity and efficiency through this application,” said Jake Salgado, Director of IT, SEGA of America, “We selected FileCatalyst technology because of the speed gains it offered – previously it would take longer than a day to transfer our project files, now we can do it in hours. The time we shave off in our transfers translate to immense productivity gains when you realize we have a global work force in different time zones.”

FileCatalyst® acceleration technology is completely software-based, and is designed to overcome network performance bottlenecks to maximize use of available bandwidth. This technology is packaged into FileCatalyst Workflow, providing file-based workflow. The process is completely customizable and offers automated and scheduling features, email notifications and LDAP/Active Directory integration.

“We are pleased to be an integral part of the applications deployed by the SEGA development group,” said Chris Bailey, CEO of Unlimi-Tech Software, “The gaming industry is experiencing huge issues relating to delivery of the massive amounts of content in today’s games. FileCatalyst allows game development companies to minimize file transfer times, and thus improve productivity across the game development workflow.”

About SEGA of America

SEGA of America, Inc. is the American arm of Tokyo, Japan-based SEGA Corporation, a worldwide leader in interactive entertainment both inside and outside the home. The company develops, publishes and distributes interactive entertainment software products for a variety of hardware platforms including PC, wireless devices, and those manufactured by Nintendo, Microsoft and Sony Computer Entertainment Inc. SEGA of America Web site is located at www.sega.com

About Unlimi-Tech Software

Located in Ottawa, Canada, Unlimi-Tech Software is the creator of FileCatalyst®, the world’s leading file transfer solution. Founded in 2000, the company has more than seven hundred and fifty commercial and government clients. FileCatalyst® technology is a software-based solution designed to accelerate and optimize file transfers across global networks. FileCatalyst® is immune to the effects that latency and packet loss have on traditional file transfer methods like FTP, HTTP or CIFS. FileCatalyst® addresses these issues for enterprise WAN, satellite and wireless communications, media and content providers, and government and military organizations.

For more information, visit the FileCatalyst website at: www.filecatalyst.com

Tags: Fast File Transfer, file transfer, FileCatalyst

FileCatalyst Direct and Express have been updated to version 2.7.1. As an incremental release, 2.7.1 is primarily a maintenance release, tightening up a few features and adding a few small improvements. 2.7 introduced progressive transfers on file deltas (see our recent article on minimizing data sets) but for uploads only. This release adds the functionality on downloads as well, good news for people with a 2-way workflow who are transferring large files.

Here are the release notes in brief:

• Transfer of deltas file may now be done progressively (start to transfer data while delta file is being built) for both upload and download
• Ability for administrators to see client type and version in Server Remote Admin
• Confirmation dialog before canceling transfers using the applets
• Ability in upload applet to remove browser pane
• “slow start” renamed to a more accurate “congestion control” and default control aggression lowered
• Improved: resuming large delta transfers
• Improved: compatibility of MonitoringAgent with particular SMTP servers

This is a recommended upgrade for any organizations using 2.7.0 or below. Please contact your FileCatalyst representative, or log into the download area to obtain your upgrade.

Please note that the above notes cover the FileCatalyst Server and Client applications as a whole; however, not all features appear in the entry-level FileCatalyst Express solution.

Tags: Acceleration, Direct, Express, Fast File Transfer, FileCatalyst, HotFolder

As discussed on Feb 23, one of the most common questions people ask is, “How do you accelerate file transfer? Are the files compressed or something like that?” The response is that compression is only part of our offering, and an optional part at that. But that doesn’t mean it’s not important. You just have to know exactly what compression can offer in terms of benefits, as well as be aware of what it cannot do.

In terms of acceleration, the benefit is quite simple: if there is less data to be sent, the transfer will be “virtually” faster. A 10MB compressed file will take roughly half as long to send as the original 20MB file, even though the actual line speed is the same. This “virtual” speed gain will depend on the compression ratio, ie. how compressible the files are. Text-based files such as word processor documents tend to be more compressible than binary files such as executables; compressed filetypes such as JPG and MP3 cannot be compressed further at all.

A common way to use compression in file transfer is to compress (zip, rar, stuffit, etc) each file before it is sent, and then decompress the file at the destination. Not a bad method, and it can potentially yield results. But frankly if this is all an acceleration solution is doing with compression, you’re almost as well off just using a script built in-house. Instead, an acceleration solution should build upon the concept of compression and add value to it. Here are a few of the things FileCatalyst does in terms of compression:

• on-the fly compression: Rather than compressing each separate complete file, data is compressed at the block level. A block is compressed, sent, uncompressed, and appended to the file being built on the destination. By compressing on-the-fly there is no wait for the transfer to start, and when the last byte arrives the transfer is also finished—no waiting for a final decompression operation. The only caveat to this method is that it takes up more CPU than scenarios not involving compression.
• compress as single archive: This option becomes useful if you are transferring many small files. By creating just one archive for hundreds of files, you save a huge amount of set-up and tear-down. Saving these operations saves some virtual speed. But then there’s raw speed: the nature of small files means that they often arrive at the destination before line speed is reached; by sending a single archive (ie. one larger file!), the transfer can ramp up to line speed. To maximize efficiency, FileCatalyst’s progressive transfers also allow the file to start transferring even before the compression is finished.
• algorithm options: although the default compression method will work in any circumstance, you may choose from a few different algorithm options that may work just a bit better with particular file types.

There are some creative ways to use compression, with different FileCatalyst client applications/applets offering a number of ways to capitalize on the benefits of compression. These various options mean that you do not simply select “Use THIS kind of compression” globally throughout your application, but are free to pick and choose at a granular level, depending on the needs of any particular task. These options will be explored in later articles.

“Is your file accleration some sort of compression?” No, but compression sure can help!

Greg

Tags: Acceleration, compression, Fast File Transfer, FileCatalyst, HotFolder