MarmotMirror Utility Demostrates Marmot Cloud Storage API

In the past few years so-called cloud storage services such as Amazon Simple Storage Service (S3) and Nirvanix promise to turn data storage into a limitless utility expandable on an as-need basis. The use of cloud storage up until now has required lock-in to a particular vendor: if you choose one service, it may not be straightforward to port your data to another service, and your existing software may not work with the new service's API. GlobalMentor has created a unifying cloud storage API named the Marmot™ API that removes the need for your application to write vendor-specific code. Marmot adds extensive metadata capabilities build on GlobalMentor's Uniform Resource Framework (URF) that work across all underlying implementations. GlobalMentor's Marmox™ Network (including this site) is built on Marmot technology; the Marmox site provides more information on how it uses Marmot.

Although Marmot source code is available now, extensive details and documentation on the Marmot API will be released later. Today I'm releasing GlobalMentor's Java-based MarmotMirror synchronization utility, which demonstrates the power of Marmot by providing a useful way to keep data in sync. MarmotMirror is a command-line Java utility that takes two storage sources called repositories and, based upon a set of options, synchronizes the resources in those repositories, along with their metadata. Today's release of MarmotMirror supports file repositories and WebDAV repositories, meaning you can synchronize two file system directories (with NTFS support), two WebDAV servers (with Subversion support), or back and forth between the two types. Future versions of Marmot will add support Amazon S3 and Nirvanix.

To give you an idea of the usefulness of MarmotMirror, consider my current file backup needs. I have over 200GB of data I've accumulated on a local hard drive, including audio files, books, and photographs. I need to be able to mirror this data (adding and/or removing files on the destination as needed) to an external backup hard drive. In addition, this data needs to be mirrored to a WebDAV server that, besides acting as a backup for the data, also serves as the live repository feeding portions of this very site through the Marmox Network.

For several years I've performed these synchronizings using the highly recommended SyncBackSE. In order to synchronize this with WebDAV, I mounded the WebDAV server as a local hard drive using WebDrive and then ran SyncBackSE. This all performed reasonably well, except for a few hiccups with WebDrive over the years, as well as the irritating fact that WebDrive doesn't store resource modified times to millisecond precision. This whole arrangement is no longer needed with MarmotMirror, plus I now have extensive metadata and cross-platform support—anywhere Java runs.

To use MarmotMirror, just install Java 6, download the com.globalmentor.jar library, and start MarmotMirror with no arguments to see the various options:

java -classpath com.globalmentor.jar com.globalmentor.marmot.MarmotMirror

MarmotMirror performs synchronization on three levels: on a per-resource basis, a resource content basis, and a resource metadata basis. You can set the resolution for all of these aspects or for each aspect independently.

Let's start with the file backup. I want to take my data on D:\ and make an exact mirror of it on U:\data. That is, when I add files to D:\, I want them to be added to U:\data, and when I remove files from D:\, I want them to removed from U:\data. MarmotMirror defaults to a resolution of backup, so this is simple:

java -classpath com.globalmentor.jar com.globalmentor.marmot.MarmotMirror --source-repository=D:\ --destination-repository=U:\data

So far we only have hard drive backup functionality, utilities for which are a dime a dozen. But MarmotMirror can backup to WebDAV just as easily:

java -classpath com.globalmentor.jar com.globalmentor.marmot.MarmotMirror --source-repository=D:\ --destination-repository=https://dav.example.com/data/ --destination-username=admin --destination-password=secret

And don't forget metadata. The Marmot API has very rich metadata facility using the Uniform Resource Framework (URF) that rivals or surpasses the capabilities of the W3C's Resource Description Framework (RDF). The Marmox Network makes extensive use of URF metadata using the Marmot API. This blog entry (hosted on Subversion) uses Marmot URF metadata for its title and user comments. My photography section (hosted on WebDAV) uses Marmot URF metadata for photograph titles and to set user access levels. When I add new photographs to my local hard drive, I want all the new photographs to be copied to the WebDAV server. But when I go to my site and update a photograph title, I want that metadata to be copied backed down to my local hard drive. With MarmotMirror, this is easy, too:

java -classpath com.globalmentor.jar com.globalmentor.marmot.MarmotMirror --source-repository=D:\ --destination-repository=https://dav.example.com/data/ --destination-username=admin --destination-password=secret --resource-resolution=backup --content-resolution=synchronize --metadata-resolution=consume

Notice how I've specified custom resolutions for each level. For the per-resource level (when a resource exists only on the source or only on the destination, but not both) I choose backup, which will effectively mirror the file to the WebDAV server. For the resource content level (that is, if a resource exists in both locations but the content is different), I choose synchronize, which will update both the source or the destination with whichever has the latest version. Lastly, for resource metadata-level synchronization I choose consume, which means that any changed metadata added at the destination will be added to the metadata of the source resource, but no source metadata will be removed if removed from the server.

Yes, that's right—MarmotMirror stores extensive per-resource metadata, even when stored on a file system! Remember the Windows Future Storage (WinFS) that Microsoft promised for Longhorn and then finally killed off? Marmot brings some of the same capabilities, but with a loosely-coupled design that works across platforms and transparently with cloud storage. WebDAV already comes with a method for saving resource-level metadata; Marmot rides on top of this and brings the namespaces, the robust types, and the directed graphs of URF. File systems don't have such a metadata capability, so Marmot by default creates metadata sidecars. But if your file system is NTFS, you can ask Marmot to store its metadata in NTFS Alternate Data Streams (ADS) that are hidden inside the file, automatically getting transported whenever you move the file manually! A single option is all it takes to specify that the source and/or destination repository is an NTFS file system:

java -classpath com.globalmentor.jar com.globalmentor.marmot.MarmotMirror --source-repository=D:\ --source-repository-type=ntfs destination-repository=https://dav.example.com/data/ --destination-username=admin --destination-password=secret

Marmot even supports Subversion running on WebDAV if autoversioning is turned on, which means that you can backup up an entire file tree to Subversion, and keep track of all modifications to it over time—even changes to resource metadata through the common Marmot API!

java -classpath com.globalmentor.jar com.globalmentor.marmot.MarmotMirror --source-repository=D:\ --source-repository-type=ntfs destination-repository=https://svn.example.com/data/ --destination-repository-type=svn --destination-username=admin --destination-password=secret

This is just a taste of the sort of power the Marmot Cloud Storage API can bring. Marmot support for Amazon S3 and/or Nirvanix will be available in the future, and you'll be able to access them with the same API, as well as use MarmotMirror to synchronize your resources and metadata across cloud storage vendors.

You can download the entire GlobalMentor library, including MarmotMirror, from https://dav.globalmentor.com/public/software/com.globalmentor.jar. You can get the source code from https://dav.globalmentor.com/public/software/com.globalmentor.src.jar, or access the latest revision directly from Subversion. Marmot and the rest of the GlobalMentor library source code is licensed under the open-source Apache License 2.0. (As the license explains, this software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.)