TACTIC Replication System

From Lunatics
Jump to: navigation, search

TACTIC is a project-management system originally developed for a commercial animation studio.

It appears to provide a lot of features we need, but it's also very complex. One thing we're going to be doing is setting up a storage areas for our digital assets.

I originally wrote this up in a Production Log post, TACTIC Replication & Backup, which outlines the IT problems:

http://wp.lunatics.tv/wp-content/uploads/2017/08/TACTIC-replication-2017-inv-1024x791.png

OverlayFS

This is new in the Linux Kernel -- support for a multi-volume file-system. We're going to use this to keep our data archives aligned with our back-up system on BD-M media (hence the 25 GB volume size).

I found some introductory articles on setting up OverlayFS:

https://www.datalight.com/blog/2016/01/27/explaining-overlayfs-%E2%80%93-what-it-does-and-how-it-works/

https://askubuntu.com/questions/109413/how-do-i-use-overlayfs

https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt

Filesystem Replication

Some possible help on setting up file-system replication. The reason I think this can probably be done with rsync (rather than requiring git or some other version-control system), is because we are synchronizing at the FILE level (no line-by-line merges needed or possible). However, TACTIC may influence this decision. More research may be needed.

Data Replication Using rsync (Linux Magazine)

Why Replicated Filesystems Are Hard

Near-Realtime Rsync Replicated File System (Apache Helix)

Using rsync to Transfer and Synchronize Local and Remote Systems

s3fs

The "s3fs" filesystem is a filesystem implemented to run on top of Amazon Web Service's "S3" cloud storage system (or any cloud storage implementing the S3 API). What we really want to do is to set things up so that the local instance sources files from our local store, while the remote instance gets them from S3 storage, but otherwise the two behave identically (and the two stores are synchronized).

s3fs package in Debian 'Stretch'

s3fs on github

FUSE-based file system backed by Amazon S3 (github s3fs)

How to Mount S3 Bucket on CentOS and Ubuntu using S3FS

How to Mount Amazon S3 as a File System


Other FUSE Filesystem Stuff

There may also be some tools of use for this. FUSE is a user-level bridge that allows filesystems that are coded in user-space (meaning an ordinary process, as opposed to part of the kernel):

SFS Asynchronous filesystem replication (SyncFS)