SynchronEX - Advanced File Synchronizer, Backup & FTP
Documentation
Contents
0. Introduction
1. Installation & Setup
(GUI Wizards)
2. Advanced Project Examples
3.1 Command Line Syntax
3.2 Project File Syntax
3.3 Project File Syntax Details
3.4 Advanced API
3.5 Built-In File Systems
4. Technology Background & Special Topics
5. FAQ / Tips & Tricks
6. FAQ / Trouble Shooting
Quick Links:
XVS/UXVS Version Tracking ( Journaling virtual file
system )
RCY ( Recycle journaling file system )
TFS (
Time-stamp-only virtual file system )
HTTPS ( Secure WebDAV/WebFolders over HTTPS /
SSL)
SFTP ( Secure FTP over SSL/TLS )
FTPS ( Secure FTP over SSL/TLS )
XFTP(S) ( Extended FTP file system for
bi-directional sync )
IGZ ( GZip auto-compression file system -
working on any other
file system )
0. Introduction
SynchronEX = Easy-To-Use GUI Frontend + Robust File Synchronizer Engine
SynchronEX is optimized for automation.
Independent from GUI, the core engine can be called from command
line, from batch files, from scheduled jobs (Wizard
interfacing
to Windows
Job Scheduler) or from other programs. The SynchronEX GUI frontend
enables easy setup of projects
through Wizards and also displays results of simulations and
exec runs
nicely.
Extra tools like a duplicate file finder complete this software.
SynchronEX
projects can be executed directly from the Windows Explorer
simply
like shown below. The project configuration files of SynchronEX are human
readable and can also be evolved into complex automation scripts (Python
language). A most simple project script would be like this:
|
#C:/texts/config.syncpy
|
|
Windows explorer integration
|
This way you can organize all your "projects" in regular Windows link folders
|
Newsletter
Order a free newsletter to be informed about updates, improvements, tips & tricks. Your data is only used for that purpose.
Download / Update
SynchronEX Homepage: Download /
Update.
Buy Now / Registration
SynchronEX Homepage: Buy
Now.
Features
SynchronEX Homepage: Feature list
1. Installation & Setup
1.1 Installation
Run the SynchronEX-Setup.exe installer.
Later versions can be installed over previous versions without prior un-install.
1.2 Usage:
- GUI : Run SynchronEX from the Windows start menu. After a 'Welcome' dialog run the project Wizard in order to set up your first project. Press the "Simulation" button. If ok, press the "Run" button.
- Scheduled Jobs : Consider Menu/Tools/Schedule Task after your projects works upon direct execution.
- Parametric scripting & programming: Consider
reading the project file syntax, the example.syncpy
& template.syncpy in the SynchronEX directory (GUI Wizard:
Script Template) and the example scripts below and
the
- Command Line Usage: command line
syntax
Windows Explorer Annotation : Changing the default Explorer (double-click) action for .syncpy projects (context menu):
- Go to "Windows Explorer/Extras/Folder Options/Types/.syncpy/Advanced ..."
- Select a new default action.
- Also new commands can be added : See command line
syntax
Annotations for command line usage:
-
The command line option '-s' runs everything as simulation: You can play save without really executing file operations. See
command line syntax - SynchronEX can be installed/copied to your favorite 'bin' directory (e.g. C:\bin). Consider re-running Menu/Config/Re-Register .syncpy Explorer Context Commands or run 'synctree.exe --setup'
General Annotations:
- Unless you are doing simple one-way incremental backups, the usage of a synchronization database (-y / syncdb=1) is recommended in order to detect concurrent change collisions. Hard synchronization (syncremove=1) automatically sets syncdb=1.
- Slash & Backslash usage ( \ / ) : In the manual scripted project files usage of slash / is preferred for directory notation like in roots = ['c:/home/texts','//remote/texts']. Otherwise use double-backslash like in 'c:\\home\\texts' or python-raw(r)-string notation like r'c:\home\texts'.
1.3 Project Setup Using GUI Wizards
The SynchronEX Project Wizards can invoked by Menu/File/NewProject(Wizard)
Wizard :: Directory-to-Directory Backup
Enter the source root directory and the target root directory. Optionally follow "Next" to enter files & directories for exclusion/inclusion and optionally follow "Next" to the advanced options.
Path names can be entered either in Windows notation with
back-slash (
\ e.g. C:\data\some\files
) or in
modern Unix notation with normal slash (
C:/data/some/files ) or mixed.
Note: In the resulting project script back-slashes are usually escaped
as
double-backslash in conformity with the Python script language style.
Wizard :: Fast Incremental FTP / FTPS Upload
SynchronEX' fast incremental FTP uses a shadow folder
for computing the changed files. Direct
comparison over FTP would be slow & weak because file dates are not
preserved during FTP transfer, ASCII
transmission alters the file, file names may be mangled etc ...
(See TFS option in order to avoid consuming extra disk
space).
If you do not already have a shadow folder, simply create an empty one like C:/www-sync. You can also copy the current source folder contents to the shadow folder in order to start incremental FTP from current state. All file which are not present in the shadow folder or which are changed in comparison to the shadow folder are transfered to the FTP remote tree and after successful transfer synchronized to the shadow folder.
Thus, the shadow folder stands as an exact copy of the current FTP remote tree. ( As a side effect this shadow folder is also very useful as kind of backup copy of the current remote tree, while editing the "edit tree". )
Password Encryption: When checking this option, the password is not stored in plain manner in the project script, but encrypted. The encryption is done using a finger-print of the current PC. Thus this encrypted password cannot be used or decrypted on an other PC.
TFS: Consider also using the "TFS:" file system (TimeStampOnlyVFS) for use with the shadow folder. Thus the shadow files on TFS virtual file system have zero-length, do not consume disk space and carry only time stamp information. Example:
roots = ["C:/www", "tfs:C:/www-shadow"]
roots_no_src = [1]
shadow_copy={ 1:"ftp://...." }
FTP File System
FTPS File System
XFTP File System
TFS File System
Wizard :: XFTP ( Bi-Directional FTP/FTPS Synchronization )
XFTP is is an eXtended FTP file system, which stores the original file modification time stamp for a file like
ftp:<dirname>/<filename>
in an auxiliary file
ftp:<dirname>/XVS/.time.<filename>
(see also XVS File system).
For other options of this Wizard see FTP
Upload Wizard
The extra time stamp information is written
as GMT flat time value. Thus the XFTP file system is suitable for
full-featured bi-directional
synchronization throughout all time-zones.
XFTPS is to XFTP as FTPS to FTP : it uses SSL/TLS as transport
layer.
Wizard :: Bi-directional synchronization (e.g. Laptop <-> PC/Server)
Use this wizard to set up a true bi-directional synchronization. E.g. between a Laptop and a PC folder.
Enter the 2 root folders and follow the wizard. An especially important selection has to be done on the 3rd wizard page: Detect collisions & synchronize removals. See the extra help of that wizard page.
Note: if you check the "target only" option for a root, that root is never a source (master) folder but only a target (backup) location.
Wizard :: Native DAV Synchronization (HTTP / HTTPS)
WebDAV is a modern full featured bi-directional WAN file system, often considered as more reliable replacement for FTP. This file system is supported by current web servers like Apache with mod_dav and other DAV servers. DAV over HTTPS is excellent for secure encrypted WAN file exchange.
Password Encryption: When checking this option, the password is not stored in plain manner in the project script, but encrypted. The encryption is done using a finger-print of the current PC. This encrypted password cannot be used or decrypted on an other PC.
DAV File System
XFTP File System
Wizard Page :: Exclude & Include Files & Directories
Filename wildcards ( * ? ) can be used anywhere in exclude & include masks. The * wildcard matches also across directories.
Annotations:
Relative path excludes & includes match in any
directory.
Relative paths are usually used like *.tmp;*.bak;special.file (Files)
or
CVS;resv (Directories).
For relative paths you can use also patterns mixed with
relative subdirectories like ( e.g.
relative file exclude pattern "resv/*.dat"
matches "C:/texts/sub/sub/sub/resv/myfile.dat" when
"C:/texts" is a root)
'Rooted' (absolute(1)) paths
match starting at the root
folder.
Example: rooted exclude file pattern
"data/sub/my*.txt" matches
on C:/texts/data/sub/mytext.txt
when 'C:/texts' is root, but does not match
C:/texts/sub/sub/sub/data/sub/mytext.txt )
Annotation on Directory Exclusion & Inclusion:
Currently excluded directories can only be revoked by
inclusion on the
same tree level ! Inclusions at deeper levels have to be bridged - e.g.
like..
excludedirs_abs = ["*"]
includedirs_abs = ["sub","sub/data","sub/data/*"].
As intended, this example wouuld not re-include a directory
"sub/otherdata"
because it matches exclude through "*" and does not match a
re-inclusion pattern.
A revoking inclusion just like ...
includedirs_abs = ["sub/data"].
.. is not sufficient to work as intended.
[ In the future this behavior may be changed for trivial
tail-only-wildcard
patterns in order to detect "include bridges" automatically. ]
Footnotes:
"rooted/absolute": absolute in relation to the root directory (not "absolute" in the sense of absolute file system paths)
Wizard :: Options
Synchronize removals
(Script: syncremove=1)
When checking this option, file/directory removals (remove actions) are also synchronized.
The trigger for a removals is : File/directory was there during last synchronization or backup; File/directory is missing now => Remove action. Thus it is a kind of "Edge-Trigger". Thus removals can only be detected after an initial first sync run, which creates the sync database as a kind of starting point !
Detecting file removals requires the synchronzation database. Thus syncdb=1 (and detecting collisions) is implicit with syncremove=1. Thus it is not necessary, but optional to set syncdb=1 explicitly.
Track a synchronization database and detect collisions of concurrent file changes
(Script: syncdb=1)
When checking this option, the status of files during a synchronization run is remembered in a 'synchronization database' file. This enables the detection of file synchronization collisions upon concurrent file changes. It also enables the detection of removed files (see syncremove=1 above).
An example of a file change collision: You have roots C:/texts and //laptop/texts. After an (initial) successful synchronization the file C:/texts/myfile.txt and //laptop/texts/myfile.txt are both edited : a concurrent file change. Upon next synchronization run it is unclear which file should be the new master file and only the human can decide. Thus when enabled syncdb=1 option, SynchronEX reports a file collision and does not copy this file at all.
This database file has by default the name <syncpy-project-filename>.syncdb and sits by default in the same directory as the .syncpy project file itself. E.g. When you have a project file C:/texts/laptop.syncpy and syncdb=1 option, then a synchronization database file C:/texts/laptop.syncpy.syncdb is created and tracked. You can alter the location of the sync database file by force when setting e.g. ..
syncdbpath="c:/outside/mysync.syncdb"
.. in the project script (See scripting documentation).
Continue operation upon errors
The default behavior is to stop the synchronization/backup run whenever any file cannot be copied/overwritten as intended. The 2nd option allows to continue the sync operation when an access error occurs for single files.
A 3rd option allows to continue the overall project script operation (e.g. multiple syncs and sub syncs ), when the a partial sync run cannot be completed (e.g. when a root/drive/network cannot be accessed ). This option is usually used in Super-Sync project files or in more complex scripted projects.
Threshold time
Only files younger as the date/time specified are copied in synchronizations/backups.
Simple log file:
A simple list of copied (and removed) files is written to this log file. The default behavior is to overwrite this log file upon each sync/simulation run.
Note: This kind of log-file can be very useful for post-operations upon synchronized files it in scripts. E.g. after an synctree.do() do a Pythonic loop like: for file in open("_syncsrc.lst"): print file;
Full log file:
The output shown in the console window is also logged to this "full log file". The default behaviour is to append the log contents to the file upon each sync/simulation run. Thus the log file maybe reset manually by deleting it.
The full log file the the file best for tracking synchronizations and backups exactly e.g. for tracing back problems later. This is especially recommended for unsupervised automatic runs.
Note: When setting up a scheduled sync/backup task with the Scheduler Wizard, the log file is usually set up from inside the Scheduler wizard ( and passed to the project by the -L command line option ).
Verbose console output
Additional sync/copy information is displayed in the console (and full log file). During FTP uploads also a detailed (debugging level) log of all FTP commands is displayed.
Check if project file is located in one of the roots folders (helps avoid muddling sync projects)
By default this option is checked: SynchronEX wants the .syncpy project file to be located in one the sync / backup root folders. This behavior helps to avoid copying project files unintentionally to strange folders. Uncheck this option if you want to store the project file outside of a root folder.
XVS version tracking of overwritten and removed files ( on local & Windows-network-UNC paths )
XVS enables to preserve and track historic versions of your valuable files. This provides the most save sync and backup mode you can think of.
Example: An overwrite of D:/backup/texts/hello.txt first copies D:/backup/texts/XVS/001.hello.txt , 002.hello.txt , etc... as kind of "in-place" backup files.
Thus you can look up and roll back the files you changed to any prior state (manifest during a synchronization run) !
When checking this XVS option checkbox in the options dialog, the xvs: file system tag is attached to all sync roots. In case you want XVS only for one of the roots (e.g. for a backup target), then do not check this checkbox, but prepend 'xvs:' to that root folder (1st wizard page). E.g. have 2 roots like "C:/texts" and "xvs:D:/backup/texts" .
For more XVS documentation see XVS.
2. Project Examples ( Use Cases )
This chapter presents projects at project script file level (& command line examples) . Consider using a wizard in the GUI before reading here.
- Synchronize your C:/texts and all sub-directories on Laptop and PC
- Simple command-line-update-sync from . to ./backup without project file
- Manual incremental (or initial) backup to ZIP archive
- 3-point synchronization in a development project
- Super-sync binding multiple sub-syncs
- Grab new Photos from a digital camera with one-click
- 2-timer sync with different parameters
- Synchronize non-recursive (no sub directories)
- Hard sync for 2 equally ranked repositories (copy + remove)
- Working in-sync on two machines with a mobile (ZIP) disk in between
- Find duplicate files from current directory or in 'roots'
- Find empty directories starting from current directory or in 'roots'
- Rotating incremental backup every week to another zip archive
- Incremental FTP upload (of a web site)
- Script example: Recycle backup of files for remove
- Script example: Moving a source tree to target
- Auto-Periodic FTP synchronization
- Synchronization to WEBDAV (DAV; Webfolders like http://server/dav ) shares
- Sending a notification email after job
- Bidirectional X-FTP Synchronization with Read/Write-Lock
2.1 Synchronize your C:/texts and all sub-directories on Laptop and PC
|
# c:/texts/config.syncpy |
Double-click config.syncpy for simulation. If everything
seems to be ok, right-click config.syncpy for action "Execute".
or run from command line:
C:/texts> synctree -s
(simulation)
and then
C:/texts> synctree
(real execution)
The configuration file config.syncpy
it taken by
default, otherwise spec like synctree
mysync.syncpy.
2.2 Simple command-line-update-sync from
. to ./backup without
project file
synctree -z -r backup
| |
| have . (default) and ./backup as sync
roots
|
no project file
This is in fact a "nested" sync, because "./backup" is a
subdirectory of "."
Of course nested sync roots are handled correctly in order to prevent
from (endless) recursive copying.
2.3 Manual incremental (or initial)
backup to ZIP archive
|
#myzipbackup.syncpy |
roots_no_src = [1] means :
the 2nd root (zip:c:/data/backup.zip)
is target only / not a source. This defines a one-way backup.
The zip file is create on the fly if it doesn't exist. If it already
exists, it
is an incremental backup.
Zip file locations maybe use the same way as ordinary directories. Even
intra
zip directories like
zip:c:/data/backup.zip//subdir
are possible (note the double slash after .zip ! ).
The same example as manual call:
synctree -z -r
zip:c:/data/backup.zip
| |
| have . (default) and zip-file as sync
roots
|
no project file
and including the one-way backup restriction:
synctree -z -r zip:c:/data/backup.zip -n1
-n1 : sync root backup is not a source!
2.4 3-point synchronization in a software
testing project
|
#config.syncpy >>> |
syncdb=1 allows save multi-point authoring. Concurrent changes
of files
are
detected.
2.5 Super-sync binding multiple
sub-syncs
|
#C:/config.syncpy (the
super-sync) >>> |
Annotation: If you provide additional parameters in this
super-sync-file, the sub-syncs inherit those parameters. (Command line
options are effective for all sub syncs too.)
2.6 Grabbing new photos from a
digital camera
|
#D:/Photos/incoming/config.syncpy |
2.7 2-timer sync with different parameters
|
#C:/home/texts/config.syncpy
>>> |
2.8 Synchronize non-recursive (no
sub directories)
|
#C:/home/texts/config.syncpy
>>> |
Command line variant: C:/home/texts>
synctree -z -r
//laptop/c/home/texts -x *
2.9 Hard sync for 2 equally ranked
repositories (copy + remove)
|
#C:/home/texts/config.syncpy
>>> |
syncremove is an advanced option, which implicitly uses syncdb=1. If you have a file remove action on one instance (except roots_no_src members) the remove action is detected through syncdb and is also "synchronized" to the other non-roots_no_src instances. Having syncremove=1, it is strongly recommended watching a simulation run (-> remove count) before really executing. Or use syncremove option "-+" in explicit cases.
2.10 Working in-sync on two machines with a mobile (ZIP) disk in between
Also this hopping sync case without direct network connection
is no problem with
SynchronEX.
It is consistent, valid and effective practice, including collision
detection on
concurrent changes, when syncdb=1 or syncremove=1.
HOME-Computer <--->
Mobile (ZIP) Disk <--->
WORK-Computer
For example on your home computer you could have this:
|
# c:/texts/config.syncpy |
... and on your work computer you have this
|
# d:/texts/config.syncpy |
You could also place both project files central on the
ZIP-disk for example as home.syncpy & work.syncpy.
With syncdb=1 or
syncremove=1 provided, you could
also carry a piece of a server 'away' on disk, work on it and re-sync
with valid
consistent collision
detection!
2.11 Find duplicate files from
current directory or in
'roots'
C:/texts> synctree
--find_dup=fname,content --more20
C:/texts> synctree
--find_dup=content >
duplist.txt
fname : filenames must be
equal
content : the content must match exactly
--more20: stop display after one page
> duplist.txt : writes the results into a text file for comfortable
analysis
Starting recursively from the current dir (c:/texts), this lists the
duplicate
files that are equal in filename and content, but may differ in time
stamp (time
criteria missing). The second example writes the duplicates into a text
file.
Also in the second example comparison is solely based on the file
contents. This
finds duplicates equal in content, but maybe differing in filename!
Read more.
For checking duplicate files in a .syncpy configuration, right-click 'Find Duplicates' on that project file according to the installation.
2.11b Find empty directories
starting from current directory or in
'roots'
C:/texts> synctree --find_emptydirs
--more10
C:/texts> synctree --find_emptydirs > emptylist.txt
fname : filenames must be
equal
content : the content must match exactly
--more0 : enable keyboard key 's' to stop &
page display
> emptylist.txt : writes the results
into a text file for comfortable analysis
.. or right-click 'Find Empty Dirs' on a .syncpy project file according to the installation.
2.12 Rotating incremental backup every week to another zip archive
|
#c:/texts/cycbackup.syncpy |
Rotating backup to C:/backup/work00.zip
..
work03.zip ... stepping every week. It is not necessary to
initialize the
zip files beforehand, they are created on-the-fly if not already
existing.
Idea: This script could be evolved to a hierarchical multi-period
backup
rotating for days, weeks, months ( shutil.copy2(
bkdir , "work_month%02d_backup.zip" % iMonth) ...)
Zip files can be used in same style as ordinary directories.
Also intra zip
directories like
zip:c:/data/backup.zip//subdir
are valid backup roots (note the double
slash after backup.zip ! )
2.13 Incremental FTP upload (of a web site)
|
#C:/www-edit/ftpupload.syncpy |
This project uploads (incrementally) your changed web files to your ftp-account in a rush. In this example the website tree is edited in C:/www-edit. The shadow copy is hold/maintained in C:/www-sync. Removed files are also removed on the ftp server.
Password handling: If you don't want to specify the user:pwd as shown you may also set only the user like in 'ftp://user@myserver.yyy/subdir'. Thus, the password is requested interactively. (If you have your password in the project file, don't forget to add '*.syncpy' to the excludefiles_rel list in order to not upload your password.) If your username or password contains the '@' character (example: myname@xdomain): quote it like myname\@xdomain. This is necessary, because unquoted '@' delimits user:pwd from the ftp server name. The GUI wizard takes care of this issue.
Each file is first transferred over FTP, then it is copied to the shadow dir. Thus, on ftp failure or broken transmission, the file will be re-transmitted on the next run.
ftp_ascii=ftp_executable = ['*.php','*.pl','*.py','*.cgi'] ensures that your CGI-scripts are uploaded in ASCII mode and that their executable flag is set on the server. You could also narrow down such declaration to special directories like ftp_ascii=ftp_executable = ['cgi-bin/*.pl','*.cgi']
This FTP mechanism can also be used for remote ftp data
backup on the
intranet/internet. But consider also WEBDAV for that purpose.
See also the Auto-Periodic FTP example here.
Annotation: ftp_shadow is an alias for shadow_copy (new in v3).
2.14 Script example: Recycle backup of files for remove
|
#C:/music/mp3.syncpy |
During simulation mode this sync backs up files for remove to C:/backupmusic. On exec run the hard sync is executed normally. Of course one could program arbitrary execution behavior, for example a joint recycle backup + sync job in one script by calling .do() twice and setting appropriate simulation flag.
2.14b Script example: Moving a source tree to target
|
roots =
['src_folder','dst_folder'] |
After successful copy operation, the source files are removed.
2.15 Auto-Periodic FTP synchronization
There are many possibilities to execute a SynchronEX script periodically by an external scheduler (Windows Task Schedule (Wizard in SynchronEX), crond for Windows, ...). A call like this should be scheduled:
c:\bin\synctree.exe c:\www-edit\ftpupload.syncpy >> c:\ftpupload.log
However you can also script a periodic sync inside a project file itself like this:
|
#C:/www-edit/periodicftp.syncpy |
2.16 Synchronization to Windows mounted WebDAV (DAV; Webfolders like http://server/dav ) shares
This example synchronizes to a WEBDAV folder. WEBDAV folders
are useful for WEB uploads as an alternative solution for FTP.
WEBDAV folders can also be used as WAN synchronization point for
sharing files
with other users.
This script mounts the WEBDAV folder to the drive letter Z:
|
#
C:/www-edit/davsync.syncpy |
For using DAV as WAN synchronization location consider removesync=1 and commenting out the line roots_no_src... like ##roots_no_src=...
Annotations:
- repeated trials for mounting Z: will cause a warning
message like "..device already used". You can savely ignore
such message. Or you can use alternative commands suppressing the
message:
print os.popen("net use Z: http://my.server.yyy/davsub").read() - WEBDAV shares are supported in recent Windows OS versions
(XP,2000,3000).
There is also a native DAV support (including https) built into SynchronEX. See >>DAV File System
2.17 Sending a notification email after job
Sending a notification email after the job is done:
Login and password can be omitted. Or the password can be loaded from a secured/secret file like: password=open("c:/secure/syncpwd.txt").read()
The mail text body could be rendered from the configured logfile like:
text="Log:"+open(logfile).read()
2.18 Bidirectional X-FTP Synchronization with Read/Write-Lock
In this example a stale lock has to be erased manually. (Automatic lock timeout to be done)
2.19 Catching an Error and Sending an Email
|
roots = ['C:/www-edit',
'C:/www-sync'] |
2.20 Searching the Log File for Errors and Sending an Email
|
synctree.do() |
3. Script Syntax and Command Line Syntax
For examples of syntax application see Examples.
3.1 Command Line Syntax
Usage:
synctree.exe [OPTIONs] [PRJSCRIPTFILE]
PRJSCRIPTFILE:
The ".syncpy" configuration script file. Default: config.syncpy
OPTIONs:
-s force simulation run; files are not really copied
-y use a sync database (.syncdb) and detect Y-collisions
-z dont use config file (OPTIONSs only mode)
-n0,2 1st and 3rd root directory are not sources
-r /foo directory "/foo" is added to the roots list
-e foo.c "foo.c" is added to the excludefiles_rel
-i foo.obj "foo.obj" is added to the includefiles_rel
-x tmp "tmp" is added to the excludedirs_rel
-k foo.tmp "foo.tmp" is killed everywhere
-d 1.2.02 set threshold date
-t 13:20 set threshold time
-v verbose list (incl. filemtime, size)
-w / -W don't wait for ENTER at the end (-W : wait if error)
-l src.log create log list of all source files
-L sync.log full log of console output to file (append mode)
-+ sync file removals also (syncremove=1)
-c read project file from console/stdin
-p don't prompt for interactive questions, just exit with error
-h small help
--help big help
--remove src.lst :remove files in the list! (create with '-l -s')
--more stop display after every 10 lines (--more0/20/40/60) --remove=src.lst remove files listed in src.lst! (create with '-l -s') --find_dup=fname,size,time,content,exact find duplicate files based on selected criteria --find_dups equal to --find_dup=fname,size --find_emptydirs find empty directories in roots or current working directory --regkey enter license key directly --setup bind file classes --template open a project template for edit --help big help --gui start the SynchronEX GUI --guirun run project in GUI
More annotations
on command line options.
3.2 Project File Syntax : Quick Reference
A Complete Synchronex Project File Example
#C:/src/mysrc/test.syncpy
roots = ['C:/www-edit', '//remotecomputer/mysrc', 'd:/backup/mysrc',
'sftp://usr@server.dom/subdir/']
roots_no_src = [2] # 2 --> 'd:/backup/mysrc' is target only
excludefiles_rel = ['config.syncpy', '*.pyc', '*.obj', '*.bak', '~$*']
includefiles_rel = ['special.obj','my*.pyc']
killfiles_rel = ['*.bak']
excludefiles_abs = ['dist/myexe.exe']
excludedirs_rel = ['CVS','tmp']
excludedirs_abs = ['data/bigdata']
# don't copy files older than this:
thres_time = "1.1.2002/14:00"
# record a synchronization database for advanced collision detection (concurrent
# changes), remove control etc.
syncdb = 1
# run these SynchronEX projects in addition:
sub_syncs = ['C:/somewhereelse/backup.syncpy', '../relative/serversync.syncpy']
Overview of Parameters in SynchronEX Project Files (.syncpy) / By Example
(See also the file "template.syncpy" in your SynchronEX program folder or open the "General Template Wizard" in the SynchronEX GUI)
roots = ['C:\\www-edit\\', 'Z:/backup/www-edit/']
roots = ['.', '//UNC_remotecomputer/mysrc', 'd:/backup/mysrc/', u'unicode/dir\u043a/]
roots = ['C:\\www-edit\\', 'sftp://user@server.dom/public_html/',
'https://me@davserver.dom/sub/dir/]
config.RegisterLimitedVFS(
"volX:",
iter_root="c:/backup/mbackup/media%INDEX%", wait_media=0,
auto_mkdir=1, initial_remove=0, count_copy=1,
limit_bytes=14400000, sector_size=512
)
roots = ['C:\\www-edit\\', 'volX:']
roots_no_src = []
roots_no_src = [0,2] # 1st and 3rd root are not source (target only/backup)
automkdir_targets = [0,2] # auto-create this roots if not existing
# (must be target-only too)
excludefiles_rel = ['*.syncpy', '*.log', '*.bak', 'secret.txt']
includefiles_rel = ['special*.bak']
excludefiles_abs = ['presentations/*big*.ppt', 'pagefile.sys']
includefiles_abs = ['presentations/out/big*.ppt']
# note: '_abs' (absolute) means: path from the sync root (not totally absolute)
excludedirs_rel = ['CVS', 'logfiles', 'backup??', '_*']
includedirs_rel = ['backup01']
excludedirs_abs = ['data/bigfiles*', 'install/3rdParty', 'System Volume Information']
includedirs_abs = ['data/bigfiles1']
# those files are removed upon each sync exec! :
killfiles_rel = ['~*.tmp']
killfiles_abs = ['data/temp*.dat']
# define Python callback functions for sophisticated file/dir exclusion:
def cb_is_exclude(rpath):
return os.path.getsize(os.path.join(root[0], rpath)) > 100000
# thus copy only files smaller than 100000 bytes)
def cb_is_exclude_dir(rpath):
return 'SUBSTRING' in rpath
# thuss copy only dirs not having 'SUBSTRING' in the name
# don't xvs-autoarchive versions of these files:
xvs_excludefiles = ['C:/data/big*.dat','D:/cache/*','*.tmp']
# note: These are really absolute paths; or paths relative to the current working directory
# (Thus files in different roots can be addressed sperately)
# track synchronization in database (which allows collision detection etc.)
syncdb = 1
# non-default path for the sync database file:
syncdbpath = 'c:/outside/mysync.syncdb'
# synchronizes file-deletions as well (syncdb=1 is set implicitly)
syncremove = 1
# don't sync files older that this date/time:
thres_time = "2.3.2002/14:00:03"
# parameter for seeing time stamps withing 2s granularity as equal
# (use this only for old FAT Filesystems)
even_time = 1
# force simulation mode ON by project script
simulation = 1
# create new directories only when files have to be created there too
lazy_mkdir = 1
verbose = 1 # extra verbose log output
# continue upon more or less severe error types:
errorcontinue = 4 # currently known levels: 0,1,2,3,4
logfile = '_backup.log' # define a log file (append mode)
logfile = '_backup.log', 'w' # define a log file (overwrite mode)
src_listfile = '_backup.lst' # a short/easy parseable log file (overwrite mode)
src_listfile = '_backup.lst', 'a' # a short/easy parseable log file (append mode)
# check if the project file (prjname.syncpy) resides in one of the roots
check_cwd = 1 # (DEFAULT)
# shadow actions (copy/remove) to certain roots to other target root paths too
# (usually used for fast WAN upload syncs by comparing local roots)
shadow_copy = { 1 : 'ftp://user@myserver1.yyy/subdir/' }
2 : 'https://user@myserver2.yyy/subdir/' }
# note: '1' means 2nd root, '2' means 3rd root, ('0' would mean 1st root)
# example roots: roots = ['C:/www-edit', 'C:/sync/server1', 'C:/sync/server2']
# define which files are transfered as ASCII and which are executable
ftp_ascii = ['*.py', '*.pl', '*.php', '*.tcl', '*.cgi', '*.txt']
ftp_executable = ['*.py', '*.pl', '*.php', '*.tcl', '*.cgi']
ftp_execumask = '755' # umask used to set on executables
# force sync copy even for files which show a collision (concurrent changes since
# last sync on more than root). The younger file is copied over the older.
collision_override = 1
symlinks = 1 # don't ignore symbolic links
# follow symlinks and copy the content:
symlink_follow = 1 # 0/DEFAULT : copy symlinks as symlinks
# explicit call to start synchronization (only need when multiple
# syncs have to be run; or for sophisticated error control and log processing):
synctree.do()
# run other SynchronEX project files in addition to the current one:
sub_syncs = ['C:/texts/backup.syncpy','D:/images/backup.syncpy']
3.3 Project File Syntax : Details
roots = [ "C:/web", "C:/web-sync" ]
2 or more roots folders for synchronization/backup.
Wide unicode root folder names (on unicode file systems like NTFS) can be coded like this:
roots = ['.', u'wide-unicode/dir-\u043a\u043a\u043a/] # /dir-ккк
( u in front of string-delimiter and '\uXXXX' 16-bit HEX character code )
syncdb=1 /
commandline: -y
A synchronization database "<project-file>.syncdb" is tracked.
This enables high
quality synchronization with detection of Y-collisions. A collision
is
detected, when 2 or more instances of a file have changed since
last synchronization run.
syncremove=1 / commandline: -+
This option defines a hard synchronization: positive and negative. If
you remove a file
somewhere (except roots_no_src members), the remove action is also
synchronized
to the other instances. Usage of a sync database (syncdb=1) is defined
implicitely
by this
option.
Defines one-way copying (backup mode). roots_no_src is a list of root indices - starting from 0 - for roots not allowed as sources (master folders). Those roots are 'target-only'.
Example: you have
roots = ['d:/backup0','c:/mydir']
.. then
roots_no_src = [0]
defines that the d:/backup0 directory is not a source, only
c:/mydir is a source.
excludefiles_rel /
excludefiles_abs / commandline: -e
excludedirs_rel /
excludedirs_abs / commandline: -x
includefiles_rel /
includefiles_abs
/ commandline: -i
includedirs_rel /
includedirs_abs
List of file/directory patterns for exclusion & inclusion. see also
Wizard Page Exclude & Include.
See also annotation on wildcards.
cb_is_exclude / cb_is_exclude_dir
Extra
pythonic callback function for checking exclude/include of files or
directories. If this callback is a valid non-None function, this
callback is
called for each file / directory. The callback function must take the
rooted
path as 1st and only argument. The function must return one of these
values:
0 or False : don't force exclude
but check other
exclude masks
1 or True : exclude ! (don't check other masks)
2 : include
! (don't
check other masks)
Example: Copy only files smaller than 100000 bytes) :
def cb_is_exclude(rpath):
return 100000 <
os.path.getsize(os.path.join(root[0],rpath))
def cb_is_exclude_dir(rpath):
return 'ZZZ' in rpath # ZZZ string in
dirname?
Precedence of exclude/include patterns / rules:
- by default all is included (except XVS files) - lowest priority
- except excludes
- but includes included = excludes from exclude = force (overrides)
- dynamic exclude by functions cb_is_exclude / cb_is_exclude_dir (exclude with highest priority)
thres_time='Dec-31-03/14:00' / commandline: -t -d
If this threshold time parameter is set, no files before that time are copied. The thres_time value may eigther US or European string format like "31.01.2003/15:00" or a flat 'seconds since 1970' (c-time/time.time()) value.
An example for a dynamic threshold time (copy only files of the last 24 hours):
thres_time = time.time() - 24*60*60
sub_syncs = ['other.syncpy','d:/data/mirror.syncpy']
Defines a super project calling other projects. If you don't provide 'roots=' in the super-project, the
super-project doesn't copy anything itself. Anyway if you set
additional parameters in the super-project, those parameters are
inherited by the
sub-projects (including the simulation flag). But in most practical
cases a
super-project file contains only a single "sub_sync="
line.
logfile="my.synclog"
logfile="my.synclog","w"
Configures a log file. All text which appears in the console output also goes to this log file By default new content is appended to an already existing log file (open mode "a"). The second example (open mode "w") overwrites the existing log file.
shadow_copy={ 1 : 'd:/backup/diff-dir' }
shadow_copy={ 1 : 'ftp://user:pwd@myserver.xxx:25/subdir' }
shadow_copy copies/removes files which are written/removed at the attached root. The integer-key [1 in the example] defines the index of the attached root.
Example:
roots = ['c:/www-edit', c:/www-shadow]
shadow_copy={ 1 : 'ftp://user:pwd@myserver.xxx:25/subdir' }
Thus changed files, which are synchronized from C:/www-edit
(root #0) to the
C:/www-shadow (root #1) folder, are also copied to (or removed at) the
ftp
server (attached to root #1 by shadow_copy=... ).
Annotations on FTP-URL Format:
The FTP-URL format including username (user) and optional password (pwd) and optional port number (25) is the same as used in internet browsers. If the password (pwd) is omitted, then it is requested interactively.
If your username or password contains the '@' character
(example: myname@xdomain),
then it has to be quoted by backslash like myname\@xdomain. This is
necessary, because unquoted '@' delimites
the "user:pwd"
from the ftp server name.
FTP Passive Mode (PASV):
Some old ftp servers do not support passive ftp mode (PASV). In that case an error like "command not supported" is yielded. Also some network routers may cut off passive ftp transfers ("connection reset by peer", ...). You may watch FTP debug output and/or switch to non-passive mode by inserting:
#C:\www-edit\ftp-nonpassive.syncpy
verbose = 1
ftp_passive = 0
roots=...
...
ftp_ascii = [ '*.cgi', '*.pl', 'cgi-bin/*.py', ... ]
Files matched by this mask are schedule for ASCII FTP transfer. ASCII transfer is usually required by CGI scripts.
ftp_executable = [ '*.cgi', '*.pl', 'cgi-bin/*.py', ... ]
Files matched by this mask are set as executable on the FTP server by "SITE CHMOD 755 <fname>" command. In case your FTP server (like Microsoft IIS FTP Server) doesn't support unix executable flags or SITE CHMOD and errors are reported when CHMODing masked files, set ftp_executable = []
ftp_execumask=755
Sets the "umask" file attributes, which are set by the FTP SITE CHMOD command for files matching the "ftp_executable" mask. Default: 755
lazy_mkdir=0
When lazy_mkdir is set to 1, the directories are only created during synchronization, when files are created. Thus empty directories, would not be re-created on the other side.
errorcontinue=0
Continue operation upon errors? :
- 0 = never
- 1 = upon single file errors
- 2 = <reserved value>
- 3 = if project continuable
- 4 = even if root error in subprojects ( see "sub_sync" )
Python Scripting / synctree.do()
You may add any python code to
.syncpy project
file. The python modules os,shutil,fnmatch,glob,string,re,time and more
are available (and imported).
If you do not have at least one call to synctree.do()
or synctree.dontdo(),
a synctree.do() call is done automatically after parsing the
project script.
Annotation on filename wildcards in exclude & include masks:
The mask filters match using filename wildcards ?, * and
[seq].
seq: character ranges, e.g. [A-Za-z] .
A more complex example: excludefiles_abs
= ['data/my*/test[A-Z]/db??store*.dat'].
Real filenames cannot contain ? or * but [ or ]. if you want to match
an explicit file/dirname like synctest[4] you have to quote the
brackets like excludefiles_abs =
['data/synctest[[]4[]]/*']. In roots
and sub_syncs you must not quote, because these are direct paths, not
patterns.
3.4 Project File Syntax : Advanced API
xfile.X_remove_recursive
Removes files and folders recursively! By default a confirmation dialog
protects
this operation.
Example:
xfile.X_remove_recursive("/my/path/to/folder", removethis=0, confirm=1)
Annotations on Command Line Parameters:
--more/--more0/--more20/--more40/--more60
Enables the integrated 'more' console display filter. Thus console output stops after 10,20,40 or 60 lines. --more0 switches on a more-filter, which is initially unlocked ('f' key), but is the same as --more10 + pressing key 's'/SPACE.
The more-filter reacts to the following keyboard keys:
- any key during hold [except Ctrl-C and 'f'] :: display next page
- 'f' during hold :: (follow) unlock the more-filter
- 's' / SPACE during operation :: (stop) lock the more-filter again
- Ctrl-C : break execution ( single files are always copied completely before the break is done )
--find_dup=fname,size,time,content,exact
Finds duplicate files based on the selected criteria
(exiting
option).
| fname | the filenames must be equal |
| size | the file sizes must be equal |
| time | the file time stamps must be equal |
| content | the file contents must be equal (implies 'size') |
| exact | all criteria above must match |
See also the graphical duplicate file finder in the GUI !
Finds duplicate files. same as --find_dup=fname,size. This is a exiting option not starting a sync.
Finds empty directories in roots or '.'
This is a exiting option not starting a synchronization.
3.5 Built-In File Systems
This chapter presents a list and short description of current built-in file systems. ( SynchronEX' Extensible Virtual File System Technology enables also custom made file systems. )
Normal Local or Windows UNC Network File System
Examples:
roots = [ 'c:/www', 'd:/web' ]
roots = [ 'c:/www', '//server/web' ]
ZIP File System (ZIP:)
SynchronEX synchronizes seamless into zip archives or even into intra-ZIP paths
Examples:
roots = [ 'c:/www', 'zip:d:/backup/web.zip' ]
roots = [ 'c:/www', 'zip:d:/backup/web.zip//sub/subdir' ]
roots_no_src = [1]
XVS File System (XVS:)
See also XVS Overview, Recycle (RCY) File System.
XVS is a sort of journaling file system on top of the Local File
System, which keeps past versions of
files
by automatically backing them up to a XVS folder ( ./XVS or ./.XVS
relative to the file folder), whenever a file is to be overwritten or
(re)moved by SynchronEX. The XVS folder is created automatically on
demand.
An XVS backup file like is created like : /sub/folder/XVS/005.normal-file-name.txt .
The index number (005 in this example) is counted upwards for each archived version of a file. An XVS backup file is created, when the original file is about to be (1) replaced by a copy-operation, (2) written/overwritten, (3) removed/moved.
The default XVS backup folder "XVS" can be changed via config.SetXVSFolder. Use XVS folder
names other '.XVS' or 'XVS' only,
when there is a good reason for. (These 2 standard XVS folder names are
on internal folder exclusion in the style of "excludedirs_rel" by
default.)
Example 1:
roots = [ '/work/www', 'xvs:/dist/www' ]
Example 2:
config.SetXVSFolder('.XVS') # changes
default 'XVS' or '.XVS'
roots = [ 'c:/www',
'xvs://server/web' ] # XVS on a UNC share
UXVS File System (UXVS:)
Operates the same way as the XVS File System, just it can work on top of ANY other filesystem - including Wide Area Network file systems like sftp, DAV, etc.Note: On the local file system UXVS it is a little slower - thus prefer XVS there.
Note: The default backup folder for UXVS is ./.XVS (from SynchronEX 4).
Example:
roots = [ 'c:/www', 'uxvs:sftp://www@myserver.dom/htdocs' ]
Recycle File System (RCY:)
See also XVS
File System.
On Windows: Archives past versions of file which are about to
be overwritten or (re)moved by automatically
moving them to the Windows Recycle-Bin folder ("Trash basket").
On UNIX: Archives past versions of overwritten or (re)moved
files on local file systems by automatically
moving them to a "./.recycle"
folder. Same operation as with the ./XVS or ./.XVS folders of the XVS
File System, just uses a differently named folder (which is
intended to be cleaned less considerate than .XVS).
Example:
roots = [ 'c:/www', 'rcy:d:/www-sync' ]
Time-Stamp-Only File System (TFS:)
TimeStampOnlyVFS upon Local File System. Usually only used for target-only roots. Files on TFS are created with zero-length (no content), but carry time stamp information. E.g. Consider using the TFS file system for use with FTP/WEBDAV or other shadow folders. Thus the shadow files on TFS have zero-length, do not consume disk space and carry only time stamp information.
Example:
roots = ["C:/www", "tfs:C:/www-shadow"]
roots_no_src = [1]
shadow_copy={ 1:"ftp://...." }
NULL File System (NULL:)
Dummy sink file system similar to "/dev/null" on UNIX, but it is a sink for folder trees.
Examples:
#example: MO/Zip/Floppy drive read test
roots = [ 'z:/data', 'null:' ]#example: full non-incremental copy / mirroring
roots = [ 'c:/www', 'null:' ]
shadow_copy={ 1: '//remote/non-incremental-mirror-dir' }
#example: force FTP full non-incremental copy
##roots = [ 'c:/www-edit', 'c:/www-shadow' ]
roots = [ 'c:/www-edit', 'null:' ]
shadow_copy={ 1: 'ftp://me:pwd@webserver.com/www' }
FTP File system (FTP:)
The plain FTP File system. Suitable mainly for using it with shadow_copy / ftp_shadow. Files on bare FTP files systems cannot be compared directly, because FTP is a weak directory protocol for synchronization: File time stamps cannot be preserved during copy and the files itself are (partially) exposed to ASCII conversion etc.
Example:
roots = [ 'c:/www', 'c:/www-shadow' ]
roots_no_src = [1]
shadow_copy={ 1: 'ftp://me@webserver.com/www' } # password asked interactively
shadow_copy={ 1: 'ftp://me:pwd@webserver.com/www' }
Advanced Configuration:
- Maximum number of retry attempts on interrupted FTP
transfer (default=2):
config.setFTPRetry(3)
config.setFTPRetry(retrymax=3,retrywindow=55)
retrywindow is the number of seconds after which the retry-counter during long FTP file transfers is reset (anew retrymax attempts)
- Using an extra transfer file during FTP(S),SFTP,DAV in
up-direction is enabled by default (in order to avoid partial files on
live servers). the use of transfer file can be switched off with
config.useXFERFile(0)
- Change the connection timeout (Time after which a new
connection will be used)
config.setFTPConnTimeout(300.0) # change timeout to 300 seconds
(Default is 120 seconds)
See also: XFTP filesystem for direct bi-directional transfer compatible with plain FTP.
FTPS / FTP over SSL/TLS File system (FTPS:)
FTPS is a FTP File system using secure SSL/TLS as transport layer. FTPS over SSL/TLS is supported by most modern FTP servers. If it doesn't work with your FTP server you will get an error like "550 command not understood" after the FTP "TLS" command (set "verbose=1").
Example:
roots = ['c:/www', 'c:/www-shadow']
roots_no_src = [1]
verbose = 1 # verbose display of FTP commands
shadow_copy = {1:'ftps://me@webserver.com/www'} # password asked interactively
shadow_copy = {1:'ftps://me:pwd@webserver.com/www' }
See also: XFTPS file system
Advanced Configuration:
- Set x509 Client Keys & Certificates by adding this to the
project script:
config.SetSSLKeyCertFiles('www.davserver.dom',
key_file='/path/keyfile.x509', cert_file='/path/cerfile.x509')
(The host name must be the same as used in the connection URL)
For advanced configuration see also FTP file system documentation.
XFTP / XFTPS File system (XFTP: / XFTPS:)
This is an eXtended FTP File system, which stores extra
original file modification time stamp
information in ftp:<dirname>/XVS/.time.<filename>
(see XVS File system). The extra time stamp information
is written
as GMT flat time value. Thus the XFTP file system is suitable for
full-featured bi-directional
synchronization throughout all time-zones. XFTPS is to XFTP as FTPS to
FTP :
uses SSL/TLS as transport layer.
Example Project Script:
roots = [ 'C:/www-edit', 'xftp://me@myserver/root' ]
excludedirs_rel = ['.ftplock']
ftp_ascii = []
flock=roots[1]+'/.ftplock'
if not simulation: xfile.X_mkdir( flock )
try:synctree.do()
finally:
if not simulation: xfile.X_rmdir( flock )
It is not recommended to use XFTP for normal incremental FTP uploads, because direct file comparison over (X)FTP is rather slow. Thus if you have many upload files and a small number of download files, it is recommended to split the job into 2 parts and use a multi-project.
For advanced configuration see also FTP file system documentation.
SFTP / SSH File Transfer Protocol ( SFTP: )
SFTP is the secure encrypted SSH(2) File Transfer Protocol. It is a modern, very reliable, secure and fast WAN file transfer protocol. Despite its name, SFTP is technically very different from FTP / FTPS and uses different software.
Most UNIX Servers offer SSH/SFTP access through the "sshd" daemon. Yet SFTP support often has to be enabled extra in the configuration for sshd in a file like /etc/ssh/sshd_config - by a line like:
#sshd config file /etc/ssh/sshd_config
...
Subsystem sftp /usr/lib/sftp-server
Logon can happen by password or by RSA/DSA private key (with optional key password). Required passwords are asked interactively if the are not set inside the script - and can interactively entered passwords can optionally be saved machine-encrypted after confirmation.
Example Project Scripts:
# wizard_SFTP_Sync
# open password logon (not recommended unless "sycntree -c" from other software)
roots = ['C:/www-edit', 'sftp://myself:directpassword@sftpserver.dom/subdir_from_myself']
# wizard_SFTP_Sync
# password will be asked interactively and optionally be stored machine-encrypted after confirmation
roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom/subdir_from_myself']
# how to point to a absolute server folder (relative from the root folder / ):
roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom//subdir_from_root']
# override the the default '~/.ssh/known_hosts' file
config.setSFTPKnownHostsFile('~/sshsx/known_hosts.txt')
# trigger private key logon (per server)
config.setSFTPPrivateKey('sftpserver.dom', '~/sshsx/private_RSA_key_sftpserver.txt')
config.setSFTPPrivateKey('otherserver.dom', '~/sshsx/private_DSA_key_otherserver.txt', 'funny_pwd')
# change connection timeout to 120 seconds (default: 60)
config.setSFTPConnTimeout(120.0)
# suppress interactive questions for keys, known hosts etc. / just raise errors
config.setInteractive(0) # like -p from commandline
roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom:2277/subdir_from_myself']
Private RSA/DSA Key Logon (with or without extra key password)
The following default locations are searched for private SSH keys: ~/.ssh/id_rsa , ~/.ssh/id_dsa
Otherwise you can set a path to a private key (per server) explicitely by this function:
config.setSFTPPrivateKey(host, priv_file_or_key, password=None)
Note: in case of an encrypted private key, the key password can also be entered interactively - and optionally be stored machine-encrypted by confirmation for later auto-use.
Note: the priv_file_or_key can be an immediate RSA/DSA key string as well (instead of a path to a file).
A Known Hosts File different from default '~/.ssh/known_hosts' can be set like:
config.setSFTPKnownHostsFile(know_hosts_file)
Advanced Configuration:
Change the connection timeout (Time after which a new connection will be used):
config.setSFTPConnTimeout(120.0)
Notes: ~ is %USERPROFILE% on Windows. Thus '~/.ssh/known_hosts' on Windows for example expands to %USERPROFILE%/.ssh/known_hosts . Check the latter location by copying the folder path into the Windows Explorer address navigation. Typically this ends up in a folder like "C:\Documents and Settings\user\.ssh"
WebDAV File System ( DAV: / HTTP: / HTTPS: )
This is the DAV over HTTP(S) file system - also called "WebDAV" or WebFolders". WebDAV is a modern full-featured bi-directional WAN file transfer protocol extending the one-way Hyper Text Transfer Protocol (HTTP). Thus DAV file/folder URLs have the same layout than normal HTTP-URLs - e.g.: 'https://myself@davserver.com/davroot' . DAV is often considered to be a more reliable replacement for FTP. DAV is supported by current web servers like Apache (with mod_dav installed) and other DAV servers. A DAV server like Apaches mod_dav usually has to be configured (".htaccess", "http_conf") in order to allow (password protected) DAV read/write/property-change access to (certain) folders. If the web server of your ISP enables DAV, you should have documentation from your ISP about how to access and use DAV. DAV over HTTPS is excellent for secure encrypted WAN file exchange.
Example:
#wizard_Native_DAV_Sync
roots = ['C:/www-edit', 'https://myself@www.davserver.dom/davroot']
Advanced Configuration:
Proxy support for DAV (HTTP & HTTPS):
- Windows: A proxy server is used automatically, if there is
a http proxy Windows setting (in MS InternetExplorer) - or
alternatively, if the environment variable http_proxy is set:
SET http_proxy="http://www.someproxy.com:3128" - UNIX: A proxy server is used on UNIX automatically, if the
environment variable http_proxy (case
sensitive) is set:
% http_proxy="http://www.someproxy.com:3128"
% export http_proxy - Use of proxy can be forced inside the project script like:
config.setProxy('http','http://www.someproxy.com:3128') #forces proxy ON
config.setProxy('http',None) # forces proxy OFF - Proxy with authentication:
config.setProxy('http','http://user@pwd:www.someproxy.com:3128')
Extra Transfer File:
- By default now (v3.0.7) SynchronEX uses an extra transfer
file during DAV upload (in order to avoid partial files on live
servers).
That method employs the DAV HTTP MOVE method, which is supported by almost any DAV server ( required in the minimal DAV standard RFC 2518 )
- Usage of the transfer file method can be switched off with:
config.useXFERFile(0)
DAV over HTTPS: Set x509 Client Keys & Certificates:
Set x509 Client Keys & Certificates by adding this to the project script:
config.setDAVKeyCertFiles('www.davserver.dom',
key_file='/path/keyfile.x509', cert_file='/path/cerfile.x509')
(the host name must be the same as used in your DAV URL)
Change the connection timeout (Time after which a new connection will be used)
config.setDAVConnTimeout(20.0) # change timeout to 20 seconds
(Default is 10 seconds)
XWebDAV File System ( XDAV: / XHTTP: / XHTTPS: )
A special DAV File system, which stores file time stamps in extra files and not in DAV-properties. Suitable for simple DAV-Servers which do not support the DAV-Method "PROPPATCH" (for example Tomcat). Similar to XFTP vs. FTP
Example:
roots = ['C:/www-edit', 'xhttps://myself@davserver/davroot']
For special configuration see also (normal) DAV file system documentation.
GZip Auto-Compression File System
Compresses and decompresses files individually on the fly on any other file system.
The (created) compressed files on the target file system have an extra extension '.sx.gz' attached by default.
Examples:
roots = ['C:/www-edit', 'igz:http://myself@davserver/www-gz-backup']
roots = ['C:/www-edit', 'igz:sftp://myself@server/gz-backup']
Change the default file extension for compressed files.
config.SetIGZExtension('.gz')
Auto-Encryption / Batch-Conversion File System
Encrypts and decrypts or batch-converts files individually on the fly by using any command line stream coder (pipe), which is installed on the system (e.g. GPG). Works on top of any other SynchronEX file system
The encrypted files on the target file system have an extra extension '.sx.enc' added by default.
Examples:
config.SetIENCPipe('ienc:',
'gpg -e -r USER1 --batch --always-trust',
'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
)
roots = ['/home/myself/www-edit', 'ienc:http://myself@davserver/www-encrypted-backup']config.SetIENCPipe('ienc:',
'gpg -e -r USER1 --batch --always-trust',
'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
key='_mygpgpassphrase_'
)
roots = ['C:/original', 'ienc:D:/encrypted']config.SetIENCPipe('ienc:',
'gpg -e -r USER1 --batch --always-trust',
'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
keyfile='C:/mygpgpassphrase_is_in_here.txt'
)
roots = ['C:/www-edit', 'ienc:igz:sftp://myself@server/encrypted-gz-backup']
.
4. Technology Background Knowledge
The SynchronEX software is about incremental multi-root file&object-tree synchronization, backup, updating backup, FTP, incremental ZIP archive update, file tree mirroring, reliable copying of file trees, scheduling projects, finding file duplicate files, finding empty directories, advanced browsing and finding of files & other file tree related work.
SynchronEX = robust engine + GUI frontend. The engine can work independently and silently. E.g. can be called from batch jobs, schedules or other software.
SynchronEX internally uses a sophisticated extensible virtual
file system (VFS)
technology. Thus SynchronEX bridges seamlessly into FTP, WEBDAV, ZIP
and other
widely used file system protocols. Other VFS add special capabilities
to local
file systems. E.g.: XVS,LimitedMediaVFS,TFS,NULL .... : see File Systems
Synchronization is
based on file time stamps.
A Simulation mode enables evolving arbitrary complex projects without risk.
SynchronEX can use a synchronization database file when
advanced comparisons
have to be done, which cannot be accomplished simply by direct
file/directory
comparison: A Y-collision detection feature (-y /syncdb=1)
provides protection against
dangerous overwrites of concurrent file changes. Hard
synchronization
including the synchronization of file/directory removals also makes use
of the
synchronization database.
Usage of
SynchronEX is very simple for common tasks (GUI Wizards). On the other
hand you are able to schedule arbitrary complex patterns, or even
evolve your
project script into a full blown Python
automated program.
SynchronEX uses human readable project scripts which can be set up with
wizards
and can be evolved to more complex scripts/programs. It is recommended
to place
projects files into one of the root folders of interest.
Exclude & Include:
There exist 5 channels of exclude/include masks: For files OR
directories, relative OR
rooted (absolute) + a logical callback function. SynchronEX' mask
policy is
"Take all minus excludes but take includes.". Rooted (absolute)
masks have priority over relative masks.
In addition a logical callback function, which is called for
each file or
directory, can bias exclusion 3-state: force exclude, force include,
delegate
decision to upper masks.
Complex python scripts can be directly executed inside the
project script.
An arbitrary number of synch-jobs can be executed programmatically
(e.g.
loops; synctree.do() calls ) in a project script. ( If no such
call is done explicitly, a
synctree.do() is automatically scheduled after parsing the project
script.
)
Copy directions in synchronizations are scheduled based on file time
stamps. Exact 1-hour distances with no change in file size, are
scheduled as "IN SYNC" because ( common incompatible daylight savings
handling on different
computers ).
SynchronEX takes special mean to defend against
synchronization
inconsistencies upon cable interrupts/OS crash. E.g. it uses special
time stamps
during copy and block OS interrupt signals.
Commandline paramaters take priority over project file settings.
4.1 Special Topics
XVS/UXVS Version Tracking / Journaling File System
XVS enables auto-back-up of former versions of files which are changed or deleted during a synchronization run or backup run. This is the most save mode at all for file synchronizations and backup tasks: You can not anymore loose files or loose old versions of your files unintentionally. XVS can be used together with syncremove=1 option ( hard synch mode ) in order to set up extremely effective synchronizations.
The principle of XVS is simple: If any file on an XVS tagged file system (e.g. root "xvs:C:/texts" ) is overwritten or removed during a synchronization run, the old file is first moved to a XVS backup location. E.g. file C:/texts/mytext.txt is backed up as C:/texts/XVS/000.mytext.txt , then on next overwrite to C:/texts/XVS/001.mytext.txt, then C:/texts/XVS/002.mytext.txt and so on. The ./XVS sub-folder is generated automatically on the fly when necessary.
You can tag any local or network root path in SynchonEX for XVS processing. ( This does not alter anything in your real file system of that disk drive. It is just a virtual file system inside SynchronEX itself. )
Thus you can prepend a "xvs:" tag to a root directory in order to declare it as XVS virtual file system. In the GUI you can also simply use the XVS checkbox or the dedicated XVS wizard which activates the XVS for all roots by default. Example:
roots = [ "xvs:C/texts", "xvs://server/texts" ]
This way you get historic backups for all versions of the files. At any time you can remove XVS backup files, e.g. when they consume a lot of disk space. The GUI tool in Menu/Tools/XVS Cleanup can be used to quickly bulk sweep XVS backup files.
The default maximum of backup versions of a file is 10000. If more versions are to be generated, an error is thrown. You can change this number e.g. by
config.SetXVSLimit(100)
.. in a script.
XVS system can be applied to local/UNC filesystems only. The UXVS file system ("Universal XVS") is the same as the SXV file system, but works on _any_ other filesystem. UXVS works a little slower generally:
roots = [ "uxvs:sftp://user@server/texts", "xvs://server/texts" ]
"What has XVS to do with CVS or MS SourceSafe?"
Answer: Not much. XVS works completely automatic without
needing to know
anything about repositories, tags, check-in/check-out. Is is
probably the
most easy-to-use method for version tracking at all. It works without
any
problems in any office or development environment where valuable
documents of any nature are edited iteratively.
But of course it is not a total replacement for CVS with controlled
version
tracking (
multi-year, branches etc. )
However even in source code development environments there are situations where it is reasonable to combine XVS and a version tracking system like CVS. E.g. XVS could take care of fast iterations, personal iterations and automatic daily versioned backups while CVS/SourceSafe does the long-term manual controlled version tracking.
Another special application with CVS repositories: SynchronEX & XVS are reported to successfully and consistently keep a CVS repository itself mirrored on 2 or more locations for certain applications, where there is no permanent network connetions (see also "collision detection" for consistency ).
5. FAQ / Tips & Tricks
Forcing synchronization upon collisions (syncdb=1) ?
First execute the project, overview the collisions and do all
necessary manual
merge work.
In order to really force synchronization solely based upon file times,
simply erase the .syncdb database file of that project and
re-execute the
project.
Finding files duplicate in content but differing in filename ?
- install a diff, filecompare, fc, .. tool
- cd to your root. e.g. cd /myfiles
- execute: synctree --find_dup=content,fname > lA.txt
- execute: synctree --find_dup=content > lB.txt
- then for example: diff lA.txt lB.txt
Synchronizing two computers with a mobile (e.g. ZIP-Drive) Disk in between ?
Also this case is no problem with SynchronEX and very save,
valid and effective with
syncdb=1
/ syncremove=1
HOME-Computer <--->
ZIP-Disk <---> WORK-Computer
For example on your home computer you have this:
|
# c:/texts/config.syncpy |
... and on your work computer you have this
|
# d:/texts/config.syncpy |
You could also place both project files central on the ZIP-disk for example as home.syncpy & work.syncpy. With syncdb=1 provided, you could also carry a pice of a server 'away' and work on it and re-sync with valid collision detection.
6. FAQ / Trouble Shooting
" Warning Message: project / working directory not in roots ; consider check_cwd=0 "
- The checkbox is on the 3rd option page of the wizard. or write "check_cwd=0" in the script.
- "not in roots": By default SynchronEX wants the project
script file (.syncpy) to be placed in one of the roots folders of the
project in order to prevent from executing wrong projects. check_cwd=0
allows the project to be executed from anywhere
" I have lots of collisions (syncdb=1). How do I get rid of them? "
- You have real collisions if you have edited/changed files on both (more than 2) sides. In that case you'd have to create a correct synchronized state by manually copying the files appropriately.
- Maybe the daylight saving settings of two machines, which are used during synchronization, are incompatible. In case you want to force copying the most youngest files: Clear the .syncdb database file of this project, simulate & execute if directions ok.
" How to log on to a Windows folder on a remote machine automatically (with password)"
- Insert the following on top of your project script:
#force auto-logon on remote windows folder:
os.system(r"net use \\COMPUTERNAME\RemoteFolder /u:user pwd")
roots = ['c:/myfolder', r'\\COMPUTERNAME\RemoteFolder']
....#force auto-logon on remote windows folder with drive letter mapping:
More on Windows command "net use".
os.system(r"net use Z: \\COMPUTERNAME\RemoteFolder /u:user pwd")
roots = ['c:/myfolder', 'Z:/']
....
" I want to call a SynchronEX project from C, Delphi, Script ... with console window hidden "
- use e.g. a function like "_popen" (C/C++) of that programming language and call "synctree.exe <options>..."
" How do I get a detailed log of FTP commands issued? "
- Select the "verbose" mode in the options wizard page (script: "verbose=1").
" SynchronEX won't create directories on my FTP server "
- In order to be fast, SynchronEX creates directories over FTP upon "directory missing error"
- Maybe your server yields a non-standard error message => please set verbose=1 and report the error message to us.
" Is it possible to compare two directories with SynchronEX but copy the changed files to a third directory? "
- Yes, use the differential (shadow) synchronization wizard ( shadow_copy={...} )
" A backup to ZIP archive always appends? "
- A replacing would be done, by forcing deletion of the ZIP
file in the script:
os.remove( root[1] )
A true overwrite/delete of a file intra-ZIP is not a job for a fast sync software like SynchronEX, because the whole ZIP file had to be rewritten anyway. E.g. try to delete a intra-zip file in XP explorer or winzip: it takes a long time. Thus SynchronEX always appends to the ZIP file.
When you force the removal + complete rewrite as above, its (nearly) as fast as an in-place deletion of files in a zip file would be
" I have problems with SynchronEX on local area networsk & Windows firewall"
Go to: Control Panel -> Network Connections -> Local Area Network -> Properties
On the General Tab:
Internet Protocol (TCP/IP) -> Properties -> Advanced -> WINS -> NetBIOS setting is either set to Default or Enable NetBios over TCP/IP
File and Printer Sharing for Microsoft Networks must be checked
On the Advanced Tab:
Enable Microsoft Firewall by checking the checkbox.
Press Settings push button to configure advanced firewall settings:
On the Services tab:
Uncheck all checkboxes to disable all default services.
If your computer hosts an FTP site, or web server, etc. you’ll have to leave the appropriate check box checked.
Press the Add push button to add Port 137 UDP to firewall
Press the Add push button to add Port 138 UDP to firewall
Press the Add push button to add Port 139 TCP to firewall
On the ICMP tab
Uncheck all checkboxes to disable all ICMP message support. You can check Echo (ping) if you want PING to work.
" How to synchronize file from very long paths (>260 chars in path name)? "
There are 2 alternative options:
- use config.SetAutoLongPaths() .
- use Windows path escapes r"\\?\" and r"\\?\UNC\"
