smsync keeps huge music collections in sync and is takes care of conversions between different formats. It’s an easy-to-use command line application for Linux.
smsync is made for use cases where you have a folder structure for your high quality lossless or lossy but high bit rate music that acts as a source. From this source you replicate your music to targets, such as a smartphone or an SD card / hard drive for your car etc. On a smartphone or in the car you either don’t have or you don’t want to spend that much storage capacity that you might have for you source music storage. Thus, the replication step from the source to the targets is not a simple copy, it’s in fact a conversion step. Music that is stored on the source in the lossless FLAC format, for example, shall be converted to MP3 while being replicated to a target.
Normally, you want to keep the folder structure during replication. This means, that a certain music file on the target shall have the same relative folder path as its counterpart has on the source.
New music is typically added to the source only. If that happened, you want to update the targets accordingly with minimal effort. If you deleted files or folders on the source for whatever reason, these deletions shall be propagated to the target as well. And, last not least, as we are talking about huge music collections (several thousands or even ten thousands of music files), the whole synchronization and replication process must happen in a highly automated and performant way.
smsync takes care of all this:
Conversions can be configurated per target and file type (i.e. for each file extension/suffix) separately. Currently, smsync supports:
-
Conversions to FLAC, from WAV and FLAC.
-
Conversions to MP3, from WAV, FLAC, MP3, OGG (Vorbis) and OPUS.
-
Conversions to OGG (Vorbis), from WAV, FLAC, MP3, OGG(Vorbis) and OPUS.
-
Conversions to OPUS, from WAV, FLAC, MP3, OGG(Vorbis) and OPUS.
For all these conversions, ffmpeg is used. In addition, a simple file copy without any format conversion is supported as well.
The synchronization between source and target is done based on timestamps. If new music has been added to the source since the last synchronization, smsync only replicates / converts the added files. If you have deleted files or folders on the source since the last synchronization, smsync deletes its counterparts on the target.
To make the synchronization as efficient as possible, the determination of changes since the last synchronization and the replication / conversion of files are done in parallel processes. The number of CPUs that is used for this as well as the number of parallel processes can be configured.
smsync is written in Golang and thus requires the installation of Go. Make sure that you’ve set the environment variable GOPATH accordingly, and make also sure that git is installed.
To download smsync and all dependencies, open a terminal and enter
$ go get gitlab.com/mipimipi/smsync
After that, build smsync by executing
$ cd $GOPATH/src/gitlab.com/mipimipi/smsync $ make
Finally, execute
$ make install
as root to copy the smsync binary to /usr/bin.
For Arch Linux (and other Linux distros, that can install packages from the Arch User Repository) there’s a smsync package in AUR.
A target has to have a configuration file with the name smsync.yaml in its root folder. This file contains the configuration for that target in YAML format.
Example:
source_dir: /home/musiclover/Music/SOURCE num_cpus: 4 num_wrkrs: 4 exclude: - 'Rock/Eric*' rules: - source: flac target: mp3 conversion: vbr:5|cl:3 - source: mp3 conversion: copy - source: '*'
smsync interprets the configuration file. In the example, the root folder of the source is /home/musiclover/Music/SOURCE. The next two entries are optional. They tell smsync to use 4 cpus and start 4 worker processes for the conversion. Per default, smsync uses all available cpus and starts #cpus worker processes.
exclude allows to exclude a list of source folders from the conversion. The folder paths in that list are interpreted relative to the source directory. Wildcards are supported. In the example, all folders fitting to the pattern /home/musiclover/Music/SOURCE/Rock/Eric* are excluded, i.e. /home/musiclover/Music/SOURCE/Rock/Eric Clapton, /home/musiclover/Music/SOURCE/Rock/Eric Burden etc. are excluded. The exclusion feature can be helpful if the target disk space is not big enough. In such a case, some artists or even entire genres can be excluded. Another option to deal with insufficient disk space would be to configure a higher compression rate.
The rules tell smsync what to do with the files stored in the folder structure of the SOURCE.
In the example, the first rule tells smsync to convert FLAC files (i.e. files with the suffix '.flac') to MP3, using the conversion vbr:5|cl:3. These conversion parameters are strings that consist of different parts which are separated by '|'. The supported content of a conversion parameter string depends on the target format - see detailed explanation below.
The second rule of the example tells smsync to simply copy MP3 files without converting them. Another possibility was to convert MP3 to MP3 by reducing the bit rate. This can be achieved by defining a dedicated conversion rule as explained above (instead of copy).
The third rule tells smsync to copy all other files by using the wild card '*' as source file suffix. This is helpful, for example, to copy pictures. Without this rule, files that do neither have the suffix .flac nor .mp3 would have been ignored in this example.
Basically, a rule consists of a source suffix, a target suffix and a conversion. In some cases, it’s not necessary to configure all of these:
-
A source suffix is always necessary
-
The target suffix can be omitted, if it’s identical to the source suffix
-
The conversion can be omitted if it’s
copy. I.e. a copy conversion can either be specified explicitly withconversion: copy(like in the second rule) or implicitly without any conversion line (like in the third rule)
Basically, two things can be determined with a conversion parameter string:
-
The target bit rate.
Here, it’s often distinguished between
-
a constant bit rate (CBR), where the bit rate is constant - a special case is the "hard constant bitrate" (HCBR), which is specific to the OPUS format and guarantees that all frames have the same size,
-
an average bit rate (ABR), where the bit rate of the files varies, but in average it reaches a certain value,
-
or a variable bit rate (VBR), where the bit rate also varies, but the compression is done according to a certain quality.
-
-
The compression quality
Many (but not all) target formats support a "compression level" (CL). With this parameter, the compression quality can be steered.
The available or supported conversion parameters depend on the target format. The table summarizes the different options.
| Format | Conversion Parameters |
|---|---|
FLAC |
FLAC only supports a compression level (parameter
See also: FFMpeg Codec Documentation. |
MP3 |
MP3 supports ABR, CBR, both with bit rates from 8 to 500 kbps (kilo bit per second), and VBR with a quality from 0 to 9 (where 0 means highest quality). In addition, MP3 supports a compression level (parameter
See also: FFMpeg Codec Documentation. |
OGG (Vorbis) |
This format supports conversions with average and variable bit rate. For ABR, bit rates from 8 to 500 kbps are supported. For VBR, possible values are -1.0, …, 10.0 where 10.0 means the best quality. VBR with quality 3.0 is the default. Thus, for a conversion to OGG (Vorbis), if no conversion rule is specified in
See also: FFMpeg Codec Documentation. |
OPUS |
OPUS supports conversions with variable (VBR), constant (CBR) and hard constant bit rate (HCBR). The latter guarantees that all frames have the same size. Allowed values are 6 to 510 kbps. In addition, OPUS supports a compression level that ranges from 0 to 10, where 10 is the highest quality. If no compression level is specified, `cl:10`is assumed. Consequently, allowed conversions are:
See also: FFMpeg Codec Documentation or opusenc documentation. |
Coming back to the example above. Let’s assume the config file smsync.yaml is stored in /home/musiclover/Music/TARGET. To execute smsync for the target, open a terminal and enter
$ cd /home/musiclover/Music/TARGET $ smsync
The synchronization process is executed in the following steps:
-
smsync reads the configuration file in
/home/musiclover/Music/TARGET. A summary of the configuration is shown and (if smsync hasn’t been called with the option ' --yes`) the user is asked for confirmation. -
smsync determines all files and directories of the source, that have changed since the last synchronization. In our example, there was no synchronization before (as otherwise the configuration file would have an entry
last_syncthat contained the time stamp of the last synchronization). Depending on the number of files, this could take a few minutes. smsync displays how many directories and files need to be synchronized and again, the user is asked for confirmation (if smsync hasn’t been called with the option ' --yes`). -
The replication / conversion of files and directories is executed. smsync shows the progress:
Elapsed Remain #Conv Avg Avg Estimated Estimated #TODO Time Time / min Durat Compr Target Size Free Space #Errs ---------------------------------------------------------------------------- 37290 00:06:06 13:50:10 38.8 11.36s 9.4% 126069 MB 79075 MB 0
Besides the number of files that still need to be converted, not only the elapsed and the remaining time is displayed, but also
-
the number of conversions per minute (which represents the "throughput" of smsync and can be used to optimize the number of assigned cpu’s and the number of workers in the configuration file)
-
the average duration of a conversion
-
the average compression rate,
-
the estimated target size
-
and the estimated free diskspace (which is an estimation of the available diskspace on the target device AFTER all files will have been converted and can be very helpful to see at an early stage if the available space will be suffient).
With the command line option
--verbosethe progress is displayed in more detail, i.e. each file is displayed after it has been converted.
-
-
After the synchronization is done, a success message is displayed and the current time is stored as
last_syncin the configuration file.
In the example, the synchronization would convert such a source folder structure:
/home/musiclover/Music/SOURCE
|- ...
|- Rock
|- ...
|- Dire Straits
| |- ...
| |- Love Over Gold
| |- ...
| |- Private Investigations.flac
| |- ...
| |- cover.jpg
|- ...
|- Eric Clapton
|- Unplugged
|- ...
|- Layla.mp3
|- ...
|- folder.png
to such a target folder structure:
/home/musiclover/Music/TARGET
|- ...
|- Rock
|- ...
|- Dire Straits
| |- ...
| |- Love Over Gold
| |- ...
| |- Private Investigations.mp3
| |- ...
| |- cover.jpg
|- ...
The folder /home/musiclover/Music/SOURCE/Rock/Eric Clapton hasn’t been converted because the directory path matches the exclusion pattern.
During the conversion with FFMPEG, errors can occur. Unfortunately, there’s not much information about the exit codes of FFMPEG (all I could find is this. In particular, it seems to be impossible to find out if an error occured during the audio conversion or if it only had to do with the cover art. Therefore, smsync reports an error every time the exit code of FFMPEG is not zero. In addition to that, a file with the detailed log information of FFMPEG (-loglevel verbose) is stored in the directory smsync.cv.err. This file is named <name-of-the-music-file-that-was-converted>.log.
In case of a huge music collaction (tens of thousands of songs), the synchronization process might take very long (10+ hours is normal for a first run). For such cases, smsync offers the possibility to interrupt the process by pressing <ESC>. The process finalizes the conversions that have already started and stops afterwards. The next synchronization run selects only the remaining source files.
|
Warning
|
Please use only this option to interrupt the process. Interruption via <CTRL-C>, closing the terminal window etc. can lead to incomplete/inconsistent target files
|
smsync has only a few options:
-
--init/-i: Do initial sync:-
Existing files and directories in the target folder are deleted (except the smsync files
smsync.yamland - if existing -smsync.log). -
A possibly existing
last_syncin the config file is ignored. I.e. files and folders in the source directory are taken into account independent from their change time.
-
-
--log/-l: Write a log file.The file
smsync.logis stored in the root folder of the target. A log file is always written in case of an error. -
--verbose/-v: Print detailed progress.Instead of the normal output, where only the aggregated progress in displayed, the name of each file and directory is displayed immediately after it has been converted or copied.
-
--yes/-y: Don’t ask for confirmation.smsync starts directly without asking for user confirmations. With this option, it’s possible to run smsync automatically via cron job.
As long as the configuration file is not changed, smsync keeps track of the consistency between source and target. If it’s changed after a synchronization happened, manual steps are necessary. Depending on the changes that have been made to the configuration, different actions need to be taken to keep source and target consistent. Important is the "scope" that is specified in the configuration. In this context, scope means the set of source file types and the source directories (i.e. the sub directories of the configured source directory and potential exclusions).
I.e. source file types have been removed from or exclusions have been added to the configuration. In this case, you have to remove the corresponding files and directories from the target manually. Under certain circumstances the find command in combination with the rm command can help. The following command removes all MP3 files from the target:
find <target-directory> -name "*.mp3" -exec rm {} \;
I.e. source file types have been added to or exclusions have been removed from the configuration. In this case, a feasible apporoach is to update the change time of the "added" (i.e. added to the scope) source files and directories. The next execution of smsync will then update the target accordingly. Also here the find command can help, but this time in combination with the touch command.
If you have added a conversion rule for WAV files, the following command updates the change time of all WAV files in the source directory tree:
find <source-directory> -name "*.wav" -exec touch {} \;
If you have removed an exclusion, the following command updates the change time of the corresponding directory:
find <directory-you-want-to-include> -exec touch {} \;
In this case, the source of a conversion rule remains unchanged.
If also the target remains unchanged but only the conversion is changed (e.g. you still want to convert FLAC to MP3 with average bitrate conversion, but since the capacity of the target device is too small you want to reduce the bitrate),
-
touch the corresponding source files (see Case 2) and
-
execute smsync.
If the target is changed (e.g. so far you converted FLAC to MP3, but now you want to convert to OGG instead),