Skip to content
This repository has been archived by the owner on Sep 15, 2021. It is now read-only.

Commit

Permalink
Fix issue Parchive#50 (scan failure) / Correct fix for issue Parchive#31
Browse files Browse the repository at this point in the history 
 (slow scan)

Issue Parchive#31 was reported as a problem whereby cache collision of crc32
values resulted in the calculation of md5 values at a large number of
file positions for certain types of data files.

The fix for issue Parchive#31 was to introduce code that would only only scan
forwards 10 bytes whilst searching for a good datablock, and then skip
the next blocksize-10 bytes before continuing searching.

This code will never find blocks that are positioned earlier in the file
than they should be, and will also never find blocks that are positioned
more than 10 bytes later in the file than they should be.

Issue Parchive#50 reported that a single byte erasure at the start of a data
file resulted in the complete failure to find any good data block in
the file.

The fix for issue Parchive#50 is an improvement of the code intended to deal
with issue Parchive#31 such that instead of only finding datablocks that are
either exactly at the correct position in the file or no more than
10 bytes after the correct position: it will now (by default) check
up to 64 bytes before and up to 64 bytes after the expected position
of the next block (where the 'expected' position is calculated relative
to the position where the last block was found).

Additionally, two new commandline parameters were added to enable this
code to be tuned:

  -N    : No data skipping (every byte of data will be scanned for blocks)
  -S<n> : Skip leaway n bytes (check for the next block within +/- n
          bytes of the expected position before skipping forward)

Finally, where the code would previously skip forward a whole block
if it found data that it thinks is a duplicate of a block that has
already been found, it will now start scanning from that position.
  • Loading branch information
PeterBClements committed May 26, 2015
1 parent 20f2412 commit dd9c4e6
Show file tree
Hide file tree
Showing 7 changed files with 218 additions and 66 deletions.
98 changes: 50 additions & 48 deletions Makefile.am
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,56 +50,58 @@ LDADD = -lstdc++
AM_CXXFLAGS = -Wall

EXTRA_DIST = PORTING ROADMAP par2cmdline.sln par2cmdline.vcproj \
par2.1 \
tests/flatdata.tar.gz \
tests/flatdata-par1files.tar.gz \
tests/flatdata-par2files.tar.gz \
tests/subdirdata.tar.gz \
tests/subdirdata-par2files-unix.tar.gz \
tests/subdirdata-par2files-win.tar.gz \
tests/smallsubdirdata.tar.gz \
tests/smallsubdirdata-par2files.tar.gz \
tests/100blocks.tar.gz \
tests/readbeyondeof.tar.gz \
tests/par2-0.6.8-crash.tar.gz \
tests/bug44.tar.gz \
tests/test1 \
tests/test2 \
tests/test3 \
tests/test4 \
tests/test5 \
tests/test6 \
tests/test7 \
tests/test8 \
tests/test9 \
tests/test10 \
tests/test11 \
tests/test12 \
tests/test13 \
tests/test14 \
tests/test15 \
tests/test16 \
tests/test17 \
tests/test18
par2.1 \
tests/flatdata.tar.gz \
tests/flatdata-par1files.tar.gz \
tests/flatdata-par2files.tar.gz \
tests/subdirdata.tar.gz \
tests/subdirdata-par2files-unix.tar.gz \
tests/subdirdata-par2files-win.tar.gz \
tests/smallsubdirdata.tar.gz \
tests/smallsubdirdata-par2files.tar.gz \
tests/100blocks.tar.gz \
tests/readbeyondeof.tar.gz \
tests/par2-0.6.8-crash.tar.gz \
tests/bug44.tar.gz \
tests/test1 \
tests/test2 \
tests/test3 \
tests/test4 \
tests/test5 \
tests/test6 \
tests/test7 \
tests/test8 \
tests/test9 \
tests/test10 \
tests/test11 \
tests/test12 \
tests/test13 \
tests/test14 \
tests/test15 \
tests/test16 \
tests/test17 \
tests/test18 \
tests/test19

TESTS = tests/test1 \
tests/test2 \
tests/test3 \
tests/test4 \
tests/test5 \
tests/test6 \
tests/test7 \
tests/test8 \
tests/test9 \
tests/test10 \
tests/test11 \
tests/test12 \
tests/test13 \
tests/test14 \
tests/test15 \
tests/test16 \
tests/test17 \
tests/test18
tests/test2 \
tests/test3 \
tests/test4 \
tests/test5 \
tests/test6 \
tests/test7 \
tests/test8 \
tests/test9 \
tests/test10 \
tests/test11 \
tests/test12 \
tests/test13 \
tests/test14 \
tests/test15 \
tests/test16 \
tests/test17 \
tests/test18 \
tests/test19

install-exec-hook :
ln -sf $(bindir)/par2$(EXEEXT) $(DESTDIR)$(bindir)/par2create$(EXEEXT)
Expand Down
2 changes: 2 additions & 0 deletions README
View file
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ The command line parameters for par2cmdline are as follow:
-p : Purge backup files and par files on successful recovery or
when no recovery is needed
-R : Recurse into subdirectories (only useful on create)
-N : No data skipping (find badly misspositioned data blocks)
-S<n> : Skip leaway (distance +/- from expected block position)
-- : Treat all remaining CommandLine as filenames

If you wish to create par2 files for a single source file, you may leave
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ The command line parameters for par2cmdline are as follow:
-p : Purge backup files and par files on successful recovery or
when no recovery is needed
-R : Recurse into subdirectories (only useful on create)
-N : No data skipping (find badly misspositioned data blocks)
-S<n> : Skip leaway (distance +/- from expected block position)
-- : Treat all remaining CommandLine as filenames

If you wish to create par2 files for a single source file, you may leave out the name of the par2 file from the command line. par2cmdline will then assume that you wish to base the filenames for the par2 files on the name of the source file.
Expand Down
57 changes: 57 additions & 0 deletions commandline.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ CommandLine::CommandLine(void)
, memorylimit(0)
, purgefiles(false)
, recursive(false)
, skipdata(true)
, skipleaway(0)
{
}

Expand Down Expand Up @@ -132,6 +134,8 @@ void CommandLine::usage(void)
" -p : Purge backup files and par files on successful recovery or\n"
" when no recovery is needed\n"
" -R : Recurse into subdirectories (only useful on create)\n"
" -N : No data skipping (find badly misspositioned data blocks)\n"
" -S<n> : Skip leaway (distance +/- from expected block position)\n"
" -- : Treat all remaining CommandLine as filenames\n"
"\n";
}
Expand Down Expand Up @@ -657,6 +661,49 @@ bool CommandLine::Parse(int argc, char *argv[])
}
break;

case 'N':
{
if (operation == opCreate)
{
cerr << "Cannot specify No Data Skipping unless reparing or verifying." << endl;
return false;
}
if (skipleaway > 0)
{
cerr << "Cannot specify no skipping and skip leaway." << endl;
return false;
}
skipdata = false;
}
break;

case 'S': // Set the skip leaway
{
if (operation == opCreate)
{
cerr << "Cannot specify skip leaway when creating." << endl;
return false;
}
if (!skipdata)
{
cerr << "Cannot specify skip leaway and no skipping." << endl;
return false;
}

char *p = &argv[0][2];
while (skipleaway <= 429496729 && *p && isdigit(*p))
{
skipleaway = skipleaway * 10 + (*p - '0');
p++;
}
if (*p || skipleaway == 0)
{
cerr << "Invalid skipleaway option: " << argv[0] << endl;
return false;
}
}
break;

case '-':
{
argc--;
Expand Down Expand Up @@ -762,6 +809,16 @@ bool CommandLine::Parse(int argc, char *argv[])
noiselevel = nlNormal;
}

// Default skip leaway
if (operation != opCreate
&& skipdata
&& skipleaway == 0)
{
// Expect to find blocks within +/- 64 bytes of the expected
// position relative to the last block that was found.
skipleaway = 64;
}

// If we a creating, check the other parameters
if (operation == opCreate)
{
Expand Down
13 changes: 11 additions & 2 deletions commandline.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,12 @@ class CommandLine
CommandLine::NoiseLevel GetNoiseLevel(void) const {return noiselevel;}

string GetParFilename(void) const {return parfilename;}
string GetBasePath(void) const {return basepath;}
string GetBasePath(void) const {return basepath;}
const list<CommandLine::ExtraFile>& GetExtraFiles(void) const {return extrafiles;}
bool GetPurgeFiles(void) const {return purgefiles;}
bool GetRecursive(void) const {return recursive;}
bool GetRecursive(void) const {return recursive;}
bool GetSkipData(void) const {return skipdata;}
u64 GetSkipLeaway(void) const {return skipleaway;}

protected:
void SetParFilename(string filename);
Expand Down Expand Up @@ -165,6 +167,13 @@ class CommandLine
bool purgefiles; // purge backup and par files on success
// recovery
bool recursive; // recurse into subdirectories

bool skipdata; // Whether we should assume that all good
// data blocks are within +/- bytes of
// where we expect to find them and should
// skip data that is too far away.
u64 skipleaway; // The maximum leaway +/- that we will
// allow when searching for blocks.
};

typedef list<CommandLine::ExtraFile>::const_iterator ExtraFileIterator;
Expand Down
Loading

0 comments on commit dd9c4e6

Please sign in to comment.