wyng-extract: non-working cleanup

[tlaurion]

Hey @tasket

Here is my current attempt to fix incompatibilities between bash/ash, where busybox is more on the dash side even when bash compatibility is on.

I would love you to be able to pick up on that work, and take whatever you like and merge it to upstream.

I made it not work at all, but

• cleaned things like <<< which are not supported, replaced with cut calls
• realpath is not supported
• broke ( ) functions and indent, but ash was complaining, not sure how to fix that but rewrite those parts
• case changing ${var,,} not supported
• etc

In current state, console stops output showing megachunk added echo and dies without going further.....
Any help appreciated.

I will point WiP heads PR that I finally gave some love to. It now includes functional blake2 and zstd, thin provisioning support without complaining of thin-provisioning tools missing and newer busybox version (making ash more compatible to bash but quite not there yet, so bashisms should be replaced everywhere).

[tlaurion]

@tasket : if you want to play with busybox, the current 1.35.0 config I was using to add trunctate and bashisms support is here: tlaurion/heads@5d82834#diff-7906e5e771d8da6406d4081c2f8e0b51443c1dd41b0300f55f50c016034e9077

[tasket]

Looking into this now.

[tlaurion]

@tasket awesome.

[tlaurion]

Commented on changes needed on my side under
tlaurion/heads@5d82834

[tasket]

I used busybox ash to simulate the environment with interactive commands. Using subshell ( ) didn't seem to cause any problem?

TBH I'm having trouble sorting out what is ash behavior and what is dash.

[tlaurion]

TBH I'm having trouble sorting out what is ash behavior and what is dash.

@tasket I know, same for me. Shellcheck only supports dash, which is what I've used to target what was not compliant changing header to parse it.

I used busybox ash to simulate the environment with interactive commands. Using subshell ( ) didn't seem to cause any problem?

I would have to test again to see what was the error thrown

[tlaurion]

@tasket you are under matrix? Can send whatever you have right now and can test it

[tasket]

@tlaurion I don't have any revisions yet. I'm only testing bits of example code now.

[tasket]

@tlaurion One bug I spotted is on line 110. Two variables should be set here (m_last and m_therest) but only one is set. It should look like:

m_last=$(echo "$ln" | cut -d " " -f1)
m_therest=$(echo "$ln" | cut -d " " -f2-)

I haven't checked if this omission was repeated elsewhere.

[tasket]

Also noticed that set -o pipefail is not available. IIRC this was necessary to get one or more pipes to work correctly.

[tlaurion]

Also noticed that set -o pipefail is not available. IIRC this was necessary to get one or more pipes to work correctly.

It is used basically everywhere under Heads scripts and should work.

[tasket]

@tlaurion I've made some correction comments in the pr. The script might actually work now and I'll try to test it sometime today.

[tlaurion]

@tasket heads PR modified to bind zstd to 1.4.9, let me know where I can see testing version of wyng-extract, was going to put that into my tests today as well.

[tlaurion]

The script might actually work now and I'll try to test it sometime today.

@tasket let me know :)

[tasket]

@tlaurion Tested without sparse, it nearly works now except for the dreaded pipe interruption at the DECOMPRESS cmd in the final pipe segment. My memory was right about that one. In my scenario pigz receives a signal 13 early.

See branch 'extractor' for the test code.

[tasket]

BTW, sig 13 means EOF. I need to be able to ignore or suppress that to make the final volume creation work. pipefail option causes the shell to only use signals from the last process, which is why this works in bash.

[tlaurion]

@tasket may I suggest you turn the shell back to bash? afae272#diff-e874df9a730390d6d1f5991b7249cdeac695104b2f152d6319c5bc3e334de965L1

Should work if bash compat on, which is the case per heads busybox compiled options:
https://github.com/mirror/busybox/blob/master/shell/ash.c#L229
https://github.com/mirror/busybox/blob/master/shell/ash.c#L4618

Busybox comfig file:
https://github.com/osresearch/heads/blob/25ce08a052b9280d9a16f185294ac0f19035e6b0/config/busybox.config#L1128

Will test tomorrow on my side, reverting script shell declared environment.

[tasket]

@tlaurion I guess this is a compile-time option. I've been using the Debian busybox ash v1.35.

Also, there were bugs introduced by adding quotes around variables. This should not be done in most cases where a variable holds a multi-word string such as $COMPRESS.

Normally, I only add quotes when I think a user (or the system) supplies a datum (such as a file path) that may or may not contain whitespace.

[tasket]

@tlaurion The busybox find command does not support -prinf. Don't know what to do about that yet. Maybe feed the find output through sed.

[tlaurion]

busybox find command does not support -prinf.

find -print should be supported.
https://github.com/mirror/busybox/blob/master/findutils/find.c#L872

Must be enabled by ENABLE_FEATURE_FIND_PRINT0

Confused. Not a configurable option for find.

Unrelated.

[tlaurion]

@tasket Confirmed. busybox find -printf is not a thing

[tlaurion]

feed the find output through sed.

@tasket find -exec stat seems to be a good solution? https://stackoverflow.com/a/3425540

[tlaurion]

@tasket also just realized that only local volumes are supported.... Not ssh without sshfs (which is not maintained and requires fuse and is not usable, really).

Can test with local wyng lvm qube private volume, but goal is definitely ssh.

Could be paid as part of https://opencollective.com/insurgo/expenses/23902 ?

[tlaurion]

Doh.
busybox dd is not supporting neither nocreat neither sparse:

wyng-backup/misc/wyng-extract.sh

Line 251 in 81788f8

| dd of="$outvol" obs="$chunksize" conv=sparse,notrunc,nocreat

https://github.com/mirror/busybox/blob/1ce60699c15e5fc2ecff5db40ea1e1ae583b8b34/coreutils/dd.c#L352

Basically, I had to change the following for it to run without error:

--- /home/user/heads/initrd/bin/wyng-extract.sh
+++ /media/user/f5e8d325-a8dd-4c52-9610-604cb329463f/wyng-extract.sh
@@ -210,7 +210,7 @@
 
     echo -en "\nChecking volume hashes..."
     sed -E '/^0 x/ d; s|'"$mregex"'|\1 \4/\2/x\2\3|;' diff-manifest  \
-    |  $HASH_CHECK -c --status
+    |  $HASH_CHECK
 
     # Change the merge-sort inputs to use the differential versions during extraction.
     m_last=diff-manifest
@@ -222,7 +222,7 @@
 
     sort -umd -k2,2 $m_last $m_therest  |  sed -E "/ x$lastchunk/q" \
     |  sed -E '/^0 x/ d; s|'"$mregex"'|\1 \4/\2/x\2\3|;' \
-    |  $HASH_CHECK -c --status
+    |  $HASH_CHECK
 
     exit 0
   fi
@@ -248,7 +248,7 @@
   sort -um -k2,2 $m_last $m_therest  |  sed -E "/ x$lastchunk/q" \
   |  sed -E 's|^0 x.*|ZERO|; t; s|'"$mregex"'|\4/\2/x\2\3|'  \
   |  xargs $DECOMPRESS -f  \
-  |  dd of="$outvol"  obs="$chunksize" conv=sparse,notrunc,nocreat
+  |  dd of="$outvol"  obs="$chunksize" conv=notrunc
 
   sync
 

But then the script does not write anything neither in sparse or normal mode:

[tasket]

@tlaurion Also busybox fallocate does not support either -z or -p option.

But lack of dd options is the biggest issue. That whole section of the script would have to be re-written.

[tasket]

@tlaurion A work-around for these limitations would involve:

• re-write everything from while [ $i ...] loop on down
• not being able to create sparse-allocated image files; only block devices like lvm can be sparse
• slower 'dig holes' loop to detect zeroed areas and de-allocate them