Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS remote fails to copy file with " Received error: unexpected EOF" #7827

Open
ptallada opened this issue May 7, 2024 · 5 comments
Open

HDFS remote fails to copy file with " Received error: unexpected EOF" #7827

ptallada opened this issue May 7, 2024 · 5 comments
Labels
bug Remote: HDFS

Comments

@ptallada
Copy link

ptallada commented May 7, 2024

Hi,

I've configured the latest version of rclone to interface with our Hadoop cluster.
Running on one of the nodes of the cluster I can ls, mkdir and rmdir, but I cannot upload files to HDFS.
I'm using Kerberos.

$ rclone version
rclone v1.66.0
- os/version: almalinux 9.3 (64 bit)
- os/kernel: 5.14.0-362.24.1.el9_3.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

The config:

[shepherd]
type = hdfs
namenode = NN_HOSTNAME
username = MY_USERNAME
service_principal_name = hdfs/NN_HOSTNAME
data_transfer_protection = authentication

The tests:

$ rclone lsd shepherd:/tmp
          -1 2024-05-07 12:02:17        -1 backup-hbase
          -1 2024-04-04 15:40:43        -1 hive
          -1 2024-05-03 13:20:28        -1 logs
          -1 2024-04-07 21:46:14        -1 out
          -1 2024-04-04 15:40:36        -1 root
          -1 2024-05-03 13:20:26        -1 spark-staging
          -1 2024-03-05 13:37:25        -1 yarn-staging
$ rclone mkdir shepherd:/tmp/test
$ rclone lsd shepherd:/tmp
          -1 2024-05-07 12:02:17        -1 backup-hbase
          -1 2024-04-04 15:40:43        -1 hive
          -1 2024-05-03 13:20:28        -1 logs
          -1 2024-04-07 21:46:14        -1 out
          -1 2024-04-04 15:40:36        -1 root
          -1 2024-05-03 13:20:26        -1 spark-staging
          -1 2024-05-07 18:27:25        -1 test
          -1 2024-03-05 13:37:25        -1 yarn-staging
$ rclone rmdir shepherd:/tmp/test
$ rclone lsd shepherd:/tmp
          -1 2024-05-07 12:02:17        -1 backup-hbase
          -1 2024-04-04 15:40:43        -1 hive
          -1 2024-05-03 13:20:28        -1 logs
          -1 2024-04-07 21:46:14        -1 out
          -1 2024-04-04 15:40:36        -1 root
          -1 2024-05-03 13:20:26        -1 spark-staging
          -1 2024-03-05 13:37:25        -1 yarn-staging

But...

$ rclone copy /etc/group shepherd:/tmp
2024/05/07 18:31:20 ERROR : group: Failed to copy: unexpected EOF
2024/05/07 18:31:20 ERROR : Attempt 1/3 failed with 1 errors and: unexpected EOF
2024/05/07 18:31:20 ERROR : group: Failed to copy: unexpected EOF
2024/05/07 18:31:20 ERROR : Attempt 2/3 failed with 1 errors and: unexpected EOF
2024/05/07 18:31:20 ERROR : group: Failed to copy: unexpected EOF
2024/05/07 18:31:20 ERROR : Attempt 3/3 failed with 1 errors and: unexpected EOF
2024/05/07 18:31:20 Failed to copy: unexpected EOF

If I increase the verbosity:

$ rclone -vvv copy /etc/group shepherd:/tmp
2024/05/07 18:31:26 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "-vvv" "copy" "/etc/group" "shepherd:/tmp"]
2024/05/07 18:31:26 DEBUG : Creating backend with remote "/etc/group"
2024/05/07 18:31:26 DEBUG : Using config file from "~MY_USERNAME/.config/rclone/rclone.conf"
2024/05/07 18:31:26 DEBUG : fs cache: adding new entry for parent of "/etc/group", "/etc"
2024/05/07 18:31:26 DEBUG : Creating backend with remote "shepherd:/tmp"
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: new [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Need to transfer - File not found at Destination
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 0/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 1/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 2/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 3/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 4/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 5/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 6/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 7/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 8/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 9/10
2024/05/07 18:31:26 ERROR : group: Failed to copy: unexpected EOF
2024/05/07 18:31:26 ERROR : Attempt 1/3 failed with 1 errors and: unexpected EOF
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: new [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Need to transfer - File not found at Destination
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 0/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 1/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 2/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 3/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 4/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 5/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 6/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 7/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 8/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 9/10
2024/05/07 18:31:26 ERROR : group: Failed to copy: unexpected EOF
2024/05/07 18:31:26 ERROR : Attempt 2/3 failed with 1 errors and: unexpected EOF
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: new [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Need to transfer - File not found at Destination
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 0/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 1/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 2/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 3/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 4/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 5/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 6/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 7/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 8/10
2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 9/10
2024/05/07 18:31:26 ERROR : group: Failed to copy: unexpected EOF
2024/05/07 18:31:26 ERROR : Attempt 3/3 failed with 1 errors and: unexpected EOF
2024/05/07 18:31:26 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2024/05/07 18:31:26 DEBUG : 6 go routines active
2024/05/07 18:31:26 Failed to copy: unexpected EOF

I can upload the file using the native client:

$ hdfs dfs -ls /tmp
Found 7 items
drwxrwxrwt   - spark hadoop          0 2024-05-03 13:20 /tmp/spark-staging
drwxrwxrwt   - yarn  hadoop          0 2024-03-05 13:37 /tmp/yarn-staging
$ hdfs dfs -copyFromLocal /etc/group /tmp
$ hdfs dfs -ls /tmp
Found 8 items
-rw-r--r--   1 MY_USERNAME hadoop       2963 2024-05-07 18:37 /tmp/group
drwxrwxrwt   - spark   hadoop          0 2024-05-03 13:20 /tmp/spark-staging
drwxrwxrwt   - yarn    hadoop          0 2024-03-05 13:37 /tmp/yarn-staging

When looking at the namenode logs, I don't see anything strange:

May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,645 INFO ipc.Server: Auth successful for [email protected] (auth:KERBEROS) from 192.168.102.50:56752
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,646 INFO authorize.ServiceAuthorizationManager: Authorization successful for [email protected] (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,648 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975440_47870444, replicas=192.168.102.50:9866, 192.168.102.52:9866, 192.168.102.51:9866, 192.168.102.55:9866, 192.168.102.59:9866, 192.168.102.60:9866, 192.168.102.62:9866, 192.168.102.66:9866, 192.168.102.69:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,651 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975424_47870445, replicas=192.168.102.50:9866, 192.168.102.55:9866, 192.168.102.54:9866, 192.168.102.51:9866, 192.168.102.57:9866, 192.168.102.66:9866, 192.168.102.64:9866, 192.168.102.69:9866, 192.168.102.61:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,653 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975408_47870446, replicas=192.168.102.50:9866, 192.168.102.55:9866, 192.168.102.52:9866, 192.168.102.53:9866, 192.168.102.57:9866, 192.168.102.65:9866, 192.168.102.64:9866, 192.168.102.61:9866, 192.168.102.63:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,655 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975392_47870447, replicas=192.168.102.50:9866, 192.168.102.57:9866, 192.168.102.58:9866, 192.168.102.56:9866, 192.168.102.52:9866, 192.168.102.61:9866, 192.168.102.63:9866, 192.168.102.62:9866, 192.168.102.66:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,657 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975376_47870448, replicas=192.168.102.50:9866, 192.168.102.53:9866, 192.168.102.58:9866, 192.168.102.55:9866, 192.168.102.56:9866, 192.168.102.67:9866, 192.168.102.68:9866, 192.168.102.63:9866, 192.168.102.69:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,659 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975360_47870449, replicas=192.168.102.50:9866, 192.168.102.51:9866, 192.168.102.53:9866, 192.168.102.55:9866, 192.168.102.58:9866, 192.168.102.61:9866, 192.168.102.69:9866, 192.168.102.64:9866, 192.168.102.63:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,661 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975344_47870450, replicas=192.168.102.50:9866, 192.168.102.56:9866, 192.168.102.54:9866, 192.168.102.58:9866, 192.168.102.57:9866, 192.168.102.60:9866, 192.168.102.61:9866, 192.168.102.63:9866, 192.168.102.62:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,663 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975328_47870451, replicas=192.168.102.50:9866, 192.168.102.51:9866, 192.168.102.55:9866, 192.168.102.59:9866, 192.168.102.54:9866, 192.168.102.69:9866, 192.168.102.68:9866, 192.168.102.65:9866, 192.168.102.67:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,665 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975312_47870452, replicas=192.168.102.50:9866, 192.168.102.59:9866, 192.168.102.54:9866, 192.168.102.51:9866, 192.168.102.56:9866, 192.168.102.68:9866, 192.168.102.62:9866, 192.168.102.61:9866, 192.168.102.65:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,666 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975296_47870453, replicas=192.168.102.50:9866, 192.168.102.59:9866, 192.168.102.54:9866, 192.168.102.55:9866, 192.168.102.57:9866, 192.168.102.65:9866, 192.168.102.64:9866, 192.168.102.63:9866, 192.168.102.61:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,669 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975280_47870454, replicas=192.168.102.50:9866, 192.168.102.54:9866, 192.168.102.58:9866, 192.168.102.56:9866, 192.168.102.55:9866, 192.168.102.62:9866, 192.168.102.60:9866, 192.168.102.66:9866, 192.168.102.68:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,670 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975264_47870455, replicas=192.168.102.50:9866, 192.168.102.55:9866, 192.168.102.59:9866, 192.168.102.52:9866, 192.168.102.51:9866, 192.168.102.63:9866, 192.168.102.64:9866, 192.168.102.66:9866, 192.168.102.67:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,672 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975248_47870456, replicas=192.168.102.50:9866, 192.168.102.53:9866, 192.168.102.52:9866, 192.168.102.59:9866, 192.168.102.58:9866, 192.168.102.69:9866, 192.168.102.62:9866, 192.168.102.68:9866, 192.168.102.60:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,674 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975232_47870457, replicas=192.168.102.50:9866, 192.168.102.59:9866, 192.168.102.52:9866, 192.168.102.53:9866, 192.168.102.57:9866, 192.168.102.69:9866, 192.168.102.62:9866, 192.168.102.65:9866, 192.168.102.66:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,676 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975216_47870458, replicas=192.168.102.50:9866, 192.168.102.51:9866, 192.168.102.52:9866, 192.168.102.54:9866, 192.168.102.59:9866, 192.168.102.65:9866, 192.168.102.64:9866, 192.168.102.68:9866, 192.168.102.60:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,678 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975200_47870459, replicas=192.168.102.50:9866, 192.168.102.54:9866, 192.168.102.56:9866, 192.168.102.51:9866, 192.168.102.57:9866, 192.168.102.63:9866, 192.168.102.64:9866, 192.168.102.68:9866, 192.168.102.66:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,679 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975184_47870460, replicas=192.168.102.50:9866, 192.168.102.52:9866, 192.168.102.58:9866, 192.168.102.56:9866, 192.168.102.51:9866, 192.168.102.67:9866, 192.168.102.62:9866, 192.168.102.66:9866, 192.168.102.63:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,682 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975168_47870461, replicas=192.168.102.50:9866, 192.168.102.56:9866, 192.168.102.58:9866, 192.168.102.54:9866, 192.168.102.57:9866, 192.168.102.69:9866, 192.168.102.67:9866, 192.168.102.62:9866, 192.168.102.64:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,683 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975152_47870462, replicas=192.168.102.50:9866, 192.168.102.52:9866, 192.168.102.51:9866, 192.168.102.53:9866, 192.168.102.58:9866, 192.168.102.62:9866, 192.168.102.67:9866, 192.168.102.65:9866, 192.168.102.66:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,685 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975136_47870463, replicas=192.168.102.50:9866, 192.168.102.56:9866, 192.168.102.58:9866, 192.168.102.53:9866, 192.168.102.57:9866, 192.168.102.62:9866, 192.168.102.68:9866, 192.168.102.66:9866, 192.168.102.65:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,687 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975120_47870464, replicas=192.168.102.50:9866, 192.168.102.54:9866, 192.168.102.56:9866, 192.168.102.55:9866, 192.168.102.51:9866, 192.168.102.66:9866, 192.168.102.62:9866, 192.168.102.63:9866, 192.168.102.60:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,689 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975104_47870465, replicas=192.168.102.50:9866, 192.168.102.53:9866, 192.168.102.58:9866, 192.168.102.59:9866, 192.168.102.55:9866, 192.168.102.63:9866, 192.168.102.68:9866, 192.168.102.60:9866, 192.168.102.61:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,691 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975088_47870466, replicas=192.168.102.50:9866, 192.168.102.53:9866, 192.168.102.51:9866, 192.168.102.58:9866, 192.168.102.55:9866, 192.168.102.64:9866, 192.168.102.61:9866, 192.168.102.68:9866, 192.168.102.65:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,693 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975072_47870467, replicas=192.168.102.50:9866, 192.168.102.54:9866, 192.168.102.58:9866, 192.168.102.51:9866, 192.168.102.53:9866, 192.168.102.62:9866, 192.168.102.65:9866, 192.168.102.64:9866, 192.168.102.68:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,695 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975056_47870468, replicas=192.168.102.50:9866, 192.168.102.51:9866, 192.168.102.52:9866, 192.168.102.53:9866, 192.168.102.54:9866, 192.168.102.62:9866, 192.168.102.64:9866, 192.168.102.69:9866, 192.168.102.63:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,696 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975040_47870469, replicas=192.168.102.50:9866, 192.168.102.57:9866, 192.168.102.59:9866, 192.168.102.51:9866, 192.168.102.58:9866, 192.168.102.68:9866, 192.168.102.66:9866, 192.168.102.63:9866, 192.168.102.60:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,698 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975024_47870470, replicas=192.168.102.50:9866, 192.168.102.57:9866, 192.168.102.56:9866, 192.168.102.55:9866, 192.168.102.54:9866, 192.168.102.64:9866, 192.168.102.65:9866, 192.168.102.61:9866, 192.168.102.69:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,700 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103975008_47870471, replicas=192.168.102.50:9866, 192.168.102.59:9866, 192.168.102.53:9866, 192.168.102.57:9866, 192.168.102.54:9866, 192.168.102.60:9866, 192.168.102.65:9866, 192.168.102.67:9866, 192.168.102.62:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,702 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103974992_47870472, replicas=192.168.102.50:9866, 192.168.102.51:9866, 192.168.102.58:9866, 192.168.102.52:9866, 192.168.102.56:9866, 192.168.102.60:9866, 192.168.102.64:9866, 192.168.102.66:9866, 192.168.102.62:9866 for /tmp/group
May 07 18:39:56 NN_HOSTNAME shepherd-hdfs-namenode[1025695]: 2024-05-07 16:39:56,704 INFO hdfs.StateChange: BLOCK* allocate blk_-9223372036103974976_47870473, replicas=192.168.102.50:9866, 192.168.102.58:9866, 192.168.102.54:9866, 192.168.102.52:9866, 192.168.102.56:9866, 192.168.102.61:9866, 192.168.102.68:9866, 192.168.102.64:9866, 192.168.102.63:9866 for /tmp/group

Any idea what is going on here?
I'm happy to provide more debugging info :)
@ptallada
Copy link
Author

ptallada commented May 7, 2024

Ah

OS: AlmaLinux release 9.3 (Shamrock Pampas Cat)
HADOOP_VERSION=3.2.3

@ncw
Copy link
Member

ncw commented May 9, 2024

2024/05/07 18:31:26 DEBUG : hdfs://NN_HOSTNAME:9000: update [/tmp/group]
2024/05/07 18:31:26 DEBUG : group: Received error: unexpected EOF - low level retry 0/10

My guess would be that rclone is connecting to that node and being immediately rejected due to some firewalling/auth/other problem.

Perhaps you can wireshark it to get some insight?

It would be worth trying the latest beta - I don't think it will behave any differently but it is worth a try.

@ptallada
Copy link
Author

ptallada commented May 9, 2024

Hi,

I don't think it is a firewall issue, in principle the ports are open.
In any case, I've tested it also on our development setup (cluster in a single host) and it fails the same way.
I cannot read or upload files, but I can create and remove directories.

I can get a tcpdump dump, but I'm unable to analyze it. I'm not very experienced in Wireshark and it all seems garbage to me :/

Is there a docker image with kerberos that I can test?

@ncw
Copy link
Member

ncw commented May 11, 2024

The library that rclone uses has a command line client - maybe you could give that a try?

Also it says this

Like hadoop fs, the commandline client expects a ccache file in the default location: /tmp/krb5cc_. That means it should 'just work' to use kinit:

$ kinit [email protected]
$ hdfs ls /
If that doesn't work, try setting the KRB5CCNAME environment variable to wherever you have the ccache saved.

Now that doesn't mean a lot to me, but it might be worth trying with rclone.

Is there a docker image with kerberos that I can test?

The docker image we use for testing rclone/test-hdfs supports kerboros

@ptallada
Copy link
Author

I works with the command line client :(

[MY_USERNAME@HOSTNAME gohdfs]$ kinit
Password for MY_USERNAME@REALM:

[MY_USERNAME@HOSTNAME gohdfs]$ ./hdfs ls /
apps
data
hbase
tmp
user
warehouse

[MY_USERNAME@HOSTNAME gohdfs]$ ./hdfs cat /user/USERNAME/transfer_wht.sh | head
#!/bin/bash

OBS_N=$(sudo -u ...........

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Remote: HDFS
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ncw @ptallada