Expand cluster attempts keep running into "disk already being used in another erasure deployment"

[dbhirud]

I am trying to deploy MinIO on an on-prem K8S cluster, which is intended to start as a single node deployment, which then can be extended by adding more nodes.
I have started with deploying MinIO in distributed mode on node 1 (with attaching 4 separate PVs). Things work as expected until this point. (Following this : https://docs.minio.io/docs/distributed-minio-quickstart-guide)

Now, when I get to expanding my cluster by adding a node, I would like to add storage to the MinIO cluster as well. So, I added 2 more PVs (of the same size as the first 4) which are bound to local paths on the second node.
My attempts to run minio server (with the 6 pods in command line) ran into error saying something to the effect "The port is already in use".
Then I tried to recreate the statefulsets with replicaCount set to 6, and then I am seeing all the statefulsets bound to the original 4 PVs erroring out saying :
"ERROR Unable to initialize backend: /data disk is already being used in another erasure deployment. (Number of disks specified: 6 but the number of disks found in the 4th disk's format.json: 4)"

Question is : Is this type of expansion of cluster supported ? What are the recommendations for gradually adding storage to the MinIO cluster on a need basis. ?
Help appreciated.

[harshavardhana]

@dbhirud expansion doesn't work like that - expansions only happen in a collection of pools and each collection is self-contained. Feel free to read our docs https://docs.minio.io/docs/distributed-minio-quickstart-guide.html

Moving this into a discussion instead of an actual issue.

Also adding one node at a time is not the use-case MinIO is built for so you should look elsewhere if that is what you want with rebalancing etc.

[dbhirud]

Thank you. I am now attempting to basically copy all buckets and objects from a standalone instance of minio to a distributed one (which has N nodes pre-configured). Is there a client command which copies ALL buckets (all objects) without having to enumerate buckets and then having to create them on the dest cluster ?

I have tried
mc cp --recursive local new-minio
and it keeps erroring saying the buckers dot not exist. Assuming that it expecting the buckets to be pre-created on the destination.

Is there another command I can use to get past this ?

[harshavardhana]

You should use mc mirror not mc cp