Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Kadalu upgrading not working properly and PV data not correct #1045

Open
Fevereiro6 opened this issue Feb 7, 2024 · 2 comments
Open
Labels
on-user pending on user

Comments

@Fevereiro6
Copy link

After updating Kadalu following the upgrade guide, everything was updated except the CLI:

kubectl-kadalu version

kubectl-kadalu plugin: 1.0.0  <-------- THIS WAS NOT UPDATED
kadalu pod(s) versions
pod/kadalu-csi-nodeplugin-2s8wp: 1.2.0
pod/kadalu-csi-nodeplugin-6lpf7: 1.2.0
pod/kadalu-csi-nodeplugin-87ldz: 1.2.0
pod/kadalu-csi-provisioner-0: 1.2.0
pod/operator-58fdbb58b6-87j4k: 1.2.0
pod/server-storage-pool-charity-1-0-0: 1.2.0
pod/server-storage-pool-charity-1-1-0: 1.2.0
pod/server-storage-pool-charity-1-2-0: 1.2.0

And did not solve my problem of PV management, as I have less than 22 PVs (16 PVs) and the sum of storage does not add up to 141 Gi:

kubectl kadalu storage-list --status

Name             Type        Utilization            Pvs Count      Min PV Size      Avg PV Size      Max PV Size
pool1  Replica1    141 Gi/149 Gi (94%)           22           100 Mi             6 Gi            30 Gi
@Yitozu
Copy link

Yitozu commented Feb 21, 2024

I also had this problem with 1.2.0: common-kadalu has 99 PVS, but there are actually not that many, I also found doing

[root@openstack-hdp01 ~]# kubectl -n kadalu exec -it kadalu-csi-provisioner-0 -c kadalu-provisioner -- bash
root@kadalu-csi-provisioner-0:/# df -h /mnt/common-kadalu
The result of the command is inconsistent with that of kubectl kadalu storage-list --status
[root@openstack-hdp01 ~]# kubectl kadalu storage-list --status
Failed to get size details of the storage "conf-kadalu"
error 1 command terminated with exit code 1


Failed to get size details of the storage "demo-kadalu"
error 1 command terminated with exit code 1



Name             Type        Utilization            Pvs Count      Min PV Size      Avg PV Size      Max PV Size
common-kadalu    Replica2    8 Ti/14 Ti (61%)              99             1 Gi            92 Gi           500 Gi  
conf-kadalu      Replica2    0/0 (0%)                       0                0                0                0  
demo-kadalu      Replica2    0/0 (0%)                       0                0                0                0  
graph-kadalu     Replica2    1 Ti/14 Ti (12%)               6           300 Gi           300 Gi           300 Gi  
project-kadalu   Replica3    454 Gi/14 Ti (3%)             19             5 Gi            23 Gi           100 Gi  
[root@openstack-hdp01 ~]# kubectl kadalu remove-archived-pv server-common-kadalu-0-0
Aborting.....
Invalid name. No such storage 'server-common-kadalu-0-0' in Kadalu configmap.
[root@openstack-hdp01 ~]# kubectl -n kadalu exec -it kadalu-csi-provisioner-0 -c kadalu-provisioner -- bash 
root@kadalu-csi-provisioner-0:/# df -h /mnt/common-kadalu
Filesystem            Size  Used Avail Use% Mounted on
kadalu:common-kadalu   15T  4.5T  9.5T  32% /mnt/common-kadalu
root@kadalu-csi-provisioner-0:/# exit
exit
You have mail in /var/spool/mail/root
[root@openstack-hdp01 ~]# kubectl kadalu storage-list --status
Failed to get size details of the storage "conf-kadalu"
error 1 command terminated with exit code 1


Failed to get size details of the storage "demo-kadalu"
error 1 command terminated with exit code 1



Name             Type        Utilization            Pvs Count      Min PV Size      Avg PV Size      Max PV Size
common-kadalu    Replica2    8 Ti/14 Ti (61%)              99             1 Gi            92 Gi           500 Gi  
conf-kadalu      Replica2    0/0 (0%)                       0                0                0                0  
demo-kadalu      Replica2    0/0 (0%)                       0                0                0                0  
graph-kadalu     Replica2    1 Ti/14 Ti (12%)               6           300 Gi           300 Gi           300 Gi  
project-kadalu   Replica3    454 Gi/14 Ti (3%)             19             5 Gi            23 Gi           100 Gi  
[root@openstack-hdp01 ~]# kubectl -n kadalu exec -it kadalu-csi-provisioner-0 -c kadalu-provisioner -- bash 
root@kadalu-csi-provisioner-0:/# df -h /mnt/common-kadalu
Filesystem            Size  Used Avail Use% Mounted on
kadalu:common-kadalu   15T  4.5T  9.5T  32% /mnt/common-kadalu
root@kadalu-csi-provisioner-0:/# 

@leelavg
Copy link
Collaborator

leelavg commented Apr 12, 2024

Failed to get size details of the storage "conf-kadalu"
error 1 command terminated with exit code 1


Failed to get size details of the storage "demo-kadalu"
error 1 command terminated with exit code 1
  • so you have removed these volumes (kds) but these aren't cleaned up in the backend configmap, need to look at it

kubectl-kadalu plugin: 1.0.0 <-------- THIS WAS NOT UPDATED

  • I don't think the plugin is self updated, you need to do that manually or via krew

May I know what are the pools that you have now, the listing differs above so it's confusing, what's k get kds -A, k get pv | grep -i kadalu

I believe the underlying issue is kadalu configmap and CRs are out of sync. Let's tackle one by one.

@leelavg leelavg added the on-user pending on user label Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
on-user pending on user
Projects
None yet
Development

No branches or pull requests

3 participants