Performance impact of refreshing each datastore info at every scrape

[pryorda]

From @wtip on September 12, 2017 21:47

I've been running into an issue where one of my esxi hosts appears to be unreachable by the exporter.
I think what's actually happening is that the scrape is timing out.

The jump in scrape duration coincides with about the time that I deployed this code change #16
and it looks like I might be bumping up against my default 10 second timeout.
I've increased my timeout but I'm not sure if this is a great solution for everyone. Maybe the datastore refresh should be a configurable option.

Copied from original issue: rverchere/vmware_exporter#18

[pryorda]

From @rverchere on September 13, 2017 9:55

Hi,

yes I have the same issue, need to find another way to refresh datastore.

[pryorda]

From @wtip on September 13, 2017 14:25

My scrape interval is 60 seconds and after my discovery yesterday I changed the timeout from 10s to 30s.
Interestingly the average scrape duration also improved from around 3.5 seconds to 2.5 seconds after the change. My guess is that this has something to do with request blocking.
In the graph above the 1 node with a significantly higher scrape duration has 2 iscsi datastores instead of 1 so my guess is that the refresh duration increases with every datastore that is attached.

There's also the RefreshDatastore() method https://github.com/vmware/pyvmomi/blob/master/docs/vim/Datastore.rst maybe this is faster. This is not supported on the free esxi license. I got a 'Current license or ESXi version prohibits execution of the requested operation.',

I'm also wondering if there's a good way to make the refresh not block the rest of the request. I don't really care if the capacity info is refreshed before the request is returned. I just need it to be refreshed on a regular basis.

[pryorda]

From @nivshlomokaltura on September 26, 2017 14:14

Hello,

Exporter is getting vcenter (5.5\6.0.2) to refresh datastore information as a vcenter task each query cycle and flood the task viewer.

i would suggest to remove ds.RefreshDatastoreStorageInfo() from vmware_exporter.py

[pryorda]

It appears this was removed. Let us know if you have any other issues with this.