-
-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RetryCount does not seem to work #290
Comments
I'm unable to reproduce this issue. Are you able to confirm what version of PrtgAPI you're running? Has the above screenshot been manipulated, or is it the output of a script? (as opposed to raw commands being typed in a PowerShell prompt) Can you also please provide the output of |
The screenshot is not manipulated in any form. It was just a half output of the get-prtgclient. Sure, here we go with some additional informations: Get-PrtgClient -Diagnostic
PSVersion : 5.1.17763.2803
PSEdition : Desktop
OS : Microsoft Windows Server 2019 Standard
PrtgAPIVersion : 0.9.17
Culture : de-DE
CLRVersion : .NET Framework 4.7.2 (461814)
PrtgVersion : 22.2.76.1705
PrtgLanguage : german.lng The "call" of the actual Add-Sensor/Add-Device Sensor is quite complex and i just can't copy paste it in three lines. This is my Setup: Script: Update-PRTGDevices.ps1Import-Module prtghelper.psm1 -force # This is a custom helper module for us internal to share some methods between scripts
Connect-PrtgServer "https://$($PRTGServer):$($PRTGPort)" (New-Credential $PRTGUser $PRTGPassHash) -PassHash -RetryCount 50 -RetryDelay 50
# Gather some devices from the crm
foreach($erpdevice in $erpdevices){
$myDevice = Set-PRTGDevice -Parent $someParentGroup -ERPDevice $erpdevice
# now we add / replace some sensors with the "same" method as on the the device.
} Custom Module: prtghelper.psm1function Set-PRTGDevice{
param($Parent, $ERPDevice)
$device = $Parent | Get-Device -Name $DeviceName
# Sometimes this just return nothing even if there is a correct device
if(($device | Measure-Object).Count -gt 0){
if(($device | Measure-Object).Count -gt 1){
throw "$DeviceName in $($Parent.Name) not unique"
}
$device | Set-ObjectProperty Host $Hostname
}else{
$device = $Parent | Add-Device $DeviceName $($Hostname)
# THIS will "fail" and take aged on adding multiple devices. and i return a wrong result
}
return $device
} |
I have the same problem 😞 With PrtgApi 0.9.19 RetryCount when using |
Variable is correctly set: Connect-PrtgServer -Server $PrtgServer -Credential $credentials -PassHash -IgnoreSSL -RetryCount 10 -RetryDelay 5
Set-PrtgClient -RetryCount 10 -RetryDelay 30
Get-PrtgClient -Diagnostic | fl *
Get-PrtgClient | fl *
...
$device = Add-Device -Name $name -Host $_.album.url -Destination $group Output:
|
Ok I think I found the cause:
It looks hardcoded here. And it it's not then probably that PrtgAPI/src/PrtgAPI/Request/RequestEngine.cs Line 244 in e3fcb40
|
@lordmilko do you have a workaround, other than rolling our own sleep-then-check-in-a-loop? |
Thanks @Silex, I think I understand what's happening here. There are two levels of retries
I have effectively moved onto other projects from PrtgAPI, and am mostly providing basic level support for the product as-is until Paessler break API V1 compatibility. I think exposing something to allow specifying an object-creation-resolution-retry-count should be easy enough however, so I will work on doing this |
Hi all, Can you please try the latest pre-release build and advise how you go? PrtgAPI now takes the |
Thanks! So, the countdown "works" but it never actually resolves... Connect-PrtgServer -Server $PrtgServer -Credential $credentials -PassHash -IgnoreSSL -RetryCount 10 -RetryDelay 5
...
$device = Add-Device -Name $name -Host $_.album.url -Destination $group
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 10
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 9
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 8
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 7
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 6
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 5 The time it takes between tries is not linear, at first it takes 5 seconds before retrying... then 10... then 15... then 20... it looks like you don't just sleep Then the other problem is that it never detects the device was actually added. |
Ok, by replacing with the following workaround it works (notice the -Resolve:$false): $device = Add-Device -Name $name -Host $_.album.url -Destination $group -Resolve:$false
while ($true) {
$device = $group | Get-Device | Where-Object Name -eq $name
if ($device) {
break
}
Write-Warning 'wait'
Start-Sleep 5
} That said, I found a weird bug: C:\> $name.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
C:\> $name.length
53
C:\> $device = Get-Device -Name "$name*"
C:\> $device.name -eq $name
True
C:\> (Get-Device -Name $name).Count
0
C:\> (Get-Device | ? Name -eq $name).Count
1 I cannot show you the name because it's a client's name, but as you can see The name has a value similar to this (maybe pipes/quotes cause troubles):
Maybe that bug is related to the |
What type of sensor are you adding? If it's something to do with SNMP, the created type may differ from what you're adding (see the wiki for more info). You have to specify If flt name contains "Hello [world]" | Get-Device |
Here's a testcase: $PrtgServer = 'localhost'
$PrtgUserName = '[email protected]'
$PrtgUserHash = '123456789'
$credentials = New-Credential -Username $PrtgUsername -Password $PrtgPasshash
Connect-PrtgServer -Server $PrtgServer -Credential $credentials -PassHash -IgnoreSSL -RetryCount 10 -RetryDelay 5
$group = Get-Group -Name 'Test group'
if (!$group) {
throw 'Could not find group'
}
$name = 'Foo "BAR-BAZ | BAK | Lorem Ipsum KEWK"'
$device = Add-Device -Name $name -Host '1.2.3.4' -Destination $group
# This outputs this:
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 10
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 9
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 8
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 7
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 6
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 5 I don't mention the sensor because the error is when I create the device, so it's the As you see the name has no special characters besides quotes & pipes. |
If you do $device = Add-Device -Name $name -Host '1.2.3.4' -Destination $group -Verbose what does it respond with? And while it's timing out creating the device, if you look in the PRTG UI has the device been created? |
While it's timeing out I see the device created in PRTG, but PrtgAPI is still waiting for it. I'll report what the "-LogLevel All" says soonish. |
The |
Ok, I found out it's the quotes and not the pipes. It used to work. I guess latest updates from PRTG broke something, the good news is that I can use any string to trigger it, will post traces soon. |
So, here are the trace of a working request and a failing request. In both case, the device is created just fine, it's just the resolving of the new device that works in the first case and fails in the second case. Here's the trace of a request working (name PS C:\> $device = Add-Device -Name 'Test' -Host '1.2.3.4' -Destination $group -Verbose
VERBOSE: Performing the operation "Add-Device" on target "Test (Host: 1.2.3.4) (Destination: Test (ID: 5912))".
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="0" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
</devices>
VERBOSE: Add-Device: Synchronously executing request https://localhost/adddevice2.htm?name_=Test&host_=1.2.3.4&ipversion_=0&discoverytype_=0&discoveryschedule_=0&id=5912&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <!DOCTYPE html>
<html>
<!--
_____ _______ _______ _______ _______ _______ ______
|_____] |_____| |______ |______ |______ | |______ |_____/
| | | |______ ______| ______| |_____ |______ | \_
We are hiring! https://jobs.paessler.com
-->
<head>
<-- snip long HTML -->
VERBOSE: Add-Device: Synchronously executing request https://localhost/api/table.xml?content=devices&columns=objid,name,location,host,group,probe,favorite,condition,upsens,downsens,downacksens,partialdownsens,warnsens,pausedsens,unusualsens,undefinedsens,totalsens,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=5912&filter_name=Test&username=prtgadmin&passhash=RETRACTED
VERBOSE: Add-Device: <?xml version="1.0" encoding="UTF-8"?>
<devices totalcount="1" listend="1">
<prtg-version>24.2.96.1375</prtg-version>
<item>
<objid>7269</objid>
<name>Test</name>
<location><a href="/devices.htm?filter_location=@sub(RETRACTED)">RETRACTED.</a></location>
<location_raw>RETRACTED</location_raw>
<host>1.2.3.4</host>
<group>Test</group>
<probe>Local Probe</probe>
<favorite><span class="objectisnotfavorite icon-gray ui-icon ui-icon-flag" title="Flag/Unflag as
Favorite" id="fav-7269"
onclick="_Prtg.objectTools.faveObject.call(this,7269,'toggle');return
false;"></span></favorite>
<favorite_raw>0</favorite_raw>
<upsens/>
<upsens_raw>0000000000</upsens_raw>
<downsens/>
<downsens_raw>0000000000</downsens_raw>
<downacksens/>
<downacksens_raw>0000000000</downacksens_raw>
<partialdownsens/>
<partialdownsens_raw>0000000000</partialdownsens_raw>
<warnsens/>
<warnsens_raw>0000000000</warnsens_raw>
<pausedsens/>
<pausedsens_raw>0000000000</pausedsens_raw>
<unusualsens/>
<unusualsens_raw>0000000000</unusualsens_raw>
<undefinedsens/>
<undefinedsens_raw>0000000000</undefinedsens_raw>
<totalsens>0</totalsens>
<totalsens_raw>0000000000</totalsens_raw>
<schedule/>
<basetype>device</basetype>
<baselink>/device.htm?id=7269</baselink>
<baselink_raw>7269</baselink_raw>
<notifiesx>Inherited</notifiesx>
<intervalx>Inherited (600)</intervalx>
<intervalx_raw>0000000060</intervalx_raw>
<access>Full access</access>
<access_raw>0000000400</access_raw>
<dependency>Parent</dependency>
<dependency_raw>Test</dependency_raw>
<position>810</position>
<position_raw>0000000810</position_raw>
<status>Up </status>
<status_raw>3</status_raw>
<priority>3</priority>
<message><div class="status">OK<div
class="moreicon"></div></div></message>
<message_raw>OK</message_raw>
<parentid>5912</parentid>
<type>Device</type>
<type_raw>device</type_raw>
<active>True</active>
<active_raw>-1</active_raw>
</item>
</devices> And here's the failing one (name
|
For the record I'm completely reinstalling PRTG because it behaves weirdly, will check if the bug is still happening in 24.3.98.1210 |
Thanks @Silex, On the basis that I'm escaping the quotes correctly, I feel like PrtgAPI is doing the right thing. If you paste this URL into your web browser it will also give the XML response. You can then potentially fiddle around specifying different values to the
Alternatively, if you find that names with single quotes work, potentially modifying your names to use single quotes instead of double quotes would work? This can easily be done with a one-liner with PrtgAPI |
Yeah, it looks like a PRTG regression. Still what's weird is that PRTG creates the node with quotes just fine. And my workaround where I query the name using So my guess is that the regression is subtle... like only |
For the record single quotes don't have this problem. |
Anyway, about the original issue I think your fix works, but I think you'd not make it so the delay increases with each tries. Even with the default values it's like it takes 3 seconds for the first try, then 6, then 9... About the issue with the quotes, I'll reopen another case if needed. |
The purpose of the request retry feature is that in large PRTG installations (10,000 sensors) there can be quite a delay in attempting to perform certain actions, in particular creating new objects. Thus, when PrtgAPI repeatedly queries for the given object, and find again and again it hasn't been created yet, the assumption is that your sever must be under intense load; we want to do our best to guarantee that your request will succeed, hence why PrtgAPI will wait longer and longer and longer to maximize its chance at success. The implication of this however is that you should not go too crazy modifying the default retry parameters; I believe the defaults might result in a total wait of around 30 seconds or so. The scaling behavior ensures that PrtgAPI's default retry rules will generally function well on both small and large PRTG installations, scaling up the delay as required for your system. |
Understood. I'd maybe consider renaming |
What's going on?
Hello,
we have a quite large PRTG Installation and sometimes it takes a long time that the prtg api does it thing ;)
No i tried to set RetryCount and RetryDelay higher but it does not seem to work:
I set RetryCount and RetryDelay to 20:
Connect-PrtgServer "https://$($PRTGServer):$($PRTGPort)" (New-Credential $PRTGUser $PRTGPassHash) -PassHash -RetryCount 20 -RetryDelay 20
And now i expected that i see something like Retries remaining 20 instead of 4 ?
Am i just doing it wrong?
thank you, best regards
xasz
Due Dilligance
The text was updated successfully, but these errors were encountered: