randomly failures when using Erratum to fetch rpmdiff test results for advisories

[lmilbaum]

Every once in a while, hitting this message. It is not consistent. No clear understanding what is going on and how to troubleshoot it.

This is the stacks-trace:

 File "/home/lmilbaum/src/code.engineering.redhat.com/et-reporter/lib/gateslib/rpmdiff.py", line 24, in rpmdiff
    results = [result for result in e.externalTests(test_type='rpmdiff')
  File "/home/lmilbaum/.virtualenvs/venv3/lib/python3.6/site-packages/errata_tool/erratum.py", line 462, in externalTests
    response = self._get(url)
  File "/home/lmilbaum/.virtualenvs/venv3/lib/python3.6/site-packages/errata_tool/connector.py", line 185, in _get
    'Pigeon crap. Did it forget to run kinit?')```

[yazug]

this seems to be an issue that kerberos authentication is not being recognized. try running klist to see if you have a valid ticket.

[ktdreyer]

Maybe we should change that error message to suggest what you suggested: "klist"

[lmilbaum]

I have reason to believe it is related to the ssl verification. Verifying it.

[ktdreyer]

If it was an SSL problem, you should have seen an SSLException from python-requests.

The code is here:

            elif ret_data.status_code in [401]:
                raise ErrataException(
                    'Pigeon crap. Did it forget to run kinit?')

It looks to me like the SSL handshake completed, and you got a real HTTP response code from the server.

[lmilbaum]

@ktdreyer I see your point. The program which uses it, is running in a mock environment. I have difficulties to debug it. Replacing the usage of errata_tool with the errata REST API just to verify it.

[ktdreyer]

Would you please say more about the mock environment?

[lmilbaum]

Sure.
https://github.com/rpm-software-management/mock/wiki

[yazug]

Thanks lioramilbaum for talking with me and providing more details for this bug report.

Updated summary of failure. The script/usage would work most of the time but randomly it would fail and spit out the above error message.

Based on the behavior I think we can rule out krb auth and certs being there as the primary failure cause. I think there is likely an edge case that is being caught by errata-tool and reporting just one failure mode back.

The code being substituted in is something like the following:

def rpmdiff(errata_id):
    e = Erratum(errata_id)
    results = [result for result in e.externalTests(test_type='rpmdiff') ...]

Which is replacing the following from their existing working code:

def rpmdiff(errata_id):
    res = requests.get( .../v1/external_tests/?filter....)
    results = [result for result in res.json()['data'] ... ]

An initial guess is that ET is getting overloaded with extra calls that Erratum init method is doing to prep the object upfront when the only thing you are really wanting to process at this point is the external test results.

[ktdreyer]

It is really weird that the ET would be giving a 401 status code. @lioramilbaum can you break up that res.json()['data'] line into several more lines, so we can see the exact problem?

Like you can make your rpmdiff(errata_id) method do the following:

    res = requests.get( .../v1/external_tests/?filter....)
    res.raise_for_status()  # If we get an HTTP 401, we will raise here
    json_result = res.json()
    data = json_result['data']
    results = [result for result in data ... ]

[ktdreyer]

Also, with the external_tests stuff, the Errata Tool does have to wait for the external test system to report back to it. In this case with RPMDiff, the ET waits for the RPMDiff systems to report the results back to the ET. It's still strange that we would get an HTTP 401 error here, but anyway, we might be hitting some small corner case here.

[yazug]

what you have is essentially what is already there and it works. I think the issue might be the extra overhead that Setting up the Erratum object adds to ET but not certain. We need to look closer and likely instrument up a test case for this to figure out the issue.

[ktdreyer]

Yeah, I would like to confirm in particular that this truly is a 401 error.

[ktdreyer]

@lioramilbaum , a while ago we talked IRC about Kerberos. You had a Jenkins job that was wrapping some calls in withKerberosCredentialsCache.groovy. I think we discovered that Groovy script was not operating the way you expected. The Jenkins job you mentioned was not using Errata Tool specifically, but I'm wondering if that same environmental problem could be applying in this case as well.

[lmilbaum]

@ktdreyer We wrap the Jenkins job script with the function - 'withKrbAuth'. Have you noticed issues with this function in particular?

[ktdreyer]

That was just wild speculation - I'm not familiar with Groovy or withKerberosCredentialsCache or withKrbAuth :)