Impact of the bug
All services.

Describe the bug
We were contacted to check an HTTP error 500 coming from our services related to a single DN. While checking the logs I managed to confirm that certain DNs do trigger an exception in our authentication code at the REST module. [1] (user related information has been stripped off for security reasons).

Upon further debugging the root of the problem has been identified, and even the time of the OPS activities causing it matched perfectly with our observations.

• In WMCore there is this function, which checks all groups and roles associated to a user in order to authenticate him:
  

  WMCore/src/python/WMCore/REST/Auth.py

  Line 127 in 747c70a

  def authz_user(role=None, group=None, site=None, key=None, verbose=False):

• This function lives in the REST module and does these checks by iterating through all HTTP headers of the request, both:

  • those coming from the user's original request and
  • those added by the Front End upon fetching the relevant user information from CRIC.
    

    WMCore/src/python/WMCore/REST/Auth.py

    Lines 17 to 20 in 747c70a

    	def user_info_from_headers(key, verbose=False):
    	"""Read the user information HTTP request headers added by front-end.
    	Validates the HMAC on them to check for tampering, and if all is ok,
    	returns user info object with the data from the headers."""

In one of those headers added by the FE we now see an unknown type of group: iam_group. Checking with @drkovalskyi and @Panos512 it became clear that the new type/category of groups has been added to CRIC yesterday related to some new development for enabling token based authentication. And only a particular set of users have been associated with it. The error we observe is simply because we know nothing about this new category of role/group. If it was just a new group... that wouldn't have broken anything, but being it a completely new type, then few lines of code need to be added on our side.

Here is the Jira Ticket associated with this: [2]

And here is a really important twiki page explaining what it is all about [3]

How to reproduce it
Anybody who has this new type of group/role associated with his account in CRIC would receive Internel server error HTTP 500 when trying to connect to WMCore services, regardless of the client or the method of authentication he uses (so far x509 only).

Expected behavior
Not to have the service broken because of a change in CRIC.

Additional context and error message
[1]

[23/Feb/2023:00:24:17]  
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 117, in run_hooks
    hook()
  File "/usr/local/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 65, in __call__
    return self.callback(**self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/WMCore/REST/Auth.py", line 138, in authz_user
    cherrypy.request.user = user_info_from_headers(key, verbose)
  File "/usr/local/lib/python3.8/site-packages/WMCore/REST/Auth.py", line 59, in user_info_from_headers
    user['roles'][hkname][site_or_group].add(name)
KeyError: 'iam_group'
[23/Feb/2023:00:24:17] HTTP 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/usr/local/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 688, in _do_respond
    self.hooks.run('before_request_body')
  File "/usr/local/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 95, in run
    self.run_hooks(iter(sorted(self[point])))
  File "/usr/local/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 117, in run_hooks
    hook()
  File "/usr/local/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 65, in __call__
    return self.callback(**self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/WMCore/REST/Auth.py", line 138, in authz_user
    cherrypy.request.user = user_info_from_headers(key, verbose)
  File "/usr/local/lib/python3.8/site-packages/WMCore/REST/Auth.py", line 59, in user_info_from_headers
    user['roles'][hkname][site_or_group].add(name)
KeyError: 'iam_group'
[23/Feb/2023:00:24:17] HTTP 
Request Headers:
  Remote-Addr: 137.138.158.176
  HOST: cmsweb-k8s-prodsrv.cern.ch
  X-REQUEST-ID: 3cb59b9a52a286789efecda1345777e6
  X-REAL-IP: 188.185.101.116
  X-FORWARDED-FOR: 188.185.101.116
  X-FORWARDED-HOST: cmsweb-k8s-prodsrv.cern.ch
  X-FORWARDED-PORT: 80
  X-FORWARDED-PROTO: http
  X-FORWARDED-SCHEME: http
  X-SCHEME: http
  X-ORIGINAL-FORWARDED-FOR: 188.184.104.18
  USER-AGENT: PycURL/7.29.0
  ACCEPT: */*
  HTTPS: on
  CMS-REQUEST-URI: /t0_reqmon/data/requestcache
  CMS-AUTH-STATUS: OK
  CMS-AUTH-CERT: 
  ....
  X-FORWARDED-SERVER: localhost

[2]
https://its.cern.ch/jira/browse/CMSKUBERNETES-220

[3]
https://twiki.cern.ch/twiki/bin/viewauth/CMS/IAMTokens