regression with 0.11 version: hardcoded super() breaks the python3 class extensibility

[sem-geologist]

I actually am not sure if this is regression introduced by kaitaistruct python runtime or rather kaitaistruct python compiler.

I was happily using kaitaistruct 0.10 python runtime for few years. However after trying to update my environment to kaitaistruct 0.11 runtime, and my initial ksy recompiled through web-ide of kaitaistruct for kaitaistruct 0.11, the parser no more works. Parser source ksy code is here:
https://github.com/sem-geologist/peaksight-binary-parser

The error I am running into after running this:

import cameca
qtifile = cameca.Cameca.from_file('Apatite_F-LTAP.qtiSet')

is this:

---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
/tmp/ipykernel_13843/3962983913.py in <module>
      3 import cameca
----> 4 qtifile = cameca.Cameca.from_file('../RE-qtiset/Apatite_F-LTAP.qtiSet')

~/[redacted]/kaitaistruct.py in from_file(cls, filename)
     43         f = open(filename, 'rb')
     44         try:
---> 45             return cls(KaitaiStream(f))
     46         except Exception:
     47             # close file descriptor, then reraise the exception

~/[redacted]/cameca.py in __init__(self, _io, _parent, _root)
    193         self._parent = _parent
    194         self._root = _root or self
--> 195         self._read()
    196 
    197     def _read(self):

~/[redacted]/cameca.py in _read(self)
    196 
    197     def _read(self):
--> 198         self.header = Cameca.SxfHeader(self._io, self, self._root)
    199         _on = self.header.file_type
    200         if _on == Cameca.FileType.calibration_results:

~/[redacted]/cameca.py in __init__(self, _io, _parent, _root)
   2201             self._parent = _parent
   2202             self._root = _root
-> 2203             self._read()
   2204 
   2205         def _read(self):

~/[redacted]/cameca.py in _read(self)
   2209                 raise kaitaistruct.ValidationNotEqualError(b"\x66\x78\x73", self.magic, self._io, u"/types/sxf_header/seq/1")
   2210             self.sxf_version = self._io.read_u4le()
-> 2211             self.comment = Cameca.CSharpString(self._io, self, self._root)
   2212             self.reserved_0 = self._io.read_bytes(24)
   2213             if self.sxf_version >= 3:

~/[redacted]/cameca.py in __init__(self, _io, _parent, _root)
    314     class CSharpString(KaitaiStruct):
    315         def __init__(self, _io, _parent=None, _root=None):
--> 316             super(Cameca.CSharpString, self).__init__(_io)
    317             self._parent = _parent
    318             self._root = _root

... last 1 frames repeated, from the frame below ...

~/[redacted]/cameca.py in __init__(self, _io, _parent, _root)
    314     class CSharpString(KaitaiStruct):
    315         def __init__(self, _io, _parent=None, _root=None):
--> 316             super(Cameca.CSharpString, self).__init__(_io)
    317             self._parent = _parent
    318             self._root = _root

RecursionError: maximum recursion depth exceeded while calling a Python object

I am attaching zipped binary file.

python 3.10.12
Apatite_F-LTAP.qtiSet.zip

[sem-geologist]

hmm... fast check with older ksy version and compile with ksc 0.10 made it working. So it is rather not issue with 0.11 kaitaistruct python runtime, but with kaitaistruct compiled python code. It looks to me that due to super() it somehow loops recursively...
at 0.10 the class on which it gets hiccuped at 0.11 looked like this:

    class CSharpString(KaitaiStruct):
        def __init__(self, _io, _parent=None, _root=None):
            self._io = _io
            self._parent = _parent
            self._root = _root if _root else self
            self._read()

        def _read(self):
            self.str_len = self._io.read_u4le()
            self.text = (self._io.read_bytes(self.str_len)).decode(u"CP1252")

where on 0.11 compiled python code it looks like this:

    class CSharpString(KaitaiStruct):
        def __init__(self, _io, _parent=None, _root=None):
            super(Cameca.CSharpString, self).__init__(_io)
            self._parent = _parent
            self._root = _root
            self._read()

        def _read(self):
            self.str_len = self._io.read_u4le()
            self.text = (self._io.read_bytes(self.str_len)).decode(u"windows-1252")


        def _fetch_instances(self):
            pass

[generalmimon]

@sem-geologist I cannot reproduce this:

Dockerfile

FROM python:3.10.12-slim
RUN pip install kaitaistruct==0.11
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
	apt-get install -y --no-install-recommends curl unzip default-jre && \
	curl -fsSLO https://github.com/kaitai-io/kaitai_struct_compiler/releases/download/0.11/kaitai-struct-compiler_0.11_all.deb && \
	apt-get install ./kaitai-struct-compiler_0.11_all.deb && \
	rm -f kaitai-struct-compiler_0.11_all.deb && \
	rm -rf /var/lib/apt/lists/*
RUN curl -fsSLO https://github.com/sem-geologist/peaksight-binary-parser/raw/6d16a273d317a783f98ea38ab709a770b7fc1ced/cameca.ksy
RUN kaitai-struct-compiler -- --verbose file -t python cameca.ksy
RUN curl -fsSLO https://github.com/user-attachments/files/23699041/Apatite_F-LTAP.qtiSet.zip
RUN unzip Apatite_F-LTAP.qtiSet.zip
CMD ["python3", "-i", "-c", "import cameca, pprint; qtifile = cameca.Cameca.from_file('Apatite_F-LTAP.qtiSet'); pprint.pp(vars(qtifile))"]

---

$ docker build --no-cache -t ks-python-issue-83 .
[+] Building 53.8s (11/11) FINISHED                                                              docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                             0.0s
 => => transferring dockerfile: 959B                                                                             0.0s
 => [internal] load metadata for docker.io/library/python:3.10.12-slim                                           0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                  0.0s
 => CACHED [1/7] FROM docker.io/library/python:3.10.12-slim                                                      0.0s
 => [2/7] RUN pip install kaitaistruct==0.11                                                                     2.0s
 => [3/7] RUN apt-get update &&  apt-get install -y --no-install-recommends curl unzip default-jre &&  curl -f  45.4s
 => [4/7] RUN curl -fsSLO https://github.com/sem-geologist/peaksight-binary-parser/raw/6d16a273d317a783f98ea38a  0.6s
 => [5/7] RUN kaitai-struct-compiler -- --verbose file -t python cameca.ksy                                      1.5s
 => [6/7] RUN curl -fsSLO https://github.com/user-attachments/files/23699041/Apatite_F-LTAP.qtiSet.zip           1.0s
 => [7/7] RUN unzip Apatite_F-LTAP.qtiSet.zip                                                                    0.3s
 => exporting to image                                                                                           2.9s
 => => exporting layers                                                                                          2.8s
 => => writing image sha256:bcadf11bd14e59e082dfa4c1cf87cd9304ba61cc1f9f19ac7a8ab15ad7c99a77                     0.0s
 => => naming to docker.io/library/ks-python-issue-83                                                            0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/guys1o2zv0fpbmvsh4op6b90f

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview

$ docker run -it --rm ks-python-issue-83
{'_io': <kaitaistruct.KaitaiStream object at 0x749f87ac3c10>,
 '_parent': None,
 '_root': <cameca.Cameca object at 0x749f87a26e30>,
 'header': <cameca.Cameca.SxfHeader object at 0x749f878b42e0>,
 'content': <cameca.Cameca.QtiSetup object at 0x749f878b5000>}
>>> 

[sem-geologist]

Oh... I tried it in more isolated env and it also works for me. So probably I was getting it in not clean python environment where it is being overridden with sub-classes extending features (i.e. adding __str__() and __repr__ methods for that CSharpString class). Prior 0.11 it was perfectly working as there was no super() method used everywhere, I guess this push super() into loop due to this type override of mine. Shame on me, probably I should deal with it differently. I think this can be closed then.

[sem-geologist]

I see now exactly where lies my problem and what would be a fix. The problem is super(AnyClassDefinedInKaitai, self).__init__(_io) instead of super().__init__(_io). So my class extension overriding breaks and falls into circular loop at __init__ for extended and overwritten classes when super(SomeClass) is called. I guess the reason behind why the class and self is hardcoded inside the super() is as only that works on python2, where super() without hardcoded values inside would work only on python3.

That practically makes class extensions impossible at 0.11 (which was possible at 0.10).

solution to my problem I think are these:

1. refactor my code, and do a monkey patch of classes with added functions instead of class extensions and overriding.
2. freeze my requirements of app to kaitaistruct 0.10 and wait until kaitai struct will drop ancient python2, and then bother here again, to fix the super() usage to not brake class expansionability of python3.

P.S. If it is possible this issue should be moved to Kaitaistruct compiler, as it is clearly not runtime issue.