roadlib setup.py creates libs in wrong directories

[D3vil0p3r]

Hello,
when I try to build and install roadlib, I note that the .py files are placed in the wrong non-standard python path. Specifically, after the install, the roadlib structure appears to be:

[  30]  /usr/lib/python3.12/site-packages/roadlib-0.27.0-py3.12.egg/
├── [ 152]  EGG-INFO/
│   ├── [   1]  dependency_links.txt
│   ├── [   1]  not-zip-safe
│   ├── [ 710]  PKG-INFO
│   ├── [  32]  requires.txt
│   ├── [ 307]  SOURCES.txt
│   └── [   8]  top_level.txt
└── [ 144]  roadtools/roadlib/
    ├── [ 628]  __pycache__/
    │   ├── [ 74K]  auth.cpython-312.opt-1.pyc
    │   ├── [ 74K]  auth.cpython-312.pyc
    │   ├── [6.2K]  constants.cpython-312.opt-1.pyc
    │   ├── [6.2K]  constants.cpython-312.pyc
    │   ├── [ 11K]  dbgen.cpython-312.opt-1.pyc
    │   ├── [ 11K]  dbgen.cpython-312.pyc
    │   ├── [ 42K]  deviceauth.cpython-312.opt-1.pyc
    │   ├── [ 42K]  deviceauth.cpython-312.pyc
    │   ├── [ 176]  __init__.cpython-312.opt-1.pyc
    │   ├── [ 176]  __init__.cpython-312.pyc
    │   ├── [3.9K]  metagen.cpython-312.opt-1.pyc
    │   └── [3.9K]  metagen.cpython-312.pyc
    ├── [ 67K]  auth.py
    ├── [6.4K]  constants.py
    ├── [ 12K]  dbgen.py
    ├── [ 38K]  deviceauth.py
    ├── [   0]  __init__.py
    └── [2.7K]  metagen.py

so the libs are placed inside the EGG directory that is not correct. In this manner, any attempt to create a Linux package for it could fail.

Is it possible to implement for roadlib and other roadtools the correct installation path as two separated directories, one for libs and one for egg-info, structured as:

[  30]  /usr/lib/python3.12/site-packages/roadlib-0.27.0-py3.12.egg/
│   ├── [   1]  dependency_links.txt
│   ├── [   1]  not-zip-safe
│   ├── [ 710]  PKG-INFO
│   ├── [  32]  requires.txt
│   ├── [ 307]  SOURCES.txt
│   └── [   8]  top_level.txt
 [ 144] /usr/lib/python3.12/site-packages/roadtools/roadlib/
    ├── [ 628]  __pycache__/
    │   ├── [ 74K]  auth.cpython-312.opt-1.pyc
    │   ├── [ 74K]  auth.cpython-312.pyc
    │   ├── [6.2K]  constants.cpython-312.opt-1.pyc
    │   ├── [6.2K]  constants.cpython-312.pyc
    │   ├── [ 11K]  dbgen.cpython-312.opt-1.pyc
    │   ├── [ 11K]  dbgen.cpython-312.pyc
    │   ├── [ 42K]  deviceauth.cpython-312.opt-1.pyc
    │   ├── [ 42K]  deviceauth.cpython-312.pyc
    │   ├── [ 176]  __init__.cpython-312.opt-1.pyc
    │   ├── [ 176]  __init__.cpython-312.pyc
    │   ├── [3.9K]  metagen.cpython-312.opt-1.pyc
    │   └── [3.9K]  metagen.cpython-312.pyc
    ├── [ 67K]  auth.py
    ├── [6.4K]  constants.py
    ├── [ 12K]  dbgen.py
    ├── [ 38K]  deviceauth.py
    ├── [   0]  __init__.py
    └── [2.7K]  metagen.py

Not sure if roadlib should be inside roadtools folder or not...

[dirkjanm]

hey, that must be the way you install the tool. Installing via setup.py install is deprecated in general and shouldn't be used. Installing via pip install as listed in the readme doesn't create egg folders.

[D3vil0p3r]

hey, that must be the way you install the tool. Installing via setup.py install is deprecated in general and shouldn't be used. Installing via pip install as listed in the readme doesn't create egg folders.

I'm trying to package the tools for several Linux distros and I cannot use pip install sadly. Since setup.py is deprecated, what if you migrate to PEP517?

[dirkjanm]

Both python setup.py sdist and python -m build create sdists and wheels that confirm to the desired directory structure, would that work for your purposes?

[D3vil0p3r]

@noraj what do you think?

The current PKGBUILD I'm working on is:

# This file is part of BlackArch Linux ( https://www.blackarch.org/ ).
# See COPYING for license details.

pkgbase=roadtools
_pkgbase=ROADtools
pkgname=('python-roadlib' 'roadrecon' 'roadoidc' 'roadtx')
pkgver=310.03f15f5
pkgrel=1
pkgdesc='Azure AD and O365 exploration framework'
arch=('any')
groups=('blackarch' 'blackarch-networking' 'blackarch-recon')
url='https://github.com/dirkjanm/ROADtools'
license=('MIT')
makedepends=('python-setuptools' 'git')
source=("git+https://github.com/dirkjanm/$_pkgbase.git")
sha512sums=('SKIP')

pkgver() {
  cd $_pkgbase

  ( set -o pipefail
    git describe --long --tags --abbrev=7 2>/dev/null |
      sed 's/\([^-]*-g\)/r\1/;s/-/./g' ||
    printf "%s.%s" "$(git rev-list --count HEAD)" \
      "$(git rev-parse --short=7 HEAD)"
  )
}

build_python-roadlib() {
  cd "$_pkgbase/roadlib"

  python setup.py build
}

build_roadrecon() {
  cd "$_pkgbase/roadrecon"

  python setup.py build
}

build_python-roadtx() {
  cd "$_pkgbase/roadtx"

  python setup.py build
}

package_python-roadlib() {
  depends=('python' 'python-adal' 'python-sqlalchemy' 'python-pyjwt')

  cd "$_pkgbase/roadlib"

  python setup.py install --root="$pkgdir" --prefix=/usr -O1 --skip-build
}

package_roadoidc() {
  depends=('python' 'python-roadlib' 'roadtx' 'python-flask' 'python-cryptography' 'python-requests' 'python-werkzeug')

  cd "$_pkgbase/roadoidc"

  install -dm 755 "$pkgdir/usr/bin"
  install -Dm 644 requirements.txt "$pkgdir/usr/share/$pkgname/requirements.txt"

  cp -a * "$pkgdir/usr/share/$pkgname/"

  cat > "$pkgdir/usr/bin/genconfig" << EOF
#!/bin/sh
exec python /usr/share/$pkgname/genconfig.py "\$@"
EOF

  chmod a+x "$pkgdir/usr/bin/genconfig"
}

package_roadrecon() {
  depends=('python' 'python-roadlib' 'python-flask' 'python-sqlalchemy'
           'python-marshmallow' 'python-flask-sqlalchemy'
           'python-flask-marshmallow' 'python-flask-cors'
           'python-marshmallow-sqlalchemy' 'python-aiohttp')

  cd "$_pkgbase/roadrecon"

  python setup.py install --root="$pkgdir" --prefix=/usr -O1 --skip-build
}

package_roadtx() {
  depends=('python' 'python-requests' 'python-selenium' 'python-selenium-wire' 'python-pyotp' 'python-pycryptodomex' 'python-blinker')

  cd "$_pkgbase/roadtx"

  python setup.py install --root="$pkgdir" --prefix=/usr -O1 --skip-build
}

[noraj]

If python setup.py <build|install> (setuptools) won't work, then try PEP517 PKGBUILD template. It may work even without pyproject.toml.

[D3vil0p3r]

@dirkjanm I was able to install the files in the right directory. I'm using the last commit. I'm getting issues on using roadrecon-gui and roadtx.

roadrecon-gui

When I run roadrecon-gui, the server starts on localhost:5000. When I open the browser and I access to it, I get Internal Server Error message and logs on console report:

[2024-11-17 15:12:06,204] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask_cors/extension.py", line 194, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/roadtools/roadrecon/server.py", line 188, in get_index
    return send_file('dist_gui/index.html')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/helpers.py", line 504, in send_file
    return werkzeug.utils.send_file(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/werkzeug/utils.py", line 428, in send_file
    stat = os.stat(path)
           ^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3.12/site-packages/roadtools/roadrecon/dist_gui/index.html'
127.0.0.1 - - [17/Nov/2024 15:12:06] "GET / HTTP/1.1" 500 -

indeed, the directory /usr/lib/python3.12/site-packages/roadtools/roadrecon/dist_gui/ is empty and does not contain any index.html file.

roadtx

When I run roadtx, I get:

Traceback (most recent call last):
  File "/usr/bin/roadtx", line 5, in <module>
    from roadtools.roadtx.main import main
  File "/usr/lib/python3.12/site-packages/roadtools/roadtx/main.py", line 14, in <module>
    from roadtools.roadtx.selenium import SeleniumAuthentication
  File "/usr/lib/python3.12/site-packages/roadtools/roadtx/selenium.py", line 8, in <module>
    from seleniumwire.webdriver import FirefoxOptions
  File "/usr/lib/python3.12/site-packages/seleniumwire/webdriver.py", line 28, in <module>
    from seleniumwire import backend, utils
  File "/usr/lib/python3.12/site-packages/seleniumwire/backend.py", line 4, in <module>
    from seleniumwire.server import MitmProxy
  File "/usr/lib/python3.12/site-packages/seleniumwire/server.py", line 5, in <module>
    from seleniumwire.handler import InterceptRequestHandler
  File "/usr/lib/python3.12/site-packages/seleniumwire/handler.py", line 5, in <module>
    from seleniumwire import har
  File "/usr/lib/python3.12/site-packages/seleniumwire/har.py", line 11, in <module>
    from seleniumwire.thirdparty.mitmproxy import connections
  File "/usr/lib/python3.12/site-packages/seleniumwire/thirdparty/mitmproxy/connections.py", line 10, in <module>
    from seleniumwire.thirdparty.mitmproxy.net import tls, tcp
  File "/usr/lib/python3.12/site-packages/seleniumwire/thirdparty/mitmproxy/net/tls.py", line 15, in <module>
    import seleniumwire.thirdparty.mitmproxy.options
  File "/usr/lib/python3.12/site-packages/seleniumwire/thirdparty/mitmproxy/options.py", line 5, in <module>
    from seleniumwire.thirdparty.mitmproxy import optmanager
  File "/usr/lib/python3.12/site-packages/seleniumwire/thirdparty/mitmproxy/optmanager.py", line 9, in <module>
    import blinker._saferef
ModuleNotFoundError: No module named 'blinker._saferef'

blinkerpython module is installed, but it does not contain _saferef.

How can we fix these two issues?

[dirkjanm]

For roadrecon, you need to build the GUI using node if you're building it from source, unless you download one of the builds from Azure Pipelines. See the project README for links.

For roadtx, the blinker version that works is <1.8.0 as is in the setup.py for roadtx

[D3vil0p3r]

@dirkjanm Since roadrecon-gui binary is created by setup.py, could it be a good idea if the npm install on frontend for roadrecon-gui is run by setup.py instead of doing it separetely?

Because I install roadrecon by pip, and it creates files in /usr/lib/python3.12/site-packages/roadtools/roadrecon, but it forces me to run npm installmanually and I'm not sure where should I do, or if I should clone manually roadrecon, and apply npm install on frontend directory and then copy frontend content to /usr/lib/python3.12/site-packages/roadtools/roadrecon/dist_gui? Cannot be automated?

[dirkjanm]

If you install roadrecon from source, you're supposed to build the GUI before you install the python package. This is already done in the build pipeline on Azure Pipelines, so for packaging it might make more sense to get the built zip from the artifacts.

[D3vil0p3r]

@dirkjanm On packaging, we prefer to build always from source instead of getting artifacts. Currently the approach is the following on fakeroot env for roadrecon:

build() {
  cd "$_pkgbase/roadrecon/frontend"

  npm install

  cd ".."

  python -m build --wheel --outdir="$startdir/dist"
}

package_roadrecon() {
  depends=('python' 'roadlib' 'python-flask' 'python-sqlalchemy'
           'python-marshmallow' 'python-flask-sqlalchemy'
           'python-flask-marshmallow' 'python-flask-cors'
           'python-marshmallow-sqlalchemy' 'python-aiohttp')

  pip install \
    --verbose \
    --disable-pip-version-check \
    --no-warn-script-location \
    --ignore-installed \
    --no-compile \
    --no-deps \
    --root="$pkgdir" \
    --prefix=/usr \
    --no-index \
    --find-links="file://$startdir/dist" \
    $pkgname
}

but despite I run npm install before running Python setup, I'm still getting the following error when I run the GUI and I access to localhost:5000:

 * Serving Flask app 'roadtools.roadrecon.server'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
[2024-11-18 18:56:22,143] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask_cors/extension.py", line 194, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/roadtools/roadrecon/server.py", line 188, in get_index
    return send_file('dist_gui/index.html')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/helpers.py", line 504, in send_file
    return werkzeug.utils.send_file(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/werkzeug/utils.py", line 428, in send_file
    stat = os.stat(path)
           ^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3.12/site-packages/roadtools/roadrecon/dist_gui/index.html'
127.0.0.1 - - [18/Nov/2024 18:56:22] "GET / HTTP/1.1" 500 -
[2024-11-18 18:56:23,940] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask_cors/extension.py", line 194, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/roadtools/roadrecon/server.py", line 188, in get_index
    return send_file('dist_gui/index.html')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/flask/helpers.py", line 504, in send_file
    return werkzeug.utils.send_file(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/werkzeug/utils.py", line 428, in send_file
    stat = os.stat(path)
           ^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3.12/site-packages/roadtools/roadrecon/dist_gui/index.html'
127.0.0.1 - - [18/Nov/2024 18:56:23] "GET / HTTP/1.1" 500 -

[dirkjanm]

You're missing the npm run build after the npm install. As per the readme:

To build the JavaScript files into ROADrecon's dist_gui directory, run npm run build.

[D3vil0p3r]

You're missing the npm run build after the npm install. As per the readme:

To build the JavaScript files into ROADrecon's dist_gui directory, run npm run build.

@dirkjanm It works. When I run the npm run build to create the package for Arch, I get this Angular prompt:

? Would you like to share pseudonymous usage data about this project with the Angular Team
at Google under Google's Privacy Policy at https://policies.google.com/privacy. For more
details and how to change this setting, see https://angular.io/analytics.

could you please add the following code in angular.json, or another equivalent solution, to prevent this prompt (and disabling analytics for Privacy reason) so that the build can run with no prompt interruption?

"cli": {
  "analytics": false
}

[dirkjanm]

I've added it to angular.json as requested

[D3vil0p3r]

It works. Thank you very much!

[dirkjanm]

awesome, thanks for taking the time to getting this into more distros!