Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

urllib camera numpy issue #1351

Open
iqgmoritz opened this issue Nov 8, 2023 · 21 comments
Open

urllib camera numpy issue #1351

iqgmoritz opened this issue Nov 8, 2023 · 21 comments

Comments

@iqgmoritz
Copy link

Describe the problem/bug

The urllib camera always fails when trying to capture a still image and the web-UI shows the error: "Could not acquire image"

Versions:

  • Mycodo Version: 8.15.12
  • Raspberry Pi Version: 4B
  • Raspbian OS Version: RaspberryPi OS (Legacy) Lite

Reproducibility

Please list specific setup details that are involved and the steps to reproduce the behavior:

  1. Go to the Camera Menu
  2. Add a Camera with the urllib library
  3. Install dependencies
  4. Try to capture a still image (standard url or any other)
  5. Error shows up

The log shows this:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 130, in
from numpy.config import show as show_config
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/config.py", line 4, in
from numpy.core._multiarray_umath import (
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/init.py", line 50, in
raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  • The Python version is: Python3.9 from "/var/mycodo-root/env/bin/python"
  • The NumPy version is: "1.26.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/var/mycodo-root/mycodo/controllers/controller_function.py", line 79, in loop
self.run_function.loop()
File "/home/moritz/Mycodo/mycodo/functions/equation_multi.py", line 167, in loop
last_measurement_a = self.get_last_measurement(
File "/var/mycodo-root/mycodo/abstract_base_controller.py", line 533, in get_last_measurement
return get_last_measurement(device_id, measurement_id, max_age=max_age)
File "/var/mycodo-root/mycodo/utils/influx.py", line 322, in get_last_measurement
last_measurement = read_influxdb_single(
File "/var/mycodo-root/mycodo/utils/influx.py", line 372, in read_influxdb_single
data = query_string(
File "/var/mycodo-root/mycodo/utils/influx.py", line 301, in query_string
ret_value = query_flux(
File "/var/mycodo-root/mycodo/utils/influx.py", line 185, in query_flux
from influxdb_client import InfluxDBClient
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/init.py", line 382, in
from influxdb_client.client.delete_api import DeleteApi
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/delete_api.py", line 7, in
from influxdb_client.client._base import _BaseDeleteApi
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/_base.py", line 22, in
from influxdb_client.client.write.dataframe_serializer import DataframeSerializer
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/dataframe_serializer.py", line 12, in
from influxdb_client.client.write.point import _ESCAPE_KEY, _ESCAPE_STRING, _ESCAPE_MEASUREMENT, DEFAULT_WRITE_PRECISION
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/point.py", line 40, in
import numpy as np
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 135, in
raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python interpreter from there.
2023-11-08 19:06:41,678 - ERROR - mycodo.controllers.controller_function_2ce6801e - Exception while running loop()
Traceback (most recent call last):
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/init.py", line 24, in
from . import multiarray
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/multiarray.py", line 10, in
from . import overrides
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/overrides.py", line 8, in
from numpy.core._multiarray_umath import (
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 130, in
from numpy.config import show as show_config
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/config.py", line 4, in
from numpy.core._multiarray_umath import (
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/init.py", line 50, in
raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  • The Python version is: Python3.9 from "/var/mycodo-root/env/bin/python"
  • The NumPy version is: "1.26.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/var/mycodo-root/mycodo/controllers/controller_function.py", line 79, in loop
self.run_function.loop()
File "/home/moritz/Mycodo/mycodo/functions/equation_multi.py", line 167, in loop
last_measurement_a = self.get_last_measurement(
File "/var/mycodo-root/mycodo/abstract_base_controller.py", line 533, in get_last_measurement
return get_last_measurement(device_id, measurement_id, max_age=max_age)
File "/var/mycodo-root/mycodo/utils/influx.py", line 322, in get_last_measurement
last_measurement = read_influxdb_single(
File "/var/mycodo-root/mycodo/utils/influx.py", line 372, in read_influxdb_single
data = query_string(
File "/var/mycodo-root/mycodo/utils/influx.py", line 301, in query_string
ret_value = query_flux(
File "/var/mycodo-root/mycodo/utils/influx.py", line 185, in query_flux
from influxdb_client import InfluxDBClient
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/init.py", line 382, in
from influxdb_client.client.delete_api import DeleteApi
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/delete_api.py", line 7, in
from influxdb_client.client._base import _BaseDeleteApi
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/_base.py", line 22, in
from influxdb_client.client.write.dataframe_serializer import DataframeSerializer
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/dataframe_serializer.py", line 12, in
from influxdb_client.client.write.point import _ESCAPE_KEY, _ESCAPE_STRING, _ESCAPE_MEASUREMENT, DEFAULT_WRITE_PRECISION
File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/point.py", line 40, in
import numpy as np
File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 135, in
raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python interpreter from there.

it just loops the "The above exception was the direct cause of the following exception" basically for >10.000 lines of code
@kizniche
Copy link
Owner

kizniche commented Nov 8, 2023

Have you attempted to upgrade the library or install the dependency it lists?

@iqgmoritz
Copy link
Author

Yes I've tried to install everything by hand even before installing mycodo on a new system. Still the same error

@iqgmoritz
Copy link
Author

In the web log it's actually a bit better to understand:
Nov 08 22:07:09 raspberrypi python[978]: ImportError:
Nov 08 22:07:09 raspberrypi python[978]: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Nov 08 22:07:09 raspberrypi python[978]: Importing the numpy C-extensions failed. This error can happen for
Nov 08 22:07:09 raspberrypi python[978]: many reasons, often due to issues with your setup or how NumPy was
Nov 08 22:07:09 raspberrypi python[978]: installed.
Nov 08 22:07:09 raspberrypi python[978]: We have compiled some common reasons and troubleshooting tips at:
Nov 08 22:07:09 raspberrypi python[978]: https://numpy.org/devdocs/user/troubleshooting-importerror.html
Nov 08 22:07:09 raspberrypi python[978]: Please note and check the following:
Nov 08 22:07:09 raspberrypi python[978]: * The Python version is: Python3.9 from "/var/mycodo-root/env/bin/python"
Nov 08 22:07:09 raspberrypi python[978]: * The NumPy version is: "1.26.1"
Nov 08 22:07:09 raspberrypi python[978]: and make sure that they are the versions you expect.
Nov 08 22:07:09 raspberrypi python[978]: Please carefully study the documentation linked above for further help.
Nov 08 22:07:09 raspberrypi python[978]: Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory
Nov 08 22:07:09 raspberrypi python[978]: The above exception was the direct cause of the following exception:
Nov 08 22:07:09 raspberrypi python[978]: Traceback (most recent call last):
Nov 08 22:07:09 raspberrypi python[978]: File "/home/moritz/Mycodo/mycodo/devices/camera.py", line 372, in camera_record
Nov 08 22:07:09 raspberrypi python[978]: import cv2
Nov 08 22:07:09 raspberrypi python[978]: File "/var/mycodo-root/env/lib/python3.9/site-packages/cv2/init.py", line 11, in
Nov 08 22:07:09 raspberrypi python[978]: import numpy
Nov 08 22:07:09 raspberrypi python[978]: File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 135, in
Nov 08 22:07:09 raspberrypi python[978]: raise ImportError(msg) from e
Nov 08 22:07:09 raspberrypi python[978]: ImportError: Error importing numpy: you should not try to import numpy from
Nov 08 22:07:09 raspberrypi python[978]: its source directory; please exit the numpy source tree, and relaunch
Nov 08 22:07:09 raspberrypi python[978]: your python interpreter from there.
Nov 08 22:07:09 raspberrypi python[978]: 2023-11-08 22:07:09,473 Exception raised in 'camera_record' when setting user grp: [Errno 2] No such file or directory: '/home/moritz/Mycodo/cameras/6fd9fa62-5a7b-4491-b806-3827ad9e10c5/still/Still-2-Camera_Name-2023-11-08_22-07-06.jpg'
Nov 08 22:07:09 raspberrypi python[978]: Traceback (most recent call last):
Nov 08 22:07:09 raspberrypi python[978]: File "/home/moritz/Mycodo/mycodo/devices/camera.py", line 488, in camera_record
Nov 08 22:07:09 raspberrypi python[978]: set_user_grp(path_file, 'mycodo', 'mycodo')
Nov 08 22:07:09 raspberrypi python[978]: File "/home/moritz/Mycodo/mycodo/utils/system_pi.py", line 683, in set_user_grp
Nov 08 22:07:09 raspberrypi python[978]: os.chown(filepath, uid, gid)
Nov 08 22:07:09 raspberrypi python[978]: FileNotFoundError: [Errno 2] No such file or directory: '/home/moritz/Mycodo/cameras/6fd9fa62-5a7b-4491-b806-3827ad9e10c5/still/Still-2-Camera_Name-2023-11-08_22-07-06.jpg'
Nov 08 22:07:09 raspberrypi python[978]: 2023-11-08 22:07:09,510 No image was created. Check your settings and hardware for any issues.
Nov 08 22:07:09 raspberrypi python[978]: 2023-11-08 22:07:09,511 Could not acquire image.

@kizniche
Copy link
Owner

kizniche commented Nov 9, 2023

How did you install the dependency?

@iqgmoritz
Copy link
Author

On the initial try I did it via the Mycodo web interface, then when it did not work I did a clean install of Raspbian Lite and installed numpy via $ pip install numpy

@kizniche
Copy link
Owner

kizniche commented Nov 9, 2023 •
edited

I'm referring to the dependency numpy told you about in the log lines you pasted.

@iqgmoritz
Copy link
Author

As I said, i installed it via the mycodo web interface prompt when i tried to add the urllib camera.
I tried it again on a freshly installed Raspberry Pi by running the $ pip install numpy first and only after that installing mycodo on the system. Apparently in both cases, mycodo downloads it‘s own version of numpy to /var/mycodo-root/env/lib/python3.9/site-packages/numpy/

I‘ve also tried to somehow get rid of this installation but pip uninstall does not work and just deleting the directory and hoping for python to access the user installed numpy dependency did not either.

@kizniche
Copy link
Owner

kizniche commented Nov 9, 2023 •
edited

How did you install libopenblas (the dependency numpy is stating you're missing in the log you pasted) from the Mycodo web interface? This is not a feature of Mycodo unless you modified the code.

@iqgmoritz
Copy link
Author

I never willingly installed libatlas. Though I am running gphoto2 on the pi to take automated timelapse photos with a DSLR, which I wanted to be able to import to mycodo with the urllib camera, so this might be the issue. I will try it on a system without gohoto2 later today.

@kizniche
Copy link
Owner

Who mentioned libatlas?

@iqgmoritz
Copy link
Author

iqgmoritz commented Nov 10, 2023 •
edited

Oh, sorry excuse the confusion. I misread the log and your message. I installed it via $ sudo apt-get install libopenblas-dev

@iqgmoritz
Copy link
Author

I have now manually compiled openblas and the error message has shrunken significantly to this:

Nov 10 15:19:19 raspberrypi python[590]: Traceback (most recent call last):
Nov 10 15:19:19 raspberrypi python[590]: File "/home/moritz/Mycodo/mycodo/devices/camera.py", line 488, in camera_record
Nov 10 15:19:19 raspberrypi python[590]: set_user_grp(path_file, 'mycodo', 'mycodo')
Nov 10 15:19:19 raspberrypi python[590]: File "/home/moritz/Mycodo/mycodo/utils/system_pi.py", line 683, in set_user_grp
Nov 10 15:19:19 raspberrypi python[590]: os.chown(filepath, uid, gid)
Nov 10 15:19:19 raspberrypi python[590]: FileNotFoundError: [Errno 2] No such file or directory: '/home/moritz/Mycodo/cameras/195f9671-eb1d-4a49-8f76-1106d50557eb/still/Still-1-Camera_Name-2023-11-10_15-19-16.jpg'
Nov 10 15:19:19 raspberrypi python[590]: 2023-11-10 15:19:19,295 No image was created. Check your settings and hardware for any issues.
Nov 10 15:19:19 raspberrypi python[590]: 2023-11-10 15:19:19,295 Could not acquire image.

I cannot make much sense of this, as it seems like there is already an error beforehand when creating the jpg file, which is just not logged

@iqgmoritz
Copy link
Author

I finally got it working, so here are the steps I followed for anyone with the same problem:

  1. I compiled openblas myself with the following commands:
    wget http://github.com/xianyi/OpenBLAS/archive/v0.2.20.tar.gz
    tar zxvf v0.2.20.tar.gz
    cd OpenBLAS
    make
    mkdir ../openblas-0.2.20
    make PREFIX=/home/pi/openblas-0.2.20 install

  2. I installed opencv myself with:
    sudo apt-get install build-essential cmake pkg-config libjpeg-dev libtiff5-dev libjasper-dev libpng-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libfontconfig1-dev libcairo2-dev libgdk-pixbuf2.0-dev libpango1.0-dev libgtk2.0-dev libgtk-3-dev libatlas-base-dev gfortran libhdf5-dev libhdf5-serial-dev libhdf5-103 python3-pyqt5 python3-dev -y
    pip install opencv-python

Now it works. Thanks for your help anyways, without you I wouldn't have noticed, where the core of the problem is coming from!

@kizniche
Copy link
Owner

Although this may work for a manual fix, this isn't a viable solution for a software fix. I will attempt to replicate the issue and find an amenable solution that can be applied.

@kizniche kizniche reopened this Nov 10, 2023
@FlooSz
Copy link

FlooSz commented Nov 10, 2023

My solution was with checking journal:
sudo journalctl -u mycodo.service

Then

  1. Go into virtualenv with: 1. "cd Mycodo" 2. "source env/bin/activate" (without ")

  2. Check if your numpy installed correctly with: env/bin/pip install numpy

  3. Install openblas with: sudo apt-get install libopenblas-base (And/or other Packages you found with first journalctl command)

@kizniche
Copy link
Owner

I couldn't replicate the issue. Perhaps your issue stems from your use of a non-current operating system version. In any case, I can't do any diagnostics if it can't be replicated and I will only use the latest OS version, since it's too much trouble to support old OS versions.

@iqgmoritz
Copy link
Author

That‘s pretty weird. I got the Raspberry Pi OS (Legacy) Lite 32bit from the Rasperry Pi Flasher, so it should have been the newest

@kizniche
Copy link
Owner

By definition, legacy is not the latest

@FlooSz
Copy link

FlooSz commented Nov 13, 2023

For optimal performance with Raspi 4b, it is recommended to utilize a 64-bit operating system to unlock its full processing capabilities.

@iqgmoritz
Copy link
Author

I was only using the 32bit version, because when I installed mycodo, weirdly this version was the only one which I was even capable of installing it on. The other OS‘s always had some error show up along the way

@kizniche
Copy link
Owner

Legacy does not mean 32-bit, it means old version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kizniche @iqgmoritz @FlooSz