Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zodb install on termux linux for Android doesn't work due to no os.link #257

Open
BandungBB opened this issue Feb 25, 2019 · 12 comments · May be fixed by #390
Open

zodb install on termux linux for Android doesn't work due to no os.link #257

BandungBB opened this issue Feb 25, 2019 · 12 comments · May be fixed by #390

Comments

@BandungBB
Copy link

BandungBB commented Feb 25, 2019

Here is the traceback error resulting from an attempt to import ZODB into any python script running on termux for Android.

import ZODB
Traceback (most recent call last)
File "data/data/com.termux/files/usr/lib/python3.7/site-packages/ZODB/init.py" line 28, in from ZODB.DB import DB, connection
File "data/data/com.termux/files/usr/lib/python3.7/site-packages/ZODB/init.py" line 27, in from ZODB.Connection import Connection, TransactionMetaData, noop
File "data/data/com.termux/files/usr/lib/python3.7/site-packages/ZODB/Connection.py" line 31, in from ZODB.blob import Blob, rename_or_copy_blob, remove_commited_dir
File "data/data/com.termux/files/usr/lib/python3.7/site-packages/ZODB/blob.py" line 950, in link_or_copy = os.link
AttributeError: module os has no attribute 'link'

@BandungBB
Copy link
Author

BandungBB commented Feb 25, 2019

While a pip install zodb within the termux distro for Android completes without incident, it fails to run since any attempt to import ZODB within termux's python fails. This is because their python's os.link has been disabled for some cross platform reason related to windows. I have found two issues related to this problem that other apps experience. pip was the first one and it seems that pip was modified to test to see if os.link is available for the platform and substituting another call if it is not

termux/termux-packages#29
pimutils/vdirsyncer#744

Is there a similar fix that can be made available for zodb? I want to be able to use zodb not only within my laptop's scripts but also for these same scripts to run within the termux distro on Android.

@tseaver
Copy link
Member

tseaver commented Feb 26, 2019

@zennobjects The only ZODB library code which uses os.link is in ZODB.blob, where it is selected as the value for the link_or_copy name on Posix systems: on Windows, link_or_copy is mapped to shutil.copy. If you can figure out how to detect that you are on termux, then you can patch ZODB.blob use shutil.copy as well in that case.

The migrateblobs script does a similar dance.

@mgedmin
Copy link
Member

mgedmin commented Feb 27, 2019

hasattr(os, 'link') might be a reasonable check.

@BandungBB
Copy link
Author

BandungBB commented Feb 28, 2019

In order to identify that I'm on termux, I use the following code in my scripts. It's based upon identifying certain features from within python - my platform (aka operating system) and - my device (hardware).

Plus one more additional piece of information. Termux modifies the python that their "pkg install python" provides for a number of specific reasons, one being that their file system layout is different (because, because). Since termux is running within Google's android services, the app relies upon a specifc NDK in order to be built. Ergo there is an attribute patched into their python that tells us, we're running on top of Android system #21 as an example. Here is my code:

import` platform, sys, os
# 1. platform_type
try:
    NDK = sys.getandroidapilevel()
    platform_type = 'android' 
except AttributeError as e:
    platform_type = platform.system().lower()

# 2. device_type
device_type = 'phone' if platform.machine() == 'armv7l' else 'laptop'

# 3. device_platform
device_platform = '_'.join((device_type, platform_type))

# ---- printout----
print(f" platform_type = {platform_type}, (which is {platform.system().lower()} )")
print(f" device_type = {device_type} (which is {platform.machine()})")
print(f" device_platform = {device_platform}")

So for my purposes, putting zodb's os.link calls between

try: 
    link_or_copy = os.link
except AttributeError as e:
    if platform_type == 'android':
        sink_or_copy = shutil.copy

should suffice to run the necessary substitution of the shutil.copy command for os.link.

@BandungBB
Copy link
Author

BandungBB commented Feb 28, 2019

Looking at the code within blob.py, the change might be something like this:

try:
    hasattr(os, link)
    os_link = True
except AttributeError as e:
    os_link is False

try:
    NDK = sys.getandroidapilevel()
    platform_type = 'android' 
except AttributeError as e:
    platform_type = platform.system().lower()

if sys.platform == 'win32':
    # On Windows, you can't remove read-only files, so make the
    # file writable first.
    ...
    link_or_copy = shutil.copy

elif platform_type == 'android' and os_link is False:
    # On Android platforms, the linux distros like termux may have 
    # removed os.link
    ...
    link_or_copy = shutil.copy

else: 
    ...
    link_or_copy = os.link

@BandungBB
Copy link
Author

BandungBB commented Mar 4, 2019

In order to test the changes, I simply used vim to change blob.py residing in the user python's site-packages/ZODB, and insert the line

link_or_copy = os.link

between a

try:
    link_or_copy = os.link
except AttributeError:
    link_or_copy = shutil.copy

rather than go through the whole recompiling and wheel creation of zodb for my termux version of linux running on an Android platform. It works like a charm. So I can confirm that there are no other issues preventing zodb from working as a result of this change.

There is no need to identify that zodb is running within termux, that it's on an Android platform, whatever. It's a simple change to the blob.py script that can be easily added to zodb.

Shall I submit a patch?

@tseaver
Copy link
Member

tseaver commented Mar 4, 2019

@zennobjects Please do submit a PR. You can add a #pragma: no cover to the except AttributeError: line.

@HDembinski
Copy link

@tseaver @zennobjects Was the issue solved? I don't see a linked PR.

@BandungBB
Copy link
Author

BandungBB commented Aug 10, 2023

Wow, a long time ago. Seems like I didn’t create the pull requests. I can’t even find my clone of this app where I changed the line within the blob.py file.

What I can say is this. The solution that I wrote, the one where the value for link_or_copy gets changed in the try:except clauses, works. The quickest solution is to use an editor and change that line in the blob.py file under ZODB within the termux ZODB.

@BandungBB
Copy link
Author

I just submitted a pull request for my proposed changes.

@tseaver tseaver linked a pull request Aug 12, 2023 that will close this issue
@BandungBB
Copy link
Author

There are two test failures, 1 failure for Ubuntu.lint and another for each python version on Windows.

[1] I think the failure for Ubuntu.lint is due to a lint check wrt the use of '#'. I've made minor changes to those comments.

[2] I am somewhat at a loss to see why the following error occurs for each python version on windows.

ImportError: cannot import name 'link_or_copy' from 'ZODB.blob' (D:\a\ZODB\ZODB\src\ZODB\blob.py)

It could be that the following check is no longer valid, possibly due to the check for sys.platform == win32 failing

if sys.platform == 'win32':
...
link_or_copy = shutil.copy

in which case link_or_copy is never set. If this is the case, then the fix that I proposed is not what breaks the code for Windows Nevertheless, the following might fix this

It could be that we should remove the setting of link_or_copy completely from within the 'if' block.

if sys.platform == 'win32'
....

and instead have a stand alone setting of link_or_copy which is at the same level as the various functions within blob.py
Ie:

def rename_or_copy_blob(f1, f2, chmod=True):
"""Try to rename f1 to f2, fallback to copy.
...
...

try:
link_or_copy = os.link
except AttributeError: # pragma: no cover
# FBO termux on Android.
# See #257
link_or_copy = shutil.copy

I'm resubmitting the pull request with these two changes.

@BandungBB
Copy link
Author

Man, it doesn't pay to be fixing code just after surgery. The littlest changes result in errors due to the discomfort.
Hopefully, the link errors are fixed. Plus there should not be any more failures on Windows given the latest indentation change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants