experiment with numba/cffi/cython?

[AndrewAnnex]

can numba/cffi be used to accelerate vectorized functions in spiceypy?
https://numba.pydata.org/numba-doc/0.15.1/interface_c.html
experiment!

[AndrewAnnex]

requires c_char support in numba numba/numba#3207

[AndrewAnnex]

I wonder if something clever using numpy dtype str_ and numba can be done, could I just pass c_void_p to cspice ctypes calls and rely on the ctypeslib in numpy to help?

[AndrewAnnex]

so after some experimentation cython is appearing as a viable option as I have locally prototyped a cython extension linked to the same shared library as spiceypy and they are able to share the same kernel pool. with cython I can also just make the subset of vectorized functions I need and slowly convert the codebase, in particular everything in support types to cython.

However, it seems like numba should still be viable, so I want to go back and figure out where I got stuck and get past that point. But I am slightly blocked by numba not supporting python 3.9 yet (or is being actively worked on now numba/numba#6345)

[gaowutong]

Or is there away to use spiceypy functions inside the numba-accelerated function?

[AndrewAnnex]

I found out a few months ago that there are a few features of ctypes that are not supported by numba and I although I could get numba to use spiceypy it was slower than using pure python. I could still be using numba incorrectly, but for now I will rule out numba unless I can be shown a working proof of concept.

As for cython, I now have a working proof on concept for accelerating spiceypy with it. With a quick benchmark I see about a 2x speedup and as far as I can tell at the moment SPICE remains the bottleneck rather than the python/cython code itself. There are some issues to resolve with linking and reliable compilation across conda-forge/pypi that I want to fix before I post a wip branch

[gaowutong]

Thanks for the reply, looking forward to your work!

[medley56]

@AndrewAnnex In your prototype, are you building the .so from CSPICE source using Cython and setuptools.Extension or are you using the existing setup script to build spice.so and somehow telling Cython that the shared object already exists (after installation)? I don't know of a way to do the latter, so I'm thinking that this change would also mean changes to setup.py and get_spice.py.

[AndrewAnnex]

Hey Gavin, I was doing the former as cython has good setup tools integration. But yeah it is a big change so ensuring the source directory bootstrap continues to work is difficult without just embedding the c source into the project directly, which I want to avoid for many reasons. Iirc there is also good tooling surrounding cmake to make it work with cython. A cmake file could completely replace the get_spice script and simplify things but I haven’t had the time. Most cmake files I’ve seen don’t attempt to bootstrap a source directory from a url but it looks possible given their docs.

…

-Andrew Annex

On Mar 2, 2022, at 3:33 PM, Gavin Medley ***@***.***> wrote:  @AndrewAnnex In your prototype, are you building the .so from CSPICE source using Cython and setuptools.Extension or are you using the existing setup script to build spice.so and somehow telling Cython that the shared object already exists (after installation)? I don't know of a way to do the latter, so I'm thinking that this change would also mean changes to setup.py and get_spice.py. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[jessemapel]

You want the FetchContent function in cmake. It allows you to download stuff like git repos and release tarballs and then hook into any cmake files located in them. CSPICE unfortunately doesn't come with a cmake configuration, but it's fairly easy to make one and then have it export a target that can be included in other projects.

If you're not going to be doing CMAKE build configuration for CSPICE and just use the makefiles, then you can use ExternalProject_Add which will download at build time instead of configuration time.