Make CrysText plugin ready for ncrystal v4

[tkittel]

@XuShuqi77 @marquezj @dddijulio

These changes are needed to make the plugin ready for NCrystal 4, which will be out in a few weeks.

It can already be used in the meantime however, by pip installing the ncrystal-core and ncrystal-python packages which contains release candidates of the new code (but do NOT install the package named just ncrystal until NCrystal4 is out). For refererence, ncrystal-core contains all binaries, headers, cmake, and ncrystal-config -- and ncrystal-python contains the NCrystal python module and all the other command line tools (including nctool).

To install the plugin, you can now simply point pip directly at the repo:

pip install git+https://github.com/highness-eu/ncplugin-CrysText

And then it will work with no further need to tweak environment variables or whatnot.

Of course, until you accept this PR, the above command won't work. But you can try out the code in this PR with:

pip install git+https://github.com/mctools/ncplugin-template.git@ncplugin-CrysText-support-ncrystal4

This PR also adds a unit tests, that ensures a basic functionality test of the plugin on linux/mac/windows.

We will discuss more on monday about how to actually develop in this context, adding tests, etc.

[tkittel]

I am not 100% sure why the CI does not run on this PR, but perhaps it is because there are still no other workflows in the repo.

[XuShuqi7]

Hi @tkittel,

First I created a new environment and installed NCrystal and the plugin using

conda create -n ncrystal-v397 -y -c conda-forge jupyter numpy scipy matplotlib pip h5py
conda activate ncrystal-v397
pip install git+https://github.com/mctools/ncplugin-template.git@ncplugin-CrysText-support-ncrystal4

Then I use Python to run

import NCrystal as NC
import numpy as np
import matplotlib.pyplot as plt

print(NC.__version__)

T = 293.6 # temperature 
wavelength = np.linspace(0.1, 5, 4901)
E = NC.wl2ekin(wavelength)

Al_text     = NC.createScatter('ncplugin-CrysText_Al_sg225.ncmat;temp={}K;dir1=@crys_hkl:1,1,1@lab:1,0,0;dir2=@crys_hkl:0,0,2@lab:0,1,0;mos=1deg;dirtol=180deg'.format(T))
Al_ver      = NC.createScatter('ncplugin-CrysText_Al_sg225_ver.ncmat;temp={}K;dir1=@crys_hkl:1,1,1@lab:1,0,0;dir2=@crys_hkl:0,0,2@lab:0,1,0;mos=1deg;dirtol=180deg'.format(T))
Al          = NC.createScatter('Al_sg225.ncmat;temp={}K'.format(T))
xs_text     = Al_text.crossSection(E, direction=(1,0,0))
xs_ver      = Al_ver.crossSection(E, direction=(1,0,0))
xs          = Al.crossSection(E, direction=(1,0,0))

# plot
fig, ax = plt.subplots()

ax.plot(wavelength, xs_text, label='R=0.3', linewidth=2)
ax.plot(wavelength, xs_ver,  label='R=1',   linewidth=2,linestyle='--')
ax.plot(wavelength, xs,      label='Powder',linewidth=2)

ax.set(xlim=(0, 5), xticks=np.arange(1, 5),
       ylim=(0, 5), yticks=np.arange(1, 5))

# Add legend
plt.legend()

plt.show()

where R=0.3 in "ncplugin-CrysText_Al_sg225.ncmat", while R=1 in "ncplugin-CrysText_Al_sg225_ver.ncmat", with additional sections

@CUSTOM_CRYSTEXT
  1  1  1  0.3  0.79
  2  0  0  0.3  0.21

and

@CUSTOM_CRYSTEXT
  1  1  1  1.0  0.79
  2  0  0  1.0  0.21

The resulting plot is

It seems that the March-Dollase model is not correctly called. Instead, the single crystal model is used. Maybe the installation of plugin is not correct?

[tkittel]

@XuShuqi77 first thing is to make sure your list of conda packages also includes make cmake compilers

[tkittel]

I am trying to make this work in a conda env, hang on.

[tkittel]

The following recipe works for me. The thing is that NCrystal 4 is not out yet, so if you want to use these source-only packages in a conda environment, you have to be very careful to make sure that ncrystal-core and the new plugin gets compiled in the exactly correct environment:

conda create -n shuqi2 -c conda-forge python pip compilers cmake make numpy scikit-build-core jupyter scipy matplotlib h5py
conda activate shuqi2
pip install ncrystal-core -v --no-cache-dir --no-binary ':all:' --no-deps --no-build-isolation
pip check
pip install ncrystal-pypluginmgr ncrystal-python --no-deps --no-build-isolation
pip check
pip install -v --no-deps --no-build-isolation git+https://github.com/mctools/ncplugin-template.git@ncplugin-CrysText-support-ncrystal4
pip check

#All done, now verify:

nctool --test
nctool --plugins
nctool --browse
nctool -d "plugins::CrysText/Al_sg225.ncmat"
NCRYSTAL_REQUIRED_PLUGINS=CrysText NCRYSTAL_PLUGIN_RUNTESTS=1 nctool --plugins

[tkittel]

I am trying your plotting script now, how long should I expect it to run for?

[XuShuqi7]

I am trying your plotting script now, how long should I expect it to run for?

Thanks for testing! I am following your recipe now. It will take <<1 seconds normally.

[tkittel]

It works:

One thing you need to fix in your script is the filenames:

Al_text     = NC.createScatter('plugins::CrysText/Al_sg225.ncmat;temp={}K;dir1=@crys_hkl:1,1,1@lab:1,0,0;dir2=@crys_hkl:0,0,2@lab:0,1,0;mos=1deg;dirtol=180deg'.format(T))
Al_ver      = NC.createScatter('plugins::CrysText/Al_sg225_ver.ncmat;temp={}K;dir1=@crys_hkl:1,1,1@lab:1,0,0;dir2=@crys_hkl:0,0,2@lab:0,1,0;mos=1deg;dirtol=180deg'.format(T))
Al          = NC.createScatter('stdlib::Al_sg225.ncmat;temp={}K'.format(T))

If it worked before, you must have had the old plugin files copied into your working dir or something. To avoid this confusion, always use e.g. stdlib:: in front of stdlib filenames and the new plugins:: in front of files from plugins (for the latter, it actually will always be needed).

[tkittel]

But it did take ~2 minutes to run!

[XuShuqi7]

Great! Thank you @tkittel ! Oh, yes, I thought the wavelength began with 1 AA. If with 0.1 AA, 2 minutes are reasonable 😊

[tkittel]

So one thing which we should consider for the test function in NCTestPlugins.cc would be to verify that this is happening, e.g. that for your Al_sg225.ncmat, you get cross sections close to 0.6barn (or whatever) for 3.5Aa neutrons.

I.e. the main point of that test function is to quickly verify that the plugin is activated and doing its thing, not something more heavy than that - and not something that will break if you tweak the parameters slightly.

However, once you also have it working we should probably just go ahead and merge this - or do you think anyone else is using this in the next few weeks?

[XuShuqi7]

I follow the installation process and make it work. Your advice regarding the test function works for me.

[XuShuqi7]

However, there is un expected warning

[/Users/shuqixu/opt/miniconda3/envs/ncrystal-v397/lib/python3.13/site-packages/numpy/lib/_function_base_impl.py:2605](http://localhost:8889/Users/shuqixu/opt/miniconda3/envs/ncrystal-v397/lib/python3.13/site-packages/numpy/lib/_function_base_impl.py#line=2604): RuntimeWarning: invalid value encountered in e2xs (vectorized)
  outputs = ufunc(*inputs)

which should result in the cut in the lower wavelength:

[XuShuqi7]

I think the problem should be due to some numerical overflow, which will not impact the use of the plugin which usually focuses on longer wavelengths. I will have a look at the source code. Or do you have any advice?

[tkittel]

I don't have time to debug the plugin internals in the next few weeks for sure, but of course I am not a fan of a statement like:

I think the problem should be due to some numerical overflow, which will not impact the use of the plugin which usually focuses on longer wavelengths.

For a high quality plugin, it should also deal with edge cases. However, there are probably many other problems to fix (e.g. make the plugin technically "isotropic" again, since you are actually not using the neutron direction).

I think for this PR it is unrelated, feel free to open an issue and investigate further there.

[tkittel]

Btw., I just released ncrystal-pypluginmgr 0.0.4, so with that you can simply install your plugin and then afterwards run:

ncrystal-pluginmanager --test CrysText

To verify that the plugin exists, and run it's test function.

[XuShuqi7]

Btw., I just released ncrystal-pypluginmgr 0.0.4, so with that you can simply install your plugin and then afterwards run:

ncrystal-pluginmanager --test CrysText

To verify that the plugin exists, and run it's test function.

This comment has been tested from my side. I agree on that the plugin should not limited to certain wavelengths and should able to address all the edges. I will open an issue and investigate this problem further.

[XuShuqi7]

Btw., I just released ncrystal-pypluginmgr 0.0.4, so with that you can simply install your plugin and then afterwards run:

ncrystal-pluginmanager --test CrysText

To verify that the plugin exists, and run it's test function.

This comment has been tested from my side. I agree on that the plugin should not limited to certain wavelengths and should able to address all the edges. I will open an issue and investigate this problem further.