Skip to content

Expose grid.evolve to the C-API #344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Jun 17, 2025
Merged

Expose grid.evolve to the C-API #344

merged 55 commits into from
Jun 17, 2025

Conversation

Radonirinaunimi
Copy link
Member

@Radonirinaunimi Radonirinaunimi commented Apr 18, 2025

Using grid.evolve to evolve grid with APFEL++ evolution

In addition to performing the evolution on the fly and outputting theory predictions, APFEL++ also wants the possibility to dump the evolved grids on disk for future usage (aka FK tables). This effectively implies grid.evolve to be exposed at the C interface.

@vbertone, the idea is then for you to provide an operator which is a rank-4 tensor. To be more illustrative, let's consider a simple DIS case. The convolution, for a fixed $Q^2$, writes as:

$$\mathrm{FK}_a\left(x_i ; \mu_0^2\right)=\sum_{k, b, \ell} \alpha_{\mathrm{s}}^{n+k}\left(\mu^2\right) \mathrm{EKO}_{a, i}^{b, \ell} \sigma_b^{(k)}\left(x_\ell, \mu^2\right)$$

where:

  • $a,b$ represent respectively the flavor index of the FK table and grid
  • $i,\ell$ represent respectively the $x$-grid index of the FK table and grid

These form the dimensions of an EKO, ie EKO[a][i][b][l]. Notice that the $x$-grid coordinates of the grid and FK table do not have to be same, and they do not have to be the same as what is actually inside the grid $-$ same is true for PID bases.

The C++ function that consumes the operator from APFEL++ would therefore tentatively look as follows:! Deprecated, see evolve-grid.cpp.

struct OperatorInfo {
   std::vector<std::size_t> pids_in;
   std::vector<std::size_t> x_in;
   std::vector<std::size_t> pids_out;
   std::vector<std::size_t> x_out;
   std::double q_grid;
   std::double q0_fk;
   pid_basis fk_pid_basis;
}

void evolve_grid(
   pineappl_grid* grid,
   OperatorInfo op_info,
   std::vector<double> operators, // Flatten,
   std::vector<double> xi = {1.0, 1.0, 1.0},
) {
   ...
   grid.evolve(slices, order_mask, xi, alphas_table);
}

and this is the function that'll dump the evolved grid. I think this is the best way to do it as opposed to what we discussed during the meeting.

So the summarize, what APFEL++ needs to provide (for a given value of $Q^2$), is a rank-4 tensor representing the EKO. In addition, we need the $x$ and PID values (can be in any basis) from which the EKO was computed $-$ this way we don't rely on much information from the grids to construct the EKOs.

Does this make sense?

Extra-changes included in this PR (not fully related):

  • Introduced re-usable action to download/cache data in all of the workflows
  • Exposed a function to get the values of the subgrid nodes (needed if one wants to access the weights of the subgrids)
  • Removed references on objects which are immediately de-referenced by the compiler
  • Improved the get-subgrids.cpp examples

TODO:

  • Find the best way to download the LHCB_WP_7TEV_opt.pineappl.lz4 grid for the test which currently is available at:
wget --no-verbose --no-clobber https://data.nnpdf.science/pineappl/test-data/LHCB_WP_7TEV_opt.pineappl.lz4
  • Infer the order masking in the example from the Grid
  • Find the best way to download the EKO_LHCB_WP_7TEV.txt which contains the flattened evolution operators

@vbertone
Copy link

Hi @Radonirinaunimi, yes, it does make sense.
So, on my side, it all boils down to providing the rank-4 tensor operator flattened on a std::vector<double>, which I can easily do.
The only question I have is how you need it to be flattened. I assume following the order in EKO[a][i][b][l], but better to make sure.
Thank you!

@Radonirinaunimi
Copy link
Member Author

Hi @vbertone, apologies for the late reply, I wanted to sort out the interface first before providing you with how the order of the flattening should look like.

However, I had not anticipated how difficult the interface would be, and therefore this is not done yet. It might still takes some time.

@vbertone
Copy link

Hi @Radonirinaunimi, no problem at all. Take your time.
I'll kick in once you are done.

@Radonirinaunimi
Copy link
Member Author

Hi @Radonirinaunimi, no problem at all. Take your time. I'll kick in once you are done.

Ok, this is not completely done (needs some clean up, fix some memory leak in the example, etc.) but I think we can start to test and experiment it.

So there is a primary example in evolve-grid.cpp. The operators should be flattened following the EKO[a][i][b][l] order as you mentioned above; ie.:

std::size_t index = 0;
for a { for i { for b {for l { eko[index++] = ... } } } }

The b and l which refers to pids1 and x1 of the grid from which the EKO should be computed from need to be extracted from the grid, for example as follows:

// Get the values of the evolve info parameters. These contain, for example, the
// information on the `x`-grid and `PID` used to interpolate the Grid.
// NOTE: These are used to construct the Evolution Operator
std::vector<double> fac1(evinfo_shape[0]);
std::vector<double> frg1(evinfo_shape[1]);
std::vector<int> pids1(evinfo_shape[2]);
std::vector<double> x1(evinfo_shape[3]);
std::vector<double> ren1(evinfo_shape[4]);
pineappl_grid_evolve_info(grid, order_mask.data(), fac1.data(),
frg1.data(), pids1.data(), x1.data(), ren1.data());

Please let me know if something is wrong/doesn't make sense.

@Radonirinaunimi
Copy link
Member Author

This is now doing what we intended to do.

@vbertone please test it and let me know. You can download the Grid for the test via:

wget --no-verbose --no-clobber https://data.nnpdf.science/pineappl/test-data/LHCB_WP_7TEV_opt.pineappl.lz4

@cschwan This is not super-clean yet and would appreciate if you can have a quick look (your feedback is always quite helpful).

@vbertone
Copy link

Hi @Radonirinaunimi, thanks a lot, I will test it as soon as possible!

@vbertone
Copy link

Hi @Radonirinaunimi, I just ran the code and it does work fine. Also, starting from here I should be able to follow the steps to eventually interface it to the evolution operator computed with APFEL++. I just have one question. Here is the output I get:

   Bin           Grid        FkTable        reldiff
  ----  -------------  -------------  -------------
     0   7.545911e+02   7.603251e+02   7.598823e-03
     1   6.902834e+02   6.949192e+02   6.715820e-03
     2   6.002520e+02   6.037697e+02   5.860343e-03
     3   4.855224e+02   4.878688e+02   4.832807e-03
     4   3.619546e+02   3.634639e+02   4.169948e-03
     5   2.458669e+02   2.467891e+02   3.750676e-03
     6   1.158685e+02   1.163001e+02   3.724532e-03
     7   2.751727e+01   2.763520e+01   4.285840e-03

So differences between grid and FK-table predictions are on the verge of 1%. Since the grid that is being used is (I guess) a realistic one and that the evolution is essentially the unity, this seems a purely numerical effect related, I suppose, to interpolation. The question is: is this accuracy satisfactory? And can in principle be improved? Thank you.

@Radonirinaunimi
Copy link
Member Author

Radonirinaunimi commented Apr 29, 2025

Hey @vbertone and @felixhekhorn, there was a bug in the masking of the orders of the example (fixed in b2bf81b). Now the agreement is:

   Bin           Grid        FkTable        reldiff
  ----  -------------  -------------  -------------
     0   7.545911e+02   7.545911e+02  -3.762829e-08
     1   6.902834e+02   6.902834e+02  -3.537025e-08
     2   6.002520e+02   6.002520e+02  -3.293028e-08
     3   4.855224e+02   4.855223e+02  -3.037598e-08
     4   3.619546e+02   3.619545e+02  -2.782860e-08
     5   2.458669e+02   2.458669e+02  -2.526536e-08
     6   1.158685e+02   1.158685e+02  -2.182740e-08
     7   2.751727e+01   2.751727e+01  -1.732084e-08

@vbertone
Copy link

Thank you, @Radonirinaunimi, this looks much better!

@vbertone
Copy link

Hi @Radonirinaunimi, I'm having some difficulties in understanding the behaviour of the code. Specifically, I would like to change the output PIDs because evolution can possibly generate channels that are not originally present. To this purpose, I made the following basic changes to the code evolve-grids.cpp:

diff --git a/examples/cpp/evolve-grid.cpp b/examples/cpp/evolve-grid.cpp
index 678f42e2..f951f4c4 100644
--- a/examples/cpp/evolve-grid.cpp
+++ b/examples/cpp/evolve-grid.cpp
@@ -118,11 +118,12 @@ int main() {
 
     // ------------------ Construct the Evolution Operator ------------------
     // The Evolution Operator is a vector with length `N_conv * N_Q2_slices * Σ product(OP shape)`
+    std::vector<int> pids2{-6, -5, -4, -3, -2, -1, 21, 1, 2, 3, 4, 5, 6};
     std::vector<double> op_slices;
-    std::size_t flat_len = x1.size() * x1.size() * pids1.size() * pids1.size();
+    std::size_t flat_len = x1.size() * x1.size() * pids1.size() * pids2.size();
     for (std::size_t _i = 0; _i != conv_types.size(); _i++) {
         for (std::size_t j = 0; j != fac1.size(); j++) {
-            std::vector<double> eko = generate_fake_ekos(pids1, x1, pids1, x1);
+            std::vector<double> eko = generate_fake_ekos(pids1, x1, pids2, x1);
             for (std::size_t k = 0; k != flat_len; k++) {
                 op_slices.push_back(eko[k]);
             }
@@ -137,11 +138,11 @@ int main() {
     }
 
     std::vector<double> xi = {1.0, 1.0, 1.0};
-    std::vector<std::size_t> tensor_shape = {pids1.size(), x1.size(), pids1.size(), x1.size()};
+    std::vector<std::size_t> tensor_shape = {pids1.size(), x1.size(), pids2.size(), x1.size()};
 
     pineappl_fk_table* fktable = pineappl_grid_evolve(grid, opinfo_slices.data(),
         max_orders.data(), op_slices.data(), x1.data(),
-        x1.data(), pids1.data(), pids1.data(),
+        x1.data(), pids1.data(), pids2.data(),
         tensor_shape.data(), xi.data(), ren1.data(), alphas_table.data());
 
     // ------------------ Compare Grid & FK after convolution ------------------

The purpose is to allow for all possible channels. However, I must be doing something wrong because when I run the code I get this error that I cannot interpret:

thread '<unnamed>' panicked at pineappl_capi/src/lib.rs:2223:23:
Evolving grid failed: General("operator information (13, 39, 6, 39) does not match the operator's dimensions: (6, 39, 13, 39)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Abort trap: 6

I also tried to play around with the order of variables here and there, but still get the same error (except that sometimes the two tensor shapes in the error message are swapped). Any idea of what I'm doing wrong? Thank you.

@Radonirinaunimi
Copy link
Member Author

Hi @vbertone, the error is because a mismatched shape between the EKO tensor_shape passed and the actual shape of the operator that are computed from the Rust side. I can seed that, as it is right, this can be really confusing because it is not clear which {pids, x} goes in (from the grid) and which one goes out (FK table). Let me clean up a bit the example and implements the example that you were trying to do as that's more interesting.

@cschwan
Copy link
Contributor

cschwan commented Jun 4, 2025

Yes, the header of this function is almost (* const should be * mut) correct, although it won't (?) do anything, because you modify a cloned Grid.

@Radonirinaunimi
Copy link
Member Author

Yes, the header of this function is almost (* const should be * mut) correct, although it won't (?) do anything, because you modify a cloned Grid.

Ah, you're right (I wrote it on the fly and did). So I guess the solution is to return the fktable.

@cschwan
Copy link
Contributor

cschwan commented Jun 8, 2025

I'm not sure how long it'll take to implement the changes, but if you can live without these changes in v1.0, @Radonirinaunimi, you can release v1.0 without this PR, which can just as easily add in v1.1. The important bit for v1.0 was #343, which has been merged.

@Radonirinaunimi
Copy link
Member Author

Hi @cschwan, I think these are sensible modifications so I will try to finish them in the next days before finally making the v1.0.0 release.

@Radonirinaunimi
Copy link
Member Author

So @vbertone, we are finally going to merge this. The function signatures have changed a bit but they are now final. Please let me know in case you need my input in updating the PineAPFEL interface.

@Radonirinaunimi Radonirinaunimi merged commit 039679f into master Jun 17, 2025
10 checks passed
@Radonirinaunimi Radonirinaunimi deleted the expose-evolve-capi branch June 17, 2025 20:28
@vbertone
Copy link

Hi @Radonirinaunimi, thanks for letting me know.
I'll work on it asap, unfortunately no earlier than next week.
I'll keep you posted.

@vbertone
Copy link

Hi @Radonirinaunimi, I've just downloaded and installed the latest master of pineappl and I was trying to adapt the PineAPFEL interface.
As expected, since the interface changed, when I try to compile the PineAPFEL code I get many errors which, unfortunately, are not very explicative.
Here's what I get:

vb262523@dphnmct139:~/Codes/PineAPFEL/code$ make -j
c++ -std=c++11 -O3 -Wall -Wextra evolve-grid-apfel.cpp -I/usr/local/include -L/usr/local/lib -lLHAPDF -I/usr/local/include/pineappl_capi -L/usr/local/lib -lpineappl_capi -I/usr/local/include -O3 -I/usr/local/include -I/usr/local/include -L/usr/local/lib -lapfelxx -L/usr/local/lib -lLHAPDF -L/usr/local/lib -lyaml-cpp -o evolve-grid-apfel
c++ -std=c++11 -O3 -Wall -Wextra evolve-grid-double-apfel.cpp -I/usr/local/include -L/usr/local/lib -lLHAPDF -I/usr/local/include/pineappl_capi -L/usr/local/lib -lpineappl_capi -I/usr/local/include -O3 -I/usr/local/include -I/usr/local/include -L/usr/local/lib -lapfelxx -L/usr/local/lib -lLHAPDF -L/usr/local/lib -lyaml-cpp -o evolve-grid-double-apfel
evolve-grid-apfel.cpp:171:35: error: use of undeclared identifier 'pineappl_grid_conv_type'; did you mean 'pineappl_grid_conv_types'?
  171 |         pineappl_conv_type conv = pineappl_grid_conv_type(grid, i);
      |                                   ^~~~~~~~~~~~~~~~~~~~~~~
      |                                   pineappl_grid_conv_types
/usr/local/include/pineappl_capi/pineappl_capi.h:1400:6: note: 'pineappl_grid_conv_types' declared here
 1400 | void pineappl_grid_conv_types(const pineappl_grid *grid, pineappl_conv_type *conv_types);
      |      ^
evolve-grid-apfel.cpp:171:65: error: cannot initialize a parameter of type 'pineappl_conv_type *' with an lvalue of type 'std::size_t' (aka 'unsigned long')
  171 |         pineappl_conv_type conv = pineappl_grid_conv_type(grid, i);
      |                                                                 ^
/usr/local/include/pineappl_capi/pineappl_capi.h:1400:78: note: passing argument to parameter 'conv_types' here
 1400 | void pineappl_grid_conv_types(const pineappl_grid *grid, pineappl_conv_type *conv_types);
      |                                                                              ^
evolve-grid-apfel.cpp:180:5: error: no matching function for call to 'pineappl_grid_evolve_info_shape'
  180 |     pineappl_grid_evolve_info_shape(grid, max_orders.data(), evinfo_shape.data());
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/pineappl_capi/pineappl_capi.h:1473:6: note: candidate function not viable: no known conversion from 'const value_type *' (aka 'const unsigned char *') to 'const bool *' for 2nd argument
 1473 | void pineappl_grid_evolve_info_shape(const pineappl_grid *grid,
      |      ^
 1474 |                                      const bool *order_mask,
      |                                      ~~~~~~~~~~~~~~~~~~~~~~
evolve-grid-apfel.cpp:190:5: error: no matching function for call to 'pineappl_grid_evolve_info'
  190 |     pineappl_grid_evolve_info(grid, max_orders.data(), fac1.data(), frg1.data(), pids_in.data(), x_in.data(), ren1.data());
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/pineappl_capi/pineappl_capi.h:1485:6: note: candidate function not viable: no known conversion from 'const value_type *' (aka 'const unsigned char *') to 'const bool *' for 2nd argument
 1485 | void pineappl_grid_evolve_info(const pineappl_grid *grid,
      |      ^
 1486 |                                const bool *order_mask,
      |                                ~~~~~~~~~~~~~~~~~~~~~~
evolve-grid-apfel.cpp:277:5: error: unknown type name 'pineappl_fktable'
  277 |     pineappl_fktable* fktable = pineappl_grid_evolve(
      |     ^
evolve-grid-apfel.cpp:277:33: error: no matching function for call to 'pineappl_grid_evolve'
  277 |     pineappl_fktable* fktable = pineappl_grid_evolve(
      |                                 ^~~~~~~~~~~~~~~~~~~~
/usr/local/include/pineappl_capi/pineappl_capi.h:1524:16: note: candidate function not viable: requires 14 arguments, but 13 were provided
 1524 | pineappl_grid *pineappl_grid_evolve(const pineappl_grid *grid,
      |                ^                    ~~~~~~~~~~~~~~~~~~~~~~~~~~
 1525 |                                     size_t nb_slices,
      |                                     ~~~~~~~~~~~~~~~~~
 1526 |                                     pineappl_operator_callback slices,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1527 |                                     const pineappl_operator_info *operator_info,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1528 |                                     const int32_t *pids_in,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~
 1529 |                                     const double *x_in,
      |                                     ~~~~~~~~~~~~~~~~~~~
 1530 |                                     const int32_t *pids_out,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~
 1531 |                                     const double *x_out,
      |                                     ~~~~~~~~~~~~~~~~~~~~
 1532 |                                     const size_t *eko_shape,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~
 1533 |                                     void *state,
      |                                     ~~~~~~~~~~~~
 1534 |                                     const bool *order_mask,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~
 1535 |                                     const double *xi,
      |                                     ~~~~~~~~~~~~~~~~~
 1536 |                                     const double *ren1,
      |                                     ~~~~~~~~~~~~~~~~~~~
 1537 |                                     const double *alphas);
      |                                     ~~~~~~~~~~~~~~~~~~~~
evolve-grid-double-apfel.cpp:151:35: error: use of undeclared identifier 'pineappl_grid_conv_type'; did you mean 'pineappl_grid_conv_types'?
  151 |         pineappl_conv_type conv = pineappl_grid_conv_type(grid, i);
      |                                   ^~~~~~~~~~~~~~~~~~~~~~~
      |                                   pineappl_grid_conv_types
/usr/local/include/pineappl_capi/pineappl_capi.h:1400:6: note: 'pineappl_grid_conv_types' declared here
 1400 | void pineappl_grid_conv_types(const pineappl_grid *grid, pineappl_conv_type *conv_types);
      |      ^
evolve-grid-double-apfel.cpp:151:65: error: cannot initialize a parameter of type 'pineappl_conv_type *' with an lvalue of type 'std::size_t' (aka 'unsigned long')
  151 |         pineappl_conv_type conv = pineappl_grid_conv_type(grid, i);
      |                                                                 ^
/usr/local/include/pineappl_capi/pineappl_capi.h:1400:78: note: passing argument to parameter 'conv_types' here
 1400 | void pineappl_grid_conv_types(const pineappl_grid *grid, pineappl_conv_type *conv_types);
      |                                                                              ^
evolve-grid-double-apfel.cpp:160:5: error: no matching function for call to 'pineappl_grid_evolve_info_shape'
  160 |     pineappl_grid_evolve_info_shape(grid, max_orders.data(), evinfo_shape.data());
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/pineappl_capi/pineappl_capi.h:1473:6: note: candidate function not viable: no known conversion from 'const value_type *' (aka 'const unsigned char *') to 'const bool *' for 2nd argument
 1473 | void pineappl_grid_evolve_info_shape(const pineappl_grid *grid,
      |      ^
 1474 |                                      const bool *order_mask,
      |                                      ~~~~~~~~~~~~~~~~~~~~~~
evolve-grid-double-apfel.cpp:170:5: error: no matching function for call to 'pineappl_grid_evolve_info'
  170 |     pineappl_grid_evolve_info(grid, max_orders.data(), fac1.data(), frg1.data(), pids_in.data(), x_in.data(), ren1.data());
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/pineappl_capi/pineappl_capi.h:1485:6: note: candidate function not viable: no known conversion from 'const value_type *' (aka 'const unsigned char *') to 'const bool *' for 2nd argument
 1485 | void pineappl_grid_evolve_info(const pineappl_grid *grid,
      |      ^
 1486 |                                const bool *order_mask,
      |                                ~~~~~~~~~~~~~~~~~~~~~~
evolve-grid-double-apfel.cpp:259:5: error: unknown type name 'pineappl_fktable'
  259 |     pineappl_fktable* fktable = pineappl_grid_evolve(
      |     ^
evolve-grid-double-apfel.cpp:259:33: error: no matching function for call to 'pineappl_grid_evolve'
  259 |     pineappl_fktable* fktable = pineappl_grid_evolve(
      |                                 ^~~~~~~~~~~~~~~~~~~~
/usr/local/include/pineappl_capi/pineappl_capi.h:1524:16: note: candidate function not viable: requires 14 arguments, but 13 were provided
 1524 | pineappl_grid *pineappl_grid_evolve(const pineappl_grid *grid,
      |                ^                    ~~~~~~~~~~~~~~~~~~~~~~~~~~
 1525 |                                     size_t nb_slices,
      |                                     ~~~~~~~~~~~~~~~~~
 1526 |                                     pineappl_operator_callback slices,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1527 |                                     const pineappl_operator_info *operator_info,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1528 |                                     const int32_t *pids_in,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~
 1529 |                                     const double *x_in,
      |                                     ~~~~~~~~~~~~~~~~~~~
 1530 |                                     const int32_t *pids_out,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~
 1531 |                                     const double *x_out,
      |                                     ~~~~~~~~~~~~~~~~~~~~
 1532 |                                     const size_t *eko_shape,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~
 1533 |                                     void *state,
      |                                     ~~~~~~~~~~~~
 1534 |                                     const bool *order_mask,
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~
 1535 |                                     const double *xi,
      |                                     ~~~~~~~~~~~~~~~~~
 1536 |                                     const double *ren1,
      |                                     ~~~~~~~~~~~~~~~~~~~
 1537 |                                     const double *alphas);
      |                                     ~~~~~~~~~~~~~~~~~~~~
6 errors generated.
make: *** [evolve-grid-apfel] Error 1
make: *** Waiting for unfinished jobs....
6 errors generated.
make: *** [evolve-grid-double-apfel] Error 1

Any input from your side is welcome. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants