`DifferentialEquations.jl` and `Optimization.jl` support

[yebai]

This is required by models like: https://turinglang.org/docs/tutorials/10-bayesian-differential-equations/, or maximum_likelihood based learning.

Probably need to

• add Mooncake support to: https://github.com/SciML/SciMLSensitivity.jl
  • Testing use of rrule SciML/DiffEqBase.jl#1103
  • Add Mooncake to Alternative AD Frontends SciML/SciMLSensitivity.jl#1151
  • Mooncake Inside Problems SciML/SciMLSensitivity.jl#1152
  • Setup Mooncake rrule SciML/DiffEqBase.jl#1158
  • Get Mooncake direct adjoints working SciML/OrdinaryDiffEq.jl#2723
  • @mooncake_overlay does not seem to activate if in an extension #587
  • WIP: Try fixing Mooncake originator SciML/DiffEqBase.jl#1159
• add support for Optimization.jl (see, here and here): now works, see here.
• need to update Optimization docs to include Mooncake in the list
• add rules for NNLib, NonlinearSolve, LinearSolve, sparse linear algebra (ARPACK)
  • copied and pasted from Mooncake Inside Problems SciML/SciMLSensitivity.jl#1152 (comment)

  I see, so yeah it's probably not ready to be in the default set if it doesn't have coverage of NNLib, SciML tooling (LinearSolve), sparse linear algebra (ARPACK), etc.

Related:

• Adding A New AD SciML/SciMLSensitivity.jl#1105
• Support more AD backends SciML/BoundaryValueDiffEq.jl#288

[ChrisRackauckas]

The DiffEq integration is done. Setting up overloads on NonlinearSolve and LinearSolve would help a lot as well, but at least ODEs should be fine. There are some edge cases with callbacks but that's due to the type expectations which maybe can be relaxed?

[yebai]

@ChrisRackauckas, can you clarify the role of NonlinearSolve and LinearSolve for SciML? Also, why are NNLib and ARPACK required?

[ChrisRackauckas]

They are the linear solvers and nonlinear solvers. NNLib is required for any machine learning usage as that's what Flux and Lux use for kernels. ARPACK is an eigenvalue solver for sprase eigenvalues, we can probably skip that one if we go through the LinearSolve.jl interface for all times.