Memory management with exported model run

[bakerwy]

Hi, I checked running exported model is very fast

With simple example,
not exported model took 12,000 seconds but exported model only took 550 seconds. ( x20 fast!)

but while testing the model, I realized that memory increases very fast and easily reaches to full capacity of my computer
and it slows down significantly.

I guess the reason is that it uses parameterized approach.
Once parameterized function executed (for example, m.Projection[i].function(1200)), it doesn't disappear and stored in memory.
so.. if I run the model with many model points, it is inevitable to reach to full memory capacity.

It seems for sentence is not working in exported model. Even if I want to change model point by using the code modelpoint_id = i, that is not applied.

Is there any way of removing executed function? For example, if I run the 1 to 10,000 m.Projection[i].function(1200) and stored the result, I want to remove the executed function to make space in memory.

Otherwise, is there any way to run exported model with for sentence?

[fumitoh]

You can discard cached results when you no longer need them by deleting the model. Note that you can create as many instances of the exported model as you want.

Suppose you create an exported model using the code below.

import modelx as mx

mx.read_model('lifelib/libraries/savings/CashValue_SE').export('CashValue_SE_nomx')

The code above creates the CashValue_SE_nomx package in the current directory.

The exported model is an instance of the _c_CashValue_SE class defined in CashValue_SE_nomx._mx_model.

When importing CashValue_SE_nomx, one instance is created. This instance is named mx_model and also CashValue_SE in both CashValue_SE_nomx._mx_model and CashValue_SE_nomx. These four names refer to the same instance.

import CashValue_SE_nomx

assert (CashValue_SE_nomx.mx_model
    is CashValue_SE_nomx.CashValue_SE
    is CashValue_SE_nomx._mx_model.mx_model
    is CashValue_SE_nomx._mx_model.CashValue_SE)

But you can create a new instance of the model class to use for a subset of the entire model points, then save only the values you need, and afterward, discard the model and manually trigger garbage collection. This process can be repeated for the number of subsets of the model points.

For example, if you want to obtain Projection.pv_net_cf() for policies 1 through 100,000, but due to memory constraints on your PC, you can only effectively run 10,000 model points at a time. Here's how you can do it.

import gc  # import the standard library for garbage collection

gc.disable()   # disable gc so that you can manually trigger garbage collection

result = []   # to store the ultimate results (`pv_net_cf()`) for all model points

step = 10_000   # subset size
for i in range(0, 10 * step, step):
    m = CashValue_SE_nomx._mx_model._c_CashValue_SE()   # create a new model instance
    result.extend(m.Projection(j).pv_net_cf() for j in range(i + 1, i + step + 1))  # calculate and store `pv_net_cf()`
    del m   # delete the name bound to the model as the cached results are not needed anymore
    gc.collect()   # manually collect garbage

gc.enable()   # re-enable gc

Garbage collection (GC) is Python's mechanism that automatically collects objects that are no longer referenced, freeing up the memory used by these objects. Since you want to free up memory manually, GC is disabled at the beginning. Since the model instance created at each step is assigned only to m, by deleting m, the model instance is no longer referenced. For your reference, the original instance created when importing CashValue_SE_nomx is referenced by four names, so to delete the object, you need to delete all four names.

[bakerwy]

Wow.. that is very simple and useful! Thanks alot!