Skip to content

Include example of partial stage reloads in docs #139

@WarmCyan

Description

@WarmCyan

Curifactory has no way internally of dealing with partial stage cache hits (e.g. one thing in cache for a stage is found but not another)

A way to deal with this manually is through record.stage_cachers:

@cf.stage(None, ["thing1", "thing2"], [PickleCacher]*2)
def make_things(record):
    # in the case where thing1 is in cache but thing2 is not, the whole stage will re-run
    # if we know this is a possibility, and maybe thing1 takes a very long time to compute, we could
    # manually handle this inside the stage (assuming it is not cleaner to separate out thing1 and thing2 production
    # into separate stages)
    if record.stage_cachers[0].check():
        thing1 = record.stage_cachers[0].load()
    else:
        # create thing1 the normal way
        ...
    # create thing2
    ...
    return thing1, thing2

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions