Erroneous warning

[grnnja]

from ploomber import DAG
from ploomber.tasks import ShellScript, PythonCallable
from ploomber.products import File
from ploomber.executors import Parallel

from ploomber.spec import DAGSpec
spec = DAGSpec('./pipeline.yaml')
dag = spec.to_dag()
# dag.executor = Parallel()
build = dag.build(force=True)

When running this code I get this output and warning:

  0%|          | 0/7 [00:00<?, ?it/s]

Executing:   0%|          | 0/6 [00:00<?, ?cell/s]

Executing:   0%|          | 0/7 [00:00<?, ?cell/s]

Executing:   0%|          | 0/7 [00:00<?, ?cell/s]

Executing:   0%|          | 0/7 [00:00<?, ?cell/s]

Executing:   0%|          | 0/7 [00:00<?, ?cell/s]

Executing:   0%|          | 0/7 [00:00<?, ?cell/s]

Executing:   0%|          | 0/8 [00:00<?, ?cell/s]

/home/prem/Documents/projects/ploomber/ploomberw/ploomber/src/ploomber/executors/serial.py:149: UserWarning: 
=========================== DAG build with warnings ============================
- NotebookRunner: linear-regression -> MetaProduct({'nb': File('output/...ession.ipynb')}) -
- /home/prem/Documents/projects/ploomber/ploomber-projectsw/ploomber-projects/guides/intro-to-ploomber/tasks/linear-regression.py -
Output '/home/prem/Documents/projects/ploomber/ploomber-projectsw/ploomber-projects/guides/intro-to-ploomber/output/linear-regression.ipynb' is a notebook file. nbconvert_export_kwargs {'exclude_input': True} will be ignored since they only apply when exporting the notebook to other formats such as html. You may change the extension to apply the conversion parameters
=============================== Summary (1 task) ===============================
NotebookRunner: linear-regression -> MetaProduct({'nb': File('output/...ession.ipynb')})
=========================== DAG build with warnings ============================

  warnings.warn(str(warnings_all))

here is the rest of the source:
https://github.com/ploomber/projects/tree/master/guides/intro-to-ploomber

I'm not sure what the cause is, but I believe there should not be a warning here.

[edublancas]

thanks for opening this!

here's the background. the warning is this:

nbconvert_export_kwargs {'exclude_input': True} will be ignored since they only apply when exporting the notebook to other formats such as html. You may change the extension to apply the conversion parameters

we use nbconvert to convert ipynb files to html/pdf. nbconvert offers options such as excluding the source code in the output file (exclude_input=True); however, this setting is not applicable if the user selected ipynb as the output format (because we won't have to use nbconvert for that); that's why we show the warning: it's telling the user that the setting is on, but it wont have any effect.

however, we recently allowed users to select multiple formats from a notebook task. for example, a notebook task might generate an output ipynb, html and pdf. Under this scenario, the warning should not be displayed.

Hence, the old condition for the warning was: is the output an ipynb file? but now it should become: is the only output an ipynb file?

[grnnja]

I'm a bit confused why we don't want the warning in this situation

Here is the relevant part of the pipeline from the original issue

- source: tasks/linear-regression.py
  product:
    nb: output/linear-regression.ipynb
  papermill_params: 
    allow_nested_loop: True
  nbconvert_export_kwargs:
    # optionally hide the code from the report
    exclude_input: True

Correct me if I'm wrong, but it seems like this task only generates one output, output/linear-regression.ipynb, which is a jupyter notebook. So then don't we want the warning since the part about multiple formats from a single notebook task doesn't apply?

[edublancas]

the other way around:

• if the output is a single notebook -> show the warning
• if there are other outputs (HTML, PDF) -> do not show it

the logic is: if a user puts exclude_input: True, but the only output is an ipynb, then we warn because that part isn't doing anything, it can be removed and nothing will happen.

[grnnja]

In the snippet, isn't the only output a single notebook, so we want to show the warning? In the original issue I thought it was showing the warning when we don't want to show it

[edublancas]

In the snippet, isn't the only output a single notebook, so we want to show the warning?

if you mean this snippet:

- source: tasks/linear-regression.py
  product:
    nb: output/linear-regression.ipynb
  papermill_params: 
    allow_nested_loop: True
  nbconvert_export_kwargs:
    # optionally hide the code from the report
    exclude_input: True

yes, here we show the warning

[idomic]

@edublancas please review