--stream-buffering=0 is not behaving as expected

[dschwoerer]

Background information

I would like to see progress information in my simulation, to ensure it is not stuck, or if it is, to handle appropriately.
However only on a newline the output is printed.

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.5 (openmpi-5.0.5-2.fc41.x86_64)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From fedora package

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: Fedora 41
• Computer hardware: x86_64-v4
• Network type: Ethernet / not used

I also tried v5.0.6 from fedora 42 but on a infiniband system with the same issue

---

Details of the problem

The output is always line buffered, even if explicitly requested to not do it.
Full buffering is similarly ignored. Printing to stderr instead of stdout does not help either.

Example program:

mpirun --stream-buffering=0 -np 2 sh -c "echo a ; printf b ; sleep 1; echo c"

Expected output:

a
ba
b<delay here>c
c

Actual output:

a
ba
<delay here>
bc
c

Previous openmpi version (4.1.x) did not introduce line buffering. Mpich is also not causing linebuffering.

[rhc54]

See #12594 for the explanation - or you could just ask mpirun for help:

$ mpirun --help output
...
Supported qualifiers include "NOCOPY" (do not copy the output to the
stdout/err streams), and "RAW" (do not buffer the output into complete
lines, but instead output it as it is received).

[dschwoerer]

I found this documentation:
https://docs.open-mpi.org/en/main/man-openmpi/man1/mpirun.1.html#the-stream-buffering-option

I do expect --help to only contain a short help, and the extensive help to be available via man.

I could however not find where the doc is generated from - --stream-buffering does not seem to be mentioned in https://docs.open-mpi.org/en/main/_sources/man-openmpi/man1/mpirun.1.rst.txt
so I do not understand where I could open a PR to clarify the docs.

Thank you for your fast response and sorry I missed the other documentation.

[rhc54]

I do expect --help to only contain a short help, and the extensive help to be available via man.

Nope - at least, not here, unless someone else wants to write it. PRRTE uses a hierarchical help system, which is actually loosely modeled on what MPICH's hydra does.

I could however not find where the doc is generated from - --stream-buffering does not seem to be mentioned

I don't think that cmd line option exists any more, though the OMPI folks have the ability to re-create it (just translate it to the new option under-the-covers). However, I don't think anyone over here has interest in doing so.

[dschwoerer]

I could however not find where the doc is generated from - --stream-buffering does not seem to be mentioned

I don't think that cmd line option exists any more, though the OMPI folks have the ability to re-create it (just translate it to the new option under-the-covers). However, I don't think anyone over here has interest in doing so.

I do not think that is needed, but then it should not be mentioned in the documentation.

It is still mentioned here:
https://docs.open-mpi.org/en/main/man-openmpi/man1/mpirun.1.html#the-stream-buffering-option

If I click on edit or view source, it is not mentioned.

Is the html version outdated and needs to be rebuild?
Is the source some where else?
I would like to open a PR to update the docs, but I cannot find where.

[jsquyres]

The generation of the mpirun man page is a bit... complicated. It draws upon text from both Open MPI and Open MPI's fork of the PRRTE project (https://github.com/open-mpi/prrte/). This was an attempt to keep mpirun-specific text in Open MPI's git repo, and PRRTE-but-Open-MPI-specific text in the schizo/ompi component directory in the PRRTE repo.

Specifically, that --stream-buffering text is in https://github.com/open-mpi/prrte/blob/master/src/mca/schizo/ompi/schizo-ompi-cli.rstxt.

We'd love to have a fix for that text -- opening a PR against https://github.com/open-mpi/prrte will fix the text, and then we would need to update the git submodule here in OMPI to pull in those changes (which then automatically re-generates + re-deploys the docs).

There will likely need to be a few PRs:

• Update the master branch in https://github.com/open-mpi/prrte (i.e., the Open MPI fork of the PRTE repo)
• Update the PRRTE submodule pointer for the main branch here in this repo
• Update the v3.0 branch in https://github.com/openpmix/prrte (i.e., the main PRRTE upstream repo) with essentially the same fix as the first PR
• Update the PRRTE submodule pointer for the v5.0.x branch here in this repo

EDITED: fixed repo / branch names.

[rhc54]

Update the v3.0 branch in https://github.com/open-mpi/prrte (probably cherry-pick the commit from the first PR)
Update the PRRTE submodule pointer for the v5.0.x branch here in this repo

No, that's not correct - your v5.0.x branch points to upstream PRRTE v3.0 branch, not your local fork. So the fix would have to go upstream first.

[jsquyres]

Oops! @rhc54 is correct -- sorry for the confusion.

• Open MPI's main branch git submodule points to our PRRTE fork.
• Open MPI's v5.0.x branch git submodule points to the main / upstream PRRTE repo.

[jsquyres]

I'd like to clarify something here: the mpirun --stream-buffering option does exist in Open MPI v5.0.x. For example:

$ mpirun --stream-buffering 0 -np 1 hello_c
Hello, world, I am 0 of 1, (Open MPI v5.0.8a1, package: Open MPI [email protected] Distribution, ident: 5.0.8a1, repo rev: v5.0.7-16-g44e520e425, Unreleased developer copy, 155)

If you pass an unrecognized option to mpirun, it will complain. For example:

$ mpirun --stream-bufferingbogus 0 -np 1 hello_c
--------------------------------------------------------------------------
An unrecognized option was included on the mpirun command line:

  Option: --stream-bufferingbogus

Please use the "mpirun --help" command to obtain a list of all
supported options.
--------------------------------------------------------------------------

Hence, this is not a "we need to fix the documentation to remove the --stream-buffering option" issue. Instead, I think @dschwoerer is trying to report that --stream-buffering is not working in the way that they expect -- not that the CLI option does not exist.

Is that correct, @dschwoerer?

[jsquyres]

Using a slightly simpler example, I can replicate the behavior that @dschwoerer reported in the original description:

mpirun --stream-buffering=0 -np 1 sh -c "echo a ; printf b ; sleep 5; echo c"

Should emit a on a line by itself, then emit b, and then 5 seconds later on the same line, it should emit c.

Instead, it emits a on a line by itself, waits 5 seconds, then emits bc on a line.

@rhc54 I see #12594, but I'm still not clear on exactly what is supposed to happen. I see that both --stream-buffering is propagating its value to the ompi_stream_buffering MCA param, and that that MCA param is calling setvbuf() on stdout and stderr appropriately.

Is there something else that needs to be done w.r.t. PMIX to effect how IOF transports the output back to mpirun?

[jsquyres]

Actually, I'm not sure that the original example is a good example of this behavior. The shell adds its own layer of stdout buffering.

Instead, using a trivial C program is a better example:

#include <stdio.h>
#include <unistd.h>
#include "mpi.h"

int main(int argc, char* argv[])
{
    MPI_Init(&argc, &argv);
    printf("Hello world\n");
    printf("Hello");
    sleep(5);
    printf(" World\n");
    MPI_Finalize();
    return 0;
}

Running this program, I would expect:

• With --stream-buffering=0: Hello world immediately, followed by Hello on the 2nd line immediately, and then 5 seconds later on the same line, World.
• With --stream-buffering=1: Hello world immediately, followed by Hello World 5 seconds later on the next line.
• With --stream-buffering=2: 2 lines of Hello world after 5 seconds (assuming that 2 x lines of Hello world is smaller than one buffer unit, so the whole thing is buffered).

With --stream-buffering=2, I see the expected output behavior. This tells me that the --stream-buffering mechanism is actually working (i.e., PRTE/OMPI are reacting to the CLI and MCA mechanisms as appropriately, and OMPI is invoking setvbuf() properly).

This is important because if you run with --stream-buffering=2 with the original shell example, you do not see the "everything is buffered / you see the entire output after 5 seconds" because the shell is overriding with its own line buffering settings.

Back to the C example: I see the same output with --stream-buffering=[0,1]: both emit a "Hello world" line, wait 5 seconds, and then another "Hello world" line.

This tells me that the value of 0 isn't behaving the way we expect. This basically leads me back to my question to @rhc54 if we need to do something more than setvbuf() with respect to IOF (or something else...?). But at least I can confirm that the --stream-buffering=X mechanism is actually propagating the values around properly and OMPI is doing what it thinks it is supposed to do (it may not be doing the correct thing, but this is a very different understanding compared to what I thought when I read the original issue).

[rhc54]

I return to my original answer - you need to hook it up to the RAW setting on output. I have no idea what the different values for "stream-buffering" are supposed to do. PRRTE/PMIx only respects the two options noted in the help output. If your option isn't doing what you expect, then you probably aren't hooking it up to a recognized PRRTE option so it can set PMIx up correctly.

[rhc54]

Just to be clear:

ompi_stream_buffering MCA param, and that that MCA param is calling setvbuf() on stdout and stderr appropriately.

No idea what you are talking about here. PRRTE isn't calling setvbuf anywhere. I assume you mean you are doing that on the app side of things?

All PRRTE does is set PMIx up so it doesn't buffer full lines (up to a newline) before output. Nothing more.