Parallel write with BP5 engine writes corrupted data exactly at index = maximum BufferChunkSize value

[Vassilis-Boubis]

I am trying to store a 5 GB dataset in parallel with 2 MPI processes using the BP5 engine.

When I do this with 1,4,8, or more processes, everything works as intended.

When using 2 processes, however, rank 0 writes its partition correctly, while rank 1 writes incorrect values starting at byte 2147381248 of its partition. I verified this by inspecting the contents of what was stored through the bpls command-line tool.

When checking the ADIOS 2 documentation, I noticed that the maximum BufferChunkSize is exactly 2147381248 bytes. Could this be related to my corrupted parallel write?

Any help would be appreciated, thank you!

   void parallelWriteData(adios2::ADIOS &adios, std::vector<uint8_t> &data, 
              const std::string file_name, uint64_t shape, uint64_t count, 
              uint64_t start){

    std::string writer_name = "DataWriter"

    adios2::IO io = adios.DeclareIO(writer_name);

    auto var = io.DefineVariable<uint8_t>("binary_data", {shape}, {start}, {count});

    adios2::Engine writer = io.Open(file_name, adios2::Mode::Write);
    writer.BeginStep();
    writer.Put(var, data.data());
    writer.EndStep();
    writer.Close();}

[vicentebolea]

Most probably a bug, an integer overflow of a kind somewhere, this is the min value for an int32_t.

[pnorbert]

What adios version is this? I am using adios master branch.

[pnorbert]

I modified the examples/hello/bpWriter.cpp example to produce my dataset. Can you replicate my result on your machine?

.../ADIOS2/build.debug$ mpirun -n 2 ./bin/adios2_hello_bpWriter_mpi
.../ADIOS2/build.debug$ ./bin/bpls -l myVector_cpp.bp -D
  uint8_t  data  {5250325339} = 0 / 255
        step 0:
          block 0: [         0:2627801482] = 0 / 255
          block 1: [2627801483:5250325338] = 0 / 255

Beginning of rank 0

.../ADIOS2/build.debug$ ./bin/bpls -l myVector_cpp.bp -d data -s "0" -c "10"  -n 10
  uint8_t  data  {5250325339} = 0 / 255
    slice (0:9)
    (0)    0 1 2 3 4 5 6 7 8 9

last 3 elements of rank 0, ending in 138, and then rank 1 starting from 1

.../ADIOS2/build.debug$ ./bin/bpls -l myVector_cpp.bp -d data -s "2627801480" -c "10"  -n 10
  uint8_t  data  {5250325339} = 0 / 255
    slice (2627801480:2627801489)
    (2627801480)    136 137 138 1 2 3 4 5 6 7

Starting of rank 1

.../ADIOS2/build.debug$ ./bin/bpls -l myVector_cpp.bp -d data -s "2627801483" -c "10"  -n 10
  uint8_t  data  {5250325339} = 0 / 255
    slice (2627801483:2627801492)
    (2627801483)    1 2 3 4 5 6 7 8 9 10

rank 1 start + 2147381248 - 5 = 4775182726

.../ADIOS2/build.debug$ ./bin/bpls -l myVector_cpp.bp -d data -s "4775182726" -c "10"  -n 10
  uint8_t  data  {5250325339} = 0 / 255
    slice (4775182726:4775182735)
    (4775182726)    252 253 254 255 0 1 2 3 4 5

/*
 * Distributed under the OSI-approved Apache License, Version 2.0.  See
 * accompanying file Copyright.txt for details.
 *
 * bpWriter.cpp: Simple self-descriptive example of how to write a variable
 * to a BP File that lives in several MPI processes.
 *
 *  Created on: Feb 16, 2017
 *      Author: William F Godoy [email protected]
 */

#include <ios>       //std::ios_base::failure
#include <iostream>  //std::cout
#include <stdexcept> //std::invalid_argument std::exception
#include <vector>

#include <adios2.h>
#if ADIOS2_USE_MPI
#include <mpi.h>
#endif

int main(int argc, char *argv[])
{
    int rank, size;
#if ADIOS2_USE_MPI
    int provided;

    // MPI_THREAD_MULTIPLE is only required if you enable the SST MPI_DP
    MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
#else
    rank = 0;
    size = 1;
#endif

    size_t nGlobal = 5368709120; // global number of points for myTest
    size_t nLocal = nGlobal / size;
    size_t nStart = nLocal * rank;
    size_t rem = nGlobal - (nLocal * size);
    if (rank == size - 1)
    {
        nLocal += rem;
    }

    // different test for 2 processes
    if (size == 2)
    {
        nGlobal = 5250325339;
        nLocal = (rank == 0 ? 2627801483 : 2622523856);
        nStart = (rank == 0 ? 0 : 2627801483);
    }

    /** Application variable */
    std::vector<uint8_t> myTest(nLocal);
    uint8_t myvalue = (uint8_t)rank;
    for (size_t n = 0; n < nLocal; ++n)
    {
        myTest[n] = myvalue++;
    }
    std::vector<float> myFloats = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
    std::vector<int> myInts = {0, -1, -2, -3, -4, -5, -6, -7, -8, -9};
    const std::size_t Nx = myFloats.size();

    const std::string myString("Hello Variable String from rank " + std::to_string(rank));

    try
    {
        /** ADIOS class factory of IO class objects */
#if ADIOS2_USE_MPI
        adios2::ADIOS adios(MPI_COMM_WORLD);
#else
        adios2::ADIOS adios;
#endif

        /*** IO class object: settings and factory of Settings: Variables,
         * Parameters, Transports, and Execution: Engines */
        adios2::IO bpIO = adios.DeclareIO("BPFile_N2N");

        adios2::Variable<uint8_t> bpData = bpIO.DefineVariable<uint8_t>(
            "data", {nGlobal}, {nStart}, {nLocal}, adios2::ConstantDims);
        std::cout << "rank: " << rank << " shape = " << nGlobal << " start = " << nStart
                  << " count = " << nLocal << std::endl;

        /** global array : name, { shape (total) }, { start (local) }, {
         * count
         * (local) }, all are constant dimensions */
        adios2::Variable<float> bpFloats = bpIO.DefineVariable<float>(
            "bpFloats", {size * Nx}, {rank * Nx}, {Nx}, adios2::ConstantDims);

        adios2::Variable<int> bpInts = bpIO.DefineVariable<int>("bpInts", {size * Nx}, {rank * Nx},
                                                                {Nx}, adios2::ConstantDims);

        adios2::Variable<std::string> bpString = bpIO.DefineVariable<std::string>("bpString");
        (void)bpString; // For the sake of the example we create an unused
                        // variable

        std::string filename = "myVector_cpp.bp";
        /** Engine derived class, spawned to start IO operations */
        adios2::Engine bpWriter = bpIO.Open(filename, adios2::Mode::Write);

        bpWriter.BeginStep();
        /** Put variables for buffering, template type is optional */
        bpWriter.Put(bpData, myTest.data());
        // bpWriter.Put(bpFloats, myFloats.data());
        // bpWriter.Put(bpInts, myInts.data());
        //  bpWriter.Put(bpString, myString);
        bpWriter.EndStep();

        /** Create bp file, engine becomes unreachable after this*/
        bpWriter.Close();
        if (rank == 0)
        {
            std::cout << "Wrote file " << filename
                      << " to disk. It can now be read by running "
                         "./bin/adios2_hello_bpReader.\n";
        }
    }
    catch (std::invalid_argument &e)
    {
        std::cerr << "Invalid argument exception: " << e.what() << "\n";
#if ADIOS2_USE_MPI
        std::cerr << "STOPPING PROGRAM from rank " << rank << "\n";
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }
    catch (std::ios_base::failure &e)
    {
        std::cerr << "IO System base failure exception: " << e.what() << "\n";
#if ADIOS2_USE_MPI
        std::cerr << "STOPPING PROGRAM from rank " << rank << "\n";
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }
    catch (std::exception &e)
    {
        std::cerr << "Exception: " << e.what() << "\n";
#if ADIOS2_USE_MPI
        std::cerr << "STOPPING PROGRAM from rank " << rank << "\n";
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }

#if ADIOS2_USE_MPI
    MPI_Finalize();
#endif

    return 0;
}

[Vassilis-Boubis]

Thank you for your help.

I have run your code, and I could replicate your results.

Therefore, I went back to my code and realised the problem is not in writing.
Since in my application I do not generate the data array myself, I had printed it before writing, to check if it matched the bpls output, and I must have made some mistake with the data types while printing it.

I do still get the same error, but I was wrong in saying it occurred during writing. I now believe that it occurs during reading: the data that rank 1 reads after byte 2147381248 differs from the bpls output.

Here is my code for reading + printing:

  std::vector<uint8_t> parallelReadData(adios2::ADIOS &adios, const std::string fileName, 
                            const size_t count, const size_t start, size_t rank) {

    adios2::IO io = adios.DeclareIO("reader");
    adios2::Engine reader = io.Open(fileName, adios2::Mode::Read);

    std::cout<< " rank "<< rank << " -- in parallelReadData -- " << "start = " << start <<std::endl;
    std::cout<< " rank "<< rank << " -- in parallelReadData -- " << "count = " << count <<std::endl;
  
    reader.BeginStep();
    auto var = io.InquireVariable<uint8_t>("binary_data");

    var.SetSelection({{start}, {count}});
   
    std::vector<uint8_t> buffer(count);

    reader.Get(var, buffer);

    reader.EndStep();
    
    reader.Close();

    if( rank ==1){
            std::cout<< "Buffer in rank 1 before returning from adios read = " << ""; 
            for (size_t i = (2147381248-10); i < (2147381248+10); ++i) {
            std::cout << static_cast<int>(buffer[i]) << " ";
   }

    return buffer;
  }

Here are the corresponding prints:

  rank 0 -- in parallelReadData -- start = 0 
  rank 0 -- in parallelReadData -- count = 2627801483
  rank 1 -- in parallelReadData -- start = 2627801483
  rank 1 -- in parallelReadData -- count = 2622523856
  Buffer in rank 1 before returning from adios read = 177 215 117 0 5 213 5 47 88 29 82 121 182 126 152 173 126 163 38 142 

Here is the bpls output :

  bpls ./output -d "binary_data" -s "4775182721" -c "20" -n "10"
    uint8_t  binary_data  {5250325339}
    slice (4775182721:4775182740)
    (4775182721)    177 215 117 0 5 213 5 47 88 29
    (4775182731)    0 71 218 90 125 211 97 243 61 147

[pnorbert]

Understood. So the bpls output now matches your expectation and the parallel reader's rank 1 shows the issue, right?

[Vassilis-Boubis]

Yes, that is correct.