Slow putObject requests

[TimLand]

Describe the bug
I'm put objects to local Minio object storage connected via 127.0.0.1，Object size is 32KB and only one thread upload cyclically. The same conditions with minio-warp(S3 benchmarking tool) I can got ~14MB/Sec throughput. However, if I use aws-sdk-cpp's PutObject I get only about 0.5MB/Sec. How can I improve it?

SDK version number
aws-sdk-cpp 1.8.183

Minio version number
minio version RELEASE.2021-01-16T02019-44Z

Platform/OS/Hardware/Device
Ubuntu 16.04 (x86_64)

What compiler are you using? what version?
g++ (GCC) 7.3.0

Additional context
the code

#include <aws/core/Aws.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/auth/AWSCredentialsProviderChain.h>
#include <aws/core/utils/logging/LogLevel.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/GetObjectRequest.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <iostream>
#include <fstream>
#include <memory>
#include <string>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/time.h>

using namespace Aws;

int main(int argc, char *argv[])
{
    SDKOptions options;
    options.loggingOptions.logLevel = Utils::Logging::LogLevel::Error;
    InitAPI(options);
    {
        Auth::AWSCredentials credentials;
        credentials.SetAWSAccessKeyId("minioadmin");
        credentials.SetAWSSecretKey("minioadmin");
    
        Client::ClientConfiguration config;
        config.endpointOverride = "127.0.0.1:9000";
        config.scheme = Http::Scheme::HTTP;
        config.verifySSL = false;
        // config.maxConnections = 64;
        S3::S3Client client(credentials, config,
                Client::AWSAuthV4Signer::PayloadSigningPolicy::Never /* signPayloads */,
                false /* useVirtualAddressing */);

        int size = 32768;
        char* data = new char[size];
        memset(data, 0, size);
        for (int i = 0; i < 30000; ++i) {
            struct timeval tv_before, tv_after;
            uint64_t time;
            gettimeofday(&tv_before, nullptr);
            std::string file = std::to_string(i);
            Aws::S3::Model::PutObjectRequest request;
            request.WithBucket("juice").WithKey(file.c_str());
            auto in = Aws::MakeShared<Aws::StringStream>("", std::stringstream::in|std::stringstream::out|std::stringstream::binary);
            in->write((char*)data, size);

            request.SetContentLength(size);
            request.SetBody(in);
            auto outcome = client.PutObject(request);
            if (outcome.IsSuccess()){;
                //std::cout << "Completed!" << std::endl;
            } else {
                std::cout << "Failed with error: " << outcome.GetError() << std::endl;
                abort();
            }
            gettimeofday(&tv_after, nullptr);
            tv_after.tv_sec -= tv_before.tv_sec;
            time = tv_after.tv_sec * 1E6 + tv_after.tv_usec;
            time -= tv_before.tv_usec;
            if (time > 1000) {
                std::cout << "cost > 1ms (" << time << " ns): " << file << std::endl;
            }
        }
        delete [](data);
    }
    ShutdownAPI(options);
    return 0;
}

[klauspost]

You didn't post your warp settings, but likely "only one thread" is the reason.

[TimLand]

hi , its my warp settings , about 11.48MB/Sec throughput

[klauspost]

I am not an expert on the AWS sdk, but it could be that that cannot do HTTP keepalive and have to reconnect for every request.

Maybe ask on that project or someone with experience in it can help.

It could be doing md5 which will slow down both the client and the server. You can add --md5 to warp to check with that enabled or find something in the SDK to disable it.

warp (naturally) doesn't take any shortcuts, so there is no reason you shouldn't be able to achieve the same given a similarly performant client sdk.

Either way, increase your concurrency. You should be able to increase it with little to no negative impact.

[TimLand]

Hi:

Increase the concurrency can improve aws-sdk-cpp upload performance but it slower than warp yet. With single thread running just to simplify the comparison test conditions.

Warp bench tool add --md5 param the upload speed does not decrease much, the cpu usageis a bit higher and HTTP keepalive disabled has no effect to improve performace。

[TimLand]

hi @klauspost

Aws-SDK-cpp debug log show that maybe is slow to get the response? about 40ms per request

[TimLand]

Hi
By mc trace myminio-server. Using aws-sdk-cpp looks more time-consuming per request.The difference is that aws-sdk did not call the GetBuckeetLocation function at the beginning？
Using aws-sdk-cpp upload（ --concurrent=1）

Using minio-go-sdk upload（ --concurrent=1）

Using wrap upload （ --concurrent=1）