COG driver not working for Large Dataset and 3cm Accuracy

[AgPriyanshu]

What is the bug?

I am using gdal_translate to generate cog, below is the command that I am using
gdal_translate <input_dataset> <output_dataset> -stats -of COG -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=LZW -co NUM_THREADS=ALL_CPUS -co BIGTIFF=YES --config GDAL_CACHEMAX 512
The above command is not able to create COG from 55GB GeoTiff with 3cm accuracy but the command below it is working fine

gdal_translate <input_dataset> <output_dataset> -stats -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=LZW -co NUM_THREADS=ALL_CPUS -co BIGTIFF=YES --config GDAL_CACHEMAX 512

Steps to reproduce the issue

Run gdal_translate command for a large GEOTIFF with 3cm accuracy

Versions and provenance

GDAL 3.8.4, released 2024/02/08

Additional context

COG driver stucks at 25GB of file creation and after that it just doesn't work and only single CPU is occupied

[rouault]

Please paste the full output of gdalinfo <input_dataset>

[AgPriyanshu]

@rouault Please check the output below

Driver: GTiff/GeoTIFF
Files: merged_v1.tif
Size is 388099, 123230
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 44N",
    BASEGEOGCRS["WGS 84",
        DATUM["World Geodetic System 1984",
            ELLIPSOID["WGS 84",6378137,298.25722356049,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Transverse Mercator",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",81,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    ID["EPSG",32644]]
Data axis to CRS axis mapping: 1,2
Origin = (665757.782678909134120,2678011.960338353645056)
Pixel Size = (0.029215391111362,-0.029215391111360)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_SOFTWARE=Global Mapper
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
  PREDICTOR=2
Corner Coordinates:
Upper Left  (  665757.783, 2678011.960) ( 82d37'55.76"E, 24d12'21.92"N)
Lower Left  (  665757.783, 2674411.748) ( 82d37'54.27"E, 24d10'24.91"N)
Upper Right (  677096.247, 2678011.960) ( 82d44'37.54"E, 24d12'17.47"N)
Lower Right (  677096.247, 2674411.748) ( 82d44'35.95"E, 24d10'20.46"N)
Center      (  671427.015, 2676211.854) ( 82d41'15.88"E, 24d11'21.22"N)
Band 1 Block=388099x32 Type=Byte, ColorInterp=Red
  Overviews: 194050x61615, 97025x30808, 48513x15404, 24257x7702, 12129x3851, 6065x1926, 3033x963, 1517x482
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 194050x61615, 97025x30808, 48513x15404, 24257x7702, 12129x3851, 6065x1926, 3033x963, 1517x482
Band 2 Block=388099x32 Type=Byte, ColorInterp=Green
  Overviews: 194050x61615, 97025x30808, 48513x15404, 24257x7702, 12129x3851, 6065x1926, 3033x963, 1517x482
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 194050x61615, 97025x30808, 48513x15404, 24257x7702, 12129x3851, 6065x1926, 3033x963, 1517x482
Band 3 Block=388099x32 Type=Byte, ColorInterp=Blue
  Overviews: 194050x61615, 97025x30808, 48513x15404, 24257x7702, 12129x3851, 6065x1926, 3033x963, 1517x482
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 194050x61615, 97025x30808, 48513x15404, 24257x7702, 12129x3851, 6065x1926, 3033x963, 1517x482
Band 4 Block=388099x32 Type=Byte, ColorInterp=Alpha
  Overviews: 194050x61615, 97025x30808, 48513x15404, 24257x7702, 12129x3851, 6065x1926, 3033x963, 1517x482