Skip to content

Commit

Permalink
Cancel upload (#1332)
Browse files Browse the repository at this point in the history
* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag (#1220)

* Feature/ Challenge based on rounds (#1191)

* Fix

* Fix

* Fix

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix

* Heavy logging

* Fix

* More logging

* Changed logging

* Removed logging

* Debug

* Fix

* Fix

* Fix

* expose send in wasm (#1225)

* fix rename dir err (#1226)

* Txnfee send (#1227)

* add txnfee for send function

* return txn verify output

* add rename dir (#1230)

* Async read (#1213)

* basic timings for download

* upload timings

* to seconds

* more timings for upload processing

* display ms properly

* cleanup

* fix merge

* fix merge

* add more timing logs

* fix dur to ms

* add timings to read and build data

* async read

* range over errChan

* fix unit test

* rmv ctx cancel from multi upload

* wait for last chunk

* fix loop

* rmv readChunk timing

* increase buffer size

* rmv timing logs

* cleanup

* updated go to version 1.20

* use -buildvcs=false

---------

Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>

* Wasm for `move`, `rename`, `copy` (#1233)

* add demo for move, rename and copy

* remove debug logs

---------

Co-authored-by: Yury <[email protected]>

* Feat/streaming server in winsdk (#1228)

* fix move when srcPath is same as destPath (#1239)

* fix(upload):fixed invalid ffmpeg command in transcode feature (#1236)

* fix(upload):fixed invalid ffmpeg command in transcode feature

* fix(upload): always delete transcode output file

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(devops): fixed build-windows

* fix(mobilesdk): add logging for MultiUpload

* fix(mobilesdk): fixed file permission issue on transcode

* fixed lint error

---------

Co-authored-by: dabasov <[email protected]>

* Fix web streaming - wasm (#1237)

* Do not send GET http request with body

* Use downloadBlocks to download in player_file

* Call ListDir to get file meta

* Format

* Download 100 blocks each time by default

* Fix downloadBlocks

Track the downloaded bytes and trunk the padding 0 bytes

* Update downloadBlocks to download blocks concurretly

Use the same download method with downloadFileByBlocks

* Add ActualFileSize in PlayListFile

* Update wasm_exec.js to go 1.21.0

* Update version

* Replace alloc.DownloadBlocks with DownloadBlocksToFileHandler

* Add concurrency control (#1245)

* control file ops concurrency

* cleanup

* add batch size to multi op

* Fix/stream tests (#1247)

* fix(upload): fixed file name issue in webstreaming

* removed unused code

* fix share consensus (#1246)

* List optimization (#1240)

* list optimization

* list optimization

* fix list hash

* add to wg

* fix list test

* check consensus

* fix listDir test

* use once for mock calls

* added sharders keep list (#1231)

* added sharders keep list

* added holder to zboxcore

* removed duplicated sharder call

* fixed lint

* fixed panic

* fixed panic

* fixed panic

* fixed panic

* fixed panic

* added consensus

* added consensus

* added consensus

* refactored

* refactored

* refactored

* chec alloc and prev alloc root (#1250)

* add check for chunkNumber (#1253)

* add check for chunkNumber

* add err log

* fix save progress

* split-key handler implemented in wasm sdk

* chnaging go version for cdn

* bumped version

* minor refactoring

* bumping  wasm_exec.js  version

* Refactoring unnecessary structs

* create new conID (#1256)

* debugging consoles added

* fix statusbar update (#1257)

* lint fixes

* lint ignore

* nolint: unused added

* use httpdo for create connection (#1259)

* Add create dir in repair (#1262)

* fix delete dir err in cb

* add way to create dir in repair

* add desc for transaction data (#1263)

* Feature: provide audit zcnswap and refactor if needed (#1205)

* feature: modifies zcnswap to be consitent with zcnbridge

* fix: fixed bugs

* fix: removed dead code

* feature: extended key store logic

* fix: fixed bugs

* fix: fixed conflicts

* fix: removed dead code

* fix: fixed bugs

* Fix/remove path (#1221)

* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>

* Revert "Fix/remove path (#1221)" (#1222)

This reverts commit 464260a.

* remove path header, fix ws tag (#1223)

* remove path header, fix ws tag

* fixed broken build

---------

Co-authored-by: din-mukhammed <[email protected]>

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

(cherry picked from commit 1c9f143)

* fix

* updated wasm binary build fix.

* feature: added test case for swap logic

* fix: resolved dependency conflicts

* fix: fixed bugs

* feature: added zcn eth rate fetch logic

* Update blobber.go

* Update authorizers_query.go

* feature: replaced confusing param naming

* fix: updated bridge initialization in wasm bridge integration

* fix: replaced value amount with correct value

* fix: wrong test case values

* feature: switch to the usage of zcn token smart contract

* fix: fixed bridge tests

---------

Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Yury <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>

* Feat/exposed cli methods in winsdk (#1248)

* feat(winsdk): added CreateWallet

* feat(winsdk): added RecoverWallet

* feat(winsdk): added logging in CreateWallet and RecoverWallet

* feat(winsdk): split InitSDK into InitZCNSDK and InitWallets

* feat(winsdk): fixed zboxApiClient initialization issue

* feat(winsdk): renamed init sdks

* feat(winsdk): fixed init zboxApiClient issue

* fix(winsdk): fixed RecoverWallet naming

* fix(winsdk): fixed file name issue in GetUploadStatus for transcode feature

* feat(winsdk): added GetFileContentType in winsdk

* feat(winsdk): added ListAllocations

* feat(winsdk): added CreateFreeAllocation

* feat(winsdk): added allocation methods

* fix(winsdk): added logs

* feat(zboxapi): added GetFreeStorage

* feat(zboxapi): also send X-App-ID-Token in header

* fix(winsdk): skip TestGetFreeStorage

* fix(winsdk): added getAllocationWith

* fix(zboxapi): used local signHash instead of sys.SignFunc

* fix(zboxapi): fixed token issue in GetFreeStorage

* fix(list): exported ThumbnailHash and ThumbnailSize in ListResult

* fix(0box): fixed GetFreeStorage/CreateFreeAllocation

* fix(winsdk): renamed jwtToken with token in CreateFreeAllocation

* fix(winsdk): renamed jwtToken with token in CreateFreeAllocation

* feat(winsdk): added GetFreeMarker

* feat(winsdk): added AddSharedInfo/DeleteSharedInfo/GetSharedToMe/GetSharedByMe

* feat(winsdk): added CreateAuthTicket/DeleteAuthTicket

* feat(winsdk): fixed GetSharedToMe/GetSharedByMe

* feat(winsdk): fixed GetSharedToMe/GetSharedByMe

* feat(winsdk): exported CreateAuthTicket/DeleteAuthTicket/CreateSharedInfo/DeleteSharedInfo/GetSharedByMe/GetSharedByPublic/GetSharedToMe

* Fix/fail shardedr on bad code (#1264)

* Fix/remove path (#1221)

* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>

* Revert "Fix/remove path (#1221)" (#1222)

This reverts commit 464260a.

* remove path header, fix ws tag (#1223)

* remove path header, fix ws tag

* fixed broken build

---------

Co-authored-by: din-mukhammed <[email protected]>

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

(cherry picked from commit 1c9f143)

* updated wasm binary build fix.

* README grammar and formatting tweaks.

* Merge remote-tracking branch 'origin/feature/sharder-keep-list' into feature/sharder-keep-list

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: UncertainBadg3r <[email protected]>

* Fix/roll logs (#1266)

* Added lumberjack logger

* Added lumberjack logger

* fix:download file/thumbnail by authticket in winsdk (#1265)

* feat(winsdk): added GetWalletBalance (#1267)

* enabled gosdk update for sprint branches (#1268)

* Revert "enabled gosdk update for sprint branches (#1268)" (#1269)

This reverts commit 201b8bd.

* fix in progress callback (#1270)

* Cleanup update terms from update allocation (#1274)

* Improve download performance (#1271)

* improve download performance

* add read buffer size

* add hash option

* fix lint test

* set default to true

* log for data

* fix read

* feature: add reset nonce for bridge (#1279)

* Add registerAuthorizer to wasm (#1275)

* Hotfix/parse hostname error (#1241)

* Fix/remove path (#1221)

* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>

* Revert "Fix/remove path (#1221)" (#1222)

This reverts commit 464260a.

* remove path header, fix ws tag

* remove path header, fix ws tag (#1223)

* remove path header, fix ws tag

* fixed broken build

---------

Co-authored-by: din-mukhammed <[email protected]>

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

(cherry picked from commit 1c9f143)

* added logging

* updated wasm binary build fix.

* updated wasm binary build fix.

(cherry picked from commit 2571888)

* README grammar and formatting tweaks.

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: UncertainBadg3r <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>

* Adding RegisterAuthorizer method in wasmsdk (#1273)

* Adding RegisterAuthorizer method in wasmsdk

* Fix:wasm test fail

* Adding demo for RegisterAuthorizer handler in wasm

* Js and Go bridge implementation

* RegisterAuthorizer handler changes and adding Async Js bridge

* golangci-lint fixes

* Added error handling in setSplitKey and setAuthURL

* Refactoring wasm exposed handlers

* Setting split key wallet before setting auth url in setAuthURL's demo

* Lint Fixes after resolving conflicts

* upload optimization (#1281)

* add sha256 simd and blake3

* add hash chan

* fix lint and unit test

* fix MHash

* rmv timing log

* fix thumbnail hash

* added logging

* added logging

* hotfix

* hotfix

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* Sprint 1.11 revert (#1284)

* Revert "experiment"

This reverts commit 050270f.

* Revert "experiment"

This reverts commit d4821f8.

* Revert "experiment"

This reverts commit 9f65b2e.

* Revert "experiment"

This reverts commit 6313cb8.

* Revert "experiment"

This reverts commit b6be0b5.

* Revert "experiment"

This reverts commit 68d4190.

* Revert "experiment"

This reverts commit 3a23f3f.

* Revert "experiment"

This reverts commit 6b6578b.

* Revert "experiment"

This reverts commit 65a0152.

* Revert "experiment"

This reverts commit 6834b05.

* Revert "experiment"

This reverts commit 8446894.

* Revert "hotfix"

This reverts commit 48cf793.

* Revert "hotfix"

This reverts commit 895c7f0.

* Revert "added logging"

This reverts commit fafce3c.

* Revert "added logging"

This reverts commit 10b6a4a.

* rmv channel hasher (#1285)

* Fix submit txn (#1287)

* Feature: add BNT, USDC and EURC tokens for Swap (#1277)

* feature: add multiple token support for swap

* feature: add correct Bancor API calls

* feature: added approve for swap of bancor token

* fix: removed debug lines

* fix: fixed swap approval

* fix: fixed bug

* fix: fixed bugs

* change hasher (#1288)

* change hasher

* set max proc to 1

* set batch size to 2

* revert hasher lib changes

* increase batch size to 4

* set batch size to 2

* fix build

* update chunk reader

* run gc

* add sleep and decrease batch size

* change blake3

* fix operation slice

* fix batch size

* increase sleep time

* add memory pool

* increase swg for wasm

* reuse chunk reader

* fix read chunk

* rmv unused package

* Remove sleep and GOMAXPROCS=1

* Increase batch size a little

---------

Co-authored-by: peterlimg <[email protected]>

* Import eth wallet with index (#1292)

* Add import account address index

* Make it optional

* Feat/import-account-idx (#1294)

* Add import account address index

* Make it optional

* Add account index option

* Add account bip32 option (#1295)

* Fix allocation min lock (#1297)

* create PRs to dependent repos when a PR is merged to sprint branches (#1283)

* create PRs to dependent repos when a PR is merged to sprint branches

* add branch name

* move wasm-build to build-sdk.yml

* cleanup sdk-release.yml

* Removed padding from the CryptoJS Encrypt and Decrypt functions functions. (#1300)

Co-authored-by: Manohar Reddy <[email protected]>

* Fix allocation min lock

* change chunkNumber for wasm (#1304)

* change numBlock for wasm

* change numBlocks

* revert sdk changes

* Feature: implement fork per run strategy (#1299)

* feature: added fork creation

* fix: fixed bug

* feature: improved Tenderly fork creation flow

* fix parse error (#1309)

* fix parse error

* fix log

* add swg

* Fix memory usage (#1311)

* fix memory usage

* change hasher to sha2 256

* fix unit test

* increase numBlocks

* fix thumbnail hash

* Fix iso header (#1314)

* add ISO check

* remove filename field from authticket

* fix typo

* log auth token

* cleanup

* Fix zboxcore min submit (#1315)

* Fix zboxcore min submit

* Fix min submit and min confirmation default configs

* Fix unit tests

* Fix unit tests

* Revert "Fix iso header (#1314)" (#1317)

This reverts commit 218d3ed.

* Add upload for empty file (#1316)

* add upload for empty file

* fix hash

* check ref

* fix actual size

* fix empty upload (#1320)

* fix read auth ticket (#1319)

* allocation validation for data parity shards (#1321)

* fix git action to raise PRs to update gosdk (#1313)

* test gosdk update

* test-gosdk-update

* use binary operators instead of terinary

* use github.ref

* use github ref

* add branch name

* use quotes

* prepare for PR

* test work flow dispatch

* test extract branch

* print env

* prepare for PR

* feature: added custom block number during Tenderly fork creation (#1323)

* Fix/retry tx on nonce error (#1322)

* Revert "experiment"

This reverts commit 050270f.

* Revert "experiment"

This reverts commit d4821f8.

* Revert "experiment"

This reverts commit 9f65b2e.

* Revert "experiment"

This reverts commit 6313cb8.

* Revert "experiment"

This reverts commit b6be0b5.

* Revert "experiment"

This reverts commit 68d4190.

* Revert "experiment"

This reverts commit 3a23f3f.

* Revert "experiment"

This reverts commit 6b6578b.

* Revert "experiment"

This reverts commit 65a0152.

* Revert "experiment"

This reverts commit 6834b05.

* Revert "experiment"

This reverts commit 8446894.

* Revert "hotfix"

This reverts commit 48cf793.

* Revert "hotfix"

This reverts commit 895c7f0.

* Revert "added logging"

This reverts commit fafce3c.

* Revert "added logging"

This reverts commit 10b6a4a.

* retry failed transaction in zbox with nonce error

* Fix wm rollback (#1324)

* fix wm rollback

* fix rb check

* version logs

* fix prev version

* cleanup

* Cleanup free storage update allocation (#1325)

* remove batch size (#1327)

* fix retry op callback (#1329)

* Revert "remove batch size (#1327)" (#1328)

This reverts commit 310edbb.

* fix cancel upload

* Add `force` param for getAllocationBlobbers wasm (#1330)

* fix retry callback (#1331)

* Fix fill panic (#1334)

* fix fill shards panic

* add log for fill err

* fix err

* add cancel upload to wasm

* fix proxy

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

* Sprint changes (#1355)

* Update sprint 1.12 (#1341)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix trailing whitespace (#1343)

* Merge staging changes (#1346)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

---------

Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix upload select (#1351)

* fix workdir in mobile sdk (#1345)

* fix mobile workdir

* set multi op batch size

* set option to download to disk in wasm (#1348)

* fix panic in hash chan (#1352)

* Fix merge conflict in sprint-1.12 (#1354)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* repair in batches (#1347)

* repair in batches

* fix lint

* fix unit test

* fix batch size

---------

Co-authored-by: Yury <[email protected]>

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Yury <[email protected]>

* fix repair after update (#1357)

* use mo ctx as parent

* fix wasm build

* empty commit

* fix ctx

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: Yury <[email protected]>
Co-authored-by: Ash-KODES <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: UncertainBadg3r <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Tanishq Gupta <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>
Co-authored-by: GuruHubb <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>
  • Loading branch information
22 people authored Jan 22, 2024
1 parent 6f48750 commit e7db14c
Show file tree
Hide file tree
Showing 9 changed files with 61 additions and 27 deletions.
4 changes: 2 additions & 2 deletions mobilesdk/zbox/storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -711,12 +711,12 @@ func CancelDownload(allocationID, remotepath string) error {
// ## Inputs
// - allocationID
// - localPath
func CancelUpload(allocationID, localpath string) error {
func CancelUpload(allocationID, remotePath string) error {
a, err := getAllocation(allocationID)
if err != nil {
return err
}
return a.CancelUpload(localpath)
return a.CancelUpload(remotePath)
}

// StartRepair - start repair files from path
Expand Down
20 changes: 14 additions & 6 deletions wasmsdk/blobber.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package main

import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
Expand Down Expand Up @@ -37,6 +38,15 @@ func listObjects(allocationID string, remotePath string, offset, pageLimit int)
return alloc.ListDir(remotePath, sdk.WithListRequestOffset(offset), sdk.WithListRequestPageLimit(pageLimit))
}

func cancelUpload(allocationID string, remotePath string) error {
allocationObj, err := getAllocation(allocationID)
if err != nil {
PrintError("Error fetching the allocation", err)
return err
}
return allocationObj.CancelUpload(remotePath)
}

func createDir(allocationID, remotePath string) error {
if len(allocationID) == 0 {
return RequiredArg("allocationID")
Expand Down Expand Up @@ -632,12 +642,10 @@ func multiUpload(jsonBulkUploadOptions string) (MultiUploadResult, error) {
RemotePath: fullRemotePath,
}
numBlocks := option.NumBlocks
if numBlocks < 1 {
if numBlocks <= 1 {
numBlocks = 100
}
if allocationObj.DataShards > 7 {
numBlocks = 75
}

options := []sdk.ChunkedUploadOption{
sdk.WithThumbnail(option.ThumbnailBytes.Buffer),
sdk.WithEncrypt(encrypt),
Expand Down Expand Up @@ -725,7 +733,7 @@ func uploadWithJsFuncs(allocationID, remotePath string, readChunkFuncName string
numBlocks = 50
}

ChunkedUpload, err := sdk.CreateChunkedUpload("/", allocationObj, fileMeta, fileReader, isUpdate, isRepair, webStreaming, zboxutil.NewConnectionId(),
ChunkedUpload, err := sdk.CreateChunkedUpload(context.TODO(), "/", allocationObj, fileMeta, fileReader, isUpdate, isRepair, webStreaming, zboxutil.NewConnectionId(),
sdk.WithThumbnail(thumbnailBytes),
sdk.WithEncrypt(encrypt),
sdk.WithStatusCallback(statusBar),
Expand Down Expand Up @@ -801,7 +809,7 @@ func upload(allocationID, remotePath string, fileBytes, thumbnailBytes []byte, w
numBlocks = 100
}

ChunkedUpload, err := sdk.CreateChunkedUpload("/", allocationObj, fileMeta, fileReader, isUpdate, isRepair, webStreaming,
ChunkedUpload, err := sdk.CreateChunkedUpload(context.TODO(), "/", allocationObj, fileMeta, fileReader, isUpdate, isRepair, webStreaming,
zboxutil.NewConnectionId(),
sdk.WithThumbnail(thumbnailBytes),
sdk.WithEncrypt(encrypt),
Expand Down
1 change: 1 addition & 0 deletions wasmsdk/proxy.go
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ func main() {
"searchcontainer": SearchContainer,
"updateForbidAllocation": UpdateForbidAllocation,
"send": send,
"cancelUpload": cancelUpload,

// player
"play": play,
Expand Down
26 changes: 21 additions & 5 deletions zboxcore/sdk/allocation.go
Original file line number Diff line number Diff line change
Expand Up @@ -655,7 +655,7 @@ func (a *Allocation) StartChunkedUpload(workdir, localPath string,

connectionId := zboxutil.NewConnectionId()
now := time.Now()
ChunkedUpload, err := CreateChunkedUpload(workdir,
ChunkedUpload, err := CreateChunkedUpload(a.ctx, workdir,
a, fileMeta, fileReader,
isUpdate, isRepair, webStreaming, connectionId,
options...)
Expand Down Expand Up @@ -815,7 +815,7 @@ func (a *Allocation) DoMultiOperation(operations []OperationRequest, opts ...Mul
mo.operationMask = zboxutil.NewUint128(0)
mo.maskMU = &sync.Mutex{}
mo.connectionID = connectionID
mo.ctx, mo.ctxCncl = context.WithCancel(a.ctx)
mo.ctx, mo.ctxCncl = context.WithCancelCause(a.ctx)
mo.Consensus = Consensus{
RWMutex: &sync.RWMutex{},
consensusThresh: a.consensusThreshold,
Expand Down Expand Up @@ -891,16 +891,23 @@ func (a *Allocation) DoMultiOperation(operations []OperationRequest, opts ...Mul
operation = NewMoveOperation(op.RemotePath, op.DestPath, mo.operationMask, mo.maskMU, mo.consensusThresh, mo.fullconsensus, mo.ctx)

case constants.FileOperationInsert:
operation, newConnectionID, err = NewUploadOperation(op.Workdir, mo.allocationObj, mo.connectionID, op.FileMeta, op.FileReader, false, op.IsWebstreaming, op.IsRepair, op.DownloadFile, op.Opts...)
cancelLock.Lock()
CancelOpCtx[op.FileMeta.RemotePath] = mo.ctxCncl
cancelLock.Unlock()
operation, newConnectionID, err = NewUploadOperation(mo.ctx, op.Workdir, mo.allocationObj, mo.connectionID, op.FileMeta, op.FileReader, false, op.IsWebstreaming, op.IsRepair, op.DownloadFile, op.Opts...)

case constants.FileOperationDelete:
if op.Mask != nil {
operation = NewDeleteOperation(op.RemotePath, *op.Mask, mo.maskMU, mo.consensusThresh, mo.fullconsensus, mo.ctx)
} else {
operation = NewDeleteOperation(op.RemotePath, mo.operationMask, mo.maskMU, mo.consensusThresh, mo.fullconsensus, mo.ctx)
}

case constants.FileOperationUpdate:
operation, newConnectionID, err = NewUploadOperation(op.Workdir, mo.allocationObj, mo.connectionID, op.FileMeta, op.FileReader, true, op.IsWebstreaming, op.IsRepair, op.DownloadFile, op.Opts...)
cancelLock.Lock()
CancelOpCtx[op.FileMeta.RemotePath] = mo.ctxCncl
cancelLock.Unlock()
operation, newConnectionID, err = NewUploadOperation(mo.ctx, op.Workdir, mo.allocationObj, mo.connectionID, op.FileMeta, op.FileReader, true, op.IsWebstreaming, op.IsRepair, op.DownloadFile, op.Opts...)

case constants.FileOperationCreateDir:
operation = NewDirOperation(op.RemotePath, mo.operationMask, mo.maskMU, mo.consensusThresh, mo.fullconsensus, mo.ctx)
Expand Down Expand Up @@ -933,6 +940,7 @@ func (a *Allocation) DoMultiOperation(operations []OperationRequest, opts ...Mul
if err != nil {
return err
}

mo.operations = nil
}
}
Expand Down Expand Up @@ -2252,7 +2260,15 @@ func (a *Allocation) RepairAlloc(statusCB StatusCallback) (err error) {
return a.StartRepair(dir, "/", statusCB)
}

func (a *Allocation) CancelUpload(localpath string) error {
func (a *Allocation) CancelUpload(remotePath string) error {
cancelLock.Lock()
cancelFunc, ok := CancelOpCtx[remotePath]
cancelLock.Unlock()
if !ok {
return errors.New("remote_path_not_found", "Invalid path. No upload in progress for the path "+remotePath)
} else {
cancelFunc(fmt.Errorf("upload canceled by user"))
}
return nil
}

Expand Down
16 changes: 9 additions & 7 deletions zboxcore/sdk/chunked_upload.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ var (

ErrInvalidChunkSize = errors.New("chunk: chunk size is too small. it must greater than 272 if file is uploaded with encryption")
ErrNoEnoughSpaceLeftInAllocation = errors.New("alloc: no enough space left in allocation")
CancelOpCtx = make(map[string]context.CancelCauseFunc)
cancelLock = &sync.Mutex{}
)

// DefaultChunkSize default chunk size for file and thumbnail
Expand Down Expand Up @@ -88,6 +90,7 @@ const (
*/

func CreateChunkedUpload(
ctx context.Context,
workdir string, allocationObj *Allocation,
fileMeta FileMeta, fileReader io.Reader,
isUpdate, isRepair bool,
Expand Down Expand Up @@ -172,7 +175,7 @@ func CreateChunkedUpload(
}

// su.ctx, su.ctxCncl = context.WithCancel(allocationObj.ctx)
su.ctx, su.ctxCncl = context.WithCancelCause(allocationObj.ctx)
su.ctx, su.ctxCncl = context.WithCancelCause(ctx)

if isUpdate {
su.httpMethod = http.MethodPut
Expand Down Expand Up @@ -411,14 +414,13 @@ func (su *ChunkedUpload) process() error {
if chunks.isFinal {
if su.fileMeta.ActualHash == "" {
su.fileMeta.ActualHash, err = su.chunkReader.GetFileHash()
}
if err != nil {
if su.statusCallback != nil {
su.statusCallback.Error(su.allocationObj.ID, su.fileMeta.RemotePath, su.opCode, err)
if err != nil {
if su.statusCallback != nil {
su.statusCallback.Error(su.allocationObj.ID, su.fileMeta.RemotePath, su.opCode, err)
}
return err
}
return err
}

if su.fileMeta.ActualSize == 0 {
su.fileMeta.ActualSize = su.progress.UploadLength
} else if su.fileMeta.ActualSize != su.progress.UploadLength && su.thumbnailBytes == nil {
Expand Down
2 changes: 1 addition & 1 deletion zboxcore/sdk/chunked_upload_bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ func BenchmarkChunkedUpload(b *testing.B) {
RemotePath: "/test.txt",
}

chunkedUpload, err := CreateChunkedUpload("/tmp", a, fileMeta, reader, false, false, false, zboxutil.NewConnectionId())
chunkedUpload, err := CreateChunkedUpload(a.ctx, "/tmp", a, fileMeta, reader, false, false, false, zboxutil.NewConnectionId())
if err != nil {
b.Fatal(err)
return
Expand Down
2 changes: 1 addition & 1 deletion zboxcore/sdk/live_upload.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ func (lu *LiveUpload) createClipsUpload(clipsIndex int, reader LiveUploadReader)
RemotePath: lu.liveMeta.RemotePath + "/" + reader.GetClipsFileName(clipsIndex),
}

return CreateChunkedUpload(lu.homedir, lu.allocationObj, fileMeta, reader, false, false, false, zboxutil.NewConnectionId(),
return CreateChunkedUpload(lu.allocationObj.ctx, lu.homedir, lu.allocationObj, fileMeta, reader, false, false, false, zboxutil.NewConnectionId(),
WithEncrypt(lu.encryptOnUpload),
WithStatusCallback(lu.statusCallback()))
}
6 changes: 3 additions & 3 deletions zboxcore/sdk/multi_operation_worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ type MultiOperation struct {
operations []Operationer
allocationObj *Allocation
ctx context.Context
ctxCncl context.CancelFunc
ctxCncl context.CancelCauseFunc
operationMask zboxutil.Uint128
maskMU *sync.Mutex
Consensus
Expand Down Expand Up @@ -164,7 +164,7 @@ func (mo *MultiOperation) Process() error {
mo.changes = make([][]allocationchange.AllocationChange, len(mo.operations))
ctx := mo.ctx
ctxCncl := mo.ctxCncl
defer ctxCncl()
defer ctxCncl(nil)
swg := sizedwaitgroup.New(BatchSize)
errsSlice := make([]error, len(mo.operations))
mo.operationMask = zboxutil.NewUint128(0)
Expand All @@ -185,7 +185,7 @@ func (mo *MultiOperation) Process() error {
if err != nil {
l.Logger.Error(err)
errsSlice[idx] = errors.New("", err.Error())
ctxCncl()
ctxCncl(err)
return
}
mo.maskMU.Lock()
Expand Down
11 changes: 9 additions & 2 deletions zboxcore/sdk/upload_worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package sdk

import (
"bytes"
"context"
"io"

thrown "github.com/0chain/errors"
Expand Down Expand Up @@ -114,6 +115,9 @@ func (uo *UploadOperation) Completed(allocObj *Allocation) {
if uo.chunkedUpload.progressStorer != nil {
uo.chunkedUpload.removeProgress()
}
cancelLock.Lock()
delete(CancelOpCtx, uo.chunkedUpload.fileMeta.RemotePath)
cancelLock.Unlock()
if uo.chunkedUpload.statusCallback != nil {
uo.chunkedUpload.statusCallback.Completed(allocObj.ID, uo.chunkedUpload.fileMeta.RemotePath, uo.chunkedUpload.fileMeta.RemoteName, uo.chunkedUpload.fileMeta.MimeType, int(uo.chunkedUpload.fileMeta.ActualSize), uo.opCode)
}
Expand All @@ -123,12 +127,15 @@ func (uo *UploadOperation) Error(allocObj *Allocation, consensus int, err error)
if uo.chunkedUpload.progressStorer != nil {
uo.chunkedUpload.removeProgress()
}
cancelLock.Lock()
delete(CancelOpCtx, uo.chunkedUpload.fileMeta.RemotePath)
cancelLock.Unlock()
if uo.chunkedUpload.statusCallback != nil {
uo.chunkedUpload.statusCallback.Error(allocObj.ID, uo.chunkedUpload.fileMeta.RemotePath, uo.opCode, err)
}
}

func NewUploadOperation(workdir string, allocObj *Allocation, connectionID string, fileMeta FileMeta, fileReader io.Reader, isUpdate, isWebstreaming, isRepair, isMemoryDownload bool, opts ...ChunkedUploadOption) (*UploadOperation, string, error) {
func NewUploadOperation(ctx context.Context, workdir string, allocObj *Allocation, connectionID string, fileMeta FileMeta, fileReader io.Reader, isUpdate, isWebstreaming, isRepair, isMemoryDownload bool, opts ...ChunkedUploadOption) (*UploadOperation, string, error) {
uo := &UploadOperation{}
if fileMeta.ActualSize == 0 {
byteReader := bytes.NewReader([]byte(
Expand All @@ -138,7 +145,7 @@ func NewUploadOperation(workdir string, allocObj *Allocation, connectionID strin
fileMeta.ActualSize = int64(len(emptyFileDataHash))
}

cu, err := CreateChunkedUpload(workdir, allocObj, fileMeta, fileReader, isUpdate, isRepair, isWebstreaming, connectionID, opts...)
cu, err := CreateChunkedUpload(ctx, workdir, allocObj, fileMeta, fileReader, isUpdate, isRepair, isWebstreaming, connectionID, opts...)
if err != nil {
return nil, "", err
}
Expand Down

0 comments on commit e7db14c

Please sign in to comment.