{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":615811805,"defaultBranch":"main","name":"dietgpu","ownerLogin":"terrelln","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-03-18T18:43:03.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/6619134?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1679182548.0","currentOid":""},"activityList":{"items":[{"before":"2a29f7b0fee81045ccef7eddacbfd8287cd694d7","after":"637fec544ecf9785766e4871ce48e846820a747c","ref":"refs/heads/dietcpu","pushedAt":"2023-03-19T22:22:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"Add readme + fse benchmarking","shortMessageHtmlLink":"Add readme + fse benchmarking"}},{"before":"4089a82432e13d31566dca292e176d0ab4cc3ab7","after":"2a29f7b0fee81045ccef7eddacbfd8287cd694d7","ref":"refs/heads/dietcpu","pushedAt":"2023-03-19T21:15:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"Add specialized variant for maxSymbol < 16 that is ~15% faster","shortMessageHtmlLink":"Add specialized variant for maxSymbol &lt; 16 that is ~15% faster"}},{"before":"ff8b6102dc187826668178485659d7a7f746d3b2","after":"4089a82432e13d31566dca292e176d0ab4cc3ab7","ref":"refs/heads/dietcpu","pushedAt":"2023-03-19T20:26:34.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"add some notes","shortMessageHtmlLink":"add some notes"}},{"before":"d9bcead32268ec88d762460f27f390314b3cefd9","after":"ff8b6102dc187826668178485659d7a7f746d3b2","ref":"refs/heads/dietcpu","pushedAt":"2023-03-19T20:20:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"encoder is working","shortMessageHtmlLink":"encoder is working"}},{"before":"eb9fa8c68f7199606cc9404de4c612f0a6440549","after":"d9bcead32268ec88d762460f27f390314b3cefd9","ref":"refs/heads/dietcpu","pushedAt":"2023-03-19T19:55:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"working","shortMessageHtmlLink":"working"}},{"before":"031e7f10364cc3cd97d74e02c45531b548db4012","after":"eb9fa8c68f7199606cc9404de4c612f0a6440549","ref":"refs/heads/dietcpu","pushedAt":"2023-03-19T06:35:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"Add encoder, but it doesn't work yet","shortMessageHtmlLink":"Add encoder, but it doesn't work yet"}},{"before":"71d052eb24fa38a72807d8a96c030ecd16993059","after":"031e7f10364cc3cd97d74e02c45531b548db4012","ref":"refs/heads/dietcpu","pushedAt":"2023-03-19T00:31:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"cleanup","shortMessageHtmlLink":"cleanup"}},{"before":"0e2ccbac4c79aeae25684816c855e85244ad1084","after":"71d052eb24fa38a72807d8a96c030ecd16993059","ref":"refs/heads/dietcpu","pushedAt":"2023-03-18T23:36:14.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"Add dietcpu decoder + sample data","shortMessageHtmlLink":"Add dietcpu decoder + sample data"}},{"before":null,"after":"0e2ccbac4c79aeae25684816c855e85244ad1084","ref":"refs/heads/dietcpu","pushedAt":"2023-03-18T23:35:48.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"dietgpu: add optional checksum and version number information (#16)\n\nSummary:\nPull Request resolved: https://github.com/facebookresearch/dietgpu/pull/16\n\nThis diff adds two useful features for using DietGPU to store numeric data in persistent storage (e.g., for checkpointing large language model training purposes).\n\n**1** This diff adds an optional feature to calculate a checksum of the data to be compressed which is stored in the archive, as well as an option to check the stored checksum against the decompressed data post-decompression. This is an additional check to safeguard the archival integrity of DietGPU.\n\nAll compress/decompress functions add an additional optional parameter to enable the checksum feature, which is the first optional parameter for all the functions.\n\nThere are two checksum flags, one for byte-wise ANS and one for the float compression. With float compression, the ANS flag cannot be used.\n\nComputing and comparing the checksum adds another full pass to the data, so it is not recommended to use it for online, in-memory compression/decompression (e.g., shipping data over a network).\n\n**2** A version number has been added to both the ANS and float data headers inside the archive. This will be used to safeguard against data format changes in subsequent iterations of DietGPU, in order to detect when a newer (or older) version of the library is being used to unpack an older (or newer, respectively) versioned archive.\n\nA version mismatch will produce a CUDA kernel assert (as this is checked for all usages, especially with asynchronous ones where we wish to have as little d2h/h2d involvement with the CPU as possible), while a checksum mismatch error will itself be returned back to the CPU synchronously as an error status (C++), or from Python, will be thrown to Python as an error via `TORCH_CHECK`.\n\nAlso added a missing check that the float type in the archive expected upon decompression matches what is stored in the archive.\n\nIn order to add this feature, the data layout of the DietGPU archive has changed to include space for the checksum in the headers.\n\nIssues / to do:\n- the checksum at present is only 8 bits due to a (lack of) data alignment expectation, and is just a simple xor reduction at present. The ANS compressor can accept any input regardless of alignment (only byte alignment is required), while the float compressor expects float word-sized alignment (e.g., float16 requires 2 byte alignment). The checksum kernel ideally should compute a full uint32 checksum (if xor), or just use something like crc32.\n\nReviewed By: bwasti\n\nDifferential Revision: D41829599\n\nfbshipit-source-id: c265eaa6efa39a3a94a4cc0e03714d994ab0ed5f","shortMessageHtmlLink":"dietgpu: add optional checksum and version number information (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1485877645\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookresearch/dietgpu/issues/16\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookresearch/dietgpu/pull/16/hovercard\" href=\"https://github.com/facebookresearch/dietgpu/pull/16\">facebo…</a>"}},{"before":"f10bd84b854f253749899ceadeb37002e4aced22","after":"7f49d339af0d5606804ec9ce87fe9ad0f39e3bb1","ref":"refs/heads/main","pushedAt":"2023-03-18T19:53:13.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"working on a test","shortMessageHtmlLink":"working on a test"}},{"before":"487475582f150fdbfcc75751c09949a3ac90b4d4","after":"f10bd84b854f253749899ceadeb37002e4aced22","ref":"refs/heads/main","pushedAt":"2023-03-18T19:52:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"working on a test","shortMessageHtmlLink":"working on a test"}},{"before":"35f9d01586466cde77acb808506eaf81222dcec6","after":"487475582f150fdbfcc75751c09949a3ac90b4d4","ref":"refs/heads/main","pushedAt":"2023-03-18T19:51:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"working on a test","shortMessageHtmlLink":"working on a test"}},{"before":"26c9b14c6d36e6f77227edf9aec9339d398ae010","after":"35f9d01586466cde77acb808506eaf81222dcec6","ref":"refs/heads/main","pushedAt":"2023-03-18T19:50:53.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"working on a test","shortMessageHtmlLink":"working on a test"}},{"before":"63bc2193d4a5fa1da75f356c19b70096d6954076","after":"26c9b14c6d36e6f77227edf9aec9339d398ae010","ref":"refs/heads/main","pushedAt":"2023-03-18T19:26:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"working on a test","shortMessageHtmlLink":"working on a test"}},{"before":"6fc1ba50d826579944a49042ef5b22128f7021d9","after":"63bc2193d4a5fa1da75f356c19b70096d6954076","ref":"refs/heads/main","pushedAt":"2023-03-18T19:03:41.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"Try fix cmake","shortMessageHtmlLink":"Try fix cmake"}},{"before":"89c375f8d6f1ed4b612a2a70b57b04864dec7849","after":"6fc1ba50d826579944a49042ef5b22128f7021d9","ref":"refs/heads/main","pushedAt":"2023-03-18T18:57:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"Merge pull request #1 from RPrenger/fix_pytorch_tests\n\nChanges to make the PyTorch tests run","shortMessageHtmlLink":"Merge pull request <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1630510335\" data-permission-text=\"Title is private\" data-url=\"https://github.com/terrelln/dietgpu/issues/1\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/terrelln/dietgpu/pull/1/hovercard\" href=\"https://github.com/terrelln/dietgpu/pull/1\">#1</a> from RPrenger/fix_pytorch_tests"}},{"before":"0e2ccbac4c79aeae25684816c855e85244ad1084","after":"89c375f8d6f1ed4b612a2a70b57b04864dec7849","ref":"refs/heads/main","pushedAt":"2023-03-18T18:43:44.000Z","pushType":"pr_merge","commitsCount":2,"pusher":{"login":"terrelln","name":"Nick Terrell","path":"/terrelln","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6619134?s=80&v=4"},"commit":{"message":"Merge pull request #1 from RPrenger/fix_pytorch_tests\n\nChanges to make the PyTorch tests run","shortMessageHtmlLink":"Merge pull request <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1630510335\" data-permission-text=\"Title is private\" data-url=\"https://github.com/terrelln/dietgpu/issues/1\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/terrelln/dietgpu/pull/1/hovercard\" href=\"https://github.com/terrelln/dietgpu/pull/1\">#1</a> from RPrenger/fix_pytorch_tests"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyMy0wMy0xOVQyMjoyMjo1MC4wMDAwMDBazwAAAAMGvpIS","startCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wMy0xOVQyMjoyMjo1MC4wMDAwMDBazwAAAAMGvpIS","endCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wMy0xOFQxODo0Mzo0NC4wMDAwMDBazwAAAAMGU6W6"}},"title":"Activity · terrelln/dietgpu"}