Added cloud storage utility to toolkit #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
leela
changed the title
leela
force-pushed
the
18-add-cloud-storage-utility
branch
3 times, most recently
from
47d571a to
22fd201
Compare
anandology
requested changes
Jul 16, 2020
toolkit/storage.py
Outdated
| collection = bucket.objects.filter(Prefix=self.path) | ||
| return [obj.key for obj in collection.all()] | ||
|
|
||
| def read_dataframe(self): |
There was a problem hiding this comment.
I guess it is better have separate read_csv and read_paquet methods.
toolkit/storage.py
Outdated
| return False | ||
| raise | ||
|
|
||
| def delete(self, del_dir: bool=False): |
There was a problem hiding this comment.
The del_dir doesn't sound right. See what is used in pathlib for this kind of operation.
| """Read the contents of a path | ||
| """ | ||
| obj = self._object | ||
| return obj.get()['Body'].read() |
There was a problem hiding this comment.
If there is a read_text, then there should be a read_bytes as well. What about write? Shouldn't we have write_text and write_bytes methods?
There was a problem hiding this comment.
Also. i guess, [Body].read() returns bytes, not text. Please check.
leela
force-pushed
the
18-add-cloud-storage-utility
branch
from
22fd201 to
1397967
Compare
|
@anandology , It is still needs a proper testing, test cases and exception handling, I am on it currently. Meanwhile you can have a look at it and please pass your comments. |
leela
force-pushed
the
18-add-cloud-storage-utility
branch
9 times, most recently
from
8c917db to
0f6e99f
Compare
leela
changed the title
leela
force-pushed
the
18-add-cloud-storage-utility
branch
7 times, most recently
from
1e59f1e to
c02c02d
Compare
leela
force-pushed
the
18-add-cloud-storage-utility
branch
21 times, most recently
from
78692a6 to
85b0858
Compare
leela
force-pushed
the
18-add-cloud-storage-utility
branch
from
85b0858 to
d60c137
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #18