S3 protocol #895
S3 protocol #895
Conversation
|
Thanks, @LuisSanchez25. Would it be more appropriate to put these new |
|
And I generally think this is a super good idea! We should benefit from the market resources! |
|
Hey @dachengx I am a bit unfamiliar with why the decision was made to put the Rucio frontend in straxen rather than in strax, that feels like a tool that would be beneficial for others outside of XENON right? So to me it might make more sense to move the Rucio storage to straxen but maybe I am just missing something. Thanks! I am still in the process of fully testing this, I think it still needs some tweaks but I should have a working prototype soon! |
|
@LuisSanchez25 I think by design, strax is for the prototypes of all classes, like plugins and storage. straxen will inherit the classes and make the functions more specific. I think this is why they put the |
There was a problem hiding this comment.
Thanks @LuisSanchez25 . I did not look into s3.py deeply because I see there are commented-out codes so maybe it is not finished?
I still would insist on moving the S3 functionality to straxen. strax should be just basic usage and processor etc.
You should not change save_file but do what should be done only in _save_chunk function.
| result = _save_file(write_file, data, compressor) | ||
| os.rename(temp_fn, final_fn) | ||
| return result | ||
| if is_s3_path is False: |
There was a problem hiding this comment.
if is_s3_path is False: -> if not is_s3_path:
What is the problem / what does the code in this PR do
Adds S3 functionality to strax
Can you briefly describe how it works?
We can now use an S3 storage system to store data
Can you give a minimal working example (or illustrate with a figure)?
import straxst = strax.Context()s3_storage = strax.S3Frontent()st.storage = [s3_storage]Please include the following if applicable:
Please make sure that all automated tests have passed before asking for a review (you can save the PR as a draft otherwise).