adding new_pathWrite() as per Reid suggestion

[petersilva]

I worked on this last week and got distracted by ddsr-dev replacing the gswob feed to the datamart. This is kind of the complement to getContent() that is used to write local files and return a well-formed message for it.

It should be helpful in callbacks that transform data. I don't know when I will get back to this either.

[github-actions]

Test Results

244 tests   235 ✅  17s ⏱️
  1 suites    8 💤
  1 files      1 ❌

For more details on these failures, see this check.

Results for commit c5a8d8b.

♻️ This comment has been updated with latest results.

[petersilva]

@reidsunderland worry:

• if message has inlined file data... what happens?
  • current implementation ignores it. result if present... new data will be over-written by message data if download==True.

what is a good behaviour?

• if data==None

  • use data from inline, write to disk.

• if data!=None

• consider this an override for inline data...

• look at options...

  • if inlining is active, then update the inline field, in addition to writing the data.
  • if ! inlining, then remove the field.

• if download==True... potential for double file write.

  • perhaps not because mtime & checksum....

For converter plugins in general, all the options feel weird...

• shovel(flowcb/filter/wmo2msc)

  • change line termination,
  • changes file tree.
  • shovels feel weird because they are supposed to be about messages... not data.
  • renamer is natural

• sarra

  • feel less weird than shovels... not convinced they are wrong...
  • renamers are natural here also.

• sender (flowcb/filter/wmo00 )

  • wmo00_accumulate makes sense... but is it a general thing, or it just makes sense in that case?
  • renamers are natural here also.

• flow ... should transformers use a flow to denote that they aren't doing what any standard components do.

• transform/xform/convert/fx/ ... should we establish another type of flow for conversion?

  • just a sarra with download off, because we will be transforming data.
  • this makes it identical to a shovel... ;-)

This is essentially re-inventing the discussion in #494... and we the hierarchy makes sense again...

for now, conclusion is to use flow/cvt_ to represent transformers.

Also need to set msg['size'] = len(data)

[petersilva]

last commit included "size" support as well. so I guess it's good to go.

[petersilva]

I'm still puzzled as to why I haven't figured out how to leverage Transfer classes for this stuff, or have code from transfer moved here... or... Transfer/_init_.py there is read_writelocal which is exactly the same logic... but a bit more complete.