document & diagram data pipeline

[ryantanaka]

Continuation from issue #2 of survey_display_local.

this was our previous understanding of our current data pipeline

I think it should look more like this after everything we've discussed.

[carlosparadis]

Thanks, @ryantanaka. I agree the too bad local statement was left shrouded in confusion and mystery.

[ryantanaka]

Yeah, glad we have a better understanding of how it works now.

Here is the xml file for the updated diagram.

data_pipeline.xml.zip

[carlosparadis]

Did you update this on the wiki?

[ryantanaka]

I don't have permission to edit the wiki. This was the suggestion I made so that I can add to the wiki. Not sure of another way to do it. Let me know what you think.

erdl/inactive-survey_display_template#4

[ryantanaka]

Got the invite, thanks. Can you enable the permission for survey_display_server too so I can add to this project's wiki?

[carlosparadis]

it's done

[ryantanaka]

@carlosviansi thanks!

[carlosparadis]

Just for self reference, a new diagram is in the making by @ryantanaka.

[ryantanaka]

current iteration of our understanding of the data pipeline

data_pipeline .xml.zip

Moving forward, I need to look at exactly which tables both apps are operating on and document it here.

[carlosparadis]

@ryantanaka As you walk through this, you may want to do that exercise we did to observe the code flow. You don't need to make anything formal like this. Just sketch on a piece of paper where the code starts and how it navigates across the files in Elm. There is something call sequence diagram but it is over kill for our needs.

I think the main processes are depicted on your other diagram, so we just need to get some understanding how it is done code-wise I guess. Which issue was that diagram going on again?

[ryantanaka]

@carlosviansi thanks i'll take a look at the sequence diagrams, and try an informal version of it.

and yup that is the right reference

[carlosparadis]

Haha this is more for the sake of instruction and to avoid it in case you see it. I think what we have is plenty. That level of detail in a diagram usually makes it get outdated way too fast.

I think as you try to solve that button order problem you will end up getting enough understanding of the elm files to the extent we need to know!

I think at some point we will want to replace the psycopg2 to a sqlalchemy interface to the database since it is easier to remember the data model too. But more on this later, I think the top priority now is those out of order buttons.

[ryantanaka]

@carlosviansi ok gotcha!

Alright, I'll be trying to get a copy of the database onto my computer first so i can set it up to mimic our currently deployed version on the IT-VM.

[carlosparadis]

I think this task is now mature enough to be complete. I will take over from here to move the diagram to a readme file later for the landing page of this project (and when I do I will close this issue). More on this when I get the higher priority issues out of the way. If you end up creating some sort of diagram beyond doodle you would like to share we can create a new issue for it.

[carlosparadis]

Pending moving the necessary information of this issue to the wiki! Re-assigned again to Ryan! Feel free to close after you move to the wiki :-)