need to document layout of scrape-util on IT-VM server

[ryantanaka]

In response to our confusion in #10, create a diagram similar to what was done for scrape-util here.

• show what symbolic links point to
• illustrate pulling data from various API/sensors, saving raw data on disk as csv's, then placement into postgres db
• show where scrape-util is called (/etc/cron.hourly)
• show what folders scrape-util uses
• keep xml file saved here

@carlosparadis, when you get the chance can you add me to this repo so i can assign this to me

[ryantanaka]

@carlosparadis, @jygh98 can you guys give some feedback? I think it's a little messy with all those arrows (but then again the program is a little messy too hah). Thanks

Scrape_Util_diagram.xml.zip

[ryantanaka]

12/19/17 conf call notes:

• divide current image in 3 blocks
  [12:35]
  1st block) database+io folders (call it io folders if you like, no need to map every single godamn folder)
  [12:35]
  2nd block) src code. It is the one already there with the chain of soft links. just draw a rectangle or something on it
  [12:36]
  3rd block) it is not included currently in the image. add some folder icon saying configs. No need to also look into all possible files, its more as a reminder in this figure it is a big chunk of concern in scrape-util
  [12:37]
  4th) make sure to check that the folder where the hobo, egauge, webctrl icons point right now, shouldnt actually be the scrape util file, as i believe that contrary to survey display the order is first scrape util, and the output is just as a memory of the operation when specified on the config file, rather than how survey display does which first saves the file to then hope for scrape util to stop by
  [12:38]
  5th) maybe we want to revisit given the new diagram how we point its 3 portions in the terminal figures and how this mess is organized currently (or maybe we dont, after we change this, we likely will want to keep the figure of the old way apart as memory, and keep the new simplified version together in this diagram)

referring to the following diagram:

I'll be updating the diagram based on what was discussed above.

[ryantanaka]

update 12/21:

3 main blocks included:

• scrape-util source
• outputs
• configs

APIs also point to where they should.

I'm going to get scrape-util running on my computer so that I can do some debugging (will just use fake data, fake sensors, whatever). A handful of the code is understandable but there are some areas where terrible naming conventions, overuse of every possible fancy python capability imaginable, and not so descriptive comments make it impossible unless i have print statements telling me whats going on.

Here is the updated diagram:

Scrape_Util_diagram (2).xml.zip

[ryantanaka]

conf. call 12/22 notes:

• split diagram into two separate images (top vs bottom)
• as of Dec 19, 2017 we started using 5 minute intervals for retrieving data instead of 1 hour (will post this on the diagram)
• regarding toml files
  • need to express that they can show up in different forms (for example: egauge.toml, psql-config.toml, value-remap.toml, etc..)
  • need to express that they can refer to other toml files
  • for project-nameN, show how HOBO data can be manually inserted
  • for project-name2, add more detail to the config.toml files
    • show that there can be two types of errors
    • for example: /var/local/scrape-util/errors/hobo-u12 and /var/local/scrape-util/hobo-u12/errors

[ryantanaka]

updated 12/27:

Scrape_Util_diagram (3).xml.zip

• still need to split images
• still need to document start date of 5 min cronjob intervals

[ryantanaka]

scrape-util layout on server:

scrape-util diagram:

Scrape_Util_diagram (4).xml.zip

[carlosparadis]

For the record, @kathrynparadis will be adding some of the diagram on item 3 for the HOBO on this issue.

[ryantanaka]

Updated:

Scrape_Util_diagram.xml.zip

[ryantanaka]

This is done. Diagrams are located in the wiki. Closing this issue.