Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEAT: Publish Glides Events to Tableau #488

Closed
wants to merge 57 commits into from
Closed

FEAT: Publish Glides Events to Tableau #488

wants to merge 57 commits into from

Conversation

huangh
Copy link
Contributor

@huangh huangh commented Mar 18, 2025

<Asana Task>

  • publish glides trips_updated and operator_signed_in data to tableau
  • convert all datetime/times to EST in tableau job

huangh added 30 commits March 7, 2025 09:50
@huangh
add devgreen to configs to handle devgreen trip_updates and vehicle_p…
8194c1d 
…ositions
@huangh
lower number of workers from 4 to 3 to see if memory pressure decreases
e4d3307
@huangh
black lint
def1bd4
@huangh
single process runner switch implemented for ingestion debugging. mod…
d04328e 
…ified yield_check to yield at max total nbytes of table rather than a number of rows. added config argument to pass through multi-process flag and configuration flags
@huangh
initialize and pass through config from pipeline.py
27dc6b8
@huangh
entry point for poetry run ingestion is at start...not main. add conf…
fa32a32 
…ig to start
@huangh
black lint
a0154a9
@huangh
3gb per yield blows the memory usage...even though by my math, 3*4 < …
2224cbd 
…30gb. but i'm no expert. going down to 1 to test
@huangh
lowering to 100mb to test. cleanup debugging messages
e5a7c8d
@huangh
sum up nbytes for each file read to compare with concat version - yie…
c452a48 
…ld on sum instead of concat
@huangh
add default value
fc57b17
@huangh
standalone downloader for whole days worth of ingestion files
a68aa14
@huangh
print rtdata.nbytes, not the table object itself
99eeede
@huangh
deduplicate code between single and multi-threaded paths in gtfs_rt c…
875c6d4 
…onversion
@huangh
add runners, profiling, investigation files - WIP
917f0fc
@huangh
set maxtasksperchild in Pool configuration. remove processes=count to…
1ea5791 
… let the Pool default to num_cpu()
@huangh
standalone runner updates
b7965f3
@huangh
add memray to pyproject.toml - doesn't seem to be successfully instal…
a6e8f49 
…led via poetry - only pip. WIP
@huangh
poetry lock by 1.7.1
dd48bf5
@huangh
set number of converters again. remove the sort on the whole table wh…
18696da 
…ich is unnecessary - only to save pq file size
@huangh
test maxtasksperchild when spawning processes. only use once, so one …
356ff59 
…process spawned per gtfs-rt converter type
@huangh
limit processpool to 4 (ncpu) even with more than 4 converters
7a857a5
@huangh
add test for write_local_pq
d1e70e6
@huangh
updates to runners/investigation analaysis scripts
099817f
@huangh
minimal changes for converting to_table to to_batches
648b3b0
@huangh
add iteration counter logging
c97ebc1
@huangh
push lock file
63cc9b7
@huangh
properly increment counter
eb5cb01
@huangh
lower batch size - this should constrain usage immediately
dfba54f
@huangh
add process memory pct to default logging
11ed9ad
huangh and others added 27 commits March 13, 2025 00:08
@huangh
set readahead values for to_batches to limit memory usage
2823410
@huangh
convert each process mem to used_mem
88bf19b
@huangh
yield check to 16mb per yield - process logger update naming of used …
ea351a0 
…mem field
@huangh
lower max_tables_to_convert 15->10, from observing memory usage of BU…
70800ea 
…S_VEHICLE_POSITIONS max out at 30%
@huangh
black
16ff41d
@huangh
add max memory line in gtfs_rt convert to break out if memory usage e…
313bb1b 
…xceeds threshold
@huangh
rename logging to make more clear
cbf8a8e
@huangh
update file prefix temporarily to only process March /03/ files to se…
cb99b1b 
…e if rowgroups is causing issues
@huangh
add convert between bytes and nrows to satisfy rowgroup heuristics. u…
c31112c 
…pdating yield size to 128mb
@huangh
calculate rowgroups properly
433d215
@huangh
test removing max process mem usage - this should not matter
60a43db
@huangh
try to gc after yield. clean up process logger output - not passing p…
aa9aa5d 
…rocess logger around anymore
@huangh
reenable max_mem_usage_per_process hatch in convert() - set threshold…
395e9f1 
… to 20%
@huangh
update runner
262bf2e
@huangh
add tracemalloc debugging
567776f
@huangh
cherry pick e64047f - convert from dataset to ParquetFile because of …
9864b32 
…discovered memory leak in pyarrow - apache/arrow#39808
@rymarczy @huangh
fix wrong schema type
af0f4bb
@rymarczy @huangh
fix ParquteFile schema
c12003b
@huangh
remove tracemalloc debug
02b3c4a
@huangh
write glides trip updates and operator sign ins to tableau - first cut
e373c5b
@huangh
the tableau schema differs where all strings -> large_strings due to …
2c546a3 
…required conversions in polars. update schema to reflect this
@huangh
add glides jobs to hyper_updates() call
e130995
@huangh
correct naming of trip_updates and operator_signins to match exactly …
f95f7d4 
…with the glides schema names
@huangh
tableau job convert all dates and times to EST - datatype from str ->…
4d04336 
… date objects
@huangh
black
73575c2
@huangh
fix schema comments and datatypes
ebdc2b0
@huangh
black
798bc5e
@huangh huangh closed this Mar 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@huangh @rymarczy