Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Date Sort & Filter #6

Open
wants to merge 61 commits into
base: main
Choose a base branch
from
Open

Date Sort & Filter #6

wants to merge 61 commits into from

Conversation

cmbirk
Copy link
Member

@cmbirk cmbirk commented Sep 6, 2022

Adds the following to each scraper:

  • Pulls the date / datetime for all articles that have them available
  • Sorts the articles by the date / datetime
  • Filters articles to only provide articles within the last 2 days

cmbirk added 30 commits June 27, 2022 13:43
@cmbirk
initialize with yarn berry
98b0b30
@cmbirk
add puppeteer / functions-framework
d823a27
@cmbirk
add more dependencies
622159e
@cmbirk
track large chromium file with git-lfs
24ed87b
@cmbirk
remove .yarn/unplugged
3098ee7
@cmbirk
re-add yarnrc/unplugged as git lfs
fca284c
@cmbirk
update lfs, add ts config
0688422
@cmbirk
start news scraper
e2083e4
@cmbirk
install yarn sdks
0ec6a2a
@cmbirk
update cache
208f7d4
@cmbirk
build first 2 scrapers
35c7c14
@cmbirk
add business nc scraper
74eb199
@cmbirk
update dependencies
a3e040c
@cmbirk
tweak business scraper
3625f91
@cmbirk
remove git lfs entries
d25eaac
@cmbirk
remove mac chromium from git
07346a0
@cmbirk
remove console statement
b42ae2c
@cmbirk
finish triangle business journal scraper
f434fc7
@cmbirk
build ncPolicyWatch scraper
4f290fd
@cmbirk
return at end of nc policy watch
0752b9d
@cmbirk
combine functions
7feec62
@cmbirk
update triangle busines journal, carolina journal scrapers
b3e117e
@cmbirk
add more dependencies
4d915bc
@cmbirk
type scrapers correctly
dbd6bbb
@cmbirk
connect scrapers to email sends
4168cd0
@cmbirk
attempting to feed all links to transaction send
1c0ae90
@cmbirk
finish email testing
d412148
@cmbirk
add google cloud packages
ced1093
@cmbirk
add gcloud secret manager
90219d6
@cmbirk
set up function for gcloud deployments and deploy
0dd9d10
@cmbirk
Merge pull request #1 from CrossroadsCX/feature/scrapers-update
69a3b8d 
Feature/scrapers update
@cmbirk
add workspace-tools yarn plugin
32af482
@cmbirk
plug datastore into sendEmail helper
55d27a9
@cmbirk
Merge pull request #2 from CrossroadsCX/feature/datastore
48ad09a 
Pull email list from GCP Datastore
@cmbirk
add email utility scripts, add import function to datastore module
cf47b9f
@cmbirk
wrap scrapers in try/catch block in case one fails. Return empty arra…
6d81d98 
…y in that case
@cmbirk
Merge pull request #3 from CrossroadsCX/feature/datastore
1c3aab2 
Datastore / Resiliency
@cmbirk
add try / catch block around email send, add additional logging
ee3e30c
@cmbirk
check bcc length and log it
5d2061e
@cmbirk
remove test email
87e6f4a
@cmbirk
allow errors in sendEmail function
db43eef
@cmbirk
remove test email
7577689
@cmbirk
build functions
35b73a5
@cmbirk
Merge pull request #4 from CrossroadsCX/feature/more-error-checking
19a4176 
Feature/more error checking
@cmbirk
finish campaign-style api calls for wotd emails
c611f43
@cmbirk
add winston and winston slack transport
3876e09
@cmbirk
set correct list id on campaign creation
72f10f3
@cmbirk
clean up newsScraper function
9410ad5
@cmbirk
build cloud functions
e7dfd23
@cmbirk
Merge pull request #5 from CrossroadsCX/feature/create-campaign
d5215d0 
Create Campaigns
@cmbirk
update most scrapers, still a couple to finish
8c0f19a
@cmbirk
update rest of scrapers for date sorting. Still need last 2 days comp…
43933ec 
…arisons
@cmbirk
comment unreachable code
409a5a6
@cmbirk
finish date filtering for all scrapers
09e9d3b
@cmbirk
catch errors in date api
d33d3fe
@cmbirk
change node version, update puppeteer
4e861f2
@cmbirk
fix newsAndObserver time change
9045d2b
@cmbirk
add Chromium to lfs
c91f9eb
@cmbirk
actually add Chromium Framework to git lfs
c64586a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@cmbirk