feat: Remake the Scraper in TS. -- Does not touch the UI. #89
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…MrFrick/dbs-scraper into feature/Remake-to-TS-Version
TheMrFrick
changed the title
Xotl
requested changes
Aug 1, 2022
There was a problem hiding this comment.
Hi @TheMrFrick... Thanks a lot for your contribution.
I would like to merge this but seems that there're lots of files that ends on a Windows style. Can you please change those in order to end with a new line. Here's some context of the importance on doing so: https://stackoverflow.com/a/729795
| *.xlsx | ||
| reworkedCards.json | ||
| reworkedCards.json_* |
There was a problem hiding this comment.
Suggested change
| reworkedCards.json_* | |
| reworkedCards.json_* | |
| { | ||
| "docwriter.progress.trackFunctions": false, | ||
| "docwriter.progress.trackMethods": false | ||
| } |
| } | ||
|
|
||
| return result; | ||
| } |
| era: find('dl.eraCol dd').text().trim(), | ||
| cardImageUrl: getImageUrl(find('.cardimg > img')).trim(), | ||
| }; | ||
| } |
| function extractCategoryFromHref(hrefString: string): string | undefined { | ||
| const match = hrefString.match(HREF_REGEXP); | ||
| return match?.groups?.category; | ||
| } |
|
|
||
| } | ||
| console.log('programming is running'); | ||
| main(); |
There was a problem hiding this comment.
Suggested change
| main(); | |
| main(); | |
| // console.log(cards); | ||
| console.log(`${cards.length} cards found!`) | ||
| } | ||
| testCardScraper(); |
There was a problem hiding this comment.
Suggested change
| testCardScraper(); | |
| testCardScraper(); | |
| writeFileSync(DEFAULT_OUTPUT, JSON.stringify({...(cardListJSON as any)})); | ||
| } | ||
|
|
||
| main(); |
There was a problem hiding this comment.
Suggested change
| main(); | |
| main(); | |
| } | ||
| } | ||
|
|
||
| main(); |
There was a problem hiding this comment.
Suggested change
| main(); | |
| main(); | |
| } | ||
|
|
||
| // promptUser('test'); | ||
| main(); |
There was a problem hiding this comment.
Suggested change
| main(); | |
| main(); | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's the purpose of this PR?
Fixes and remakes the scraper in Typescript.
Added additional dev_utils to read an
.xlsxfile of your own collection split into sheets of colors and adds the card price and the card number.What fixes?
How can this be tested?
No tests were written.
All testing can be done by
npm ciin the subdirectory to get the npm_modules.Afterwards you can run the
npm run startscript to test the web scraper.