Skip to content
This repository was archived by the owner on Sep 29, 2023. It is now read-only.

RS-5211 asset importr - handle 200k row jobs #3

Merged
merged 6 commits into from
May 7, 2018

Conversation

stevenjonescgm
Copy link
Contributor

  • debugging improvements
  • fix some error handling
  • DRY resource processors
  • reduce processing and memory overhead by iterating the JSON hash without duplicating it
  • early quit after 15 errors for quick feedback
  • It seems ActiveRecord was scanning import.data on every import.save even when not changed. using update_columns instead for saving job progress saved ~80% of processing time when importing 40,000 assets (in development environment)

Copy link
Contributor

@mathmatomancer mathmatomancer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, I'm assuming the red in github's view for the bower.json is because it doesn't know what to do with it, not because it's an error?

@stevenjonescgm stevenjonescgm merged commit 25061e0 into master May 7, 2018
@stevenjonescgm stevenjonescgm deleted the RS-5211-asset-importr branch May 7, 2018 16:59
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants