-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split file each 4GB for BigQuery Quota Policy #6
Comments
I have encountered this problem. Caused by: org.jruby.exceptions.RaiseException: (Error) failed during waiting a Load job, get_job(myproject, embulk_load_job_513c2da9-2e73-498d-b57a-493ab53860af), errors:[{:reason=>"invalid", :message=>"Error while reading table: XXXX, error message: Input CSV files are not splittable and at least one of the files is larger than the maximum allowed size. Size is: 7505312411. Max allowed size is: 4294967296."}] |
Hello, @kosukekurimoto |
アドバイスをありがとうございます。私は該当のドキュメントを発見しました。 compression: NONEで再度トライしてみます。 |
BigQuery has following Quota Policy.
So, It's better to split output file each 4GB.
Without new-lines in strings: 5 TB
Problems
The text was updated successfully, but these errors were encountered: