Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Github: Timeout-problem importing a big repository #60

Open
01-Scripts opened this issue May 5, 2013 · 12 comments
Open

Github: Timeout-problem importing a big repository #60

01-Scripts opened this issue May 5, 2013 · 12 comments

Comments

@01-Scripts
Copy link
Contributor

Hello,

I'm facing a problem when I try to import a kind of big repository for the first time with the source-integration-plugin for Github.
I've just installed the newest version of mantis, applied the patch to get it working together with SourceGithub and tried to do a complete import of one of my repositories.

But after around 60s my provider ends the process.
This leads in some cases to a new project enry named "Imported ....current date&time". And in some other cases (if I only import som special branches), the commits of these branches are imported but now connection between the issues and the commits are made.

Any idea how I can solve that problem for me? Is it possible to break the import process into smaller pieces?

Regards,
Michael

@amyreese
Copy link
Contributor

amyreese commented May 5, 2013

If the import times out, you should be able to continue the import using the "import latest option", where it will start importing where it left off before the timeout. The "Imported " is caused by a failed "import all", which is a poorly-named way to re-import the entirety of the repository, which will just have the same timeout issues. Import latest will not cause this though, so it might be tedious, but it will eventually get all of the changesets.

Of course, the alternate solution is to temporarily remove the PHP execution timeout, perform the import, and then reinstate the timeout, but import latest will work regardless.

@morgant
Copy link
Collaborator

morgant commented May 5, 2013

Additionally, you can try using the Source plugin's import.php page which will use the long_process_timeout from your Mantis config file (the default value of '0' means that it will never time out). This is what I do when running via cron to automate imports. I can give further details tomorrow, if you'd like.

@01-Scripts
Copy link
Contributor Author

Thank you for your hints. I will try using the "Import latest" option this evening and come back to you when I got some results.
Thank you so far.

@01-Scripts
Copy link
Contributor Author

Using the "import latest" option I was able to import all commits of my repository. But there are no references between the commits and the issues. Although I mentioned all bug ids in the commit message. Any chance to manually parse the commit messages to create the references?

@01-Scripts
Copy link
Contributor Author

Issue closed.
I wrote a small function to manually reparse all my entries in my database and create the proper connections between bugid and changeid.
For every connection there has to be an entry in the 'mantis_plugin_Source_bug_table'-table.

@dregad
Copy link
Member

dregad commented May 30, 2013

If you don't mind sharing, you might want to make your code available (e.g. as a gist) as it may be useful to others, and potentially something that could be integrated into the plugin itself

@01-Scripts
Copy link
Contributor Author

For sure.
Here it is:
https://gist.github.com/01-Scripts/5731171
Just quick and dirty but it worked for me like a charm.

@sverhagen
Copy link

FYI, the $g_long_process_timeout seems to affect a part of the process, and not the process as a whole. When setting it to e.g. "10" (seconds implied) it takes a minute or so (>>10s) to fail:

Fatal error: Maximum execution time of 10 seconds exceeded in .../mantis/plugins/SourceBitBucket/SourceBitBucket.php on line 397

Setting it to 0 does not cause the above error anymore, but still runs me into the (as @01-Scripts described it) "my provider ends the process", or in other words:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, [email protected] and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

As for the script in the gist, that's great, thank you for that! One remark, my database might be case-sensitive (MySQL is a bit wonky in that area), unlike yours, and "mantis_plugin_source_changeset_table" needed to be "mantis_plugin_Source_changeset_table" for me to work.

@sverhagen
Copy link

Here's another approach that I'm tinkering with:

  • Add $t_repo->save(); to plugins/Source/pages/repo_import_latest.php (near the end, where it says post_stats). How else will my "Import Latest Data" ever save it's data?
  • That probably only works for an "Import Latest Data" at the tail of things. Intermediate ones will never reach the save() then, due to the timeout. So my guess is that only the last run of "Import Latest Data" that you do (and that presumably doesn't time out, if it's small enough) gets to this point
  • If we were to fix that, that probably only works for the "Import Latest Data", and not for whatever is imported during the original "Import Everything", that also broke due to the timeout.
  • Talking to my provider to find out more about the (for me) two minutes timeout.

After I deleted a repository, which should clean up the database, should it not, I ended up still seeing a lot of data in the "mantis_plugin_Source_bug_table". I'm afraid that's somehow to blame on the script provided in this thread. It would be better to just fix the plugin 😄

@01-Scripts
Copy link
Contributor Author

Because of some new input to this issue (an because I run into the same timeout problem from time to time) I repoen the issue. Maybe we can fix that generally.

@01-Scripts 01-Scripts reopened this Aug 3, 2015
@luzpaz
Copy link

luzpaz commented Dec 7, 2015

we tried to get source integration working for Scribus bugs.scribus.net without success as well.

@luzpaz
Copy link

luzpaz commented Jun 28, 2017

#92

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants