Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't allow multiple initdb processes on OSX. #42

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jameshilliard
Copy link

@jameshilliard jameshilliard commented Sep 17, 2020

Fixes #157:

[initdb:pid(95804)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 2020-09-17 12:59:51.996 MDT [95916] DETAIL:  Failed system call was shmget(key=5432002, size=56, 03600).

@mdavydau You might be interested in this.

This issue only seems to appear when running parallel tests where multiple initdb processes could be active.

From my understanding the reason this error happens is because initdb does shared memory allocation tests to determine the appropriate value for the shared_buffers config option, however on OSX these tests themselves allocate a high enough percentage of the total default usable system shared memory that having more than one initdb process results in a very high likelihood of failure due to the processes allocating too much memory for concurrent shared memory tests.

@tomix26
Copy link
Collaborator

tomix26 commented Oct 12, 2020

I encountered a similar problem that was caused by the fact that postgres processes were not terminated properly if a fatal error occurred during the test execution. Otherwise, everything works fine for me, even when I am running multiple tests at once. So could you please check the postgres processes?

There is a related issue: zonkyio/embedded-database-spring-test#105

@jameshilliard
Copy link
Author

I encountered a similar problem that was caused by the fact that postgres processes were not terminated properly if a fatal error occurred during the test execution.

I thought I fixed that bug in #39, I haven't seen it after updating to the latest release.

Otherwise, everything works fine for me, even when I am running multiple tests at once.

I see this issue when running tests on a project with a gradle test worker for every virtual core, as my macbook pro has 8 cores and 16 virtual cores this results in a very large number of initdb and postgresql processes being launched simultaniously which greatly increases the chances of hitting the conditions necessary to reproduce this bug(multiple initdb processes running at the same time).

My test suite for this project has a very large amount of tests using postgresql as well which is also required to reproduce this from my understanding.

This is the gradle option I'm using to spin up the parallel test workers:

maxParallelForks = Runtime.runtime.availableProcessors()

So could you please check the postgres processes?

I've checked that already and it is not the cause, I can reproduce this bug very reliably(I hit it effectively 100% of the time in one of my projects on my macbook pro) and confirmed that this change fully fixes the test failures.

@tomix26
Copy link
Collaborator

tomix26 commented Oct 12, 2020

I thought I fixed that bug in #39, I haven't seen it after updating to the latest release.

I thought it too, but when I checked it out, there were still some not properly terminated processes.

I see this issue when running tests on a project with a gradle test worker for every virtual core, as my macbook pro has 8 cores and 16 virtual cores this results in a very large number of initdb and postgresql processes being launched simultaniously which greatly increases the chances of hitting the conditions necessary to reproduce this bug(multiple initdb processes running at the same time).

My test suite for this project has a very large amount of tests using postgresql as well which is also required to reproduce this from my understanding.

This is the gradle option I'm using to spin up the parallel test workers:

maxParallelForks = Runtime.runtime.availableProcessors()

So could you please check the postgres processes?

I've checked that already and it is not the cause, I can reproduce this bug very reliably(I hit it effectively 100% of the time in one of my projects on my macbook pro) and confirmed that this change fully fixes the test failures.

Could you please provide some simple reproducer?

Fixes:
[initdb:pid(95804)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 2020-09-17 12:59:51.996 MDT [95916] DETAIL:  Failed system call was shmget(key=5432002, size=56, 03600).
@jameshilliard
Copy link
Author

Could you please provide some simple reproducer?

Pushed a reproducer and an attempt at covering additional conditions discovered that trigger this, the reproducer is pretty aggressive and I haven't been able to fix all the issues so far.

@tomix26 tomix26 force-pushed the master branch 3 times, most recently from dc23c88 to d343a6f Compare May 30, 2021 18:38
@tomix26 tomix26 force-pushed the master branch 5 times, most recently from 616f7f3 to 69c9b08 Compare February 7, 2022 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mac OS "No space left on device" on local tests
2 participants