Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(scan): scan fails to create new OS thread #6934

Open
nierz opened this issue Mar 6, 2024 · 0 comments · May be fixed by #6935
Open

bug(scan): scan fails to create new OS thread #6934

nierz opened this issue Mar 6, 2024 · 0 comments · May be fixed by #6935
Labels
bug Something isn't working community Community contribution kubernetes Kubernetes query

Comments

@nierz
Copy link

nierz commented Mar 6, 2024

Hi,

It seems like KICS fails to scan a large repository (about 9000 files).
The reason seems to be unlimited concurrent workers in the Preparing Scan Assets stage.

Assuming there's no way to limit the amount of workers spawned - here's the function that goes crazy in spawning workers: pkg/analyzer/analyzer.go:Analyze
More specifically, this part in the code:

	// Start the workers
	for _, file := range files {
		wg.Add(1)
		// analyze the files concurrently
		a := &analyzerInfo{
			typesFlag:        a.Types,
			excludeTypesFlag: a.ExcludeTypes,
			filePath:         file,
		}
		go a.worker(results, unwanted, locCount, &wg)
	}

Our use-case is quite unique which is probably why this hasn't been a problem so far:
We're trying to scan a remote repository on a network mount, and I'm assuming the slower responsiveness makes the workers spawn more quickly than the previous ones are able to finish their job.
Even though this use-case isn't the obvious one, I still think it's a problem for this part to be unlimited (assuming I didn't just miss how to limit it).

The other constraining part is that we're running this scan on a Kubernetes pod, which is limited to 1000 PIDs.

Expected Behavior

Limit the amount of workers - if not as a customizable variable then at least so KICS doesn't crash

Actual Behavior

Scan fails due to failing to create a new thread. Here's the log:

Preparing Scan Assets:  \runtime: failed to create new OS thread (have 989 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc 

Steps to Reproduce the Problem

kics scan -p .

Run on a really big repository or a slow network drive.
Alternatively, artificially limit the maximum number of threads.

Specifications

(N/A if not applicable)

  • Version: v1.7.12
  • Platform: Kubernetes
  • Subsystem: N/A
@nierz nierz added bug Something isn't working community Community contribution labels Mar 6, 2024
@github-actions github-actions bot added the kubernetes Kubernetes query label Mar 6, 2024
@nierz nierz changed the title bug(scan): scan fails on creating new thread bug(scan): scan fails to create new OS thread Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working community Community contribution kubernetes Kubernetes query
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant