Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Flaky Test]: TestFleetAirGappedUpgradeUnprivileged – Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory #4377

Open
rdner opened this issue Mar 7, 2024 · 5 comments · May be fixed by #4482
Assignees
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team

Comments

@rdner
Copy link
Member

rdner commented Mar 7, 2024

Failing test case

TestFleetAirGappedUpgradeUnprivileged

Error message

Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory

Build

https://buildkite.com/elastic/elastic-agent/builds/7621#018e1986-3969-42c4-91d2-071745693aa4

OS

Linux

Stacktrace and notes

upgrade_fleet_test.go:234: Creating Agent policy...
    upgrade_fleet_test.go:239: Creating Agent enrollment API key...
    upgrade_fleet_test.go:246: Getting default Fleet Server URL...
    upgrade_fleet_test.go:250: Installing Elastic Agent (unprivileged: true)...
    fixture_install.go:130: [test TestFleetAirGappedUpgradeUnprivileged] Inside fixture install function
    fixture.go:632: >> running binary with: [/tmp/TestFleetAirGappedUpgradeUnprivileged1821186027/003/elastic-agent-8.14.0-SNAPSHOT-linux-arm64/elastic-agent install --force --non-interactive --unprivileged --url https://3da74bc2f27846ea9f056981dbca1fbb.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token N0NLWUdZNEIxa0wzTzdlbUl4NTA6QWRVTEEwbF9Sb2VwcmJ3U1IzejJvZw==]
    upgrade_fleet_test.go:265: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:265
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:195
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:115
        	Error:      	Received unexpected error:
        	            	error running agent install command: exit status 1
        	Test:       	TestFleetAirGappedUpgradeUnprivileged
        	Messages:   	failed to install start agent [output: Unprivileged installation mode enabled; this is an experimental and currently unsupported feature.
        	            	Installing in non-interactive mode.
        	            	
[    ] Creating group elastic-agent  [0s] 
                                          

[   =] Successfully created group elastic-agent  [0s] 
                                                      

[   =] Creating user elastic-agent  [0s] 
                                                      

[   =] Successfully created user elastic-agent  [0s] 
                                                      

[   =] Copying install files  [0s] 
                                                      

[=== ] Successfully copied files  [10s] 
                                                      

[==  ] Installing service  [11s] 
                                                      

[ ===] Installed service  [11s] 
                                                      

[ ===] Starting Service  [11s] 
                                                      

[====] Service Started  [11s] Elastic Agent successfully installed, starting enrollment.
        	            	
                                                      

[====] Enrolling Elastic Agent with Fleet  [11s] 
                                                      

[====] Waiting For Enroll...  [11s] {"log.level":"info","@timestamp":"2024-03-07T15:46:39.865Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://3da74bc2f27846ea9f056981dbca1fbb.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
        	            	{"log.level":"info","@timestamp":"2024-03-07T15:46:41.330Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
        	            	{"log.level":"error","@timestamp":"2024-03-07T15:46:41.474Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = error reading from server: EOF'. Waiting for 2s","ecs.version":"1.6.0"}
        	            	{"log.level":"info","@timestamp":"2024-03-07T15:46:43.474Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 1","ecs.version":"1.6.0"}
        	            	{"log.level":"error","@timestamp":"2024-03-07T15:46:43.475Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 4s","ecs.version":"1.6.0"}
        	            	{"log.level":"info","@timestamp":"2024-03-07T15:46:47.475Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 2","ecs.version":"1.6.0"}
        	            	{"log.level":"error","@timestamp":"2024-03-07T15:46:47.476Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 8s","ecs.version":"1.6.0"}
        	            	{"log.level":"info","@timestamp":"2024-03-07T15:46:55.479Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 3","ecs.version":"1.6.0"}
        	            	{"log.level":"error","@timestamp":"2024-03-07T15:46:55.479Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 16s","ecs.version":"1.6.0"}
        	            	{"log.level":"info","@timestamp":"2024-03-07T15:47:11.480Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 4","ecs.version":"1.6.0"}
        	            	{"log.level":"error","@timestamp":"2024-03-07T15:47:11.481Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 32s","ecs.version":"1.6.0"}
        	            	{"log.level":"error","@timestamp":"2024-03-07T15:47:43.481Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":296},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"","ecs.version":"1.6.0"}
        	            	Something went wrong while enrolling the Elastic Agent: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory"
        	            	Error: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory"
        	            	For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.14/fleet-troubleshooting.html
        	            	
                                                      

[    ] Failed to Enroll  [1m16s] 
                                                      

[    ] Stopping Service  [1m16s] 
                                                      

[    ] Successfully Stopped Service  [1m16s] 
                                                      

[    ] Uninstalling  [1m16s] 
                                                      

[    ] Stopping service  [1m16s] 
                                                      

[    ] Successfully stopped service  [1m16s] 
                                                      

[   =] Stopping upgrade watcher; none found  [1m16s] 
                                                      

[====] Removing service  [1m17s] 
                                                      

[==  ] Successfully uninstalled service  [1m17s] 
                                                      

[==  ] Removing install directory  [1m17s] 
                                                      

[  ==] Removed install directory  [1m17s] 
                                                      

[  ==] Uninstalled  [1m17s] Error uninstalling.  Printing logs
        	            	{debug 2024-03-07 15:47:43.510876076 +0000 UTC m=+76.943096555 processes Error fetching PID info for 2, skipping: FillPidMetrics: error getting metadata for pid 2: error fetching exe from pid 2: readlink /proc/2/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.517504754 +0000 UTC m=+76.949725233 processes Error fetching PID info for 3, skipping: FillPidMetrics: error getting metadata for pid 3: error fetching exe from pid 3: readlink /proc/3/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.517689194 +0000 UTC m=+76.949909793 processes Error fetching PID info for 4, skipping: FillPidMetrics: error getting metadata for pid 4: error fetching exe from pid 4: readlink /proc/4/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.517848194 +0000 UTC m=+76.950068713 processes Error fetching PID info for 5, skipping: FillPidMetrics: error getting metadata for pid 5: error fetching exe from pid 5: readlink /proc/5/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.517984434 +0000 UTC m=+76.950204953 processes Error fetching PID info for 6, skipping: FillPidMetrics: error getting metadata for pid 6: error fetching exe from pid 6: readlink /proc/6/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.518117154 +0000 UTC m=+76.950337633 processes Error fetching PID info for 7, skipping: FillPidMetrics: error getting metadata for pid 7: error fetching exe from pid 7: readlink /proc/7/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.518255594 +0000 UTC m=+76.950476073 processes Error fetching PID info for 8, skipping: FillPidMetrics: error getting metadata for pid 8: error fetching exe from pid 8: readlink /proc/8/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.518389674 +0000 UTC m=+76.950610153 processes Error fetching PID info for 11, skipping: FillPidMetrics: error getting metadata for pid 11: error fetching exe from pid 11: readlink /proc/11/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.518515714 +0000 UTC m=+76.950736233 processes Error fetching PID info for 12, skipping: FillPidMetrics: error getting metadata for pid 12: error fetching exe from pid 12: readlink /proc/12/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.518673314 +0000 UTC m=+76.950893793 processes Error fetching PID info for 13, skipping: FillPidMetrics: error getting metadata for pid 13: error fetching exe from pid 13: readlink /proc/13/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.518803914 +0000 UTC m=+76.951024393 processes Error fetching PID info for 14, skipping: FillPidMetrics: error getting metadata for pid 14: error fetching exe from pid 14: readlink /proc/14/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.518932074 +0000 UTC m=+76.951152593 processes Error fetching PID info for 15, skipping: FillPidMetrics: error getting metadata for pid 15: error fetching exe from pid 15: readlink /proc/15/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519068194 +0000 UTC m=+76.951288713 processes Error fetching PID info for 16, skipping: FillPidMetrics: error getting metadata for pid 16: error fetching exe from pid 16: readlink /proc/16/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519196594 +0000 UTC m=+76.951417073 processes Error fetching PID info for 17, skipping: FillPidMetrics: error getting metadata for pid 17: error fetching exe from pid 17: readlink /proc/17/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519328794 +0000 UTC m=+76.951549273 processes Error fetching PID info for 18, skipping: FillPidMetrics: error getting metadata for pid 18: error fetching exe from pid 18: readlink /proc/18/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519453394 +0000 UTC m=+76.951673873 processes Error fetching PID info for 19, skipping: FillPidMetrics: error getting metadata for pid 19: error fetching exe from pid 19: readlink /proc/19/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519602034 +0000 UTC m=+76.951822553 processes Error fetching PID info for 20, skipping: FillPidMetrics: error getting metadata for pid 20: error fetching exe from pid 20: readlink /proc/20/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519739354 +0000 UTC m=+76.951959873 processes Error fetching PID info for 21, skipping: FillPidMetrics: error getting metadata for pid 21: error fetching exe from pid 21: readlink /proc/21/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519866593 +0000 UTC m=+76.952087112 processes Error fetching PID info for 22, skipping: FillPidMetrics: error getting metadata for pid 22: error fetching exe from pid 22: readlink /proc/22/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.519990953 +0000 UTC m=+76.952211432 processes Error fetching PID info for 24, skipping: FillPidMetrics: error getting metadata for pid 24: error fetching exe from pid 24: readlink /proc/24/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.520118873 +0000 UTC m=+76.952339352 processes Error fetching PID info for 26, skipping: FillPidMetrics: error getting metadata for pid 26: error fetching exe from pid 26: readlink /proc/26/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.520247033 +0000 UTC m=+76.952467512 processes Error fetching PID info for 27, skipping: FillPidMetrics: error getting metadata for pid 27: error fetching exe from pid 27: readlink /proc/27/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.520377873 +0000 UTC m=+76.952598352 processes Error fetching PID info for 28, skipping: FillPidMetrics: error getting metadata for pid 28: error fetching exe from pid 28: readlink /proc/28/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.520503913 +0000 UTC m=+76.952724432 processes Error fetching PID info for 29, skipping: FillPidMetrics: error getting metadata for pid 29: error fetching exe from pid 29: readlink /proc/29/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.520651473 +0000 UTC m=+76.952871992 processes Error fetching PID info for 30, skipping: FillPidMetrics: error getting metadata for pid 30: error fetching exe from pid 30: readlink /proc/30/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.520781153 +0000 UTC m=+76.953001632 processes Error fetching PID info for 31, skipping: FillPidMetrics: error getting metadata for pid 31: error fetching exe from pid 31: readlink /proc/31/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.520906833 +0000 UTC m=+76.953127352 processes Error fetching PID info for 32, skipping: FillPidMetrics: error getting metadata for pid 32: error fetching exe from pid 32: readlink /proc/32/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521029993 +0000 UTC m=+76.953250512 processes Error fetching PID info for 33, skipping: FillPidMetrics: error getting metadata for pid 33: error fetching exe from pid 33: readlink /proc/33/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521165793 +0000 UTC m=+76.953386272 processes Error fetching PID info for 34, skipping: FillPidMetrics: error getting metadata for pid 34: error fetching exe from pid 34: readlink /proc/34/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521291553 +0000 UTC m=+76.953512032 processes Error fetching PID info for 35, skipping: FillPidMetrics: error getting metadata for pid 35: error fetching exe from pid 35: readlink /proc/35/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521454113 +0000 UTC m=+76.953674592 processes Error fetching PID info for 36, skipping: FillPidMetrics: error getting metadata for pid 36: error fetching exe from pid 36: readlink /proc/36/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521601313 +0000 UTC m=+76.953821792 processes Error fetching PID info for 37, skipping: FillPidMetrics: error getting metadata for pid 37: error fetching exe from pid 37: readlink /proc/37/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521732273 +0000 UTC m=+76.953952752 processes Error fetching PID info for 38, skipping: FillPidMetrics: error getting metadata for pid 38: error fetching exe from pid 38: readlink /proc/38/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521859233 +0000 UTC m=+76.954079712 processes Error fetching PID info for 39, skipping: FillPidMetrics: error getting metadata for pid 39: error fetching exe from pid 39: readlink /proc/39/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.521991273 +0000 UTC m=+76.954211752 processes Error fetching PID info for 40, skipping: FillPidMetrics: error getting metadata for pid 40: error fetching exe from pid 40: readlink /proc/40/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.522117553 +0000 UTC m=+76.954338032 processes Error fetching PID info for 41, skipping: FillPidMetrics: error getting metadata for pid 41: error fetching exe from pid 41: readlink /proc/41/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.522247473 +0000 UTC m=+76.954467952 processes Error fetching PID info for 42, skipping: FillPidMetrics: error getting metadata for pid 42: error fetching exe from pid 42: readlink /proc/42/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.522374833 +0000 UTC m=+76.954595312 processes Error fetching PID info for 43, skipping: FillPidMetrics: error getting metadata for pid 43: error fetching exe from pid 43: readlink /proc/43/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.522498873 +0000 UTC m=+76.954719352 processes Error fetching PID info for 44, skipping: FillPidMetrics: error getting metadata for pid 44: error fetching exe from pid 44: readlink /proc/44/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.522644473 +0000 UTC m=+76.954864992 processes Error fetching PID info for 45, skipping: FillPidMetrics: error getting metadata for pid 45: error fetching exe from pid 45: readlink /proc/45/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.522780473 +0000 UTC m=+76.955000992 processes Error fetching PID info for 46, skipping: FillPidMetrics: error getting metadata for pid 46: error fetching exe from pid 46: readlink /proc/46/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.522909513 +0000 UTC m=+76.955129992 processes Error fetching PID info for 47, skipping: FillPidMetrics: error getting metadata for pid 47: error fetching exe from pid 47: readlink /proc/47/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523036833 +0000 UTC m=+76.955257312 processes Error fetching PID info for 48, skipping: FillPidMetrics: error getting metadata for pid 48: error fetching exe from pid 48: readlink /proc/48/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523160753 +0000 UTC m=+76.955381232 processes Error fetching PID info for 49, skipping: FillPidMetrics: error getting metadata for pid 49: error fetching exe from pid 49: readlink /proc/49/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523298073 +0000 UTC m=+76.955518592 processes Error fetching PID info for 50, skipping: FillPidMetrics: error getting metadata for pid 50: error fetching exe from pid 50: readlink /proc/50/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523426233 +0000 UTC m=+76.955646712 processes Error fetching PID info for 51, skipping: FillPidMetrics: error getting metadata for pid 51: error fetching exe from pid 51: readlink /proc/51/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523575392 +0000 UTC m=+76.955795911 processes Error fetching PID info for 52, skipping: FillPidMetrics: error getting metadata for pid 52: error fetching exe from pid 52: readlink /proc/52/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523706632 +0000 UTC m=+76.955927111 processes Error fetching PID info for 53, skipping: FillPidMetrics: error getting metadata for pid 53: error fetching exe from pid 53: readlink /proc/53/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523834912 +0000 UTC m=+76.956055431 processes Error fetching PID info for 54, skipping: FillPidMetrics: error getting metadata for pid 54: error fetching exe from pid 54: readlink /proc/54/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.523962752 +0000 UTC m=+76.956183231 processes Error fetching PID info for 55, skipping: FillPidMetrics: error getting metadata for pid 55: error fetching exe from pid 55: readlink /proc/55/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.524095312 +0000 UTC m=+76.956315791 processes Error fetching PID info for 56, skipping: FillPidMetrics: error getting metadata for pid 56: error fetching exe from pid 56: readlink /proc/56/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.524225752 +0000 UTC m=+76.956446231 processes Error fetching PID info for 57, skipping: FillPidMetrics: error getting metadata for pid 57: error fetching exe from pid 57: readlink /proc/57/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.524362112 +0000 UTC m=+76.956582591 processes Error fetching PID info for 58, skipping: FillPidMetrics: error getting metadata for pid 58: error fetching exe from pid 58: readlink /proc/58/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.524493592 +0000 UTC m=+76.956714071 processes Error fetching PID info for 59, skipping: FillPidMetrics: error getting metadata for pid 59: error fetching exe from pid 59: readlink /proc/59/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.524645752 +0000 UTC m=+76.956866231 processes Error fetching PID info for 60, skipping: FillPidMetrics: error getting metadata for pid 60: error fetching exe from pid 60: readlink /proc/60/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.524775552 +0000 UTC m=+76.956996031 processes Error fetching PID info for 67, skipping: FillPidMetrics: error getting metadata for pid 67: error fetching exe from pid 67: readlink /proc/67/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.524898352 +0000 UTC m=+76.957118911 processes Error fetching PID info for 69, skipping: FillPidMetrics: error getting metadata for pid 69: error fetching exe from pid 69: readlink /proc/69/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.525030592 +0000 UTC m=+76.957251071 processes Error fetching PID info for 73, skipping: FillPidMetrics: error getting metadata for pid 73: error fetching exe from pid 73: readlink /proc/73/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.525156432 +0000 UTC m=+76.957376911 processes Error fetching PID info for 74, skipping: FillPidMetrics: error getting metadata for pid 74: error fetching exe from pid 74: readlink /proc/74/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.525288672 +0000 UTC m=+76.957509151 processes Error fetching PID info for 75, skipping: FillPidMetrics: error getting metadata for pid 75: error fetching exe from pid 75: readlink /proc/75/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.525447992 +0000 UTC m=+76.957668471 processes Error fetching PID info for 76, skipping: FillPidMetrics: error getting metadata for pid 76: error fetching exe from pid 76: readlink /proc/76/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.537904789 +0000 UTC m=+76.970125268 processes Error fetching PID info for 133, skipping: FillPidMetrics: error getting metadata for pid 133: error fetching exe from pid 133: readlink /proc/133/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.538058869 +0000 UTC m=+76.970279388 processes Error fetching PID info for 134, skipping: FillPidMetrics: error getting metadata for pid 134: error fetching exe from pid 134: readlink /proc/134/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.538205349 +0000 UTC m=+76.970425828 processes Error fetching PID info for 152, skipping: FillPidMetrics: error getting metadata for pid 152: error fetching exe from pid 152: readlink /proc/152/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.538347469 +0000 UTC m=+76.970567988 processes Error fetching PID info for 154, skipping: FillPidMetrics: error getting metadata for pid 154: error fetching exe from pid 154: readlink /proc/154/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.538479629 +0000 UTC m=+76.970700108 processes Error fetching PID info for 155, skipping: FillPidMetrics: error getting metadata for pid 155: error fetching exe from pid 155: readlink /proc/155/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.538638388 +0000 UTC m=+76.970858907 processes Error fetching PID info for 156, skipping: FillPidMetrics: error getting metadata for pid 156: error fetching exe from pid 156: readlink /proc/156/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.539306788 +0000 UTC m=+76.971527307 processes Error fetching PID info for 164, skipping: FillPidMetrics: error getting metadata for pid 164: error fetching exe from pid 164: readlink /proc/164/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.539447508 +0000 UTC m=+76.971667987 processes Error fetching PID info for 293, skipping: FillPidMetrics: error getting metadata for pid 293: error fetching exe from pid 293: readlink /proc/293/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.544865227 +0000 UTC m=+76.977085746 processes Error fetching PID info for 1919, skipping: FillPidMetrics: error getting metadata for pid 1919: error fetching exe from pid 1919: readlink /proc/1919/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.547984546 +0000 UTC m=+76.980205025 processes Error fetching PID info for 23668, skipping: FillPidMetrics: error getting metadata for pid 23668: error fetching exe from pid 23668: readlink /proc/23668/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.548139066 +0000 UTC m=+76.980359545 processes Error fetching PID info for 23669, skipping: FillPidMetrics: error getting metadata for pid 23669: error fetching exe from pid 23669: readlink /proc/23669/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.548278146 +0000 UTC m=+76.980498625 processes Error fetching PID info for 23786, skipping: FillPidMetrics: error getting metadata for pid 23786: error fetching exe from pid 23786: readlink /proc/23786/exe: no such file or directory /go/pkg/mod/github.com/elastic/[email protected]/metric/system/process/process.go:173 }
        	            	{debug 2024-03-07 15:47:43.653798958 +0000 UTC m=+77.086019477  Loaded configuration from /tmp/TestFleetAirGappedUpgradeUnprivileged1821186027/003/elastic-agent-8.14.0-SNAPSHOT-linux-arm64/elastic-agent.yml /go/src/github.com/elastic/elastic-agent/internal/pkg/config/loader.go:45 }
        	            	{debug 2024-03-07 15:47:43.656901557 +0000 UTC m=+77.089122116  Merged configuration from /tmp/TestFleetAirGappedUpgradeUnprivileged1821186027/003/elastic-agent-8.14.0-SNAPSHOT-linux-arm64/elastic-agent.yml into result /go/src/github.com/elastic/elastic-agent/internal/pkg/config/loader.go:57 }
        	            	{debug 2024-03-07 15:47:43.656918797 +0000 UTC m=+77.089139276  Merged all configuration files from [/tmp/TestFleetAirGappedUpgradeUnprivileged1821186027/003/elastic-agent-8.14.0-SNAPSHOT-linux-arm64/elastic-agent.yml], no external input files /go/src/github.com/elastic/elastic-agent/internal/pkg/config/loader.go:64 }
        	            	{debug 2024-03-07 15:47:43.660806716 +0000 UTC m=+77.093027195 composable Starting controller for composable inputs /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:118 }
        	            	{debug 2024-03-07 15:47:43.660851796 +0000 UTC m=+77.093072275 composable Started controller for composable inputs /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:161 }
        	            	{debug 2024-03-07 15:47:43.669333514 +0000 UTC m=+77.101553993 composable Variable state changed for composable inputs; debounce started /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:197 }
        	            	{debug 2024-03-07 15:47:43.669490114 +0000 UTC m=+77.101710633 docker Docker client will negotiate the API version on the first request. /go/pkg/mod/github.com/elastic/[email protected]/docker/client.go:49 }
        	            	{info 2024-03-07 15:47:43.673087193 +0000 UTC m=+77.105307672 composable.providers.docker Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker/docker.go:44 }
        	            	{debug 2024-03-07 15:47:43.679043471 +0000 UTC m=+77.111263990 composable kubernetes_secrets provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetessecrets/kubernetes_secrets.go:81 }
        	            	{debug 2024-03-07 15:47:43.679052831 +0000 UTC m=+77.111273430 composable Kubernetes leaderelection provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go:53 }
        	            	{debug 2024-03-07 15:47:43.68301835 +0000 UTC m=+77.115238829 composable.providers.kubernetes Kubernetes provider for resource pod skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes/kubernetes.go:106 }
        	            	{debug 2024-03-07 15:47:43.68313487 +0000 UTC m=+77.115355389 composable.providers.kubernetes Kubernetes provider for resource node skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes/kubernetes.go:106 }
        	            	{debug 2024-03-07 15:47:43.770163567 +0000 UTC m=+77.202384086 composable Computing new variable state for composable inputs /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:213 }
        	            	{debug 2024-03-07 15:47:43.770344967 +0000 UTC m=+77.202565486 composable Stopping controller for composable inputs /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:165 }
        	            	{debug 2024-03-07 15:47:43.882744097 +0000 UTC m=+77.314964576 composable Stopped controller for composable inputs /go/src/github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:194 }
        	            	Error: enroll command failed for unknown reason: exit status 1
        	            	For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.14/fleet-troubleshooting.html
        	            	]
--- FAIL: TestFleetAirGappedUpgradeUnprivileged (128.11s)
@rdner rdner added Team:Elastic-Agent Label for the Agent team flaky-test Unstable or unreliable test cases. labels Mar 7, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@rdner rdner changed the title [Flaky Test]: TestFleetAirGappedUpgradeUnprivileged – Error: enroll command failed for unknown reason: exit status 1 [Flaky Test]: TestFleetAirGappedUpgradeUnprivileged – Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory Mar 7, 2024
@rdner
Copy link
Member Author

rdner commented Mar 7, 2024

Might be related to #4127

@blakerouse
Copy link
Contributor

This seems unrelated to #4127. Based on the logs it seems that the Elastic Agent process never started. Doesn't seem like the tests collected diagnostics (which should fail), but should fallback to getting a copy of the installation directory.

I need logs of the Elastic Agent or from journald to be able to determine why it was not running.

@rdner
Copy link
Member Author

rdner commented Mar 12, 2024

Related failures: TestProxyURL_EnrollProxyAndNoProxyInThePolicy and TestProxyURL_RemoveProxyFromThePolicy failed in https://buildkite.com/elastic/elastic-agent/builds/7481#018de487-bbd5-4596-8a58-5b3e2856b046

@AndersonQ AndersonQ self-assigned this Mar 12, 2024
@AndersonQ
Copy link
Member

Just for the record, we'll wait for another failure like that as now we have more logs in place to help to understand what is happening

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants