Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After last update, the plugin does not start new machines (or seems to be doing anything at all) #528

Open
ohadcohenanjuna opened this issue May 1, 2024 · 11 comments
Labels

Comments

@ohadcohenanjuna
Copy link

Jenkins and plugins versions report

Jenkins Version: Jenkins 2.440.3. (can confirm also a problem on 2.440.2)
Azure VM Agent: working: 901.ved986df424b_3
not working: 921.v76424a_e24594

What Operating System are you using (both controller, and any agents involved in the problem)?

Agents are Ubuntu machines; the controller is a docker image running on AWS Linux2. the docker image is: jenkins:lts-jdk21

Reproduction steps

After upgrade and restart of the Jenkins server, jobs are just waiting for agents but the plugin does not kick in, nothing in the logs (so it seems that the plugin is not seeing jobs are pending at all).

Expected Results

Agents would spin when jobs are pending,

Actual Results

nothing happens. no longs, just jobs in the queue and no new agents

Anything else?

Downgrading the plugin fixed the issue.

Are you interested in contributing a fix?

No response

@lukaszkargol
Copy link

lukaszkargol commented May 14, 2024

To extend @ohadcohenanjuna description:

Got same behaviour. Issue seems to be in validation of template:

May 14, 2024 1:56:45 PM SEVERE com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate verifyTemplate
Error validating template
java.lang.NullPointerException: Cannot invoke "java.nio.file.Path.register(java.nio.file.WatchService, java.nio.file.WatchEvent$Kind[])" because the return value of "java.nio.file.Path.getParent()" is null
	at com.sshtools.common.logger.FileWatchingService.register(FileWatchingService.java:76)
	at com.sshtools.common.logger.DefaultLoggerContext.<init>(DefaultLoggerContext.java:61)
	at com.sshtools.common.logger.Log.getDefaultContext(Log.java:38)
	at com.sshtools.common.logger.Log.isLevelEnabled(Log.java:103)
	at com.sshtools.common.logger.Log.isInfoEnabled(Log.java:87)
	at com.sshtools.common.ssh.components.jce.JCEProvider.enableBouncyCastle(JCEProvider.java:237)
	at com.sshtools.common.ssh.components.jce.JCEComponentManager.<init>(JCEComponentManager.java:110)
	at com.sshtools.common.ssh.components.ComponentManager.getInstance(ComponentManager.java:131)
Caused: java.lang.RuntimeException: Unable to locate a cryptographic provider
	at com.sshtools.common.ssh.components.ComponentManager.getInstance(ComponentManager.java:135)
	at com.sshtools.common.publickey.OpenSSHPrivateKeyFile.getRSAKeyPair(OpenSSHPrivateKeyFile.java:608)
	at com.sshtools.common.publickey.OpenSSHPrivateKeyFile.toKeyPair(OpenSSHPrivateKeyFile.java:158)
	at com.sshtools.common.publickey.SshKeyUtils.getPrivateKey(SshKeyUtils.java:113)
	at com.microsoft.azure.vmagent.util.KeyDecoder.getPublicKey(KeyDecoder.java:17)
	at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.verifySSHKey(AzureVMManagementServiceDelegate.java:2454)
	at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.verifyTemplate(AzureVMManagementServiceDelegate.java:2070)
	at com.microsoft.azure.vmagent.AzureVMAgentTemplate.verifyTemplate(AzureVMAgentTemplate.java:1405)
	at com.microsoft.azure.vmagent.AzureVMCloudVerificationTask.verify(AzureVMCloudVerificationTask.java:83)
	at com.microsoft.azure.vmagent.AzureVMCloud.provision(AzureVMCloud.java:628)
	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:726)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Any suggestions ?

@ysmaoui
Copy link

ysmaoui commented May 15, 2024

we started also having the same issue after the last plugin update
and our guess is that this is an issue with a dependency of the plugin, specifically this issue: sshtools/maverick-synergy#86

which was introduced by #507

@timja
Copy link
Member

timja commented May 15, 2024

Any specific steps to reproduce or key types needed?

@ysmaoui
Copy link

ysmaoui commented May 15, 2024

we are using ssh keys generated by ssh-keygen ( rsa key 4096 bytes )

@timja
Copy link
Member

timja commented May 17, 2024

I've tried with:

  1. ssh-keygen -t rsa -b 4096 -m PEM
  2. My original RSA formatted key I had
  3. An openssh format RSA key

On each of them I've tried:

  1. Verifying the agent template
  2. Spawning an agent and running a command on them

I didn't hit any issues

Is it possible to create a minimal reproduction?

I've tried this on:

Java 11 and Java 21
Jenkins 2.452.1
All latest plugins

@timja
Copy link
Member

timja commented May 17, 2024

Could you provide a full plugin list? Its possible you have a dependency conflict and the dependency update exposed it.

I can't see any other plugin using this dependency though

@ohadcohenanjuna
Copy link
Author

I can't post it publicly here for security reasons :(

@ysmaoui
Copy link

ysmaoui commented May 18, 2024

this is a list of plugins where we have seen the issue

analysis-model-api:12.3.0
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
artifactory:4.0.6
asm-api:9.7-33.v4d23ef79fcc8
audit-trail:361.v82cde86c784e
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
azure-credentials:312.v0f3973cd1e59
azure-keyvault:251.vcfe31c013dc7
azure-sdk:157.v855da_0b_eb_dc2
azure-vm-agents:921.v76424a_e24594
basic-branch-build-strategies:81.v05e333931c7d
bitbucket-build-status-notifier:1.4.2
bitbucket-push-and-pull-request:3.0.2
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.12
blueocean-commons:1.27.12
blueocean-config:1.27.12
blueocean-core-js:1.27.12
blueocean-dashboard:1.27.12
blueocean-display-url:2.4.2
blueocean-events:1.27.12
blueocean-git-pipeline:1.27.12
blueocean-github-pipeline:1.27.12
blueocean-i18n:1.27.12
blueocean-jwt:1.27.12
blueocean-personalization:1.27.12
blueocean-pipeline-api-impl:1.27.12
blueocean-pipeline-editor:1.27.12
blueocean-pipeline-scm-api:1.27.12
blueocean-rest-impl:1.27.12
blueocean-rest:1.27.12
blueocean-web:1.27.12
blueocean:1.27.12
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1163.va_f1064e4a_a_f3
build-discarder:139.v05696a_7fe240
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_
cloudbees-folder:6.901.vb_4c7a_da_75da_3
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-109.vfe16c66636eb_
conditional-buildstep:1.4.3
config-file-provider:973.vb_a_80ecb_9a_4d0
configuration-as-code-groovy:1.1
configuration-as-code:1775.v810dc950b_514
coverage:1.14.0
credentials-binding:657.v2b_19db_7d6e6d
credentials:1337.v60b_d7b_c7b_c9f
dashboard-view:2.508.va_74654f026d1
data-tables-api:2.0.5-1
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
dtkit-api:3.0.2
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
email-ext:1806.v856a_01a_fa_39a_
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.2-1
forensics-api:2.4.0
generic-webhook-trigger:2.2.0
git-client:4.7.0
git-forensics:2.1.0
git-server:114.v068a_c7cc2574
git:5.2.1
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1787.v8b_8cd49a_f8f1
github:1.38.0
gradle:2.11
greenballs:1.15.1
groovy:457.v99900cb_85593
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hashicorp-vault-plugin:367.v8a_1ee1cccf3a
htmlpublisher:1.33
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jacoco:3.3.6
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
jaxb:2.3.9-1
jenkins-design-language:1.27.12
jersey2-api:2.42-147.va_28a_44603b_d5
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
job-dsl:1.87
job-restrictions:0.8
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:0.11
kubernetes-pipeline-devops-steps:1.6
kubernetes:4203.v1dd44f5b_1cf9
ldap:725.v3cb_b_711b_1a_ef
locale:431.v3435fa_8f8445
lockable-resources:1246.v28b_e4cc6fa_16
mailer:472.vf7c289a_4b_420
mask-passwords:173.v6a_077a_291eb_5
matrix-auth:3.2.2
matrix-groovy-execution-strategy:1.0.8
matrix-project:822.824.v14451b_c0fd42
maven-plugin:3.23
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
multibranch-scan-webhook-trigger:1.0.11
multiple-scms:0.8
oic-auth:4.250.v5a_d993226437
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
parameterized-scheduler:262.v00f3d90585cc
parameterized-trigger:787.v665fcf2a_830b_
parasoft-findings:10.7.1
performance:957.v658a_7065b_92a_
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:241.v3d46398a_328c
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2198.v41dd8ef6dd56
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2198.v41dd8ef6dd56
pipeline-model-extensions:2.2198.v41dd8ef6dd56
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:4.1.0
prism-api:1.29.0-13
prometheus:763.vf8c26b_a_34b_d6
pubsub-light:1.18
rebuild:332.va_1ee476d8f6d
remote-file:1.24
resource-disposer:0.23
rich-text-publisher-plugin:1.5
rocketchatnotifier:1.5.2
role-strategy:717.v6a_69a_fe98974
run-condition:1.7
schedule-build:559.ve9fb_2e719354
scm-api:690.vfc8b_54395023
script-security:1335.vf07d9ce377a_e
sidebar-link:2.4.1
simple-theme-plugin:176.v39740c03a_a_f5
skip-notifications-trait:313.vd1337c8f8134
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-agent:367.vf9076cd4ee21
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
startup-trigger-plugin:2.9.4
structs:337.v1b_04ea_4df7c8
test-results-analyzer:0.4.1
text-finder:1.26
thinBackup:2.0
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
warnings-ng:11.3.0
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps-global-lib:612.v55f2f80781ef
workflow-cps:3894.vd0f0248b_a_fc4
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45
xunit:3.1.4

and this is an SBOM of the jenkins image:
sbom.txt

@timja
Copy link
Member

timja commented May 18, 2024

Thanks I’ll try reproduce with that list ~next week

@timja
Copy link
Member

timja commented May 19, 2024

I wasn't able to reproduce it with that list =/

@yoghurtpower
Copy link

yoghurtpower commented May 22, 2024

In addition to @ysmaoui maybe there could be an issue with the specific template used as casc:

jenkins:
  clouds:
  - azureVM:
      azureCredentialsId: "az-vm-sp"
      deploymentTimeout: 1200
      existingResourceGroupName: "rg-test-1"
      maxVirtualMachinesLimit: 16
      name: "VM-Cloud"
      resourceGroupReferenceType: "existing"
      vmTemplates:
      - templateName: "docker"
        builtInImage: "Ubuntu 22.04 LTS"
        credentialsId: "ssh-jenkins"
        diskType: "managed"
        doNotUseMachineIfInitFails: false
        executeInitScriptAsRoot: false
        existingStorageAccountName: "satestjenkinsjj001"
        imageReference:
          galleryImageDefinition: "docker"
          galleryImageSpecialized: true
          galleryImageVersion: "1.0.0"
          galleryName: "sig_test_jenkins"
          galleryResourceGroup: "rg-test-1"
          gallerySubscriptionId: "xxx hidden xxx"
        imageTopLevelType: "advanced"
        installGit: true
        javaPath: "java"
        labels: "docker"
        launcher:
          ssh:
            sshConfig: |-
              Host 192.168*
                   HostkeyAlgorithms +ssh-rsa
                   PubkeyAcceptedAlgorithms +ssh-rsa
        location: "West Europe"
        maxVirtualMachinesLimit: 4
        maximumDeploymentSize: 4
        noOfParallelJobs: 1
        osDiskSize: 512
        osDiskStorageAccountType: "Standard_LRS"
        osType: "Linux"
        retentionStrategy: "azureVMCloudOnce"
        storageAccountNameReferenceType: "existing"
        storageAccountType: "Standard_LRS"
        subnetName: "snet-jenkins-vms"
        usageMode: NORMAL
        usePrivateIP: true
        virtualMachineSize: "Standard_F16s_v2"
        virtualNetworkName: "vnet-jenkins-test"
        virtualNetworkResourceGroupName: "rg-test-1"
        tags:
        - name: "JenkinsVMType"
          value: "docker"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants