latest release on minishift results in failed DCs in the jenkins namespace

[jstrachan]

I get this:

$ oc get dc
NAME                 REVISION   DESIRED   CURRENT   TRIGGERED BY
content-repository   1          1         0         config
jenkins              1          1         0         config

$oc describe dc jenkins
Name:		jenkins
Namespace:	developer-jenkins
Created:	About a minute ago
Labels:		app=jenkins
		group=io.fabric8.tenant.apps
		provider=fabric8
		version=4.0.37
Annotations:	configmap.fabric8.io/update-on-change=jenkins
		fabric8.io/git-branch=release-v4.0.37
		fabric8.io/git-commit=406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4
		fabric8.io/iconUrl=https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg
		fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=jenkins&var-version=4.0.37
		fabric8.io/scm-con-url=scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
		fabric8.io/scm-devcon-url=scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
		fabric8.io/scm-tag=fabric8-services/tenant-jenkins-1.0.0
		fabric8.io/scm-url=http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins
		fabric8.io/target-platform=openshift
		kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"DeploymentConfig","metadata":{"annotations":{"configmap.fabric8.io/update-on-change":"jenkins","fabric8.io/git-branch":"release-v4.0.37","fabric8.io/git-commit":"406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4","fabric8.io/iconUrl":"https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg","fabric8.io/metrics-path":"dashboard/file/kubernetes-pods.json/?var-project=jenkins\u0026var-version=4.0.37","fabric8.io/scm-con-url":"scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-devcon-url":"scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-tag":"fabric8-services/tenant-jenkins-1.0.0","fabric8.io/scm-url":"http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins","fabric8.io/target-platform":"openshift","maven.fabric8.io/source-url":"jar:file:/home/jenkins/workspace/bric8-tenant-jenkins_master-AGRITQKDOGHX6ZQLEFCNJVZP7VWPA5P7B6D2RTTKZ2KN4XI2LSCA/apps/jenkins/target/jenkins-4.0.37.jar!/META-INF/fabric8/openshift.yml"},"labels":{"app":"jenkins","group":"io.fabric8.tenant.apps","provider":"fabric8","version":"4.0.37"},"name":"jenkins","namespace":"developer-jenkins"},"spec":{"replicas":1,"revisionHistoryLimit":2,"selector":{"app":"jenkins","group":"io.fabric8.tenant.apps","provider":"fabric8"},"strategy":{"recreateParams":{"timeoutSeconds":10000},"type":"Recreate"},"template":{"metadata":{"annotations":{"configmap.fabric8.io/update-on-change":"jenkins","fabric8.io/git-branch":"release-v4.0.37","fabric8.io/git-commit":"406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4","fabric8.io/iconUrl":"https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg","fabric8.io/metrics-path":"dashboard/file/kubernetes-pods.json/?var-project=jenkins\u0026var-version=4.0.37","fabric8.io/scm-con-url":"scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-devcon-url":"scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-tag":"fabric8-services/tenant-jenkins-1.0.0","fabric8.io/scm-url":"http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins","fabric8.io/target-platform":"openshift"},"labels":{"app":"jenkins","group":"io.fabric8.tenant.apps","provider":"fabric8","version":"4.0.37"}},"spec":{"containers":[{"env":[{"name":"PROJECT_NAMESPACE","value":"developer"},{"name":"KUBERNETES_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"GIT_COMMITTER_EMAIL","value":"[email protected]"},{"name":"GIT_COMMITTER_NAME","value":"fabric8"},{"name":"OPENSHIFT_ENABLE_OAUTH","value":"true"},{"name":"OPENSHIFT_ENABLE_REDIRECT_PROMPT","value":"true"},{"name":"KUBERNETES_TRUST_CERTIFICATES","value":"true"},{"name":"KUBERNETES_MASTER","value":"https://kubernetes.default:443"},{"name":"JAVA_GC_OPTS","value":"-XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90"},{"name":"JAVA_OPTS","value":"-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true"},{"name":"OPENSHIFT_JENKINS_JVM_ARCH","value":"i686"},{"name":"CONTAINER_INITIAL_PERCENT","value":"0.07"},{"name":"CONTAINTER_INITIAL_PERCENT","value":"something-non-empty"},{"name":"JENKINS_OPTS","value":"-Dgroovy.use.classvalue=true"},{"name":"JAVA_HOME","value":"/etc/alternatives/java_sdk"},{"name":"MAVEN_OPTS","value":"-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"},{"name":"RECOMMENDER_API_TOKEN","valueFrom":{"secretKeyRef":{"key":"token","name":"jenkins-recommender-api-token"}}}],"image":"fabric8/jenkins-openshift:vb6fc097","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sh","/var/lib/jenkins/pre-shutdown.sh"]}}},"livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/login","port":8080},"initialDelaySeconds":420,"timeoutSeconds":10},"name":"jenkins","ports":[{"containerPort":50000,"name":"slave"},{"containerPort":8080,"name":"http"}],"readinessProbe":{"httpGet":{"path":"/login","port":8080},"initialDelaySeconds":10,"timeoutSeconds":10},"resources":{"limits":{"cpu":"2","memory":"512Mi"},"requests":{"cpu":"0"}},"volumeMounts":[{"mountPath":"/var/lib/jenkins","name":"jenkins-home","readOnly":false},{"mountPath":"/opt/openshift/configuration/","name":"jenkins-config"}]}],"serviceAccountName":"jenkins","volumes":[{"name":"jenkins-home","persistentVolumeClaim":{"claimName":"jenkins-home"}},{"configMap":{"name":"jenkins"},"name":"jenkins-config"}]}},"triggers":[{"type":"ConfigChange"}]}}

		maven.fabric8.io/source-url=jar:file:/home/jenkins/workspace/bric8-tenant-jenkins_master-AGRITQKDOGHX6ZQLEFCNJVZP7VWPA5P7B6D2RTTKZ2KN4XI2LSCA/apps/jenkins/target/jenkins-4.0.37.jar!/META-INF/fabric8/openshift.yml
Latest Version:	1
Selector:	app=jenkins,group=io.fabric8.tenant.apps,provider=fabric8
Replicas:	1
Triggers:	Config
Strategy:	Recreate
Template:
Pod Template:
  Labels:		app=jenkins
			group=io.fabric8.tenant.apps
			provider=fabric8
			version=4.0.37
  Annotations:		configmap.fabric8.io/update-on-change=jenkins
			fabric8.io/git-branch=release-v4.0.37
			fabric8.io/git-commit=406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4
			fabric8.io/iconUrl=https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg
			fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=jenkins&var-version=4.0.37
			fabric8.io/scm-con-url=scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
			fabric8.io/scm-devcon-url=scm:git:[email protected]:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
			fabric8.io/scm-tag=fabric8-services/tenant-jenkins-1.0.0
			fabric8.io/scm-url=http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins
			fabric8.io/target-platform=openshift
  Service Account:	jenkins
  Containers:
   jenkins:
    Image:	fabric8/jenkins-openshift:vb6fc097
    Ports:	50000/TCP, 8080/TCP
    Limits:
      cpu:	2
      memory:	512Mi
    Requests:
      cpu:	0
    Liveness:	http-get http://:8080/login delay=420s timeout=10s period=10s #success=1 #failure=30
    Readiness:	http-get http://:8080/login delay=10s timeout=10s period=10s #success=1 #failure=3
    Environment:
      PROJECT_NAMESPACE:		developer
      KUBERNETES_NAMESPACE:		 (v1:metadata.namespace)
      GIT_COMMITTER_EMAIL:		[email protected]
      GIT_COMMITTER_NAME:		fabric8
      OPENSHIFT_ENABLE_OAUTH:		true
      OPENSHIFT_ENABLE_REDIRECT_PROMPT:	true
      KUBERNETES_TRUST_CERTIFICATES:	true
      KUBERNETES_MASTER:		https://kubernetes.default:443
      JAVA_GC_OPTS:			-XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
      JAVA_OPTS:			-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true
      OPENSHIFT_JENKINS_JVM_ARCH:	i686
      CONTAINER_INITIAL_PERCENT:	0.07
      CONTAINTER_INITIAL_PERCENT:	something-non-empty
      JENKINS_OPTS:			-Dgroovy.use.classvalue=true
      JAVA_HOME:			/etc/alternatives/java_sdk
      MAVEN_OPTS:			-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
      RECOMMENDER_API_TOKEN:		<set to the key 'token' in secret 'jenkins-recommender-api-token'>	Optional: false
    Mounts:
      /opt/openshift/configuration/ from jenkins-config (rw)
      /var/lib/jenkins from jenkins-home (rw)
  Volumes:
   jenkins-home:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:	jenkins-home
    ReadOnly:	false
   jenkins-config:
    Type:	ConfigMap (a volume populated by a ConfigMap)
    Name:	jenkins
    Optional:	false

Deployment #1 (latest):
	Name:		jenkins-1
	Created:	about a minute ago
	Status:		Failed
	Replicas:	0 current / 0 desired
	Selector:	app=jenkins,deployment=jenkins-1,deploymentconfig=jenkins,group=io.fabric8.tenant.apps,provider=fabric8
	Labels:		app=jenkins,group=io.fabric8.tenant.apps,openshift.io/deployment-config.name=jenkins,provider=fabric8,version=4.0.37
	Pods Status:	0 Running / 0 Waiting / 0 Succeeded / 0 Failed

Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  1m		1m		1	deploymentconfig-controller			Normal		DeploymentCreated	Created new replication controller "jenkins-1" for version 1

Took a long while to figure out whats up; saw these events:

7:52:42 AM | jenkins-1-deploy | Pod | Normal | Killing | Killing container with id docker://deployment:Need to kill Pod

i.e. openshift killed the deploy pods. I guess we now need more memory / CPU?

[jstrachan]

that was running minishift via minishift start --vm-driver=xhyve --memory=9000 --cpus=5 --disk-size=50g too btw

[jstrachan]

Triggering the DeploymentConfig's by hand in the openshift console seemed to fix it.

I guess we should ditch DeploymentConfigs for the jenkins tenant!

[rawlingsj]

Yeah! As a first step lets just switch the jenkins DC to a Deployment.