Yum repository poller plugin for GoCD

The Yum repository poller is a package material plugin capable of polling yum repositories for rpm packages. GoCD server interacts with this plugin via package material plugin interfaces. The plugin makes use of a command similar to the following to poll the server. So it does not depend on the files that yum depends on e.g. files under /etc/yum.repos.d

dnf repoquery --repofrompath=uuid,$REPO_URL --repoid=uuid -q $PACKAGE_SPEC -qf "%{LOCATION}..."

A given instance of polling is considered successful if dnf repoquery returns a single package as output.

Installation

From GoGCD 23.1.0 onwards, you should download the latest release jar and install into your server's plugins/external directory per the plugin user guide.

Prior to GoCD 23.1.0, this plugin came bundled along with the GoCD server, and a separate installation is not required.

Note: This plugin is available for GoCD servers running on Linux nodes having dnf installed.

Repository definition

Repo URL must be a valid http, https or file URL. This plugin looks for the presence of ${REPO_URL}/repodata/repomd.xml to ascertain validity. Basic authentication (https://user:[email protected]/repo) is supported for http and https repositories.

Package definition

In case of this plugin, the package definition is completely determined by the package spec. The package spec may be in any of the following formats. Please refer to the dnf repoquery docs for details.

name
name.arch
name-ver
name-ver-rel
name-ver-rel.arch
name-epoch:ver-rel.arch
epoch:name-ver-rel.arch

Shell glob patterns may also be used. For example, say we have a component under development getting ready for release of version 1.2.0. We cut a branch for the release and bump up the version on trunk/master to 1.3.0. Thus, a package generated by trunk/master may look like mycomp-1.3.0-b72349-noarch.rpm while that generated by branch may look like mycomp-1.2.0-b72344-noarch.rpm. Now if we have a deployment pipeline that is only interested in 1.2 series packages, the package spec needs to be mycomp-1.2.* rather than just mycomp.

Package Metadata

The following rpm metadata is accessed by the plugin

1. BuildTime (required, automatically set by rpmbuild) - Used by the plugin to validate if the package is newer than what was last seen by GoCD. GoCD displays this field as Modified On.
2. Packager - GoCD displays this field as Modified By. If not provided, it is shown as anonymous
3. URL - Displayed as a Trackback URL by GoCD. Use this as a means to trace back to the job that published the package (within GoCD or outside) to the yum repository.

Published Environment Variables

The following information is made available as environment variables for tasks:

GO_PACKAGE_< REPO-NAME >_< PACKAGE-NAME >_LABEL
GO_REPO_< REPO-NAME >_< PACKAGE-NAME >_REPO_URL
GO_PACKAGE_< REPO-NAME >_< PACKAGE-NAME >_PACKAGE_SPEC
GO_PACKAGE_< REPO-NAME >_< PACKAGE-NAME >_LOCATION

Individual plugins may provide additional info via additional environment variables.

Downloading RPMs

Let's say we set up a package repository named ORA and define a package gcc with a spec of gcc-4.* and set it up as material for a pipeline. To download the package locally on the agent, we could write a task like this:

[go] Start to execute task: <exec command="/bin/bash" >
<arg>-c</arg>
<arg>curl -o /tmp/gcc.rpm $GO_PACKAGE_ORA_GCC_LOCATION</arg>
</exec>

When the task executes on the agent, the environment variables get subsituted as below:

[go] Start to execute task: <exec command="/bin/bash" >
<arg>-c</arg>
<arg>curl -o /tmp/$GO_PACKAGE_ORA_GCC_LABEL.rpm $GO_PACKAGE_ORA_GCC_LOCATION</arg>
</exec>.
...
[go] setting environment variable 'GO_PACKAGE_ORA_GCC_LABEL' to value 'gcc-4.4.7-3.el6.x86_64'
[go] setting environment variable 'GO_REPO_ORA_GCC_REPO_URL' to value 'http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64'
[go] setting environment variable 'GO_PACKAGE_ORA_GCC_PACKAGE_SPEC' to value 'gcc-4.*'
[go] setting environment variable 'GO_PACKAGE_ORA_GCC_LOCATION' to value 'http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/gcc-4.4.7-3.el6.x86_64.rpm'
...

Or, to simply pass it as an argument to a deploy script on a remote server

<exec command="/bin/bash">
  <arg>-c</arg>
  <arg>ssh server "cd /to/dest/dir;deploy.sh $GO_PACKAGE_ORA_GCC_LOCATION"</arg>
</exec>

Installing RPMs

For self contained packages (no external dependencies other than what is already installed on the target node), it is just enough to do:

rpm -U /path/to/downloaded/pkg.rpm

On the other hand, if the package isn't self-contained, we'd run:

yum install $GO_PACKAGE_ORA_GCC_LABEL

This would require that /etc/yum.repos.d contain the repository definitions.

Creating and Publishing RPMs

Although the support for package as material in GoCD isn't concerned with how the packages are created and published, here is a short set of pointers to information on the web.

• Building an RPM using rpmbuild and SPEC file
• Building using fpm
• Tutorial to set up a local yum repository using createrepo. Publishing to a yum repo simply involves uploading/copying over a new package revision at the correct location and running createrepo --update

Notes

1. This plugin will detect at max one package revision per minute (the default interval at which GoCD materials poll). If multiple versions of a package get published to a repo in the time interval between two polls, GoCD will only register the latest version in that interval.
2. This plugin makes use of buildtime in rpm metadata to determine if a poll has returned a new result. If for some reason (e.g. timezone misconfiguration), the buildtime of pkg-1.1 is less than that of pkg-1.0, then the plugin will not register pkg-1.1 as a newer package.
3. The only way to update an rpm is to change the version or release. Republishing a different file with the same name and different buildtime won't do.
4. Package groups are not supported.
5. The GoCD command repository has a bunch of commands related to rpm packages.

Building the code base

To build the jar, run ./gradlew clean test assemble

Troubleshooting

Enable Debug Logs

If you are on GoCD version 19.6 and above:

Edit the file wrapper-properties.conf on your GoCD server and add the following options. The location of the wrapper-properties.conf can be found in the installation documentation of the GoCD server.

# We recommend that you begin with the index `100` and increment the index for each system property
wrapper.java.additional.100=-Dplugin.yum.log.level=debug

If you're running with GoCD server 19.6 and above on docker using one of the supported GoCD server images, set the environment variable GOCD_SERVER_JVM_OPTIONS:

docker run -e "GOCD_SERVER_JVM_OPTIONS=-Dplugin.yum.log.level=debug" ...

If you are on GoCD version 19.5 and lower:

• On Linux:

  Enabling debug level logging can help you troubleshoot an issue with this plugin. To enable debug level logs, edit the file /etc/default/go-server (for Linux) to add:

  export GO_SERVER_SYSTEM_PROPERTIES="$GO_SERVER_SYSTEM_PROPERTIES -Dplugin.yum.log.level=debug"

  If you're running the server via ./server.sh script:

  $ GO_SERVER_SYSTEM_PROPERTIES="-Dplugin.yum.log.level=debug" ./server.sh

License

Copyright 2022 Thoughtworks, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.