Skip to content

Starting out

Jump to bottom
Peter Wu edited this page Feb 7, 2024 · 21 revisions

How to start out with cloning Golang

Most of the steps here are taken from the Golang contribution guide. These are the steps we took to clone Golang:

  • Step 1: Clone the Go source code from go.googlesource.com: you can check out the Go source repo onto your local file system anywhere you want as long as it's outside your GOPATH. Make sure it's stable by compiling and testing it once:
$ git clone https://go.googlesource.com/go
$ cd go/src
$ ./all.bash                                # compile and test
  • Step 2: Prepare changes in a new branch, created from the master branch
$ git checkout -b mybranch
$ [edit files...]
$ git add [files...]
  • Step 3: Test your changes, re-running all.bash. To build under Windows use all.bat; this also requires setting the environment variable GOROOT_BOOTSTRAP to the directory holding the Go tree for the bootstrap compiler.
$ ./all.bash    # recompile and test

After running for a while and printing a lot of testing output, the command should finish by printing:

ALL TESTS PASSED

You can use make.bash instead of all.bash to just build the compiler and the standard library without running the test suite. Once the go tool is built, it will be installed as bin/go under the directory in which you cloned the Go repository, and you can run it directly from there.

For quickly testing, you can:

  • In general, you can run make.bash instead of all.bash to only rebuild the Go tool chain without running the whole test suite. Or you can run run.bash to only run the whole test suite without rebuilding the tool chain. You can think of all.bash as make.bash followed by run.bash.

  • In this section, we'll call the directory into which you cloned the Go repository $GODIR. The go tool built by $GODIR/make.bash will be installed in $GODIR/bin/go and you can invoke it to test your code. For instance, if you have modified the compiler and you want to test how it affects the test suite of your own project, just run go test using it:

$ cd <MYPROJECTDIR>
$ $GODIR/bin/go test
  • If you're changing the standard library, you probably don't need to rebuild the compiler: you can just run the tests for the package you've changed. You can do that either with the Go version you normally use, or with the Go compiler built from your clone (which is sometimes required because the standard library code you're modifying might require a newer version than the stable one you have installed).
$ cd $GODIR/src/hash/sha1
$ [make changes...]
$ $GODIR/bin/go test .
  • If you're modifying the compiler itself, you can just recompile the compile tool (which is the internal binary invoked by go build to compile each single package). After that, you will want to test it by compiling or running something.
$ cd $GODIR/src
$ [make changes...]
$ $GODIR/bin/go install cmd/compile
$ $GODIR/bin/go build [something...]   # test the new compiler
$ $GODIR/bin/go run [something...]     # test the new compiler
$ $GODIR/bin/go test [something...]    # test the new compiler

The same applies to other internal tools of the Go tool chain, such as asm, cover, link, and so on. Just recompile and install the tool using go install cmd/<TOOL> and then use the built Go binary to test it.

In addition to the standard per-package tests, there is a top-level test suite in $GODIR/test that contains several black-box and regression tests. The test suite is run by all.bash but you can also run it manually:

$ cd $GODIR/test
$ $GODIR/bin/go run run.go

Rebuild the standard library

If you want to re build the standard library, use this command $GODIR/bin/go install std. If you want to include the command-line tools, use this command $GODIR/bin/go install cmd std.

Specifics for our clone

Rebasing

(Note: This procedure can only be carried out by a member of the Cloudflare Research team.) The cf branch tracks the latest stable release with all other custom changes on top of it. Once a new release (say, 1.16.6) is made, the following procedure can be used to update the branch to a newer version:

  1. git checkout origin/cf

  2. git checkout -b username/cf-1.16.6

  3. git rebase go1.16.6 (use git rebase -i if needed, see https://github.com/cloudflare/go/pull/107 for example)

  4. Resolve the VERSION: add cf tag conflict by redoing the -devel-cf suffix on the first line:

    git reset VERSION && git checkout VERSION && sed -i '' '1s/[0-9]$/&-devel-cf/' VERSION && git add VERSION && git rebase --continue

(command for macOS, for Linux replace sed -i '' by sed -i)

  1. Build and run tests with cd src && ./all.bash
  2. Open a PR for merging your branch into cf.
  3. Get necessary approvals.
  4. git push -f -u origin cf Be careful! This force-pushes your branch to cf. (Should auto-close the PR.)
  5. Update internal documentation.

For minor releases, this should go relatively smoothly. For major releases, more conflicts may occur due to evasive changes in the standard library. In any case, merge commits will be dropped as part of the rebase.

Alternatively, you can:

# set the upstream
 git remote add upstream [email protected]:golang/go.git

# fetch the needed tag (or omit it to fetch everything)
 git fetch upstream go1.17.6

# rebase
 git rebase go1.17.6

Testing

The full test suite can be run with: cd src && ./all.bash

Since most of our changes only touch a few libraries, generally it is sufficient to: bin/go test crypto/tls crypto/x509

As part of all.bash, Go normally warns if there are new packages in the standard library. We have disabled checks for src/circl. To run these tests manually:

bin/go test -run=TestDependencies ./src/go/build

As part of all.bash, Go checks for new APIs that are not documented in api/go1.X.txt files. Since we have a couple of additions (circl, crypto/tls#Config fields, etc.), we have made this check non-fatal. If you are getting API check errors such as Error running API checker: exit status 1, you can quickly re-run the test with:

bin/go tool dist test -run api --no-rebuild

How to regenerate the TLS testdata

(Should no longer be needed, we should try as hard as possible to avoid changing the default ClientHello messages.)

If you need to regenerate the testdata for TLS, you need to:

* Install openssl1.1
* run `bin/go test -v ./src/crypto/tls/... -update`

If you are using Mac, and you need openssl1.1, you can:

$ brew update
$ brew install [email protected]
// Note that this will change to [email protected] everywhere on your system 
$ echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.bash_profile
# Verify 
$ openssl version
# OpenSSL 1.1.1c

On some cases, you will need to regenerate the tests for deprecated suites. In this case, you will need to install [email protected] from source:

Get the source code from:

  url "https://www.openssl.org/source/openssl-1.1.1i.tar.gz"
  mirror "https://dl.bintray.com/homebrew/mirror/openssl-1.1.1i.tar.gz"
  mirror "https://www.mirrorservice.org/sites/ftp.openssl.org/source/openssl-1.1.1i.tar.gz"
  sha256 "e8be6a35fe41d10603c3cc635e93289ed00bf34b79671a3a4de64fcee00d5242"

These references are taken from the brew formula.

Configure it and install it like so:

    $ ./config enable-weak-ssl-ciphers no-shared --prefix=path-for-the-bin --openssldir=path-for-the-repo
    $ make
    $ make test
    $ make install

The bin code will be find where it was defined by the --prefix flag. Then, change your path:

$ export PATH="path-to-the-bin:$PATH"

In case all of this fails, and you get a lot of timeouts, try:

1. Run it from go/src/, as '../bin/go test -v ./crypto/tls/... -update'
2. Make sure openssl is on the correct path
3. Run 'bin/go build cmd std'
4. Run './make.bash'
Clone this wiki locally