umpf: add built-in support for synchronizing topic branches

While umpf was explicitly developed with multiple developers adding
utags and sharing topic branches in mind, it is less than ideal when
there are multiple developers working on the same topic branches.

And it frequently leads to one of two issues:

  - The branch is pushed before the utag is accepted into the BSP
    repository and other developers get an unexpected addition to
    their umpf

  - The branch is not pushed after the utag is accepted into the BSP
    repository and other developers get an unexpected removal from
    their umpf

Every time this happens, it wastes a bit of time to identify what went
wrong and thus a solution built into umpf is appropriate:

  - CI will call umpf --remote=downstream --force push $BSP/series.inc
    when a PR touching a useries is accepted

  - Developers can call umpf pull to synchronize their topic branches
    or to find out when difference they have to the now upstream
    version

Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

bash_completion

@@ -38,7 +38,7 @@ _umpf_completion()
 	"")
 		COMPREPLY=( $( compgen -W "${completion_cmds[*]} help" -- $cur ) )
 		;;
-	diff|show|tag|tig|build)
+	diff|show|tag|tig|build|push|pull)
 		local -a refs
 		refs=( $( compgen -W "$( git for-each-ref --format='%(refname:short)' refs/tags refs/heads refs/remotes)" -- $cur ) )
 		if [ ${#refs[@]} -eq 0 ]; then

doc/getting-started.rst

@@ -447,6 +447,47 @@ Or tell umpf to rebase onto a new *umpf-base* when creating a fresh *utag*::
     # umpf-topic-range: 8bae5bbec8cb4599c141405e9755b7c0e42e064f..19cdc2b857e662a38c712b41ce610000a5ddc6ae
     # umpf-end
 
+Synchronizing umpf topic branch
+-------------------------------
+
+Due to Git's distributed nature, checked out topic branches can get
+out-of-sync. To compare local topic branches against those referenced
+in a *utag*, ``umpf pull`` can be used::
+
+  umpf --dry-run pull 5.0/special-customer-release/20190311-1
+  umpf: Using series from commit message...
+   * [new branch]                          02fb74aa381080855a57080138b29ecc96586788 -> v5.0/topic/most-fixes
+   ! [rejected]                            f0693b782dd026f2adc4d3c336d9ac6dfb352a73 -> v5.0/topic/more-fixes  (non-fast-forward)
+
+Following options are supported:
+
+- ``--dry-run``: compare the branches, but stop short of actually updating
+                 them
+- ``--force``: reset local branches that are not checked-out to the
+               ``umpf-hashinfo`` in the ``utag``
+- ``--update``: restrict updates to only branches available locally
+
+The counterpart to publish topic branches to a remote after creating a new
+``utag`` is ``umpf push``:
+
+  umpf --dry-run --remote=downstream push 5.0/special-customer-release/20190311-1
+  umpf: Using series from commit message...
+  To ssh://downstream
+   * [new branch]                          02fb74aa381080855a57080138b29ecc96586788 -> v5.0/topic/most-fixes
+   ! [rejected]                            f0693b782dd026f2adc4d3c336d9ac6dfb352a73 -> v5.0/topic/more-fixes  (non-fast-forward)
+  error: failed to push some refs to 'ssh:/downstream'
+
+It supports the same options as ``umpf pull``, but instead of doing local
+changes, it operates on the specified remote.
+
+``umpf push`` is especially useful when multiple developers are creating
+`utags` for the same project in parallel. Each developer will initially
+only push their `utag` to the common repository. Once the changes
+introduced by a `utag` are accepted, all topic branches can be force
+updated on the remote to this most recent `utag`, possibly via
+a server-side pull-request post-merge hook running, e.g.::
+
+  umpf --remote=downstream --force .../linux/patches/series.inc
 
 Overview
 --------

umpf

@@ -38,6 +38,7 @@ PATCH_DIR="umpf-patches"
 IDENTICAL=false
 STABLE=false
 FORCE=false
+DRYRUN=false
 UPDATE=false
 VERBOSE=false
 VERSION_SEPARATOR=-
@@ -178,6 +179,8 @@ usage() {
 	      --nix                  with format-patch: write patch series nix
 	  -h, --help
 	  -f, --force
+	      --dry-run              with push/pull: Do everything except actually send
+	                             the updates.
 	      --flags                specify/override umpf-flags
 	  -i, --identical            use exact commit hashes, not tip of branches
 	  -s, --stable               create a 'stable' tag from a branch based on an
@@ -194,6 +197,7 @@ usage() {
 	                             specified, it's interpreted as
 	                             <topic>=[<remote>/]<topic>
 	  -u, --update               with --patchdir: update existing patches in <path>
+	                             with push/pull: update only existing branches
 	  -v, --version <version>    with tag: overwrite version number [default: 1]
 
 	Commands:
@@ -218,6 +222,8 @@ usage() {
 	  build <umpf>               build an umerge from another umpf
 	  distribute <commit-ish>    push patches not yet in any topic branch
 	                             upstream
+	  push [<umpf>]              push topic branches to the given remote
+	  pull [<umpf>]              pull topic branches into the local repository
 
 	  continue                   continue a previously interrupted umpf command
 	  abort                      abort a previously started umpf command
@@ -245,7 +251,7 @@ setup() {
 	fi
 
 	o="fhilsub:n:p:r:v:"
-	l="auto-rerere,bb,nix,flags:,force,help,identical,stable,update,base:,name:,patchdir:,relative:,override:,remote:,local,version:"
+	l="auto-rerere,bb,nix,flags:,dry-run,force,help,identical,stable,update,base:,name:,patchdir:,relative:,override:,remote:,local,version:"
 	if ! args="$(getopt -n umpf -o "${o}" -l "${l}" -- "${@}")"; then
 		usage
 		exit 1
@@ -271,6 +277,9 @@ setup() {
 		-f|--force)
 			FORCE=true
 			;;
+		--dry-run)
+			DRYRUN=true
+			;;
 		--flags)
 			FLAGS="${1}"
 			shift
@@ -1855,6 +1864,198 @@ do_distribute() {
 	run_distribute
 }
 
+### namespace: push ###
+
+push_topic() {
+	echo "${content}" >> "${STATE}/topic-names"
+}
+
+push_hashinfo() {
+	echo "${content}" >> "${STATE}/topics"
+}
+
+push_release() {
+	echo "${content}" >> "${STATE}/tagname"
+}
+
+push_topic_range() {
+	[ ! -e "${STATE}/tagname" ] && return
+	[ -e "${STATE}/tagrev-flat" ] && abort "more than one 'topic-range' after 'release'!"
+
+	echo "${content##*..}" > "${STATE}/tagrev-flat"
+}
+
+### command: push ###
+
+resolve_commitish() {
+	${GIT} rev-parse --revs-only "$@" 2>/dev/null
+}
+
+shorten_commitish() {
+	resolve_commitish --short ${1}
+}
+
+resolve_tag() {
+	local remote=$1 tag=$2 commit
+	if [ -n "$remote" ]; then
+		# handles conflicting tags on remote
+		commit=$(git ls-remote -q $remote refs/tags/$tag 2>/dev/null | \
+			sed 's/\s\+.*$//')
+	else
+		commit="refs/tags/$tag"
+	fi
+
+	resolve_commitish "${commit}^{}"
+}
+
+update_local() {
+	local success=false args="${1}"
+	local opts
+
+	${FORCE} && opts+="--force"
+
+	local line
+	while read -r line; do
+		local prefix="" suffix=""
+
+		eval set -- ${line}
+		[ ${#} -eq 0 ] && continue
+
+		local refparsed=$(shorten_commitish $3)
+
+		if [ -z "${refparsed}" ]; then
+			prefix=" * [new branch]"
+		elif [[ "${1}" = *~* ]]; then
+		       prefix="   ${1}..$(shorten_commitish ${2})"
+		elif $FORCE; then
+		       prefix=" + ${refparsed}..$(shorten_commitish ${2})"
+		       suffix="  (forced update)"
+		else
+		       prefix=" ! [rejected]"
+		       suffix="  (non-fast-forward)"
+		fi
+
+		printf "%-40s %s -> %s%s\n" "$prefix" $2 $3 "$suffix"
+	done <<< "$args"
+
+	while read -r line; do
+		eval set -- ${line}
+		[ ${#} -eq 0 ] && continue
+
+		if $DRYRUN || git branch $opts $3 $2; then
+			success=true
+		fi
+	done <<< "$args"
+
+	$success || abort
+}
+
+do_push () {
+	local opts args remote
+	local -a branches branch_names
+	local -A topics
+
+	if [ -z "${GIT_REMOTE}" ]; then
+		info "Git remote must be specified. Cannot continue."
+		exit 1
+	fi
+
+	if [ "${GIT_REMOTE}" != "refs/heads/" ]; then
+		remote=${GIT_REMOTE%/}
+	fi
+
+	prepare_persistent push "${@}"
+	parse_series push "${STATE}/series"
+
+	local tagname="$(<"${STATE}/tagname")"
+	local tagrevf="$(<"${STATE}/tagrev-flat")"
+	mapfile -t branches < "${STATE}/topics"
+	mapfile -t branch_names < "${STATE}/topic-names"
+
+	if [ -n "${remote}" ]; then
+		# Needed, so git rev-parse below can check for existent branches
+		git fetch --quiet --no-tags ${remote} 2>/dev/null
+	fi
+
+	local rtagrev="$(resolve_tag "${remote}" ${tagname})"
+	local rtagrevf="$(resolve_commitish ${rtagrev}^)"
+	if [ "$tagrevf" != "$rtagrevf" ]; then
+		if [ -z "$rtagrevf" ]; then
+			abort "${remote}${remote:+/}refs/tags/$tagname not found"
+		else
+			abort "${remote}${remote:+/}refs/tags/$tagname" \
+			      "has unexpected commit-ish $rtagrev"
+		fi
+	fi
+
+	for i in "${!branch_names[@]}"; do
+		local branch=${branch_names[$i]}
+		local rbranchrev="$(resolve_commitish "${GIT_REMOTE}${branch}")"
+
+		[ -z "${remote}" ] && [ "${branches[$i]}" = "${rbranchrev}" ] && continue
+
+		# Don't touch local branches that are already on the correct revision.
+		# For remote branches, we let git push handle it.
+		[ -z "${remote}" ] && [ "${branches[$i]}" = "${rbranchrev}" ] && continue
+		$UPDATE && [ -z "${rbranchrev}" ] && continue
+
+		topics[${branch}]=$(resolve_commitish ${branches[$i]})
+	done
+
+	if [ -n "${remote}" ]; then
+		${FORCE}  && opts+="--force-with-lease"
+		${DRYRUN} && opts+="--dry-run"
+
+		for topic in "${!topics[@]}"; do
+			args+="${topics[$topic]}:refs/heads/${topic} "
+		done
+
+		if [ -z "$args" ]; then
+			info "No branches to push"
+			cleanup
+			return
+		fi
+
+		${GIT} push $opts ${remote} -- $args
+	else # local
+		for topic in "${!topics[@]}"; do
+			local ref=$topic rev=${topics[$topic]}
+			local oldval
+
+			# The old value being computed below is not needed to
+			# create the branch. We compute a suitable one anyway,
+			# so we can show how a ref's commit-ish has changed in
+			# the pull case like we do in the push case.
+			if git merge-base --is-ancestor $ref $rev &>/dev/null; then
+				local ancestors=$(git rev-list $rev ^$ref --count)
+				args+="$(shorten_commitish "$rev")~$ancestors"
+			elif $FORCE; then
+				args+="$(shorten_commitish "$ref")"
+			else
+				args+='""'
+			fi
+			args+=" $rev $ref"
+			args+=$'\n'
+		done
+
+		if [ -z "$args" ]; then
+			info "No branches to push"
+			cleanup
+			return
+		fi
+
+		update_local "$args"
+	fi
+
+	cleanup
+}
+
+### command: pull ###
+
+do_pull () {
+	GIT_REMOTE=refs/heads/ do_push "$@"
+}
+
 ### command: continue ###
 
 do_continue() {