Debian packaging for Ubuntu 20.04

[jblache]

This PR introduces the Debian packaging we've been working on; it was done on Ubuntu 20.04 and should port without too much trouble to 18.04 and later versions.

We install Swift under /opt to stay out of the way of the packaging system with this monolith package approach; a default package is produced alongside the monolith that installs symlinks in the PATH for a default Swift version. The monolith packages are co-installable, as they are versioned with the Swift release.

This produces a large amount of lintian warnings due to files in /opt, and those are unfortunately non-overridable. There are other warnings that we can address in time that are more valid, starting with the binaries that embed an RPATH pointing at the package's build directory.

The debian/copyright file remains a TODO as we have discussed.

To build the source package, use debian/getsource.sh according to the instructions within.

In the current state, the packaging carries a few patches that are a burden for build automation, as patches must apply cleanly (no fuzz) at build time. The script above attempts to help with that by refreshing the patches automatically, however, that will certainly fail at some point in time.

Some packaging files are templated and generated by the clean target in debian/rules; this is to handle the versioned package name for the monolith.

Related pull requests corresponding to the patches in debian/patches:
swiftlang/swift#41738
swiftlang/swift#41739
swiftlang/swift#41758
swiftlang/swift-integration-tests#97

[shahmishal]

Please include pull request from swift and swift-integration-test repo to keep track for future release.

[futurejones]

@jblache do you have some packages you have built with this system that we can download and test?
Also -

a default package is produced alongside the monolith that installs symlinks in the PATH for a default Swift version.

This seems to imply there are 2 deb packages being created. Is this correct or am I reading it wrong?
Thanks.

[jblache]

@jblache do you have some packages you have built with this system that we can download and test? Also -

Soon! We've got some work left to do before we can make these packages available on an experimental basis.

a default package is produced alongside the monolith that installs symlinks in the PATH for a default Swift version.

This seems to imply there are 2 deb packages being created. Is this correct or am I reading it wrong? Thanks.

You are correct. There will be a swiftlang-X.Y.Z package for each Swift version, and those will be co-installable. An unversioned swiftlang package will also be available that makes a default Swift version available in the PATH. The latter is optional to install, and can be pinned (via APT) to a particular version if so desired.

[futurejones]

@jblache could you explain who 'we' is. It would be good to have some contact details and more information.

The concept of having 2 .deb install packages is very strange for this use case.
When Swift is installed do we use
sudo apt install swiftlang
or
sudo apt install swiftlang-X.Y.Z ?

The swiftlang package is just installing the symlinks, normally this would be done with a simple post install script in the main deb package.
As the swiftlang package is unversioned how do you update Swift?

We have discussed previously about using the opt directory for the install location and it was decided not to use it as it is not allowed by Debian policy. Has this decision been changed?
Thanks.

[jblache]

The concept of having 2 .deb install packages is very strange for this use case. When Swift is installed do we use sudo apt install swiftlang or sudo apt install swiftlang-X.Y.Z ?

It depends on what you want to achieve. We're providing flexibility with the split packaging so you can run the latest and greatest as your default, easily-accessible Swift version, and have other versions handy as needed, or the other way around, or any other setup that answers your need of the moment.

The swiftlang package is just installing the symlinks, normally this would be done with a simple post install script in the main deb package. As the swiftlang package is unversioned how do you update Swift?

Unless you are thinking of alternatives here, postinst should absolutely not create symlinks. All files and symlinks must ship in packages, or be controlled by a proper mechanism (e.g. alternatives). Orphan symlinks are a usability and security nightmare.

We have discussed previously about using the opt directory for the install location and it was decided not to use it as it is not allowed by Debian policy. Has this decision been changed? Thanks.

Use of /opt in packages shipping in Debian is indeed not allowed, for good reasons, however for packages outside the distribution, there is no harm in using /opt when it makes sense.

When we get to a more modular build that allows for better integration into the system, we'll relocate out of /opt; until then, it is best to move out of the way of the rest of the system, to avoid potential conflicts and having to deal with them. In the meantime, we can provide a better experience for installing, deploying, and using Swift on Debian-based systems.

This is only the start, and not the end state.

[futurejones]

@tomerd @shahmishal @jblache

Use of /opt in packages shipping in Debian is indeed not allowed, for good reasons, however for packages outside the distribution, there is no harm in using /opt when it makes sense.

When we get to a more modular build that allows for better integration into the system, we'll relocate out of /opt; until then, it is best to move out of the way of the rest of the system, to avoid potential conflicts and having to deal with them. In the meantime, we can provide a better experience for installing, deploying, and using Swift on Debian-based systems.

This goes against everything we have discussed previously.
Yes there is harm in using opt and no it doesn't make sense as we have already discussed and approved an alternate install location.
The entire purpose of what we are doing here is to create packages that can be approved and included in the Debian and Ubuntu repositories.

If the aim now is just to get some deb packages built as quick and easy as possible, and hide all the issues so you don't have to deal with them, there are far better and efficient ways do it than what is proposed in this PR.

[tomerd]

@futurejones @jblache @shahmishal

we absolutely can and should use a different location than /opt if its is more future facing and bringing us closer to the conventional way of doing things and eventually included in formal Debian and Ubuntu repositories. the question I think remains what is that location? I think all the options had issues?

its worth exploring the two packages solution further. iiuc the idiomatic way of doing this is using alternatives so we should explore how hard it is to adopt that in the initial drop, or if it okay to leave as a second step.

[jblache]

Updates to address feedback upcoming after a full build completes.

[futurejones]

The monolith packages are co-installable, as they are versioned with the Swift release.

@jblache Have you tested this?
If you are saying, for example, that you can install swiftlang-5.5.3 and swiftlang-5.6.1 on the same system, this won't be possible.
It appears that each swiftlang-X.Y.Z package is being built with symlinks to usr/bin

# from swiftlang.links.in
opt/swift/X.Y.Z/bin/swift usr/bin/swift
opt/swift/X.Y.Z/bin/swiftc usr/bin/swiftc
opt/swift/X.Y.Z/bin/sourcekit-lsp usr/bin/sourcekit-lsp
opt/swift/X.Y.Z/share/man/man1/swift.1.gz usr/share/man/man1/swift.1.gz

This will cause the apt install to fail due the conflict with any existing usr/bin files installed from another swift version.

[jblache]

The entire purpose of what we are doing here is to create packages that can be approved and included in the Debian and Ubuntu repositories.

That is indeed the goal, it is however not the entire nor sole purpose. Packages are ultimately a convenience, and denying this convenience to users and developers until we perfect the packages to the point they meet the criteria for inclusion does not help anyone. This will be a lengthy journey as there are extensive changes to be made.

In recognition of these hard facts, we worked on a phased approach that will gradually improve the quality, correctness, and system integration of the packages as progress is made on the underlying required changes.

I understand that you disagree with this approach and I encourage you to share your feedback so we can address any issues that we might have overlooked.

As proposed here, the packages are equivalent to deploying the Swift binary tarball to /opt. They are convenient in the sense that they can be used and managed through the package manager, can be depended upon in local packages, and carry dependency information (so you won't remove dependencies by accident). As they stay out of the way of the rest of the system, they are safe and don't risk conflicting with other packages and causing trouble (as I gather happened with lld).

I don't think it is ideal, but it is a definite improvement over the current tarball situation.

As far as distribution inclusion goes, and speaking for Debian only because that is what I am most familiar with, there are serious obstacles to this in the current state of things. To name a few off the top of my head, and in no particular order:

• LLVM/Clang source duplication (and the resulting duplicated binaries)
• CMake embedding
• embedded libraries
• unversioned shared libraries
• forked external libraries (CMark)
• BlocksRuntime (this should be provided by the distribution's compiler-rt packages)

While some of these may seem easy to fix, the fixes do represent substantial changes to the build system and/or release management process that need to be worked out, scoped, scheduled, and implemented.

In the end, libdispatch, Foundation, CMark4Swift (and the ones I'm forgetting, if any) should be self-standing and packaged on their own. The build should use the system's CMake, and if necessary a backported CMake should be provided.

Resolving the LLVM situation is something I don't have a good grasp on at this time, and there are a few different routes this could go.

At the end of it all, once we'll have achieved the perfect packages, we will have one more piece to add: we'll need to define how Swift software/modules can be packaged, write up the policy, and implement the helper tools to go with it. This is largely a green field, guided by existing, similar mechanisms that exist in Debian today for other languages.

Contributions are always welcome; there is a good amount of interesting work to be done, especially on that later bit. If you have any ideas, we're happy to hear them and discuss possible solutions.

[jblache]

It appears that each swiftlang-X.Y.Z package is being built with symlinks to usr/bin

# from swiftlang.links.in

The symlinks listed in that file will be incorporated into the swiftlang package, as the filename implies. As such, there won't be conflicts, the links will get updated to point to the binaries shipped in the corresponding swiftlang-X.Y.Z package, i.e.

swiftlang_5.6.0-1.deb -> swiftlang-5.6.0_5.6.0-1.deb

Any given system can have 1..n swiftlang-X.Y.Z packages installed concurrently, but only one swiftlang package providing the default/convenience symlinks. The swiftlang package is optional to install, may be pinned to a particular version via APT pinning, or otherwise will always reflect the latest available version.

[futurejones]

@tkremenek @tomerd @shahmishal

Can you please explain who @jblache is and what position he has in the Packaging for Debian project.

His comments above read like a politicians' election speech and show a total lack of knowledge or disregard for all the people who have been working on this for many years.

That is indeed the goal, it is however not the entire nor sole purpose. Packages are ultimately a convenience, and denying this convenience to users and developers until we perfect the packages to the point they meet the criteria for inclusion does not help anyone. This will be a lengthy journey as there are extensive changes to be made.

wtf, lost for words ...

In recognition of these hard facts, we worked on a phased approach that will gradually improve the quality, correctness, and system integration of the packages as progress is made on the underlying required changes.

Who is we and where is this happening? Who are the team members, where are the meeting minutes, where are the forum discussions? Where is the project plan, timeline etc.?

I understand that you disagree with this approach and I encourage you to share your feedback so we can address any issues that we might have overlooked.

@jblache I encourage you to take some time to read other PR's, comments, forum posts and bug reports so you can find all the work that other people have done that you have overlooked.

[tomerd]

@futurejones per our discussions on forums.swift.org following your message from 4/4, we will move forward with this PR in order to make progress with this project. @shahmishal myself and the other contributors involved continue to hold the position that working collaboratively is the best path to success, and would very much like to see you be part of this project. As such things go, it requires all parties to be open to differences in opinions and achieving progress through the principal of "rough consensus and running code".