Rootless Podman Quadlet

[jennydaman]

This guide sets up Nextcloud-AIO using Podman in rootless mode and Quadlet behind a reverse proxy.

0. Install Podman

Podman version 4.8.0 or above is required. Older versions of Podman require workarounds, see edit history and discussion below.

1. Set up a reverse proxy

Consult the upstream documentation on how to do this: https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md

An example Caddyfile might look like this:

cloud.example.com {
    reverse_proxy localhost:11000
}


https://cloud.example.com:8443 {
    reverse_proxy localhost:11001 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

2. Create Systemd Unit File

Create the file ~/.config/containers/systemd/nextcloud-aio-mastercontainer.container with the following content:

[Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target
Requires=podman.socket

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:latest
PublishPort=127.0.0.1:11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1001/podman/podman.sock:/var/run/docker.sock:ro
Network=bridge
SecurityLabelDisable=true

Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=127.0.0.1
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1001/podman/podman.sock

[Install]
WantedBy=multi-user.target default.target

Important notes:

• Change /run/user/1001/podman/podman.sock to the path of your Podman socket (podman info --format '{{ .Host.RemoteSocket.Path }}')
• Nextcloud-AIO wants to create and join itself to a network called nextcloud-aio. The option Network=bridge enables this behavior, for details see podman network connect not implemented for slirp4netns containers/podman#19577 (comment)

Also, create the file ~/.config/containers/systemd/nextcloud-aio-mastercontainer.volume:

[Volume]
VolumeName=nextcloud_aio_mastercontainer

3. Start the Services

systemctl --user daemon-reload
systemctl --user start nextcloud-aio-mastercontainer

4. (Optional) Configure Containers to Restart After Reboot

Podman is daemonless so unlike Docker, containers do not restart automatically after reboot.

To enable restart of Podman containers after reboot, see containers/podman#20418 (comment)

5. Business As Usual

Go to https://cloud.example.com:8443 to access the Nextcloud AIO interface and start the Nextcloud server.

Notes

Backups and mastercontainer self-updating might not work, these details have yet to been sorted out.

[jennydaman]

Known issues

I've encountered some problems, however it is unknown whether these are actual bugs related to Podman or just usage errors.

• Built-in backup solution with BorgBackup doesn't really work.
• Master container updates might not work.

For, now, you can manually update the mastercontainer by running

systemctl --user stop nextcloud-aio-mastercontainer.service
podman pull docker.io/nextcloud/all-in-one:latest
systemctl --user start nextcloud-aio-mastercontainer.service

[loeffelpan]

Thanks. Will take some time until FCOS includes this.
I will stay with the monkey patch until then.

[p-fruck]

FYI Podman 4.8.0 has been released a few days ago and is already present in the Fedora 39 repos. Should be only a matter of the CoreOS auto updater to pick that up.
@jennydaman could you please set Podman >= 4.8.0 as the required version in the top level discussion comment? This discussion has become quite complex to follow

[loeffelpan]

@jennydaman Your quick fix doesn't work anymore as @szaimen did this commit: f935993
Maybe you can pull latest commits from upstream to your repo for the fix?

[jennydaman]

@p-fruck and @loeffelpan done

[daniarla]

Is there any workarounds for borgBackup?
Right now it only fails with some error on the directory configured (probably because is running without root permissions) or at least is only what is stated in the logs.

In fact, the database always starts with:
chmod: /var/run/postgresql: Operation not permitted
in the logs

[Verhoeckx]

Will this solution also work with a Podman compose file?

What is de advantage of using a systemd unit file?

[jennydaman]

podman-compose would probably work, how to use it is not documented here.

On the surface level, the main advantage of Quadlet is convenient for people who like systemd and are already comfortable with its syntax.
Other than that, there are a few technical advantages of systemd. For instance, the workflow for lifecycle management (creating, starting, restarting, removing) of podman containers using systemd is better than using podman directly (e.g. systemctl --user stop container both stops and removes the container). For deployment, it's nice to use systemd as the services manager directly because it's what controls everything else. In the case of Nextcloud-AIO, it's possible to specify nextcloud-aio-mastercontainer's dependence on podman.socket and have systemd make things work instead of having to start the socket and nextcloud-aio-mastercontainer manually.

[Verhoeckx]

@jennydaman: thanks for the clarification! I didn't know that it was possible to start containers (with Podman) in unit files! Seems like a clean/efficient solution!

[jennydaman]

Soft reset instructions

systemctl --user stop nextcloud-aio-mastercontainer
containers=$(podman ps -a -f 'name=^nextcloud-aio' --format='{{.Names}}')
podman stop $containers
podman rm $containers
systemctl --user start nextcloud-aio-mastercontainer

[loeffelpan]

Hey, I'm using podman (rootless) in favor of docker on FCOS. My first attempt to get this working like described above ended in an error at domain check. Maybe this is a networking issue as caddy is proxying to host-ip:11000 and the domaincheck-container is only listening at loopback interface.
So I tried to use curl from the host to domaincheck-container and that worked.

Next step was to skip domaincheck but that ended in an Slim Application Error.
Any help would be appreciated. Please feel fre to ask for more information. I will provide what you need.

Bonus:
How should the reverse proxy container be configured to allow domaincheck to be successful?
My Caddy is proxying to the host's IP as loopback with default slirp4netns is not working. Should I use bridged networking for other Containers? By now I understood that's a workaround for AIO, but in containers/podman#19577 they say that bridge networking will be default in podman 5. Any thoughts?

[p-fruck]

Hey there @loeffelpan, are you running behind cloudflare? If so, this might be the reason for your domaincheck issues. When disabling the domaincheck, do the application errors look similar to the ones described in this comment? #3487 (reply in thread) If so, this will be fixed in a newer version of podman. 4.7.0 doesn't contain this fix and I haven't tested 4.7.2 until yet, but I don't think it includes my fix already. Until the fixed podman version is released, you can also patch the source code of your nextcloud AIO as described by @jennydaman here: #3487 (reply in thread)

[kri164]

How do you turn off domaincheck? Can I use this?

Environment=SKIP_DOMAIN_VALIDATION=true

[loeffelpan]

@kri164 Yes. Correct.

[loeffelpan]

@p-fruck You're correct. With slirp4netns loopbacks are difficult. I got this now working with APACHE_IP_BINDING=0.0.0.0.
Thanks.

Is the need of Network=bridge also solved with podman 4.7.2? I understand containers/podman#19577 that podman network connect for slirp4netns will not be fixed? So bridged network is simply needed, not longer a workaround.

If I an correct this should be mentioned in the initial post for anyone who is joining this discussion.

[p-fruck]

Interestingly, my local setup is created with the network in bridge mode by default (podman 4.7.0). Might have to do with my system default (Fedora CoreOS 39 beta), though I did not check any further. @loeffelpan what exactly would you like to mention? Afaict the default network mode of podman might deviate depending on how the given linux distribution packages podman, so I would simply mention that bridge network must be used (as already done by @jennydaman )

[loeffelpan]

I think this should not differ between distros. I'm also on podman 4.7.0 with FCOS 38.
And for rootless containers slirp4netns is default. bridge is default for rootless containers. This is also mentioned in the docs of podman run:
slirp4netns[:OPTIONS,…]: use slirp4netns(1) to create a user network stack. This is the default for rootless containers.
bridge[:OPTIONS,…]: Create a network stack on the default bridge. This is the default for rootful containers.

I was pointing to this one from the initial post.

Important notes:

• Change /run/user/1001/podman/podman.sock to the path of your Podman socket (podman info --format '{{ .Host.RemoteSocket.Path }}')
• Network=bridge is a necessary workaround for a slirp4netns bug

There is Network=bridge mentioned as a workaround for a bug. But it turned out that this is not a bug and will not be changed.

Allover, I think @jennydaman should modify this as simply needed and not a workaround. Regardless of distro/package differences or rootful/rootless setups.

[kri164]

podman -v
podman version 4.7.2

podman logs nextcloud-aio-mastercontainer
[shortened]
NOTICE: PHP message: Slim Application Error
Type: GuzzleHttp\Exception\ServerException
Code: 500
Message: Server error: POST http://localhost/v1.41/networks/nextcloud-aio/connect resulted in a 500 Internal Server Error response:
{"cause":"network is already connected","message":"container ad680a72162f0b469da30b74ee46ed7a3d0948253f68905ca70bc843e7f (truncated...)
File: /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php
Line: 113
Trace: #0 /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Middleware.php(72): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response), NULL, Array, NULL)

[p-fruck]

This error seems like its the same I mentioned above, related to podman.

[loeffelpan]

I'm confused that this question has not beeing asked. Maybe I missed something.
With only the mastercontainer as quadlet unit file. All the other nextcloud containers are gone after reboot.
How to start/deploy them in this setup?

[loeffelpan]

Thats quite a good idea. Thanks! @szaimen
Another option would be something like this as cronjob (not tested yet).

@reboot containers=$(podman ps -a -f 'name=^nextcloud-aio' --format='{{.Names}}'); podman start $containers

[jennydaman]

@loeffelpan and @szaimen thanks for bringing podman-restart.service to my attention, I've made a comment to upstream containers/podman#20418 (comment) and also modified the instructions here.

[loeffelpan]

In my case (on FCOS) podman-restart.service is disabled on any reboot.
@p-fruck You are on FCOS as well. Is this default behaviour and I have to set it to enable with ignition?

[p-fruck]

Can confirm that its disabled by default. This should not be anything CoreOS specific though. You can enable it in the ignition file for new installations, but a simple systemctl --user enable podman-restart.service should be fine too, since the ignition is only applied once.

[loeffelpan]

I did this. But for some reason (maybe zincati update finaliziation?) its disabled again. Any idea?

[loeffelpan]

Anyone using collabora in this setup?
I tried but got tons of errors and login page was loading infinite.

[richdocuments] Fehler: GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 45002 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://kbw3.de/hosting/capabilities at <<closure>>

 0. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php line 158
    GuzzleHttp\Handler\CurlFactory::createRejection("*** sensitive parameters replaced ***")
 1. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php line 110
    GuzzleHttp\Handler\CurlFactory::finishError(["GuzzleHttp\\Handler\\CurlHandler"], "*** sensitive parameters replaced ***", ["GuzzleHttp\\Handler\\CurlFactory"])
 2. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php line 47
    GuzzleHttp\Handler\CurlFactory::finish(["GuzzleHttp\\Handler\\CurlHandler"], "*** sensitive parameters replaced ***", ["GuzzleHttp\\Handler\\CurlFactory"])
 3. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php line 137
    GuzzleHttp\Handler\CurlHandler->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
 4. /var/www/html/lib/private/Http/Client/DnsPinMiddleware.php line 114
    GuzzleHttp\Middleware::GuzzleHttp\{closure}("*** sensitive parameters replaced ***")
 5. /var/www/html/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php line 35
    OC\Http\Client\DnsPinMiddleware->OC\Http\Client\{closure}("*** sensitive parameters replaced ***")
 6. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php line 31
    GuzzleHttp\PrepareBodyMiddleware->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
 7. /var/www/html/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php line 71
    GuzzleHttp\Middleware::GuzzleHttp\{closure}("*** sensitive parameters replaced ***")
 8. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php line 63
    GuzzleHttp\RedirectMiddleware->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
 9. /var/www/html/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php line 75
    GuzzleHttp\Middleware::GuzzleHttp\{closure}("*** sensitive parameters replaced ***")
10. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php line 331
    GuzzleHttp\HandlerStack->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
11. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php line 168
    GuzzleHttp\Client->transfer("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
12. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php line 187
    GuzzleHttp\Client->requestAsync("*** sensitive parameters replaced ***")
13. /var/www/html/lib/private/Http/Client/Client.php line 230
    GuzzleHttp\Client->request("*** sensitive parameters replaced ***", "https://kbw3.de/hosting/capabilities", ["/mnt/ncdata/fi ... e])
14. /var/www/html/custom_apps/richdocuments/lib/Service/CapabilitiesService.php line 135
    OC\Http\Client\Client->get("https://kbw3.de/hosting/capabilities", [45,[true]])
15. /var/www/html/custom_apps/richdocuments/lib/Backgroundjobs/ObtainCapabilities.php line 40
    OCA\Richdocuments\Service\CapabilitiesService->refetch()
16. /var/www/html/lib/private/BackgroundJob/Job.php line 54
    OCA\Richdocuments\Backgroundjobs\ObtainCapabilities->run(null)
17. /var/www/html/lib/private/BackgroundJob/TimedJob.php line 60
    OC\BackgroundJob\Job->execute(["OC\\BackgroundJob\\JobList"], ["OC\\Log"])
18. /var/www/html/cron.php line 152
    OC\BackgroundJob\TimedJob->execute(["OC\\BackgroundJob\\JobList"], ["OC\\Log"])

at 2023-11-08T03:07:47+00:00

Without collabora nextcloud is working fine.

[p-fruck]

Not using collabora yet but I had some DNS issues trying to access my IdP when configuring OIDC for SSO. Can you try to curl https://kbw3.de and nslookup kbw3.de within your nextcloud-aio-nextcloud container? For me, the DNS started working again after invoking nslookup in the container manually, though this might have been an issue with my setup...

[loeffelpan]

curl https://kbw3.de from inside the nextcloud container is working fine. Seems not to be an DNS issue.

[p-fruck]

It depends. For me curl was working but the php request didn't until I executed nslookup for the domain once. Apparently the dns entry had to be cached first for some odd reason

[loeffelpan]

To solve this I started another discussion which brought me on the right track.
It seem that Collabora needs CAP_MKNOD to run. Unfortunately docker provides this capability by default. Podman does not.
But even when @szaimen adds this capability to his API commands for creating the collabora container it still needs root to use this capability.

So running collabora in our rootless setup isn't possible right now - regardless docker or podman.
And running collabora in AIO in a rootful setup is only possible with docker, not with podman as CAP_MKNOD is missing so far.

[bennypowers]

I followed the above steps on Fedora Silverblue 39 and was unable to start the nextcloud-aio-mastercontainer service

❯ podman info --format '{{ .Host.RemoteSocket.Path }}'
/run/user/1000/podman/podman.sock

❯ cat ~/.config/containers/systemd/nextcloud-aio-mastercontainer.container 
[Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target
Requires=podman.socket

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:latest
PublishPort=127.0.0.1:11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:ro
Network=bridge

Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=127.0.0.1
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1000/podman/podman.sock

[Install]
WantedBy=multi-user.target default.target

❯ cat ~/.config/containers/systemd/nextcloud-aio-mastercontainer.volume
[Volume]
VolumeName=nextcloud_aio_mastercontainer

❯ systemctl --user status nextcloud-aio-mastercontainer.service 
× nextcloud-aio-mastercontainer.service - Nextcloud AIO Master Container
     Loaded: loaded (/var/home/username/.config/containers/systemd/nextcloud-aio-mastercontainer.container;>
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Mon 2023-12-25 14:24:09 IST; 8s ago
   Duration: 963ms
       Docs: https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
    Process: 2095 ExecStart=/usr/bin/podman run --name=nextcloud-aio-mastercontainer --cidfile=/run/user/1000/n>
    Process: 2154 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1000/nextcloud-aio-mastercontain>
   Main PID: 2095 (code=exited, status=1/FAILURE)
        CPU: 2.406s

Dec 25 14:24:08 pi systemd[1814]: Started nextcloud-aio-mastercontainer.service - Nextcloud AIO Master Containe>
Dec 25 14:24:08 pi podman[1936]: 2023-12-25 14:24:08.738104124 +0200 IST m=+3.287549113 container start c7e1cfd>
Dec 25 14:24:08 pi nextcloud-aio-mastercontainer[2095]: Docker socket is not available. Cannot continue.
Dec 25 14:24:08 pi nextcloud-aio-mastercontainer[2095]: Please make sure to mount the docker socket into /var/r>
Dec 25 14:24:08 pi nextcloud-aio-mastercontainer[2095]: If you did this by purpose because you don't want the c>
Dec 25 14:24:08 pi nextcloud-aio-mastercontainer[1936]: c7e1cfdfa050b14d492a18ae4648d91ef8063c3d8ada0372831058f>
Dec 25 14:24:09 pi podman[2107]: 2023-12-25 14:24:09.606304183 +0200 IST m=+0.778314258 container remove c7e1cf>
Dec 25 14:24:09 pi systemd[1814]: nextcloud-aio-mastercontainer.service: Main process exited, code=exited, stat>
Dec 25 14:24:09 pi systemd[1814]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Dec 25 14:24:09 pi systemd[1814]: nextcloud-aio-mastercontainer.service: Consumed 2.406s CPU time.

[bennypowers]

no dice for me, unfortunately

❯ systemctl --user status podman.socket
● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: disabled)
     Active: active (listening) since Tue 2023-12-26 07:43:25 IST; 2 days ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1000/podman/podman.sock (Stream)
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/podman.socket

Dec 26 07:43:25 pi systemd[1789]: Listening on podman.socket - Podman API Socket.

❯ systemctl --user status nextcloud-aio-mastercontainer.service 
× nextcloud-aio-mastercontainer.service - Nextcloud AIO Master Container
     Loaded: loaded (/var/home/username/.config/containers/systemd/nextcloud-aio-mastercontainer.container;>
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2023-12-26 07:43:35 IST; 2 days ago
   Duration: 2.187s
       Docs: https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
    Process: 2249 ExecStart=/usr/bin/podman run --name=nextcloud-aio-mastercontainer --cidfile=/run/user/1000/n>
    Process: 2505 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1000/nextcloud-aio-mastercontain>
   Main PID: 2249 (code=exited, status=1/FAILURE)
        CPU: 2.522s

Dec 26 07:43:31 pi systemd[1789]: Started nextcloud-aio-mastercontainer.service - Nextcloud AIO Master Containe>
Dec 26 07:43:31 pi podman[1956]: 2023-12-26 07:43:31.564381165 +0200 IST m=+4.085842112 container start a26bba9>
Dec 26 07:43:31 pi nextcloud-aio-mastercontainer[1956]: a26bba9eb7db3df4f2594f8a261b24f1d9d334edd2051d63ff3f0ef>
Dec 26 07:43:31 pi nextcloud-aio-mastercontainer[2249]: Docker socket is not available. Cannot continue.
Dec 26 07:43:31 pi nextcloud-aio-mastercontainer[2249]: Please make sure to mount the docker socket into /var/r>
Dec 26 07:43:31 pi nextcloud-aio-mastercontainer[2249]: If you did this by purpose because you don't want the c>
Dec 26 07:43:33 pi podman[2262]: 2023-12-26 07:43:33.590328049 +0200 IST m=+1.897156009 container remove a26bba>
Dec 26 07:43:33 pi systemd[1789]: nextcloud-aio-mastercontainer.service: Main process exited, code=exited, stat>
Dec 26 07:43:35 pi systemd[1789]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Dec 26 07:43:35 pi systemd[1789]: nextcloud-aio-mastercontainer.service: Consumed 2.522s CPU time.
lines 1-22/22 (END)

[loeffelpan]

Is SELinux enabled on Silverblue? As it's basically Fedora, it should.
On FCOS I saw some AVC deny messages in the audit.log when sharing a socket to a container. My setup runs in permissive mode by now due to other issues and it works. I did not figure it out how to modify SELinux to allow sharing sockets with containers.

[szaimen]

Maybe this helps? https://github.com/nextcloud/all-in-one#are-there-known-problems-when-selinux-is-enabled

[loeffelpan]

Indeed that would help. But disabling security is not my way to go. I'm looking for a way to allow socket access by rule for SELinux instead.

[p-fruck]

Honestly, I feel like running with SELinux in enforcing mode and disabling security opts for the master container only might be the way to go. If you want to build your own SELinux policy, maybe this reply can help ;)

[kri164]

Some notes from my last install. Hope it helps.

cgroups

Running containers as rootless requires cgroups version 2. Version 1 or hybrid mode v1/v2 doesnt work. Check the status
mount | grep '^cgroup' | awk '{print $1}' | uniq
To change this add a kernel command line parameter to the boot loader and reboot
systemd.unified_cgroup_hierarchy=1

SELinux

Check the status with command
sestatus
If enabled you should mount volumes with big Z letter as option
Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:Z

Netavark

Todays versions of Podman comes with Netavark module, but existing installations are configured for CNI. If you have problems with intercontainer DNS resolution, switch it.
Check current status
podman info | grep networkBackend
then install "aardvark-dns netavark" packages
and finally switch
podman system reset
WARNING: It will complete WIPE of your container environment, all networks, volumes, images will be erased!

Backup restore

Backup restore doesnt work during initial installation. Error is "Permission denied". Not sure if common bug or rootless install only. As a workaround, you can symlink backup directory to /tmp and use it for import
ln -s /home/aio/backup /tmp/backup
Regular backup works fine.

[bennypowers]

Thank you for these instructions. I was able to get this working through cloudflare tunnel by adding a "Bypass - include everyone" rule for the zero trust app. (Allow - include everyone did not work because it continued to ask for cloudflare OTP, which broke the mobile apps with "Malformed server configuration").

I will need to make modifications to config.php, namely default_phone_region and perhaps also trusted_proxies. Is there an environment variable I can declare in the nextcloud-aio-mastercontainer.container unit which will set those?

[p-fruck]

Hi, I don't think so but you could use some kind of script/init-container which executes the occ command as described here

[bennypowers]

For future reference, I solved this (from the host OS) by editing the file at ~/.local/share/containers/storage/volumes/nextcloud_aio_nextcloud/_data/config/config.php

[GinoBadouri]

Fedora 39 here.
It seems I need to put the capital Z: Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:Z
However SELinux is getting in the way:

type=AVC msg=audit(1704703853.627:14274): avc: denied { connectto } for pid=37151 comm="docker" path="/run/user/1000/podman/podman.sock" scontext=system_u:system_r:container_t:s0:c287,c329 tcontext=unconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0 

Jan 08 09:55:30 fedora.smus.net systemd[35538]: Started nextcloud-aio-mastercontainer.service - Nextcloud AIO Master Container.
Jan 08 09:55:30 fedora.smus.net nextcloud-aio-mastercontainer[37842]: Trying to fix docker.sock permissions internally...
Jan 08 09:55:30 fedora.smus.net nextcloud-aio-mastercontainer[37842]: Adding internal www-data to group root
Jan 08 09:55:30 fedora.smus.net podman[37740]: 2024-01-08 09:55:30.628566465 +0100 CET m=+0.767446952 container start 2ac69323cd5d11ea278861715537c644b0b2926d4e00551ac6fe8efe51b09e>
Jan 08 09:55:30 fedora.smus.net nextcloud-aio-mastercontainer[37740]: 2ac69323cd5d11ea278861715537c644b0b2926d4e00551ac6fe8efe51b09ecd
Jan 08 09:55:30 fedora.smus.net nextcloud-aio-mastercontainer[37842]: Cannot connect to the docker socket. Cannot proceed.

Any ideas?

[kri164]

Maybe this one help https://github.com/dpw/selinux-dockersock

[p-fruck]

In case you do not want to deal with SELinux policies, you might want disable SELinux labeling for socket as explained in the docs

[GinoBadouri]

@p-fruck That seems to work.
So for Fedora you also need to set: SecurityLabelDisable=true

[Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target
Requires=podman.socket

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:latest
PublishPort=127.0.0.1:11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:Z
Network=bridge
SecurityLabelDisable=true

Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=127.0.0.1
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1000/podman/podman.sock

[Install]
WantedBy=multi-user.target default.target

[loeffelpan]

Any way to create a policy on Fedora Core OS? I can't figure it out.

[p-fruck]

I think this question should be asked in the podman discussions. There official answer seems to be disabling SELinux labels for the specified container though. Feel free to link to link your discussion here ;)

[loeffelpan]

On any mastercontainer update I got this error (in the logs of watchtower container):
crun: cannot set memory swappiness with cgroupv2

Same on starting newly created mastercontainer.
Any idea?

[loeffelpan]

Am I the only one with problems on updates (of mastercontainer or any other container of aio)?

[bennypowers]

for me, at the moment, automated updates always fail to restart containers, and occasionally crash the ncaio container as well.

[loeffelpan]

Yeah. Containers can be created but not started. Seems due to the above error concerning memory swapiness.
I found this issue but no solution.
containers/podman#13916

Maybe someone can take a look?

[bennypowers]

I'm not sure that I have the same issue as you, although the symptoms are similar (ncaio crashing or not starting containers)

here are my latest watchtower logs. note the date is feb 2 although just this morning (feb 4) I experienced aio stopping all containers and failing to start them

time="2024-02-02T04:00:37Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2024-02-02T04:00:38Z" level=debug msg="Making sure everything is sane before starting"
time="2024-02-02T04:00:38Z" level=info msg="Watchtower 1.7.1"
time="2024-02-02T04:00:38Z" level=info msg="Using no notifications"
time="2024-02-02T04:00:38Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2024-02-02T04:00:38Z" level=info msg="Running a one time update."
time="2024-02-02T04:00:38Z" level=debug msg="Checking containers for updated images"
time="2024-02-02T04:00:38Z" level=debug msg="Retrieving running containers"
time="2024-02-02T04:00:39Z" level=debug msg="Trying to load authentication credentials." container=/nextcloud-aio-mastercontainer image="docker.io/nextcloud/all-in-one:latest"
time="2024-02-02T04:00:39Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2024-02-02T04:00:39Z" level=debug msg="Got image name: docker.io/nextcloud/all-in-one:latest"
time="2024-02-02T04:00:39Z" level=debug msg="Checking if pull is needed" container=/nextcloud-aio-mastercontainer image="docker.io/nextcloud/all-in-one:latest"
time="2024-02-02T04:00:39Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2024-02-02T04:00:40Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2024-02-02T04:00:40Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2024-02-02T04:00:40Z" level=debug msg="Setting scope for auth token" image=docker.io/nextcloud/all-in-one scope="repository:nextcloud/all-in-one:pull"
time="2024-02-02T04:00:40Z" level=debug msg="No credentials found."
time="2024-02-02T04:00:40Z" level=debug msg="Parsing image ref" host=index.docker.io image=nextcloud/all-in-one normalized=docker.io/nextcloud/all-in-one tag=latest
time="2024-02-02T04:00:40Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/nextcloud/all-in-one/manifests/latest"
time="2024-02-02T04:00:41Z" level=debug msg="Found a remote digest to compare with" remote="sha256:98e17c2da1870a879fc01160a675d634664d0ae9a2f92ed1d67d8809a9dc7fc1"
time="2024-02-02T04:00:41Z" level=debug msg=Comparing local="sha256:708b48efd2479ec97b7ffb59f3d30c134556b56f4d41f4ca60323cd2687d1c7d" remote="sha256:98e17c2da1870a879fc01160a675d634664d0ae9a2f92ed1d67d8809a9dc7fc1"
time="2024-02-02T04:00:41Z" level=debug msg=Comparing local="sha256:985050250dbfa4a88b7958dadf63ee85da85d3ec901bcfbee6dbc10765e546af" remote="sha256:98e17c2da1870a879fc01160a675d634664d0ae9a2f92ed1d67d8809a9dc7fc1"
time="2024-02-02T04:00:41Z" level=debug msg="Digests did not match, doing a pull."
time="2024-02-02T04:00:41Z" level=debug msg="Pulling image" container=/nextcloud-aio-mastercontainer image="docker.io/nextcloud/all-in-one:latest"
time="2024-02-02T04:01:10Z" level=info msg="Found new docker.io/nextcloud/all-in-one:latest image (de8ab1049901)"
time="2024-02-02T04:01:10Z" level=info msg="Stopping /nextcloud-aio-mastercontainer (8e0a3c13758f) with SIGTERM"
time="2024-02-02T04:01:21Z" level=debug msg="AutoRemove container 8e0a3c13758f, skipping ContainerRemove call."
time="2024-02-02T04:01:38Z" level=error msg="container /nextcloud-aio-mastercontainer (8e0a3c13758f) could not be removed"
time="2024-02-02T04:01:38Z" level=info msg="Session done" Failed=1 Scanned=1 Updated=0 notify=no
time="2024-02-02T04:01:38Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

If the problem is docker.io limits, I can temporarily solve by authenticating, but it seems like something deeper is going on here

[loeffelpan]

You're right. Seems to be not the same error.
Mine is clearly related to swapiness like in the mentioned issue.

[daniarla]

Hey!
After a couple of days fighting with podman and nextcloud-aio on debian and getting the error:

NOTICE: PHP message: Could not start domaincheck container: Could not start container nextcloud-aio-domaincheck: Server error: `POST http://localhost/v1.41/containers/create?name=nextcloud-aio-domaincheck` resulted in a `500 Internal Server Error` response:

{"cause":"executable file not found in $PATH","message":"container create: lookup init binary: exec: \"catatonit\": exec (truncated...)

It seems that for some reason podman needs catatonit in order to create other containers. I didn't know that it could have saved a couple of days of troubleshooting.

I'm just documenting it here in case that someone has the same problem!

[MastaG]

I'm running them with:

[Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target
Requires=podman.socket

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:latest
PublishPort=127.0.0.1:11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:Z
Network=bridge
SecurityLabelDisable=true
Pull=always

Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=127.0.0.1
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1000/podman/podman.sock
Environment=NEXTCLOUD_DATADIR="/opt/ncdata"
Environment=AIO_DISABLE_BACKUP_SECTION=true

[Install]
WantedBy=multi-user.target default.target

But for some reason after 24 hours the containers seem to exit.
When I log back in and issue a "podman container ls -a" they somehow start back up.

When I look inside the logs of the nextcloud container it seems to spawn a lot of:
ERROR: Wrong IP address '' in listen.allowed_clients

any ideas?

[MastaG]

I did a couple of things.
First of all I've switched to the "beta" tag, because I've read that the "Wrong IP address" error was solved in there.
I've also removed the old nextcloud-aio-watchtower and borgbackup containers, they were created before I was using AIO_DISABLE_BACKUP_SECTION=true.

Furthermore I've also added a Restart=always to the Service section.
And finally I've enabled the AutoUpdate=registry so it automatically updates the master container using podman-update.

It now looks like this:

[Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target
Requires=podman.socket

[Service]
Restart=always

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:beta
PublishPort=127.0.0.1:11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:Z
Network=bridge
SecurityLabelDisable=true
Pull=always
AutoUpdate=registry

Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=127.0.0.1
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1000/podman/podman.sock
Environment=NEXTCLOUD_DATADIR="/opt/ncdata"
Environment=AIO_DISABLE_BACKUP_SECTION=true

[Install]
WantedBy=multi-user.target default.target

Hopefully it will continue running now.

[MastaG]

Alright I found out why my containers didn't keep on running.
It was because the linger service wasn't enabled for the user running the containers.

The linger service makes sure a login session happens during boot time:
sudo loginctl enable-linger <USERNAME>

Furthermore it's a good idea to make the systemd-logind service wait for internet connectivity:
sudo systemctl edit systemd-logind
Now add the following:

[Unit]
After=network.target network-online.target
Wants=network-online.target

Finally change the ~/.config/containers/systemd/nextcloud-aio-mastercontainer.container file to read:

[Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target network.target network-online.target
Wants=network-online.target
Requires=podman.socket

[Service]
Restart=always

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:beta
PublishPort=127.0.0.1:11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:Z
Network=bridge
SecurityLabelDisable=true
Pull=always
AutoUpdate=registry

Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=127.0.0.1
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1000/podman/podman.sock
Environment=NEXTCLOUD_DATADIR="/opt/ncdata"
Environment=AIO_DISABLE_BACKUP_SECTION=true

[Install]
WantedBy=multi-user.target default.target

This makes sure the login session which starts the systemd user services is triggered during boot time and after network-online.target has been reached :)

All problems are solved now!

[GinoBadouri]

I now found out there's an issue with podman-restart.
When you reboot the server.
Podman-restart doesn't take into account the dependencies of the various containers they have on each other.
This is because the systemd unit only controls the master container as a service.

For example: nextcloud-aio-nextcloud needs to start after several other containers such as nextcloud-aio-apache.
This way the variables inside /usr/local/etc/php-fpm.d/www.conf are set correctly.
Specifically the listen.allowed_clients value doesn't seem complete, because several other containers haven't fully started yet.
The only way to fix this after a reboot, is to go into the AIO interface, press STOP and then START.

[GinoBadouri]

I think it's not so good to use podman-restart for restarting the containers after a reboot.
Instead it's better to just add:

[Service]
Restart=always
ExecStartPost=/home/USER/restart-nextcloud-aio.sh

To the /home/USER/.config/containers/systemd/nextcloud-aio-mastercontainer.container file.

Then create the /home/USER/restart-nextcloud-aio.sh file with the following:

#!/bin/sh
echo "Waiting for containers to become healthy"
podman wait --condition healthy nextcloud-aio-mastercontainer
echo "Tell master container to start all containers.."
podman exec -it nextcloud-aio-mastercontainer sudo -u www-data php /var/www/docker-aio/php/src/Cron/StartContainers.php

(don't forget to chmod +x the file)

This way systemd will start the master container and make sure that it's restarted due to Restart=always.
After the master container becomes healthy, it will tell it to start the other containers, which will happen in the correct order :)

[Lippiece]

Trying this on Arch Linux arm on RPi 5 (it may or may not be the case) is and I get the following:

When connected to localhost:11001: Slim Application Error. Then after page refresh: Domaincheck container is not running.

In the journal:

Apr 13 22:33:40 ...[26736]: NOTICE: PHP message: Could not get current channel Server error: `GET http://localhost/v1.41/containers/nextcloud-aio-mastercontainer/json` resulted in a `500 Internal Server Error` response:
Apr 13 22:33:40 ...[26736]: {"cause":"internal libpod error","message":"network inspection mismatch: asked to join 2 network(s) map[nextcloud-aio:{[ (truncated...)

Also on podman start nextcloud-aio-domaincheck:

ERRO[0000] IPAM error: failed to open database /run/user/1000/containers/networks/ipam.db: open /run/user/1000/containers/networks/ipam.db: no such file or directory 
Error: unable to start container "6de1ff3a6e323d2095373ce8c22f2f40666446e1d37ce6e4abbb35af4e4655b9": netavark: IO error: failed to create aardvark-dns directory: No such file or directory (os error 2)

aardvark-dns is installed.

[Lippiece]

Resolved. Now this:

~  podman logs nextcloud-aio-mastercontainer
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: IPv4 forwarding is disabled
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
It seems like you did not give the mastercontainer the correct name? (The 'nextcloud-aio-mastercontainer' container was not found.)

[bennypowers]

whats in your .container file?

[Lippiece]

[Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target
Requires=podman.socket

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:latest
PublishPort=11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1001/podman/podman.sock:/var/run/docker.sock:ro
Network=bridge
SecurityLabelDisable=true

Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=0.0.0.0
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1001/podman/podman.sock
Environment=SKIP_DOMAIN_VALIDATION=true

[Install]
WantedBy=multi-user.target default.target

[Lippiece]

Resolved by stopping/disabling non-user podman.service and podman.socket, and by using PublishPort=8081:8000 to connect to AIO. Containers have started normally, I can access my Nextcloud!

[dsreyes1014]

How are you guys running caddy with this setup ? Are you running it with podman-rootless or with podman-root?

[MastaG]

I'm using nginx as the reverse proxy and ssl offloader.
The problem I'm facing is that the public IP used by nginx for listening on port 80 and 443 resides on the same server.
When I enter the nextcloud container with: podman exec -it nextcloud-aio-nextcloud sh
I cannot connect with https://my_nextcloud_domain/ because the public IP lives on the same host as my rootless containers.
I think that's a limitation of podman's bridge driver?
Or did somebody found a workaround for this?