"OpenPodcastSync"? New API specification

[thrillfall]

The name "gpodder" could also be a bit misleading, since you cannot simply connect clients that only support gpodder (different authentication). How about specifying this as a new API called something like "OpenPodcastSync"? That API specification could then state that:

• gpodder.net supports a subset of the API
• your nextcloud plugin supports the entire API
• there can be different authentication methods in front, but the API itself is the same
• there is a way to discover, for a given server, which capabilities it has

Maybe this could even be a joined work with the people from PodcastIndex (if they agree to keep the API as generic as it is now, without relying on PodcastIndex-specific IDs). In the best case, collaborating with PodcastIndex could convince even proprietary services to implement the protocol, so syncing works seamlessly across different clients.

Originally posted by @ByteHamster in AntennaPod/AntennaPod#6036 (comment)

[keunes]

Existing & missing API end-points / functionality

I fully agree. I had already proposed this to some folks over there and they were open to the idea. James was happy to host/publish agreed standards on https://www.podinfra.net (https://github.com/jamescridland/podinfra.net).

@thrillfall, would you be able to share a (simple) list of calls that gPodder supports? Then we can see where we want/need to expand. In the AntennaPod repo we've already had a discussion somewhere about the calls we'd need, I'll try and dig that up.

[ByteHamster]

For a first specification of the API, I think we should skip anything directory/rating related. Maybe we want to add that in another version of the API or to another API, but it is not the main use case that we were discussing for OpenPodcastSync. I would say we should get started on the most requested features, not lose ourselves in discussions about features that are way less relevant for now.

[ByteHamster]

Episode API - not Episode Action, since not each action is stored individually (due to overhead) but just one entry per episode. Identification using media url/guid. Maybe useful to have a Key/Value for features here as well? Could contain is_downloaded, is_favorite, ... Alternative would be to have predefined fields for these features, wouldn't be so easy to extend.

I think key/value could be harder to implement server-side. Then you need JOINs and stuff, which could be less efficient than simple table columns when running a larger server.

[JonOfUs]

You're probably right, since it's not necessary for episodes, it would definitely perform better without it.
What do you think of the key/value stores in subscriptions and settings? They sound like a useful feature to me. Since there are always way less subscriptions than there are episodes, this would probably even be quite performant.

[ByteHamster]

Yeah, I guess we need key/value for settings and subscriptions

[keunes]

Rating podcasts + episodes locally and synchronizing ratings through the server and maybe even different visibility options for ratings (only me / others on same sync server / public?) sound like a good idea
...
Publishing ratings is another difficult thing to do.

I'm not sure I understood what exactly mean with the second paragraph, what 'publishing' entails.

In reply to @ByteHamster:

Maybe we want to add that in another version of the API or to another API, but it is not the main use case that we were discussing for OpenPodcastSync.

For me, synchronisation with Podfriend is one of the reasons why I'm enthusiastic about this initiative. It's a server with which I can sync my my listening progress. And it has ratings. Is it a 'repository'? I wouldn't say so. It'd be cool to not only sync progress but also ratings. And Martin (from Podfriend) said let's talk.

I would say we should get started on the most requested features, not lose ourselves in discussions about features that are way less relevant for now.

I know sync is more popular than ratings. But rating episodes/podcasts I feel is something users might be interested in. If not publishing reviews, then certainly consume them to find new podcasts. Just for my own reference:

small gets audio shows but cannot rate directly each show (April '22)
great solution. everything works well. I haven't found a rating for podcasts yet. (April '20)

[keunes]

Backwards compatibility / breaking changes

Creating a new thread for this comment by @JonOfUs, about using guid as an episode identifier:

What do you think, how should this change be implemented in gpoddersync?
Should it just be a v2.0 that is no longer compatible with v1.0?
This will change a lot, e.g. the unique constraint violation usage probably needs to be reworked if backwards compability is wanted. Or a strict change to GUID as unique index and the deletion of all EpisodeActions without a GUID. But this definitely sounds like a lot of thinking.

Personally I would see backward compatibility as a 'nice to have'. If need be, we should adopt breaking changes: gPodder web is not active any-more and holding back for an inactive (and not super-widely used) service will slow everyone down. Let's focus on active projects like gPodder desktop, kasts, AntennaPod, etc. I have high hopes they'll move along if we engage them.

For example, podcast guid (based on first known URL hash) should probably be adopted (replacing the podcast identifier that has been causing issues in the past). That'd be a breaking change, too.

[ByteHamster]

Publishers will never be consistent about their GUIDs. Not the podcast GUIDs, not the episode GUIDs. Many don't even use episode GUIDs, let alone the podcast ones. So I don't think we should base anything solely on that. I think the current system where we transmit both is the best we can get.

[JonOfUs]

To remove gpodder references, an "OpenPodcastSync" implementation would need to be a completely new Nextcloud app as the app ID contains "gpodder". This way, backwards compability with gpoddersync wouldn't be necessary, both sync endpoints could just coexist.

If some podcasts don't provide any episode GUIDs, the current solution to allow episode URL and GUID as identificator at the same time (while GUID superseds URL) would still be necessary. Is that what you mean by the current system?

[JonOfUs]

I created a small list of differences from the gpodder.net API, maybe it is better to directly compare this spec to that API since Nextcloud Gpodder is only a subset of it:

1. No episode actions but only "time played" and "last played" fields for statistics, leads to smaller database usage
2. GUID in episode API (optional), used preferably above media url (less duplicates)
3. GUID in subscription API? (optional) also used preferably if available, but way less common, so not that useful
4. Possibly more flexible authentification allowing HTTP Auth and maybe even OAuth2 by default? And more auth methods like Nextcloud if implemented by podcast clients
5. Queue(s) sync, very useful since many podcast clients have one
6. No devices API leading to a less complex design, which should make implementing servers (and using the sync as end-users, as ByteHamster pointed out) much easier
7. Possibly more flexible settings with the open Key/Value stores, even on a per-subscription base (gpodder.net also has such a store, but with some official settings keys). Nearly every config could be synced this way. And since the most common settings could be published, this could lead to nice interoperability between podcast clients.

[thp]

Not sure if "podcast GUID" is a good idea -- it means one cannot derive the URL from just the GUID alone. Why didn't they just go for storing the "first known URL" (normalized like in the GUID generation, without scheme and trailing slashes) and store that somehow?

This way, it'd be more obvious to a human what a URL could be, and maybe it'd motivate people to use better URIs in the first place.

Supporting the case of "the podcast was at a different URL in the past that we don't control anymore" could be done with the "rel" value of link items, either an existing one, or a new one (original-podcast-feed-url? or podcast-guid-url): https://microformats.org/wiki/existing-rel-values

The Atom format (which can be mixed with RSS): has atom:link, e.g. you could have:

<atom:link href="www.example.com/podcast.rss" rel="podcast-guid-url" type="application/rss+xml" />

(which can be used as-is, as well as deriving a GUID from it, if needed)

[JonOfUs]

Element identification

I think this topic deserves its own thread. It's about how podcasts and episodes are identified across different podcast clients.
Since no internal IDs can be used, publicly available information about these podcasts have to be used.

gpodder.net identifies subscriptions by feed url and episodes by media url, which lead to many duplicates in the past since both of them can change.
Episode GUID is quite common, but not each podcast supports it and sometimes (Tagesschau) even guids change.
Subscription GUID is rarely used, but may be used more in the future, if I understand it correctly.
Our approach is to make GUID in both cases optional and use URLs for identification if they are missing, but use GUIDs for identification if possible. This approach, however, leads to not very pretty APIs since everywhere where episodes or podcasts are referenced, two separate identification fields have to be available, URL and guid. Episode referencing is, for example, necessary in the queue API.

An alternative approach could be to have a single field combining media URL and guid. But that sounds not very good as well as it does not follow established standards at all.

And in each case, a way of handling detected duplicates would be nice as well since they can still occur. I had the idea of a 'duplicates' endpoint where podcast clients post duplicate episodes (and maybe even subscriptions, if they suddenly 301 their feed). Other clients could then use this information to remove these duplicates from their database (if they haven't already).

[thp]

Wanting to have GUID is a nice thing, but obviously (as you wrote above) it's up to the podcast publisher to put sensible values into the feed. If Tagesschau changes the GUID, there's not much one can do apart from trying to heuristically match URL, title, duration, file size, file contents, ... which will lead to other problems and false positives.

The "real" fix there is to fix Tagesschau's feed, not try to come up with a "new standard to work around wrong data". Or to take a quote attributed to Charles Babbage:

On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

Then again, you can't force all publishers to Do The Right Thing, and being able to deal with "non-optimal" feeds makes the sync protocol work for a greater set of podcasts. Podcasts are (thankfully still) decentralized, if you put them all behind a walled garden (cough cough Spotify) you could of course enforce lots of things.

Every feed needs a URL, and every episode also needs a download URL, which is why we went for those features in the protocol. How stable those URLs are, and if there's also such a thing as GUIDs, is a different topic (and apparently a source of much pain).

---

Stefan created a "best practice" guide for feeds in the past:
https://github.com/gpodder/podcast-feed-best-practice/blob/master/podcast-feed-best-practice.md

Apple also has a "feed requirements" documents for Podcasts:
https://podcasters.apple.com/support/823-podcast-requirements

Also this, "A Podcaster's Guide to RSS":
https://help.apple.com/itc/podcasts_connect/

In general, the "podcasts should have feature X" is wishful thinking, and while there's some chance of trying to lobby publishers into adopting sane defaults, there will always be the odd one out and then it's a question of can the system deal with it gracefully or at least acceptably (having duplicate episodes is annoying, but not as bad as "missing" episodes or the inability to parse feeds in general).

Also, trying to invent "The New Standard" is XKCD 927 -- better utilize existing things as much as possible and interoperate with what's out there.

You can always send as much data as you have about the episode (e.g. send the media URL, but also send the GUID if available, and let the server side deal with it in the best way possible). This allows the server side to improve in the future with matching/detecting without having to change the protocol or the clients. It also allows "simple" implementations of the server side to just ignore one or the other while still providing interoperability (similarly, the protocol could specify that URL + GUID are "recommended" for server->client messaging, but clients should be able to deal with only one of them being available).

For example:

{
    "guid": "b8de566f-04f9-4efa-b594-f5040d40f990",
    "url": "https://example.com/file.mp3"
}

But also allowed:

{
    "guid": null,
    "url": "https://example.net/podcast.ogg"
}

This way, turning a file URL into a GUID is up to the receiver, and ideally both would be stored, and GUID would be preferred over the file URL. Then, again, it's an implementation detail what happens if:

• GUID missing, URL matches
• GUID exists and matches, URL matches (very likely the same episode)
• GUID exists and matches, URL differs (probably fine, just different CDN URL?)
• GUID exists and differs, URL matches (ouch?)
• GUID exists and differs, URL differs (very likely a different episode)

[JonOfUs]

Yes, that sounds like the best way we can handle episode identification. Nextcloud Gpodder handles episode identification this way and I haven't had any issues with it up to now (haven't tested it extensively, however).
In Apple's guide it sounds like they try to actually enforce episode GUIDs. I don't know any statistics, but from my knowledge most podcast feeds provide them. So using them (optionally) sounds logical to me.

GUID exists and differs, URL matches (ouch?)

Tagesschau case here, I think. It's probably best to handle it as different episodes. We could put advanced duplicate detection into this (AntennaPod does this), but different implementations of the spec would surely decide differently sometimes which could lead to unexpected behaviour. The only way I see to conveniently prevent/remove duplicates is an actual endpoint for it (in some kind).

[ByteHamster]

GUID exists and differs, URL differs (very likely a different episode)

Very often it is still the same episode. When podcasters replace their media file (fix missing parts etc) and don't know what they are doing, they often also replace the GUID. AntennaPod does quite a bit of guessing to detect these.