Integrate new data sources

[dcaribou]

Summary

This discussion tries to reflect on the question of whether this project should become wider in scope and include other data sources, different than transfermarkt-scraper. For example,

• The project may be enhanced to include other Transfermarkt scrapers in addition to transfermarkt-scraper in the acquisition step
• The project may be enhanced to include other data sources altogether in the acquisition step

Obviously the preparation layer would be used to integrate all these disparate scrapers and data sources into one single data model, which is still the main value proposition of the project.

Essentially, the question is: should this project become football-datasets, rather than simply transfermarkt-datasets?

---

Update from december 6th 2023

It is now possible to include generic data sources in the project. Checkout the issues below for more information.

• Generalise raw/data to accommodate multiple sources #238
• Add new data source transfermarkt-api #242

[dcaribou]

One scraper that would be candidate to be included in the project is worldfootballR. This is an interesting module, because it includes functions to get data from Transfermarkt as well as other sources.

See #228 for an analysis of on datasource.

[LarchLiu]

For me personally, I found this project because I wanted to find detailed and comprehensive football related data.
It would be great if other data sources could be integrated as well.

[dcaribou]

I agree @LarchLiu. think by now I am quite convinced that the project should be enhanced so as to support other scrapers and data sources in general. Here's a couple of things that would need to happen to make it possible (that I can think of).

Generalise data/raw

This was thought to host the data from a single source (transfermarkt-scraper). If we are going to host more data sources, there are structural changes that would need to happen to that folder. For example, something like this

data/
|- raw/
|-- transfermarkt-scaper/
|-- worldfootballR/
|-- other-datasource/

Generalise acquiring orchestration

There is currently one singe script that runs the acquiring (acquire.py). In order to support multiple sources or origins, the acquiring layer would somehow need to be able to run different scripts for each of the sources. Ie, there would be multiple acquire.py scripts (imagine other extensions as well, like .R or .sh).

Review data model

The current data model has been designed to support transfermarkt-scraper data and there may be updates that would be required to support the integration of disparate data sources. For example, if we decide there needs to be a players table, what is a player_id that can identify a player across datasets?

I guess the underlying question here is: what is the data model that can best represent professional football?

[LarchLiu]

I think the data on Transfermarkt is already quite comprehensive(competitions, players, games, clubs), and other data sources can be used to supplement it on some details. The data model is still based on Transfermarkt should be simpler, and the changes will not be very significant. All we need to do is write some tools to map the data from other sources to datasets.

[dcaribou]

The data model is still based on Transfermarkt should be simpler, and the changes will not be very significant.

Yes, I agree with this. It feels like the current data model + minor additions should work. It's probably best to start with that and evolve it as needed.

All we need to do is write some tools to map the data from other sources to datasets.

Yeah, and dbt is a great tool for this.