chore!: replace direct download of nyc_taxi with hvsampledata by Azaya89 · Pull Request #1462 · holoviz/datashader

Azaya89 · 2025-09-26T13:26:36Z

This PR replaces the direct download of the large nyc_taxi data with the version from hvsampledata.

Testing with an editable install of hvsampledata

import hvsampledata as hvs
import datashader as ds
import pandas as pd

path = hvs.download("nyc_taxi_remote")
df = pd.read_parquet(path)
print(df.head())

agg = ds.Canvas().points(df, 'dropoff_x', 'dropoff_y')
print(agg.shape)

   tpep_pickup_datetime tpep_dropoff_datetime  passenger_count  ...  tip_amount  dropoff_hour  pickup_hour
0  2015-01-15 19:05:39   2015-01-15 19:23:42                1  ...        3.25            19           19
1  2015-01-10 20:33:38   2015-01-10 20:53:28                1  ...        2.00            20           20
2  2015-01-10 20:33:38   2015-01-10 20:43:41                1  ...        0.00            20           20
3  2015-01-10 20:33:39   2015-01-10 20:35:31                1  ...        0.00            20           20
4  2015-01-10 20:33:39   2015-01-10 20:52:58                1  ...        0.00            20           20

[5 rows x 12 columns]
(600, 600)

Depends on hvsampledata PR being merged first for CI to pass

hoxbro

You should remove pyct + setuptools from pyproject.toml + pixi.toml

datashader/__init__.py

hoxbro · 2025-09-26T13:35:51Z

examples/getting_started/1_Introduction.ipynb

+    "import hvsampledata as hvs\n",
    "\n",
-    "df = pd.read_csv('../data/nyc_taxi.csv', usecols=['dropoff_x', 'dropoff_y'])\n",
+    "df = hvs.nyc_taxi(\"pandas\", engine_kwargs={\"columns\": ['dropoff_x', 'dropoff_y']})\n",


Suggested change

"df = hvs.nyc_taxi(\"pandas\", engine_kwargs={\"columns\": ['dropoff_x', 'dropoff_y']})\n",

"df = hvs.nyc_taxi_remote(\"pandas\", engine_kwargs={\"columns\": ['dropoff_x', 'dropoff_y']})\n",

Also, mention that this is the first time this cell is run.

hoxbro · 2025-09-26T13:37:29Z

pixi.toml

 setuptools = "*" # distutils for pyct
 toolz = "*"
 xarray = "*"
+fastparquet = "*"


Please check that this has not indirectly pinned anything like pandas.

scripts/download_data.py

codspeed-hq · 2025-09-26T13:51:29Z

CodSpeed Performance Report

Merging #1462 will improve performances by 48.77%

_{Comparing hvsampledata (22e7fc1) with main (a5ae17c)}

Summary

⚡ 2 improvements
✅ 41 untouched

Benchmarks breakdown

	Mode	Benchmark	`BASE`	`HEAD`	Change
⚡	Simulation	`test_layout[forceatlas2_layout]`	63.9 ms	56 ms	+14.11%
⚡	Simulation	`test_quadmesh_rectilinear[1024]`	137 ms	92.1 ms	+48.77%

maximlt

I think this should be first deprecated before being removed https://holoviz.org/about/heps/hep2.html.

hoxbro · 2025-10-09T10:29:26Z

I think this should be first deprecated before being removed holoviz.org/about/heps/hep2.html.

Are you okay with not having pyct as a required dependency? but still having the entrypoint and explaining that:

the entrypoint is deprecated, but users need to install pyct for this to work and,
maybe guide people to use hvsampledata if they want to download data?

This is also a little bit weird, because we don't break running code, but "only" the CLI.

maximlt · 2025-10-10T07:46:11Z

The CLI is part of public interface so I think it deserves the same treatment as the API. I don't see an urgent reason to immediately remove pyct from the dependencies. I'm in favor of simply adding deprecation warnings, and removing the whole thing in a few releases.

hoxbro · 2025-10-15T10:42:00Z

datashader/__init__.py

 from . import transfer_functions as tf                   # noqa (API import)
 from . import data_libraries                             # noqa (API import)

+with suppress(ImportError):


This is not the right place it should be in main.py.

maximlt · 2025-10-15T15:13:00Z

pyproject.toml

    'numpy',
    'pandas',
    'param',
-    'pyct',


I don't think pyct should be removed yet from the dependencies.

It's not removed totally. Just moved to optional dependencies.

With these changes if I install datashader and run datashader example it breaks. So that's a breaking change. The goal is to deprecate it so people get a chance to stop using datashader example. Then pyct can be removed entirely from the dependencies.

codecov · 2025-10-15T16:39:55Z

Codecov Report

❌ Patch coverage is 62.50000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.31%. Comparing base (a5ae17c) to head (22e7fc1).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datashader/__init__.py	71.42%	4 Missing ⚠️
datashader/__main__.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1462      +/-   ##
==========================================
- Coverage   88.34%   88.31%   -0.03%     
==========================================
  Files          96       96              
  Lines       18932    18943      +11     
==========================================
+ Hits        16725    16730       +5     
- Misses       2207     2213       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Azaya89 · 2025-10-15T17:37:26Z

I added a temporary patch to allow the CI to pass pending a new hvsampledata release but otherwise I think it's ready for review now.

scripts/download_data.py

hoxbro · 2025-10-16T06:45:38Z

pyproject.toml

    # 2025-09
    "ignore:Signature .* for <class 'numpy.longdouble'>.*:UserWarning",
+    # 2025-10
+    "ignore:The 'pyct' package bundled as a datashader dependency is deprecated since version 0.19 and will be removed in version 0.20.*" # https://github.com/holoviz/datashader/pull/1462


I think we should remove this filtering. Running datashader in Python (and not CLI) should never show this warning.

hoxbro · 2025-11-25T17:26:30Z

examples/getting_started/1_Introduction.ipynb

-    "import hvsampledata as hvs\n",
+    "import hvsampledata as hvs, pandas as pd\n",
    "\n",
-    "df = hvs.nyc_taxi_remote(\"pandas\", engine_kwargs={\"columns\": ['dropoff_x', 'dropoff_y']})\n",


Let us keep this.

This reverts commit 70ff1e8.

hoxbro · 2025-11-27T09:53:48Z

pixi.toml

 setuptools = "*" # distutils for pyct
 toolz = "*"
 xarray = "*"
+pyarrow = "*"


This should be added back to where it previously was.

hoxbro · 2025-11-27T09:54:54Z

scripts/download_data.py

+with suppress(ImportError):
+    import pyct.cmd
+    from datashader import _warn_pyct_deprecated
+
+    _warn_pyct_deprecated(stacklevel=1)
+    pyct.cmd.fetch_data(name="data", path="examples", datasets="datasets.yml")


The scripts directory is for internal use only; please remove this.

You did not remove this...

hoxbro · 2025-11-27T09:57:18Z

datashader/__init__.py

+
+def _warn_pyct_deprecated(stacklevel=2):
+    warnings.warn(
+        "The 'pyct' package is deprecated since version 0.19 "


If a user saw this warning, do you think they would understand it?

I will also move warnings import into the function.

If a user saw this warning, do you think they would understand it?

Reads simple enough for me, yeah.

I will also move warnings import into the function.

OK.

Let me rephrase it: what does the pyct package do, and why is it affecting the deprecated functions? Without looking at the code, it seems very confusing.

Hmm, OK I get it now. A user wouldn't have used pyct directly and would have been confused seeing the mention of it in the message. Will rephrase to mention the actual functions being called.

hoxbro · 2025-11-27T09:59:14Z

datashader/__init__.py

+    """Wrapper to add deprecation warning to pyct functions."""
+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        _warn_pyct_deprecated(stacklevel=2)


I'm not sure if these stack levels are set correctly, but they are likely still emitted because the category is a FutureWarning.

You can test this out by setting the CategoryWarning to DeprecationWarning.

You're right. Thanks. The correct stacklevel is 3 in this instance. I tested it via

import datashader as ds ds.fetch_data("datashader")

hoxbro

Some small tweaks.

Will have @maximlt have the final say on the deprecations.

hoxbro · 2025-11-28T08:51:00Z

examples/getting_started/1_Introduction.ipynb

-    "The file `nyc_taxi.csv` used above is located at\n",
-    "[nyc_taxi.csv](https://github.com/holoviz/datashader/blob/main/examples/data/.data_stubs/nyc_taxi.csv) in the Datashader repository. When running this example, make sure the file is available locally and update the path accordingly.\n",
-    ":::\n"
+    "The first time this cell is run, it may take some time for the NYC taxi dataset to be downloaded remotely (about 260MB).\n",


Suggested change

"The first time this cell is run, it may take some time for the NYC taxi dataset to be downloaded remotely (about 260MB).\n",

"The first time this cell is run, it will download the NYC taxi dataset (about 260MB).\n",

hoxbro · 2025-11-28T08:51:25Z

datashader/__init__.py


 # make pyct's example/data commands available if possible
-from functools import partial
+from functools import partial, wraps


Let's del wraps as we did with the others.

hoxbro · 2025-11-28T08:53:56Z

datashader/__main__.py

+    with suppress(ImportError):
+        import pyct  # noqa: F401
+
+        from . import _warn_pyct_deprecated
+        _warn_pyct_deprecated(stacklevel=3)
    try:
        import pyct.cmd


Suggested change

with suppress(ImportError):

import pyct # noqa: F401

from . import _warn_pyct_deprecated

_warn_pyct_deprecated(stacklevel=3)

try:

import pyct.cmd

try:

import pyct.cmd

from . import _warn_pyct_deprecated

_warn_pyct_deprecated(stacklevel=3)

hoxbro · 2025-11-28T08:58:19Z

pixi.toml

 shapely = ">=2.0.0"
 spatialpandas = "*"
 streamz = "*"
+hvsampledata = ">=0.1.4"


Suggested change

hvsampledata = ">=0.1.4"

hvsampledata = ">=0.1.5a3"

Let's use a dev version to clearly indicate which version works here. We can remove the dev version once 0.1.5 is released.

hoxbro · 2025-11-28T09:04:26Z

datashader/__init__.py

+
+    warnings.warn(
+        "The 'fetch_data()', 'copy_examples()', and 'examples()' functions are "
+        "deprecated since version 0.19 and will be removed in version 0.20. "


With the release candidacy of datashader, I feel like 0.20 would be released at least 6 months after 0.19.

Seems like each minor release has been a year in between (ignoring 0.17.0, which did not correctly drop a Python version).

Ok, so do you suggest we remove it in 0.19.xx, and modify the message to that effect?

No. I suggest 0.20, and not 0.21. We would never drop something in a patch release.

OK, the message is correct then. It says 0.20 already.

Yes, but still want to highlight it as we normally do two minor releases.

FYI, as per HEP2, there's no requirement to indicate the removal version:

https://holoviz.org/about/heps/hep2.html is worth a read if you haven't done it yet.

replace direct download of nyc_data with hvsampledata

b0f4c04

Azaya89 requested a review from Copilot September 26, 2025 13:26

This comment was marked as outdated.

Sign in to view

Azaya89 self-assigned this Sep 26, 2025

hoxbro reviewed Sep 26, 2025

View reviewed changes

Azaya89 added 3 commits September 26, 2025 18:35

remove pyct completely

015d39a

use pyarrow instead

39ec30e

download data in script and update example notebook

c54c98d

hoxbro changed the title ~~chore: replace direct download of nyc_taxi with hvsampledata~~ chore!: replace direct download of nyc_taxi with hvsampledata Sep 27, 2025

maximlt reviewed Oct 9, 2025

View reviewed changes

Azaya89 added 3 commits October 14, 2025 16:38

Merge branch 'main' into hvsampledata

3ff0ebf

Revert "remove pyct completely"

17d6402

add deprecation warning for pyct

83e277e

hoxbro reviewed Oct 15, 2025

View reviewed changes

Azaya89 added 3 commits October 15, 2025 12:04

temp workaround for CI

ecfbfab

move deprecation warning to main.py

c48447c

second try

0f657bd

maximlt reviewed Oct 15, 2025

View reviewed changes

third try for example notebook

8ba61da

Try GH install of hvsampledata for CI instead

192dd62

Azaya89 marked this pull request as ready for review October 15, 2025 17:32

Azaya89 requested review from Copilot, hoxbro and maximlt October 15, 2025 17:37

This comment was marked as spam.

Sign in to view

holoviz deleted a comment from Copilot AI Oct 15, 2025

hoxbro reviewed Oct 16, 2025

View reviewed changes

remove filtering

a916499

hoxbro added this to the v0.19.0 milestone Nov 13, 2025

Azaya89 added 3 commits November 25, 2025 17:58

replace ugly boilerplate code

b94cb86

update hvsampledata version

49c3e22

update notebook

70ff1e8

hoxbro reviewed Nov 25, 2025

View reviewed changes

Azaya89 and others added 6 commits November 25, 2025 18:30

point to main

1a8b640

Revert "update notebook"

f16f4dd

This reverts commit 70ff1e8.

update test.yaml

3089271

revert deletion of pyct usage

7b4586e

add deprecation warning for all usage of pyct

db073d8

add warning for old pyct download pattern

0e7d3b5

Azaya89 requested a review from hoxbro November 26, 2025 14:43

hoxbro reviewed Nov 27, 2025

View reviewed changes

post review

78ac90f

Azaya89 requested a review from hoxbro November 27, 2025 16:54

hoxbro reviewed Nov 28, 2025

View reviewed changes

review again

22e7fc1

maximlt approved these changes Nov 30, 2025

View reviewed changes

hoxbro merged commit 9e30db1 into main Dec 1, 2025
14 checks passed

hoxbro deleted the hvsampledata branch December 1, 2025 07:53

	"df = hvs.nyc_taxi(\"pandas\", engine_kwargs={\"columns\": ['dropoff_x', 'dropoff_y']})\n",
	"df = hvs.nyc_taxi_remote(\"pandas\", engine_kwargs={\"columns\": ['dropoff_x', 'dropoff_y']})\n",

	"The first time this cell is run, it may take some time for the NYC taxi dataset to be downloaded remotely (about 260MB).\n",
	"The first time this cell is run, it will download the NYC taxi dataset (about 260MB).\n",

Uh oh!

Conversation

Azaya89 commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

hoxbro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed-hq bot commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #1462 will improve performances by 48.77%

Summary

Benchmarks breakdown

Uh oh!

maximlt left a comment

Choose a reason for hiding this comment

Uh oh!

hoxbro commented Oct 9, 2025

Uh oh!

maximlt commented Oct 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Azaya89 commented Oct 15, 2025

Uh oh!

This comment was marked as spam.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hoxbro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Azaya89 commented Sep 26, 2025 •

edited

Loading

codspeed-hq bot commented Sep 26, 2025 •

edited

Loading

codecov bot commented Oct 15, 2025 •

edited

Loading