SearchResult no longer ordered chronologically, it could break user expectation

[orionlee]

Issue:
The new sort order in v2.4.2 could break some user expectation.

In just released v2.4.2, the SearchResult sort order is changed to give priority of author (implicitly via sort_order), over the observation time (implicitly via mission as it contains sector/quarter/campaign).

lightkurve/src/lightkurve/search.py

Lines 137 to 141 in 7a7dc2a

	sort_priority = {"Kepler": 1, "K2": 1, "SPOC": 1, "KBONUS-BKG":2, "TESS-SPOC": 2, "QLP": 3}
	self.table["sort_order"] = [
	sort_priority.get(author, 9) for author in self.table["author"]
	]
	self.table.sort(["distance", "project", "sort_order", "year", "exptime"])

The old sort order for reference:

self.table.sort(["distance", "year", "mission", "sort_order", "exptime"])

Example

In the use case that an user wants to download all lightcurves of a given TIC, and use the best available product for each sector (see #1048).

I used to be able to get a SearchResult, ordered by observation time (via mission).
With v.2.4.2, the SearchResult is first ordered by author.
As a result, the LightCurveCollection obtained from SearchResult.download_all() is not ordered chronologically either.

> sr = lk.search_lightcurve("TIC229810745", mission="TESS")
> print(sr)

 #     mission     year       author      exptime target_name distance       proposal_id      
                                             s                 arcsec                         
--- -------------- ---- ----------------- ------- ----------- -------- -----------------------
  0 TESS Sector 02 2018              SPOC     120   229810745      0.0                     N/A
  1 TESS Sector 13 2019              SPOC     120   229810745      0.0                     N/A
  2 TESS Sector 69 2023              SPOC     120   229810745      0.0                  G05003
  3 TESS Sector 68 2023              SPOC     120   229810745      0.0                  G05003
  4 TESS Sector 63 2023              SPOC     120   229810745      0.0                  G05003
  5 TESS Sector 67 2023              SPOC     120   229810745      0.0                  G05003
  6 TESS Sector 01 2018         TESS-SPOC    1800   229810745      0.0                     N/A
  7 TESS Sector 02 2018         TESS-SPOC    1800   229810745      0.0                     N/A
  8 TESS Sector 13 2019         TESS-SPOC    1800   229810745      0.0                     N/A
  9 TESS Sector 29 2020         TESS-SPOC     600   229810745      0.0                     N/A
...            ...  ...               ...     ...         ...      ...                     ...
 16 TESS Sector 63 2023               QLP     200   229810745      0.0                     N/A
 17 TESS Sector 02 2018             TASOC     120   229810745      0.0 G011160_G011155_G011188
 18 TESS Sector 01 2018              TGLC    1800   229810745      0.0                     N/A
 19 TESS Sector 02 2018              TGLC    1800   229810745      0.0                     N/A
 20 TESS Sector 02 2018             TASOC    1800   229810745      0.0 G011160_G011155_G011188
 21 TESS Sector 01 2018             TASOC    1800   229810745      0.0 G011160_G011155_G011188
 22 TESS Sector 01 2018             TASOC    1800   229810745      0.0 G011160_G011155_G011188
 23 TESS Sector 02 2018 GSFC-ELEANOR-LITE    1800   229810745      0.0                     N/A
 24 TESS Sector 02 2018             TASOC    1800   229810745      0.0 G011160_G011155_G011188
 25 TESS Sector 01 2018 GSFC-ELEANOR-LITE    1800   229810745      0.0                     N/A

---

version: v2.4.2

[orionlee]

@christinahedges The sort order change was slipped in in PR #1380 . I wonder if the change could break other people's codes (in addition to mine).

My actual use case is to download the lightcurves of a given TIC for all observation time, with the best available product/author for each sector (use case in #1048).

The change makes the resulting SearchResult / correspond LightCurveCollection no longer orders chronologically. I have to explicitly re-sort the SearchResult to compensate it.

[martindevora]

I'm experiencing the same and it is breaking my code too.

[christinahedges]

@orionlee so MAST uploaded a few new HLSPs that broke other stuff in lightkurve and this was my attempt at fixing it. I'm working on 2.5.0dev right now with a much better fix.

Let me take a crack at finishing this up and let's chat on that PR about what to do with v2.4.2 and find the best solution

[orionlee]

christinahedges: I have subscribed to PR #1383 for the new new HLSPs. Ping me when you're ready.

---

In the meantime, I use a helper to resort the SearchResult chronologically in my codes.

import lightkurve as lk

def _sort_chronologically(sr: lk.SearchResult):
    res = lk.SearchResult(sr.table.copy())
    res.table.sort(["distance", "year", "mission", "sort_order", "exptime"])
    return res