Remove tags matching `org-agenda-hide-tags' when building agendas

[kevinjfoley]

Have been using this locally to respect org-agenda-hide-tags in agenda views.

[alphapapa]

This is interesting, thanks.

However, I'm not sure if I want to add support for that variable to this package. I don't intend to reimplement all agenda features--at least, not in the current implementation of org-ql-view.el. I intend to rewrite the entry-formatting code, and rather than support the variety of special-purpose agenda-tweaking variables like this, I'd prefer to design more flexible systems. And I'd prefer not to add a feature like this now when my plan would be to take it away or redesign it in the future (there's too much pulling-the-rug-out-from-under-users already in this world).

What do you think?

[kevinjfoley]

I'm not sure if I want to add support for that variable to this package.

I understand, from a quick glance at the formatting code it seems like it probably evolved to address each need as it arose and could be more flexible.

I'd prefer not to add a feature like this now when my plan would be to take it away or redesign it in the future

My question is, do you foresee that a more flexible system wouldn't be able to support this feature? Or is it that you'd like to build out the new system and then add features without having to worry about too much backwards compatible from day one? Or maybe you intend to support it in a different way?

I understand the decision not to add this feature, just hoping to gain some insight on your vision and would be happy to help implementing it if I'm able.

[alphapapa]

The existing formatting code is not much more than the original proof-of-concept from when the package was called org-agenda-ng. It wasn't designed with flexibility in mind.

It's hard to say where the line should be drawn with regard to backward compatibility (or just compatibility--it's not really backward) with existing org-agenda code and features. I don't want this project to be constrained by what already exists in org-agenda. I'd rather take this as an opportunity for a fresh approach that is more designed than evolved, and one that allows for some trial-and-error (if 1.0 means anything, this project is nowhere near that yet). At the same time, I want the package to be usable now, and reasonably stable between versions (i.e. not breaking much between versions).

So, if that makes sense, I'd generally prefer not to add support for special options that affect the display of the agenda-like views. (Options that affect how Org buffers are interpreted are a different matter.)

If it helps any, you might want to look at this branch: https://github.com/alphapapa/org-ql/tree/wip/view-section. You can see what it looks like here: https://www.reddit.com/r/orgmode/comments/d11zfj/wip_magitlike_sections_for_orgql_views/ My long-term plan is to use something like that for org-ql-view views, with recursive grouping like what https://github.com/alphapapa/bufler.el provides for Emacs buffers. That would hopefully include sortable columns within groups, etc.

Given those intentions, it probably doesn't make sense to spend too much time on the formatting of individual items now, because any new entry formatter would probably have to be significantly changed.

What do you think?

Thanks for your work.

[kevinjfoley]

Thanks for the explanation and sharing the view-section branch/gif, it clarifies things.

Do you think that "1.0" would still support displaying org-ql results in org-agenda via orq-ql-block? Or do you think that in the long term org-ql-block will be deprecated in favor of purely org-ql based agenda views?

If the intention is to keep org-ql-block support I think the project would ideally have a way to handleorg-agenda settings, decoupled from org-ql-view. What do you think?

Regardless, I agree that org-ql-view--format-element isn't the best place for this and understand if there is no place for this to be implemented for now.

Apologies for any inaccuracies or misunderstandings above. I appreciate the time you've taken so far to explain things and of course all of the work on this and many other projects.

[alphapapa]

Somehow I clicked "Edit" on your comment instead of "Quote reply", and I didn't even realize it (the message header was off the top of the screen) until after I had edited your message. Let me try to fix this... Here's what I meant to be my reply:

Hi,

Sorry for the long delay.

Do you think that "1.0" would still support displaying org-ql results in org-agenda via orq-ql-block? Or do you think that in the long term org-ql-block will be deprecated in favor of purely org-ql based agenda views?

I see no reason to remove org-ql-block support. People aren't going to stop using org-agenda, so we should continue integrating with it to the extent possible. And I generally hate regressions and removing features. :)

If the intention is to keep org-ql-block support I think the project would ideally have a way to handleorg-agenda settings, decoupled from org-ql-view. What do you think?

That's an interesting idea. On one hand, it sounds reasonable. On the other, it sounds like it might require the maintenance of two parallel implementations, at some level, which I would certainly like to avoid.

So I would say that, with regard to compatibility with org-agenda features, it would have to be handled on a case-by-case basis. Features that are relatively simple to support would probably be okay (though I don't promise to implement any of them myself, or necessarily to merge PRs implementing them--I am picky about what code I merge, as you know ;).

Regardless, I agree that org-ql-view--format-element isn't the best place for this and understand if there is no place for this to be implemented for now.

My main concerns about this PR in particular are:

1. I want this code to continue to be as fast as possible. Sometimes I do searches that return many (i.e. hundreds, or even over a thousand) results from my Org files.
2. It adds complexity.

So to merge it, I'd want to benchmark it against the existing code to measure any performance decrease. And I'd probably want to consider implementing it in a flexible way so maybe more could be done than just hiding certain tags. IOW, if we're going to add a special case of sorts to this rather inflexible function, I'd rather do it in a flexible way rather than pile on complexity and special cases--if you know what I mean.

What do you think?

Apologies for any inaccuracies or misunderstandings above. I appreciate the time you've taken so far to explain things and of course all of the work on this and many other projects.

Thanks for the kind words and for your contributions.

[kevinjfoley]

I'd want to benchmark it against the existing code to measure any performance decrease

This is good to know. Do you think it would be helpful to have some kind of performance regression test as part of the CI suite? Or are you just interested in kind of an ad hoc check?

And I'd probably want to consider implementing it in a flexible way so maybe more could be done than just hiding certain tags. IOW, if we're going to add a special case of sorts to this rather inflexible function, I'd rather do it in a flexible way rather than pile on complexity and special cases--if you know what I mean.

I think that makes perfect sense. I kind of just threw this in haphazardly in the place that made the most immediate sense but it's probably not smart for the long term.

I was actually thinking recently that ideally org-agenda would handle this. A user could pass a list of entries data, regardless of whether it came from an org-agenda search or an external tool like org-ql and then it would handle all of the formatting per the org-agenda options.

However looking through the org-agenda code this looks really difficult to accomplish. The formatting code seems spread across many places with duplicated logic in many places.

Given the challenge it would be to refactor org-agenda I do think it would be nice (and more feasible) to implement it in a clean way in org-ql.

I'm not 100% sure what that should look like but if you have any ideas I'd be happy to explore them. Otherwise I can put more thought into and share what I come up with.

[alphapapa]

I'd want to benchmark it against the existing code to measure any performance decrease

This is good to know. Do you think it would be helpful to have some kind of performance regression test as part of the CI suite? Or are you just interested in kind of an ad hoc check?

In general, that sounds like an interesting idea. I'm not sure how practical it would be to do so. It would certainly be nice to have. I'm open to suggestions about how it could be done.

In my own ad hoc tests, I use these macros to compare implementations: https://github.com/alphapapa/emacs-package-dev-handbook#bench-multi-macros

I was actually thinking recently that ideally org-agenda would handle this. A user could pass a list of entries data, regardless of whether it came from an org-agenda search or an external tool like org-ql and then it would handle all of the formatting per the org-agenda options.

That's an interesting idea. For org-ql-view, I wouldn't want to go that direction, because I started this project to get away from org-agenda.el. However, it might make sense to do that for org-ql-block's output so it would fit naturally into regular agendas. It might not even require much code, because we already generate strings that have compatible text properties.

However looking through the org-agenda code this looks really difficult to accomplish. The formatting code seems spread across many places with duplicated logic in many places.

Yes, that's why I started this project (which I originally called org-agenda-ng). :)

Given the challenge it would be to refactor org-agenda I do think it would be nice (and more feasible) to implement it in a clean way in org-ql.

I'm not 100% sure what that should look like but if you have any ideas I'd be happy to explore them. Otherwise I can put more thought into and share what I come up with.

My long-term plans are to implement a replacement for org-agenda-mode buffers as a new version of org-ql-view. It would probably use magit-section or something like it, as in the proof of concept I linked in an earlier message. I'd probably do something similar to bufler-define-column for formatting entries, although one of these days I need to look into format-spec and see if it would make sense to use that. The idea would be to have a simple, flexible way to define custom formatting for entries. And I'd probably use something like bufler-group-tree to provide a new entry-grouping system to replace the use of org-super-agenda in this package.

I'm exploring and learning as I work on these different packages that have some overlapping ideas. I still have a long way to go. :)

And, yes, please do continue sharing your ideas! Thanks very much.

[alphapapa]

I'm not sure where this is going or what its status is, so I'm removing it from the 0.6 milestone.

[alphapapa]

FYI, you can see some WIP at https://github.com/alphapapa/org-ql/tree/wip/taxy-org-ql-view and https://www.reddit.com/r/orgmode/comments/pecpsw/coming_soonish_taxyorgqlsearch_sort_of_like_an/