Locked up screen with large number of modules - how to debug?

[doctorfree]

I went overboard and loaded up my config.yml with a couple dozen modules spread over a large grid:

grid:
    columns: [54, 54, 54, 54, 54, 54, 74]
    rows: [12, 12, 12, 12, 12, 12, 12]

My wtf process occasionally hangs with this configuration. I disabled several modules and it seems to be ok now. I am in the tedious process of re-enabling modules one by one to try and provoke another hang and isolate the conditions a bit. I am running wtf on Ubuntu Linux 20.04.

My question is two-fold. First, are there known limitations on grid size, number of modules, or any other constraints? Is this system-dependent? And second, do you have any suggestions for ways to debug this?

Thanks for any advice.

The config.yml that causes a hang:

wtf:
  grid:
    columns: [54, 54, 54, 54, 54, 54, 74]
    rows: [12, 12, 12, 12, 12, 12, 12]
  refreshInterval: 1
  # Comment out or set to 'xterm-256color' if not using Kitty
  term: 'xterm-kitty'
  # openUrlUtil needed for Linux only if xdg-open is not working
  # Comment out for Mac OS or to use xdg-open
  # This execs whatever browser is configured in mimetypes
  # To specify a character browser, set to 'w3m' or 'lynx'
  openUrlUtil:
    - 'gio'
    - 'open'
  mods:
    github:
      # Set environment variable WTF_GITHUB_TOKEN
      # From https://github.com/settings/tokens
      enabled: true
      position:
        top: 0
        left: 0
        height: 2
        width: 1
      refreshInterval: 300
      customQueries:
        othersPRs:
          title: 'Others Pull Requests'
          filter: 'is:open is:pr -author:doctorfree'
        bugs:
          title: 'Bugs'
          filter: 'is:issue is:open label:bug'
        issues:
          title: 'Issues'
          filter: 'is:issue is:open'
      username: doctorfree
      repositories:
        - 'doctorfree/MusicPlayerPlus'
        - 'doctorfree/Asciiville'
        - 'doctorfree/DoctorFreeScripts'
    weather:
      # Set environment variable WTF_OWM_API_KEY
      # From https://home.openweathermap.org
      position:
        top: 0
        left: 1
        height: 1
        width: 1
      cityids:
        - 5393052
        - 6173331
        - 3128760
        - 6167865
        - 6176823
      colors:
        current: 'lightblue'
      enabled: true
      language: 'EN'
      refreshInterval: 900
      tempUnit: 'F'
      useEmoji: true
    digitalclock:
      color: orange
      enabled: true
      font: bigfont
      hourFormat: 12
      title: 'Time'
      position:
        top: 1
        left: 1
        height: 1
        width: 1
      refreshInterval: 1s
      type: 'digitalclock'
    ipapi:
      colors:
        name: red
        value: white
      # Optional  
      args:
        - 'ip'
        - 'isp'
        - 'as'
        - 'city'
        - 'regionName'
        - 'country'
        - 'continent'
        - 'coordinates'
        - 'organization'
        - 'timezone'
        - 'reverseDNS'
      enabled: true
      position:
        top: 0
        left: 2
        height: 1
        width: 1
      refreshInterval: 150s
    ipinfo:
      position:
        top: 1
        left: 2
        height: 1
        width: 1
      colors:
        name: red
        value: white
      enabled: true
      refreshInterval: 150
    security:
      enabled: true
      position:
        top: 0
        left: 3
        height: 1
        width: 1
      refreshInterval: 3600
    clocks:
      enabled: true
      position:
        top: 1
        left: 3
        height: 1
        width: 1
      colors:
        rows:
          even: 'lightblue'
          odd: 'white'
      locations:
        # From https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
        Santa Cruz: 'America/Los_Angeles'
        Enid: 'America/Chicago'
        London: 'Europe/London'
        Singapore: 'Asia/Singapore'
        Peru: 'America/Lima'
        New Zealand: 'Pacific/Auckland'
        New York: 'America/New_York'
        UTC: 'Etc/UTC'
      refreshInterval: 15
      sort: 'chronological'
    linux_subreddit:
      enabled: true
      numberOfPosts: 10
      position:
        top: 0
        left: 4
        height: 1
        width: 3
      refreshInterval: 15m
      sortOrder: top
      subreddit: 'linux'
      topTimePeriod: month
      type: subreddit
    cli_subreddit:
      enabled: true
      numberOfPosts: 10
      position:
        top: 1
        left: 4
        height: 1
        width: 3
      refreshInterval: 15m
      sortOrder: top
      subreddit: 'commandline'
      topTimePeriod: month
      type: subreddit
    prices_cryptolive:
      enabled: true
      title: 'Cryptocurrency Prices'
      position:
        top: 2
        left: 0
        height: 2
        width: 1
      updateInterval: 30
      currencies:
        ADA:
          displayName: Cardano
          to:
            - USD
        RNDR:
          displayName: Render Token
          to:
            - USD
        AGI:
          displayName: SingularityNET
          to:
            - USD
        COTI:
          displayName: COTI
          to:
            - USD
        GRT:
          displayName: The Graph
          to:
            - USD
        HNT:
          displayName: Helium
          to:
            - USD
      top:
      colors:
        from:
          name: coral
          displayName: grey
        to:
          name: white
          price: green
        top:
          from:
            name: grey
            displayName: coral
          to:
            name: red
            field: white
            value: green
      type: cryptolive
    ada_cryptolive:
      enabled: true
      title: 'Cardano (ADA) Markets'
      position:
        top: 2
        left: 1
        height: 2
        width: 1
      updateInterval: 30
      currencies:
        ADA:
          displayName: Cardano
          to:
            - USD
      top:
        ADA:
          displayName: Cardano
          limit: 5
          to:
            - USD
      colors:
        from:
          name: coral
          displayName: grey
        to:
          name: white
          price: green
        top:
          from:
            name: grey
            displayName: coral
          to:
            name: red
            field: white
            value: green
      type: cryptolive
    rndr_cryptolive:
      enabled: true
      title: 'Render Token (RNDR) Markets'
      position:
        top: 2
        left: 2
        height: 2
        width: 1
      updateInterval: 30
      currencies:
        RNDR:
          displayName: Render Token
          to:
            - USD
      top:
        RNDR:
          displayName: Render Token
          limit: 5
          to:
            - USD
      colors:
        from:
          name: coral
          displayName: grey
        to:
          name: white
          price: green
        top:
          from:
            name: grey
            displayName: coral
          to:
            name: red
            field: white
            value: green
      type: cryptolive
    agix_cryptolive:
      enabled: true
      title: 'SingularityNET (AGI) Markets'
      position:
        top: 2
        left: 3
        height: 2
        width: 1
      updateInterval: 30
      currencies:
        AGI:
          displayName: SingularityNET
          to:
            - USD
      top:
        AGI:
          displayName: SingularityNET
          limit: 5
          to:
            - USD
      colors:
        from:
          name: coral
          displayName: grey
        to:
          name: white
          price: green
        top:
          from:
            name: grey
            displayName: coral
          to:
            name: red
            field: white
            value: green
      type: cryptolive
    coti_cryptolive:
      enabled: true
      title: 'COTI Markets'
      position:
        top: 2
        left: 4
        height: 2
        width: 1
      updateInterval: 30
      currencies:
        COTI:
          displayName: COTI
          to:
            - USD
      top:
        COTI:
          displayName: COTI
          limit: 5
          to:
            - USD
      colors:
        from:
          name: coral
          displayName: grey
        to:
          name: white
          price: green
        top:
          from:
            name: grey
            displayName: coral
          to:
            name: red
            field: white
            value: green
      type: cryptolive
    grt_cryptolive:
      enabled: true
      title: 'The Graph (GRT) Markets'
      position:
        top: 2
        left: 5
        height: 2
        width: 1
      updateInterval: 30
      currencies:
        GRT:
          displayName: The Graph
          to:
            - USD
      top:
        GRT:
          displayName: The Graph
          limit: 5
          to:
            - USD
      colors:
        from:
          name: coral
          displayName: grey
        to:
          name: white
          price: green
        top:
          from:
            name: grey
            displayName: coral
          to:
            name: red
            field: white
            value: green
      type: cryptolive
    hnt_cryptolive:
      enabled: true
      title: 'Helium (HNT) Markets'
      position:
        top: 2
        left: 6
        height: 2
        width: 1
      updateInterval: 30
      currencies:
        HNT:
          displayName: Helium
          to:
            - USD
      top:
        HNT:
          displayName: Helium
          limit: 5
          to:
            - USD
      colors:
        from:
          name: coral
          displayName: grey
        to:
          name: white
          price: green
        top:
          from:
            name: grey
            displayName: coral
          to:
            name: red
            field: white
            value: green
      type: cryptolive
    news:
      type: hackernews
      title: 'HackerNews'
      enabled: true
      numberOfStories: 10
      position:
        top: 4
        left: 0
        height: 1
        width: 3
      storyType: top
      refreshInterval: 900
    disks:
      type: cmdrunner
      cmd: 'df'
      args: ['-h']
      enabled: true
      position:
        top: 4
        left: 3
        height: 1
        width: 2
      refreshInterval: 3600
    neofetch:
      type: cmdrunner
      args: ['--off']
      cmd: 'neofetch'
      pty: false
      enabled: true
      title: 'NeoFetch'
      position:
        top: 4
        left: 5
        height: 2
        width: 1
      refreshInterval: 3000
    lunarphase:
      enabled: true
      language: 'en'
      position:
        top: 4
        left: 6
        height: 2
        width: 1
    feedreader:
      enabled: true
      feeds:
      - https://rss.cbc.ca/lineup/topstories.xml
      - https://www.santacruzsentinel.com/feed/
      - https://www.mercurynews.com/location/california/bay-area/south-bay/santa-cruz-county/feed/
      - https://goodtimes.sc/feed/atom/
      - https://kion546.com/category/news/santa-cruz/feed/
      - https://www.santacruzsentinel.com/news/local-news/feed/
#     Subscription required to read articles
#     - http://feeds.washingtonpost.com/rss/national
#     - http://www.nytimes.com/services/xml/rss/nyt/HomePage.xml
      feedLimit: 10
      position:
        top: 5
        left: 0
        height: 1
        width: 2
      refreshInterval: 1h
    resources:
      type: resourceusage
      enabled: true
      position:
        top: 5
        left: 2
        height: 1
        width: 1
      refreshInterval: 1
    uptime:
      type: cmdrunner
      args: []
      cmd: 'uptime'
      enabled: true
      position:
        top: 5
        left: 3
        height: 1
        width: 2
      refreshInterval: 30

[senorprogrammer]

That's an interesting one, and thank you for posting your config and sorry you're running into this issue. I'll try to reproduce.

To answer your questions:

My question is two-fold. First, are there known limitations on grid size, number of modules, or any other constraints?

No, nothing known or intended.

And second, do you have any suggestions for ways to debug this?

The only way I can think of given the state of the system right now is what you're doing, manually removing modules and seeing what happens. Per-module performance benchmarking is definitely something that could help here, but doesn't exist yet.

I hope you'll keep this thread posted on anything you discover.

[doctorfree]

I am going through the process of enabling modules one-by-one in an effort to reliably reproduce this issue with a minimal set of modules and hopefully identify which module or modules are causing the hang. I have reproduced the issue by enabling my lunarphase module development version along with many other modules enabled. However, I cannot reproduce the issue with an enabled lunarphase either alone or in combination with a few other modules. It seems to be triggered by heavy load or the interaction with another module(s).

It is difficult to debug. It may take some time to experiment with a variety of combinations of modules enabled. But I will work on it.

I have cancelled the latest pull request for the lunarphase module until I get this sorted out. Thanks for answering my questions.

[doctorfree]

I can reproduce the hang using only the github module and/or the lunarphase module. The issue is triggered by rapid key presses that cause the module to issue another HTTP request. In the case of the github module this would be KeyLeft and KeyRight which switch to the previous or next listed repository. For the lunarphase module KeyLeft, KeyRight, KeyUp, and KeyDown can be used to change the day of the lunar phase causing another HTTP request to be made.

I don't yet know exactly where things are falling off the rails. Perhaps the flood of key presses is causing a previously made request to fail to close the Body or read to EOF before a new request is issued. Is there some asynchronous action involved in keyboard handling? I still have to investigate further. But, it appears the issue is not constrained to a single module (e.g. lunarphase) but rather to any module with keyboard input handling that generates an HTTP request. Maybe don't re-use the same client? I don't know.

Just reporting latest state of my testing. Any suggestions?

[Seanstoppable]

My initial thought here is that we shouldn't make an HTTP request when switching panes. Instead, leave those updates to happen in the background, and keypresses are then only iterating through cached content. If someone wants fresh data, there is still the refresh key that can be pressed.

[doctorfree]

I've implemented a slightly unsatisfactory workaround in the lunarphase module after reading the comment by @Seanstoppable above.

Key press methods that change the lunar phase day now simply modify widget.day and do not call widget.Refresh() which does not perform another HTTP request. The HTTP request is performed at the scheduled widget refresh at the end of the widget's RefreshInterval.

This fixes the hang caused by multiple rapid key presses triggering multiple rapid HTTP requests. However, it means the change in day is not reflected in the widget display for many seconds or minutes or however long there is to go to get through the refresh interval. The user can, of course, refresh manually by pressing r. That's not so bad I guess. Changing the lunar phase day means press some key (or multiple keys to move the day multiple days/weeks) then press r or just wait.

Perhaps there is a better solution. I will think about it.

[Seanstoppable]

Oh, if that is what the lunar module does, that indeed makes less sense for it (vs something like github which should have pretty discrete values).

For something like the lunar module, my next thought is around what about these requests is causing us to lock up? I wouldn't expect, for example, that we are causing a bunch of requests to queue up somewhere. But perhaps we are causing a number of refresh actions to queue that is causing a problem in the main refresh loop
I wonder if we should do one or more of the following:
Try using different HTTP clients for each call. I feel like this shouldn't make a difference, but seems like an easy thing to test
Block somehow such that we aren't able to stack multiple keypresses onto the queue. Essentially, wait until the keypress is resolved before allowing the next to resolve.
Pass a cancellable context to the HTTP client so when we are executing a new action that is going to cause an HTTP request, we cancel any outstanding one. Particularly for this module where we are only displaying the one record.

We may also want to consider storing a subset of fetched results in memory, given that if we want to go BACK to a given day, we shouldn't need to fetch information we've fetched previously.

[doctorfree]

Yes, getting to the root cause of this issue would be helpful. I chose to simply work around it in the lunarphase module. Some of the modules I tried did not provoke a lockup, the github module did provoke a lockup when switching between repositories rapidly. So maybe it is a timing issue and dependent upon the duration of the round trip. I did try using different HTTP clients in lunarphase and that did not help.

Blocking, context, and cache are all worth pursuing. I have other projects that require immediate attention. Thanks for your comments, they helped me get to a stable point. I will return to this as soon as I can.