VisiData Plugins

Custom plugins for https://github.com/saulpw/visidata/

• vds3: Open Amazon S3 paths and objects
  • Demo
  • Installation
    • Install VisiData
    • Install the Plugin
      • Using the Plugin Framework (Recommended)
      • Manually
  • Usage
    • Open a file stored in S3
    • List all buckets
    • Browse a bucket's contents
    • Browse all CSV files in a bucket (glob matching)
    • Browse previous versions of objects
    • Join/combine objects
  • Configuration
  • Status
• kvpairs: Toggle values between lists of Key/Value pairs and dicts
  • Overview
  • Installation
• vfake_extensions: Niche addons for vfake
  • Overview
  • Installation
  • Usage
  • Autofake Demo
• vpager: Open long cell values in the system pager
  • Overview
  • Installation
  • Usage
• debugging_helpers: Integrate VisiData with debugging packages
  • Overview
  • Workflow
  • Notes
  • Demo
• parent_navigation: Helpers for navigating a parent sheet from its child
  • Overview
  • Installation
  • Usage
  • Demos
    • Parent/Child Sheet Navigation
    • Frequency Table "Zoom" Navigation
• vd_jmespath: Evaluate JMESPath expressions
  • Overview
  • Installation
  • Usage
• Contributing
• Acknowledgements

vds3: Open Amazon S3 paths and objects

Demo

Note: This plugin's functionality has been migrated into VisiData core. If you're using VisiData v2.12dev or higher, you no longer need this plugin!

Browse S3 with an interface like a console-based file explorer:

Use glob-matching to focus your search:

Installation

Install VisiData

Via pip:

pip3 install visidata

There is a comprehensive guide to various installation methods here. I prefer using pipx for the main install, and pipx inject to add plugin dependencies. Choose whichever install method works best for you, as long as you install VisiData 2.0 or higher.

Install the Plugin

This plugin can be installed using VisiData's built-in plugin framework, or manually.

Using the Plugin Framework (Recommended)

1. Start VisiData.

vd

2. Hit <Space>, type open-plugins and hit <Enter> to open the plugins sheet.

3. Scroll to the vds3 plugin and hit a to add it.

4. Wait for installation to complete, then exit VisiData.

5. Be sure that your ~/.visidatarc file contains the line:

import plugins

6. Restart VisiData.

Manually

1. Install s3fs:

pip3 install s3fs

2. Copy vds3.py to a plugins subdirectory inside your VisiData directory (by default, ~/.visidata):

mkdir -p ~/.visidata/plugins
cd path/to/visidata-plugins
cp plugins/vds3.py ~/.visidata/plugins

3. Add this line to your ~/.visidatarc file:

import plugins.vds3

Usage

Because this plugin builds on top of s3fs and boto3, it takes advantage of standard AWS CLI configuration settings.

Be sure that the AWS CLI is installed and configured to point to your desired AWS account.

Open a file stored in S3

vd 's3://my-bucket/path/to/file.json.gz'

List all buckets

vd 's3://'

Browse a bucket's contents

vd 's3://my-bucket'
vd 's3://my-bucket/path'

When browsing a bucket, VisiData will behave like a file explorer:

Enter: Opens the current file or directory as a new sheet. g+Enter: Open all selected files and directories.

q's behavior is unchanged (closes the current sheet), but while browsing a bucket it effectively becomes the "go up one level" command.

Browse all CSV files in a bucket (glob matching)

vd 's3://my-bucket/**/*.csv.gz

Since glob-matching can return results from multiple "directories" (S3 prefixes), the glob results sheet will display full object names rather than imitating a navigable directory hierarchy.

Browse previous versions of objects

Open an S3 path:

vd 's3://my-bucket'
vd 's3://my-bucket/path'

Hit ^V to toggle support for S3 versioning. When enabled, there will be an additional Latest? column along with a Version ID column that is hidden by default. Previous versions can be opened with Enter or g+Enter as usual.

Join/combine objects

From an S3 directory listings, select multiple objects and use & to join object contents into a single sheet. This uses the native VisiData join functionality under the hood, so the available join types match those described in VisiData's join documentation.

Configuration

This plugin's behavior can be tweaked with the following options:

vds3_glob (Default: True): Enable glob matching for S3 paths. Glob-matching will only kick in for path names which contain glob patterns (*, ?, [ or ]). However, it's possible to have S3 keys which contain those characters. In those cases, set this to False to explicitly disable glob-matching.

vds3_endpoint (Default: None): Specify a custom S3 endpoint. This can be useful for local testing, or for pairing this plugin with S3-compatible endpoints such as MinIO, Backblaze B2, etc.

Note: This sample ~/.visidatarc snippet defines local S3 endpoints to be used when specific AWS profiles are active. It assumes that if the moto or localstack AWS CLI profiles are active, you have a local moto server or localstack S3 service running on a specific port. For any other AWS profile it falls back to the AWS default endpoint. A block like this can help you naturally switch between endpoints based on context, rather than requiring command line switches.

profile_endpoint_map = {
    'localstack': 'http://localhost:4572',
    'moto': 'http://localhost:3000',
}
options.vds3_endpoint = profile_endpoint_map.get(os.environ.get('AWS_PROFILE'))

Options can be configured directly in a ~/.visidatarc file:

options.vds3_glob = False

Or specified at runtime:

vd --vds3-glob false 's3://my-bucket/file[?].json'

VisiData also supports changing options from the Options sheet inside the application. Jeremy Singer-Vine's tutorial is a helpful reference for that.

Status

This plugin is in a "minimally viable" state - focused on basic S3 read operations. Reading directly from S3 into pandas/dask dataframes is not currently supported, nor is writing to S3.

kvpairs: Toggle values between lists of Key/Value pairs and dicts

Overview

This plugin adds a pair of column-level convenience functions (from_entries and to_entries), which are similar to their jq counterparts. As of this writing, they're most useful for helping to break out tags from AWS API responses. For that specific case, this custom keybinding is a handy shortcut that composes with VisiData's existing "expand column" logic:

Sheet.addCommand(
    "gz{",
    "expand-tags",
    "expand_cols_deep(sheet, [sheet.colsByName['Tags'].from_entries()], cursorRow)"
)

In that scenario, assume we have a Tags column whose data looks like this:

[
    {"Key": "Environment", "Value": "production"},
    {"Key": "Name", "Value": "my-project"}
]

from_entries() turns that into this:

{
    "Environment": "production",
    "Name": "my-project"
}

And VisiData's expand_cols_deep() function (bound by default to () breaks that into Tags.Environment and Tags.Name columns, so each tag becomes a first-class VisiData column.

Installation

The kvpairs plugin is not currently included in VisiData's plugin framework. It can be installed manually by copying kvpairs.py to your local ~/.visidata/plugins directory and including import plugins.kvpairs in your ~/.visidatarc file.

vfake_extensions: Niche addons for vfake

Overview

VisiData's vfake plugin provides interactive access to some common Faker functionality. The extra bits in vfake_extensions are some personal customizations. They skew heavily toward AWS and are probably most useful as a reference/inspiration for other vfake customizations.

Installation

This plugin won't be included in VisiData, and probably shouldn't be added manually as-is either. If you find any pieces of vfake_extensions.py useful, transplant them into your own ~/.visidatarc file or personal plugin collection inside ~/.visidata/plugins.

Usage

VdCustomProvider could be a helpful reference if you have a need to define your own custom Faker generator functions for use with vfake.

The autofake functionality can save a lot of time if you repeatedly generate fake data for values that follow predictable patterns.

Autofake Demo

vpager: Open long cell values in the system pager

Overview

For cells that contain long strings, it can sometimes be easier to pass the value into an external viewer rather than relying on VisiData's line wrapping. This plugin supports arbitrary external commands. Unless otherwise specified it relies on the PAGER environment variable, and defaults to less.

Installation

• Option 1: Include the contents of vpager.py in your ~/.visidatarc file.

• Option 2:

  • Copy vpager.py to your local ~/.visidata/plugins directory.
  • Add import plugins.vpager to ~/.visidata/plugins/__init__.py
  • Add import plugins to ~/.visidatarc if it is not there already

• Define keybindings or options in your ~/.visidatarc file. Examples:

# Use spacebar as a custom command prefix, and shuffle other bindings
# around to accommodate that.
BaseSheet.unbindkey(' ')
vd.allPrefixes.append(' ')
vd.bindkeys[':'] = {'BaseSheet': 'exec-longname'}
BaseSheet.bindkey(' ;', 'split-col')

# Tell `open-cell-pager` to invoke `bat` rather than $PAGER
vd.options.vpager_cmd = '/usr/bin/env bat --paging=always'

# Space+Enter: Open a cell's values with the default viewer - `vpager_cmd`
#              if it's defined, otherwise $PAGER.
#
# Space+m: Open a cell's value in `glow` for prettier rendered markdown
#          in the terminal.
BaseSheet.bindkey(" Enter", "open-cell-pager")
BaseSheet.addCommand(" m", "open-cell-markdown", "cursorCol.pageValue(cursorRow, cmd='glow -p')")

Usage

Navigate to a cell with a long value, and use the View --> Open cell with menu to open that cell with an external program.

If you've defined custom keys for the open-cell-* group of commands, use those instead of menus.

debugging_helpers: Integrate VisiData with debugging packages

Overview

VisiData is a multi-threaded curses application, which can trip up some traditional console-based debugging tools. For example, vanilla pdb is a terrible fit for VisiData - the output is all over the place.

This plugin adds a --debugger option, initially supporting the following debuggers:

• PuDB
• remote-pdb
• web-pdb

Since the latter two wrap pdb, they will automatically use pdb++ if it's installed.

Workflow

Install a supported debugger via pip, then run VisiData with the --debugger option:

vd --debugger pudb sample_data/benchmark.csv

VisiData should immediately trigger a breakpoint. The behavior here varies by debugger:

• PuDB: Takes over your screen immediately
• remote-pdb: Awaits a connection (telnet 127.0.0.1 4444 or nc 127.0.0.1 4444 from another pane/window)
• web-pdb: Awaits a web connection (browse to http://localhost:5555)

Once the debugger is active, you can start poking around right away or continue execution with c. At that point, the debugger will set up an event handler for the interrupt signal. This plugin binds z^C (z, Ctrl-C) as an interrupt keybinding, so that becomes your interactive "break on demand" shortcut.

Notes

• I had issues with several of PuDB's shell options (ptpython, ptipython, bpython). I had more success setting up a modified bpython shell as a PuDB custom shell.
• PuDB works great as a full-screen debugger, but debugging from a separate terminal is also handy if you need to see the debugger without hiding the VisiData screen. VisiData and PuDB in separate panes of the same tmux window is a nice setup.
• Even with some careful debugger choices and configuration, VisiData and the debugger can sometimes draw over each other. When that happens, VisiData's ^L binding to redraw the screen is helpful.
• Despite all the links and notes here, I mostly debug with the VS Code Python extension which makes this plugin completely useless! 😃

Demo

parent_navigation: Helpers for navigating a parent sheet from its child

Overview

VisiData's split window feature enables interesting use cases like displaying a data set and frequency table simultaneously, or a master list of records and a child view of details. In that second case, it can be useful to keep focus in the child/detail view while navigating up and down in the parent view. This little plugin sets up keybindings for that.

Installation

1. Copy parent_navigation.py to your local ~/.visidata/plugins directory.
2. Add import plugins.parent_navigation to ~/.visidata/plugins/__init__.py
3. Add import plugins to ~/.visidatarc if it is not there already

This plugin adds commands but does not define its own keyboard shortcuts for them, since those are a matter of personal preference and the risk of collisions is high. Instead, you can define your own shortcuts in ~/.visidatarc. For reference, mine look like this:

TableSheet.bindkey(ALT + "j", "next-parent-row")
TableSheet.bindkey(ALT + "k", "prev-parent-row")

FreqTableSheet.bindkey(ALT + "j", "zoom-next-freqrow")
FreqTableSheet.bindkey(ALT + "k", "zoom-prev-freqrow")
FreqTableSheet.bindkey(ALT + ENTER, "zoom-cur-freqrow")

So j and k move in the current sheet, but with Alt they move in a parent sheet instead.

Usage

• With a sheet open in VisiData, dive into the details of a row (Enter) or cell (z + Enter)
• Use the next-parent-row and prev-parent-row commands to navigate a parent sheet, refreshing the child sheet as you go
  • Note: This pairs nicely with VisiData's support for split views, which allow you to see the parent and child sheets at the same time.

Demos

Parent/Child Sheet Navigation

Frequency Table "Zoom" Navigation

vd_jmespath: Evaluate JMESPath expressions

Overview

JMESPath is a query language for JSON data. This plugin adds VisiData commands to add columns or select rows based on JMESPath expressions.

Installation

• Option 1: Include the contents of vd_jmespath.py in your ~/.visidatarc file.
• Option 2:
  • Copy vd_jmespath.py to your local ~/.visidata/plugins directory.
  • Add import plugins.vd_jmespath to ~/.visidata/plugins/__init__.py
  • Add import plugins to ~/.visidatarc if it is not there already

This plugin adds commands but does not define its own keyboard shortcuts for them, since those are a matter of personal preference and the risk of collisions is high. Instead, you can define your own shortcuts in ~/.visidatarc. For reference, mine look like this:

from visidata import BaseSheet, vd

# Use space as a prefix key rather than to execute a command by name.
vd.bindkeys[':'] = {'BaseSheet': 'exec-longname'}
vd.allPrefixes.append('Space')

# Define JMESPath commands by adding a custom prefix to the built-in
# addcol/select/unselect commands.
BaseSheet.bindkey(' =', 'addcol-jmespath')
BaseSheet.bindkey(' |', 'select-jmespath')
BaseSheet.bindkey(' \\', 'unselect-jmespath')

Usage

Inside a sheet containing JSON data:

• addcol-jmespath adds a new column by evaluating a given expression against each row
• select-jmespath and unselect-jmespath toggle row selection based on an expression

Contributing

Please open an issue for any bugs, questions or feature requests. Pull requests welcome!

Acknowledgements

• VisiData is a slick tool - saulpw, anjakefala and other contributors have done a great job with it.
• jsvine's intro tutorial and plugins repo are excellent references.
• Dask's s3fs is a great foundation when you need to squint and pretend S3 is a filesystem.
• Thanks to geekscrapy and frosencrantz for testing and helping to improve these plugins.