Meeting notes

February 1 2021

Sprint recap @ 10AM

General:

• Next spring Feb 18-19

Kate:

• build python notebook for PS
• finishing display code
• profiling
• gpu
• batch synthesis
• document differences btwn matlab and python of fwd pass and optimizaiton
• notebook with replication
• fix broken tests & add synthesis tests confirming solution

Lyndon:

• ask Alex about sizes of images on which frontend was trained
• ask him about masking
• remove steps from tqdm
• have frontend model have pretrained flag to update state
• typehinting uniformization

Billy:

• display code. pr not ready yet
• rgb images
• remove seaborn requirement
• remove pyrtools.imshow to po.imshow()

Pierre:

• ci build tests failing, but seems to be not our fault
• tools.signal.py has some redundant functions

January 29 2021

Sprint day 2 AM

Have post-sprint meeting on Monday Feb 01 10am instead of this afternoon.

Lyndon:

• almost done eigendist tutorial
• strange artifacts for front-end model | apply disk mask?
• internalize plotting functions

Pierre:

• merge billy's master changes
• many tests do not pass for some reason
• fair amount of documentation added

Billy:

• 2 simple PRs to review
• helping kate with optimization

Kate:

• matlab coarse-to-fine is done differently than ours
• getting to the bottom of high-freq artifacts

Nikhil:

• going thru line-by-line for pyramid

January 28 2021

Sprint day 1 AM

End of day meeting at 5pm

Priorities:

• Core tutorials!
• blitz tutorial
• advanced
• preliminary tutorials (loop Eero in at this point).

pierre:

• geodesic docs

kate:

• losses for P-S model

billy:

• go through lyndon's comments on PR

lyndon:

• tutorial cleaning

Tutorial 0/preliminaries:

• perhaps lives in its own separate folder for newcomers
• introduce ideas around linear model
• 1D signal
• how would we properly motivate geodesics for linear model?
• First principles, blitz 60minutes tutorial,

What should a blitz look like?

• one nonlinear model
• demo synthesis methods

January 22, 2021

Sprint next week:

• 11AM start
• 1 Hour discussion about tutorial 0 and arrangement of tutorials

Nikhil:

• complex tensor
• Synthesis superclass abstraction

Lyndon:

• eigendist tutorials
• display code

Billy:

• display issues
• synth display code doesnt work w rgb

Pierre:

• work on geodesic stuff from november sprint list
• tutorial0 (linear case)
• merge geodesic branch

Kate:

• plotting portilla-simoncelli stats in meaningful way
  • quantify convergence for overall and indiv statistic loss
  • quantify differences between matlab and python
• starting from white noise, end up w/ structured highfreq noise in result
  • starting with smaller Gaussian noise, result looks more reasonable
  • using plots to better assess/evaluate what is happening, e.g. divide by scale/type

Miscellaneous:

• python Black?
• docstring style

June 26, 2020

• for po.imshow, want to enforce 4d tensors because everything else requires it
  • also, handle complex tensors correctly
  • and double-check our model outputs
  • add po.pyrshow, using Nikhil's convert_pyrshow function
  • there are three cases for bchw:
    • multi-batch, 1 channel: can do automatically
    • multi-batch, RGB channel: can do automatically (but add an option to override the assumption that 3 channels means RGB)
    • multi-batch, multi-channel: can't do automatically, user must specify either 1 channel (and all batches) or 1 batch (and all channels)
• need to decide, at a hgih leve, what we want in tutorials and example folders. i.e., what content needs to be covered. this is a good thign to cover in our spring
• Nikhil is almost done with his steerable pyramid notebook, will push by next week. then tools_organizations branch will be ready for merge
• Kate's weird kernel crash is back, though there's a workaround. she'll open an issue to keep track
• Kate will start bringing normalization and coarse-to-fine into Portilla-Simoncelli texture (and look into speed)
• Billy should move generate_norm_stats and zscore_stats into tools/optim.py, on the abstractions branch. They may be useful for other cases?
• Lyndon is continuing to work on the eigendistortion tutorial, should be ready by next week (then we can merge the fix-autodiff branch?)
• we should figure out and make explicit what we do with multiple channels (i.e., we treat each separately because we care about grayscale)
• during sprint: look into CoLab / BinderHub for running tutorials in browser. CoLab has cool features, but Lyndon and Billy are a bit leery of relying on Google
• Billy wants to show code can re-generate existing MAD competition examples, so should reach out to Zhou Wang about this

July 17, 2020

• Billy should send Zhou Wang an email to see if he's comfortable sharing the MAD Competition results he sent
• Working through Nikhil's new notebook, the tutorial for steerable pyramid highlighting the differences between plenoptic and pyrtools implementations
  • spent a lot of time discussing relationship between downsampled and not-downsampled version, how to build tests for the new not-downsampled version. decided that the energy should be the same between them (relatively easy to check) and that if you sub-sample the not-downsampled version, the pyramid coefficients should be identical to the downsampled version in the pixel domain (no filtering required).
• discovered our python 3.7 build on Travis CI is broken, looks like it's a matplotlib issue that they're aware of and working on (and there's a work-around there)
• should probably move the models Pierre added to the tools_organizations branch to a new branch (and delete them from that branch) because they might need a bit more work
• pyramids are hard. we should add a link to David's handout somewhere in our documentation.

Once we add those tests, fix Travis build, and move Pierre's stuff, then we can merge that branch -- should be ready in next week. Lyndon and Billy's branches aiming to also merge in the next two weeks, before the sprint.

August Sprint

August 3, 2020

Starting the sprint! Will go from now until Tuesday, August 8.

• Will meet every day in the morning and evening for a quick stand up:
  • morning: everyone has ~5 minutes (hard cut-off at 6) to describe what they're working on in the coming day.
  • evening: everyone has ~5 minutes (hard cut-off at 6) describe what they worked on, what they accomplished, what they're stuck on, and set up potential meetings for help.
• Evening meeting is at 5:30pm EST, so will head to tea time afterwards.
• First, everyone wants to finish the stuff they were hoping on working on before the sprint started.
• Then, we'll work on documentation, tutorials, and examples.
  1. we want an intro page that basically answers "why should you use this?", so has some flashy examples for each method (pointing to literature), as well as examples of adversarial examples to motivate why it's important.
  2. "notebook 0": works through synthesis methods with a linear model y=Ax, focus on building intuition.
  3. Overview tutorials for the models: steerable pyramid, texture statistics, pooling windows (and ventral stream?)
  4. More detailed tutorial for each synthesis method (focus on use, point to papers for going into the math, defer to documentation for lots of details on attributes and arguments)
  5. Building new model, what do you need to ensure that it works well with our methods.
  6. Building new synthesis method? lowest priority, because this will probably be done by someone in the lab. would still be good to have in order to make sure that Synthesis superclass is well-documented
• 1 and 2 are probably the pages that will have the most input from Eero.
• would also be nice to have something like a review paper summarizing the whole synthesis approach.

August 4, 2020

Morning

• stand ups:
  • Billy:
    • worked on SSIM, documented plenoptic version and got it matching matlab versions, both standard and weighted (as in MAD paper)
    • meeting later this afternoon with Pierre to discuss SSIM more (don't understand all arguments, how to handle batch/channel)
    • will add tests to match matlab
    • using matlab weighted SSIM, generated images that look like those in the MAD paper, will check if plenoptic can generate similar and, if not, check the derivatives
  • Kate: NMA tech support caused issues with terminal, so will have to do clean install and re-set up
    • once that's done, will go back to Portilla-Simoncelli texture stats, making sure it can operate on 4d tensors, then run on many images, and write tests
      • for tests, won't be able to compare directly to MATLAB because of known differences. will get the whys and hows documented there, add tests comparing with known plenoptic output
  • Lyndon:
    • working on Jupyter notebook tutorial for eigendistrotions, using FrontEnd model and VGG16 to recreate results from Alex's paper
    • cleaning up documentation and inconsistencies, improving user experience for eigendistortions
  • Pierre:
    • working on animating subplots for steerable pyramid
    • wants to get geodesics working and simplify them
  • Nikhil:
    • examining pyramid, got real pyramid passing the energy-matching test
    • but discovered that complex is not -- how can both be tight frame?
• Travis CI
  • we had run into an issue where tests exceeded max time allowed, so will need to fix, probably using build matrices
  • Billy will add bugfix for new torch version to master, so others can pull in

SSIM

• batch: handle reasonably so that if both images have same number of batches, they get compared 1-to-1, otherwise can be many-to-1.
• remove size_average argument (always return mssim with batch and channel), and get weighted version to have same shape
• remove window_size and window arguments (can add later if people need)
• should normalize in 2d rather than 1d
• remove automatic ability for val_range, add more info in docs
• change the external MAD plots to say best/worst for both SSIM and MSE and have them in same order

Evening

• Travis CI stuff fixed on master, everyone should pull in.
• Nikhil figured out the complex pyramid issues, just needs to double-check reconstruction and then we can merge tomorrow.
• Lyndon needs to finish up some tutorials and then is also ready to merge.
• Billy will not get the recreation of original MAD Competition results on abstractions branch, instead finish up tests and merge tomorrow, then start new branch of that recreation.

August 5, 2020

Morning

• Billy:
  • needs to move coarse-to-fine together option over into abstractions branch if not there already
  • get msssim function working again, then add ssim/msssim tests
  • and merge!
  • (we'll probably not manage to do a full review)
• Kate:
  • for Portilla-Simoncelli texture stats, several stuff need doing:
    • working with batch and channel (currently, only 1x1xHxW)
    • make human-readable
    • tests that prove it's close enough to MATLAB, baseline for future-proofing
    • working with coarse-to-fine
    • re-run on database of images to get norm factor
  • how to prioritize? tests and human-readable first, then rest is more software-engineering-y and easier to bring others in
• Nikhil:
  • fixed pyramid should return tensor by default
  • write method to convert between tensor and dictionary
  • pyr_coeffs attribute should always be dictionray
• Lyndon: working on Jupyter Notebook tutorial
• Pierre: animations

hopefully by end of day, be read to merge branches from Nikhil, Lyndon, and Billy

August 6, 2020

Morning

• Pierre: animation is working, now needs to fix edge cases and then work on geodesics
• Lyndon: tests breaking on Travis but not locally, once that's done should be ready to merge
• Kate:
  • making sure Portilla-Simoncelli can be built on a pyramid (in addition to an image) for tests because small differences between matlab and python get amplified and lead to larger mis-matches
  • we think numpy and matlab are the same, and that the observed difference comes from small differences between numpy and torch's fft; they're within floating poitn error per element, but summed can get to like 10^-4
  • documenting why it's different with matlab
• Nikhil: making progress, but the reconstruction tests have broken
• Billy:
  • will be ready as soon as the comparison with MATLAB test is done, but how do we review? it's massive
  • docstrings are overhwhelming so: remove attribute documentation if they're the same as arguments, think about hiding attributes, remove TODOs, conceptual part of it belongs elsewhere
  • Kate will edit later today

Evening

• Nikhil: two new pyramid reconstruction errors, also exist in the pyrtools versions. they show up in odd-sized images and in pyramids with zero order. going to open an issue and document, then move on.
• Pierre: working on geodesics. would be nice to have an existing result to reproduce, not sure if we do.
• Lyndon: still working on tutorials, got sidetracked by weird error in testing. eigendistortion tests have all passed.
• Billy: tests done, still need to make changes to docs.

We'll have a meeting tomorrow morning (after stand-ups) to go over abstractions and how to use them.

August 7, 2020

Morning

• Lyndon: having trouble with FrontEnd distortions always showing up along edge. will create issue but won't wait to fix for merge, which we can do once tests pass
• Kate: still working through comparison to MATLAB, tracking down what comes from precision errors propagating and if any errors are additional
  • we could try using float64 to get more precision (MATLAB uses that precision by default), but that requires going back through plenoptic and prytools code to get everything initialized in float64 (initializing as float32 and converting to it afterwards does not help). pytorch has a set_default_dtype function, which would hopefully do this, but numpy does not
  • will look into texture branch and whether we can delete
• Nikhil: just one more test to pass, almost ready
• Pierre: geodesics code is coming together for simple example.
  • wants to move some stuff out of class, potentially into superclass? discuss about abstrations
  • wants to get spectral model working
• Billy: tests are done, still needs to finish docstrings changes

Abstractions

Billy, Pierre, and Lyndon discussing the Synthesis superclass and how to use it for geodesics and eigendistortions

• it feels like spatial masks, clamping, and an objective that penalizes range all should be considered "constraints" and grouped together in some way, but not sure the best way to do that.
• would be nice to organize arguments in some way, because there are a lot, but how?
• Billy will:
  • write up what attributes an inheriting class should have to make use of Synthesis's methods
  • rename: image -> signal, target -> base, matched -> synthesized; so names make more sense for all methods
  • change update_plot so it works for line plot as well. if there's only one line, will update that. if more than one, must specify the label (or pass an array that has the right number of observations
• then Pierre and Lyndon will start trying to adapt to their existing methods and reach out if/when any issues come up.

Evening

• Lyndon:
  • table frequency of implementation of FrontEnd for now
  • even with larger images, distortions for FrontEnd are still on edge (will open issue)
  • tutorial is almost done (will not include FrontEnd for now)
  • "it's like Zeno's Paradox"
• Nikhil: wants to avoid having to convert tensor to dict if input tensor and current pyr_coeffs don't have same number of scales/bands, should be fine.
• Pierre: fixed then broke geodesics, trying to simplify code
• Kate: stupid bug in generation of pyramids (in MATLAB), needs to see if that fixes most issues
• Billy: tests done, still working on docs

Our goal is now to merge the three open PRs by the end of the sprint

August 10, 2020

Morning

• Kate: tracking down Portilla-Simoncelli errors, setting up stuff to check systematically because they're not always consistent
• Lyndon: PR ready to merge, so everyone should look through on their own today and we'll merge at the end of the day. he'll look at adding documentation in the tools branch
• Pierre: tracking down some geodesics bugs
• Nikhil: check tests and update docs, but almost there
• Billy: finish docs

Evening

• Kate: long-standing bug was due to == instead of =, once that was fixed then everything seems to be passing when using float64 (though checkerboard error is higher than the rest. will get tests finalized, baseline version saved out, and then work on making it more human-readable in the several days after the sprint
• Nikhil: not enough time today, still cleaning up and working on tutorial, but hopes to get final push done tonight
• Lyndon: added docstrings to stuff on tools_organization branch, then helped Pierre on geodesics
• Pierre: solving issues on geodesics, but it's still not super well-behaved
• Billy: still working on docs

August 11, 2020

Final day

Morning

• Billy:
  • needs feedback on PR
  • will look at whether docs build and some plotting stuff if time
• Lyndon: merged PR!
• Kate: building tests, then trying to make things human readable. will want eyes later on the code to help with that
• Nikhil: just about to push, found bug in complex case but then will be done
• Pierre: geodesics, many small items on to do list. struggling to find good settings for a few different models.

Evening

• Billy: responded to Lyndon's suggestions, got docs working, started working on update_plot for lines
• Kate: writing tests, found and fixed a bug.
  • by end of week, will push tests into branch (and rename to replicate_portilla_simoncelli) and remove the stuff to remove
• Pierre: added progress bar, trying VGG16, still having trouble getting geodesics well-behaved
• Lyndon: got started computing eigenvectors using gradient descent (for making compatible with Synthesis)
• Nikhil: everything is pushed, merged from master. need eyes on PR and checking that everything works

Moving forward:

• by tomorrow's lab meeting, will merge Nikhil's branch, then Billy's. Then (later this week) Kate will remove the Portilla-Simoncelli stuff she wants to remove, then we'll make it public!
• we'll try to have another sprint in the fall, after Lyndon's quals in October
• more big picture goals:
  • reproduce MATLAB MAD Competition results
  • finalize Portilla-Simoncelli and reproduce results
  • get geodesics working
  • get geodesics and eigendistortions using Synthesis superclass
  • more models
  • interoperability with BrainScore
  • Nikhil's V2 model (once it's more stable, will use pre-trained version)
  • merge pooling branch (needs to be cleaned up more)
  • vision science tutorial experiment: how would you use this for a psychophysics experiment
  • getting other eyes on this (other folks in lab, Corey)
  • submitting to JOSS, writing up review paper