Files

papers
- 2016
- 2017
- 2019
- 2020
- 2021
  - 210121 Thoughts on recent papers.md
  - 210216 Non-Autoregressive Text Generation with Pre-trained Language Models.md
  - 210223 Do Transformer Modifications Transfer Across Implementations and.md
  - 210223 Linear Transformers Are Secretly Fast Weight Memory Systems.md
  - 210223 Pyramid Vision Transformer.md
  - 210223 Zero-Shot Text-to-Image Generation.md
  - 210224 Zero-Shot Text-to-Image Generation.md
  - 210225 Evolving Attention with Residual Convolutions.md
  - 210225 IBRNet.md
  - 210225 LazyFormer.md
  - 210225 Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling.md
  - 210225 Simple multi-dataset detection.md
  - 210225 SparseBERT.md
  - 210226 Transformer in Transformer.md
  - 210227 Thoughts on recent papers.md
  - 210227 Ultra-Data-Efficient GAN Training.md
  - 210302 Fixing Data Augmentation to Improve Adversarial Robustness.md
  - 210304 Barlow Twins.md
  - 210305 Fixing Data Augmentation to Improve Adversarial Robustness 2.md
  - 210305 Thoughts on recent papers.md
  - 210310 Involution.md
  - 210312 Revisiting ResNets.md
  - 210316 You Only Look One-level Feature.md
  - 210317 Learning to Resize Images for Computer Vision Tasks.md
  - 210317 Training GANs with Stronger Augmentations via Contrastive Discriminator.md
  - 210318 AlignMix.md
  - 210318 All NLP Tasks Are Generation Tasks.md
  - 210318 CrossViT.md
  - 210318 CvT.md
  - 210318 Drop the GAN.md
  - 210318 FastNeRF.md
  - 210318 Few-shot Semantic Image Synthesis Using StyleGAN Prior.md
  - 210318 GNeRF.md
  - 210318 GPT Understands, Too.md
  - 210318 HiT.md
  - 210318 Labels4Free.md
  - 210318 Looking Beyond Two Frames.md
  - 210318 MVSNeRF.md
  - 210318 Multi-Scale Vision Longformer.md
  - 210318 NeMI.md
  - 210318 TFPose.md
  - 210318 TransCenter.md
  - 210318 Transformer Trackin.md
  - 210318 TrivialAugment.md
  - 210318 Using latent space regression to analyze and leverage compositionality.md
  - 210319 ConViT.md
  - 210319 Paint by Word.md
  - 210319 Scalable Visual Transformers with Hierarchical Pooling.md
  - 210319 Thoughts on recent papers.md
  - 210323 Scaling Local Self-Attention for Parameter Efficient Visual Backbones.md
  - 210323 Thoughts on recent papers.md
  - 210324 A Broad Study on the Transferability of Visual Representations with Contrastive Learning.md
  - 210324 Can Vision Transformers Learn without Natural Images.md
  - 210324 Finetuning Pretrained Transformers into RNNs.md
  - 210324 Mip-NeRF.md
  - 210324 Vision Transformers for Dense Prediction.md
  - 210325 An Image is Worth 16x16 Words, What is a Video Worth.md
  - 210325 Contrasting Contrastive Self-Supervised Representation Learning Models.md
  - 210325 KiloNeRF.md
  - 210325 PlenOctrees for Real-time Rendering of Neural Radiance Fields.md
  - 210325 Rethinking Self-Supervised Learning.md
  - 210325 Swin Transformer.md
  - 210325 USB.md
  - 210326 Thoughts on recent papers.md
  - 210330 Dual Contrastive Loss and Attention for GANs.md
  - 210331 EfficientNetV2.md
  - 210331 Going deeper with Image Transformers.md
  - 210401 Partition-Guided GANs.md
  - 210401 Towards General Purpose Vision Systems.md
  - 210402 LeViT.md
  - 210402 Robust wav2vec 2.0.md
  - 210403 Thoughts on recent papers.md
  - 210405 An Empirical Study of Training Self-Supervised Vision Transformers.md
  - 210407 Multimodal Fusion Refiner Networks.md
  - 210407 Pushing the Limits of Non-Autoregressive Speech Recognition.md
  - 210407 Regularizing Generative Adversarial Networks under Limited Data.md
  - 210407 Revisiting Simple Neural Probabilistic Language Models.md
  - 210407 Seeing Out of tHe bOx.md
  - 210408 InfinityGAN.md
  - 210408 Modulated Periodic Activations for Generalizable Local Functional.md
  - 210408 SI-Score.md
  - 210409 Efficient Large-Scale Language Model Training on GPU Clusters.md
  - 210409 Not All Attention Is All You Need.md
  - 210410 UniDrop.md
  - 210412 Thoughts on recent papers.md
  - 210413 DatasetGAN.md
  - 210413 Few-shot Image Generation via Cross-domain Correspondence.md
  - 210413 Masked Language Modeling and the Distributional Hypothesis.md
  - 210413 The Curious Case of Hallucinations in Neural Machine Translation.md
  - 210414 Aligning Latent and Image Spaces to Connect the Unconnectable.md
  - 210415 GANcraft.md
  - 210415 Rethinking Text Line Recognition Models.md
  - 210415 See through Gradients.md
  - 210417 Demystifying the Better Performance of Position Encoding Variants for.md
  - 210417 TransVG.md
  - 210417 mT6.md
  - 210418 Data-Efficient Language-Supervised Zero-Shot Learning with.md
  - 210420 Carbon Emissions and Large Neural Network Training.md
  - 210420 PP-YOLOv2.md
  - 210420 RoFormer.md
  - 210421 Token Labeling.md
  - 210422 ImageNet-21K Pretraining for the Masses.md
  - 210422 Multiscale Vision Transformers.md
  - 210422 On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation.md
  - 210422 So-ViT.md
  - 210423 M3DeTR.md
  - 210423 VidTr.md
  - 210424 Thoughts on recent papers.md
  - 210426 EigenGAN.md
  - 210426 Improve Vision Transformers Training by Suppressing Over-smoothing.md
  - 210426 MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding.md
  - 210426 Multimodal Contrastive Training for Visual Representation Learning.md
  - 210426 Visformer.md
  - 210427 ConTNet.md
  - 210428 Twins.md
  - 210429 A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning.md
  - 210429 Decoupled Dynamic Filter Networks.md
  - 210429 Emerging Properties in Self-Supervised Vision Transformers.md
  - 210429 Ensembling with Deep Generative Views.md
  - 210429 MarioNette.md
  - 210429 Thoughts on recent papers.md
  - 210429 With a Little Help from My Friends.md
  - 210430 Thoughts on recent papers 1.md
  - 210430 Thoughts on recent papers 2.md
  - 210502 Larger-Scale Transformers for Multilingual Masked Language Modeling.md
  - 210503 ISTR.md
  - 210505 Beyond Self-attention.md
  - 210505 QueryInst.md
  - 210505 RepMLP.md
  - 210505 Thoughts on recent papers.md
  - 210506 ACORN.md
  - 210506 DiffSinger.md
  - 210506 Do You Even Need Attention.md
  - 210507 Pareto-Optimal Quantized ResNet Is Mostly 4-bit.md
  - 210508 Thoughts on recent papers.md
  - 210509 Conformer.md
  - 210509 FNet.md
  - 210510 Are Pre-trained Convolutions Better than Pre-trained Transformers.md
  - 210510 GSPMD.md
  - 210510 Poolingformer.md
  - 210510 ResMLP.md
  - 210510 Self-Supervised Learning with Swin Transformers.md
  - 210510 Visual Grounding with Transformers.md
  - 210511 Diffusion Models Beat GANs on Image Synthesis.md
  - 210511 VICReg.md
  - 210512 Enhancing Photorealism Enhancement.md
  - 210512 Segmenter.md
  - 210512 When Does Contrastive Visual Representation Learning Work.md
  - 210515 Are Convolutional Neural Networks or Transformers more like human vision.md
  - 210517 Divide and Contrast.md
  - 210517 Pay Attention to MLPs.md
  - 210517 Rethinking the Design Principles of Robust Vision Transformer.md
  - 210521 Intriguing Properties of Vision Transformers.md
  - 210524 Self-Attention Networks Can Process Bounded Hierarchical Languages.md
  - 210524 StructuralLM.md
  - 210524 True Few-Shot Learning with Language Models.md
  - 210524 Unsupervised Speech Recognition.md
  - 210526 Aggregating Nested Transformers.md
  - 210528 ByT5.md
  - 210528 Gotta Go Fast When Generating Data with Score-Based Models.md
  - 210529 Less is More.md
  - 210531 On Fast Sampling of Diffusion Probabilistic Models.md
  - 210601 Comparing Test Sets with Item Response Theory.md
  - 210601 Exploring the Diversity and Invariance in Yourself for Visual Pre-Training Task.md
  - 210601 Incorporating Visual Layout Structures for Scientific Text Classification.md
  - 210601 Towards Real-time and Light-weight Line Segment Detection.md
  - 210601 You Only Look at One Sequence.md
  - 210603 DynamicViT.md
  - 210603 Luna.md
  - 210603 The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models.md
  - 210603 When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations.md
  - 210604 RegionViT.md
  - 210604 SOLQ.md
  - 210606 On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation.md
  - 210606 Referring Transformer.md
  - 210606 Rethinking Training from Scratch for Object Detection.md
  - 210607 Learning to Efficiently Sample from Diffusion Probabilistic Models.md
  - 210607 Refiner.md
  - 210607 Shuffle Transformer.md
  - 210607 ViTAE.md
  - 210608 Compacter.md
  - 210608 DETReg.md
  - 210608 Data-Efficient Instance Generation from Instance Discrimination.md
  - 210608 FastSeq.md
  - 210608 Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks.md
  - 210608 Scaling Vision Transformers.md
  - 210608 SpeechBrain.md
  - 210609 CoAtNet.md
  - 210610 Cascaded Diffusion Models for High Fidelity Image Generation.md
  - 210610 Score-based Generative Modeling in Latent Space.md
  - 210612 D2C.md
  - 210613 Thinking Like Transformers.md
  - 210614 Delving Deep into the Generalization of Vision Transformers under Distribution Shifts.md
  - 210614 Improved Transformer for High-Resolution GANs.md
  - 210614 Non Gaussian Denoising Diffusion Models.md
  - 210614 SAS.md
  - 210615 BEiT.md
  - 210615 Dynamic Head.md
  - 210615 Revisiting the Calibration of Modern Neural Networks.md
  - 210617 Layer Folding.md
  - 210617 LoRA.md
  - 210617 Multi-head or Single-head.md
  - 210617 WaveGrad 2.md
  - 210617 XCiT.md
  - 210623 Alias-Free Generative Adversarial Networks.md
  - 210623 Charformer.md
  - 210623 Stable, Fast and Accurate.md
  - 210624 Exploring Corruption Robustness.md
  - 210624 VOLO.md
  - 210624 Video Swin Transformer.md
  - 210628 Early Convolutions Help Transformers See Better.md
  - 210630 Using AntiPatterns to avoid MLOps Mistakes.md
  - 210701 CSWin Transformer.md
  - 210701 Focal Self-attention for Local-Global Interactions in Vision Transformers.md
  - 210701 Variational Diffusion Models.md
  - 210705 Long-Short Transformer.md
  - 210705 What Makes for Hierarchical Vision Transformer.md
  - 210706 Depth-supervised NeRF.md
  - 210706 Rethinking Positional Encoding.md
  - 210712 Combiner.md
  - 210713 Per-Pixel Classification is Not All You Need for Semantic Segmentation.md
  - 210713 Visual Parser.md
  - 210714 Deduplicating Training Data Makes Language Models Better.md
  - 210714 HTLM.md
  - 210715 Beyond Goldfish Memory.md
  - 210715 Internet-Augmented Dialogue Generation.md
  - 210716 Autonomy 2.0.md
  - 210718 AS-MLP.md
  - 210718 YOLOX.md
  - 210725 H-Transformer-1D.md
  - 210726 Contextual Transformer Networks for Visual Recognition.md
  - 210726 Continental-Scale Building Detection from High Resolution Satellite Imagery.md
  - 210726 Text is Text, No Matter What.md
  - 210727 A Tale Of Two Long Tails.md
  - 210727 Is Object Detection Necessary for Human-Object Interaction Recognition.md
  - 210728 SimROD.md
  - 210729 Batch Active Learning at Scale.md
  - 210730 Perceiver IO.md
  - 210730 Soft Calibration Objectives for Neural Networks.md
  - 210731 CrossFormer.md
  - 210802 SDEdit.md
  - 210809 Making Transformers Solve Compositional Tasks.md
  - 210809 NeuralMVS.md
  - 210811 ConvNets vs. Transformers.md
  - 210811 DEMix Layers.md
  - 210812 Mobile-Former.md
  - 210813 Conditional DETR for Fast Training Convergence.md
  - 210813 Curriculum Learning.md
  - 210816 On the Opportunities and Risks of Foundation Models.md
  - 210819 Do Vision Transformers See Like Convolutional Neural Networks.md
  - 210819 ImageBART.md
  - 210824 SimVLM.md
  - 210825 Multi-Task Self-Training for Learning General Representations.md
  - 210826 Train Short, Test Long.md
  - 210827 Injecting Text in Self-Supervised Speech Pretraining.md
  - 210830 A Battle of Network Structures.md
  - 210830 Shatter.md
  - 210902 Do Prompt-Based Models Really Understand the Meaning of their Prompts.md
  - 210902 Skim-Attention.md
  - 210903 Finetuned Language Models Are Zero-Shot Learners.md
  - 210906 An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models.md
  - 210906 Parsing Table Structures in the Wild.md
  - 210907 Beyond Preserved Accuracy.md
  - 210907 Datasets.md
  - 210907 How much pretraining data do language models need to learn syntax.md
  - 210907 Rare Words Degenerate All Words.md
  - 210908 A Recipe For Arbitrary Text Style Transfer with Large Language Models.md
  - 210908 Panoptic SegFormer.md
  - 210908 Scaled ReLU Matters for Training Vision Transformers.md
  - 210909 Bag of Tricks for Optimizing Transformer Efficiency.md
  - 210909 PIMNet.md
  - 210910 Instance-Conditioned GAN.md
  - 210910 LibFewShot.md
  - 210910 Panoptic Narrative Grounding.md
  - 210910 ReasonBERT.md
  - 210913 Pose with Style.md
  - 210913 STraTA.md
  - 210914 Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition.md
  - 210914 Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding.md
  - 210917 Primer.md
  - 210922 Pix2seq.md
  - 210922 Recursively Summarizing Books with Human Feedback.md
  - 210922 Scale Efficiently.md
  - 210927 BigSSL.md
  - 210927 Compressive Visual Representations.md
  - 210927 PASS.md
  - 210927 WarpedGANSpace.md
  - 210928 VideoCLIP.md
  - 210929 Localizing Objects with Self-Supervised Transformers and no Labels.md
  - 210929 Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization.md
  - 211005 Exploring the Limits of Large Scale Pre-training.md
  - 211006 8-bit Optimizers via Block-wise Quantization.md
  - 211006 How BPE Affects Memorization in Transformers.md
  - 211008 KaraSinger.md
  - 211011 Unsupervised Neural Machine Translation with Generative Language Models Only.md
  - 211012 Open-Set Recognition.md
  - 211012 Rethinking supervised pre-training for better downstream transferring.md
  - 211012 Word Order Does Not Matter For Speech Recognition.md
  - 211013 ADOP.md
  - 211013 ByteTrack.md
  - 211014 Non-deep Networks.md
  - 211015 Breaking Down Multilingual Machine Translation.md
  - 211015 MaGNET.md
  - 211015 Multitask Prompted Training Enables Zero-Shot Task Generalization.md
  - 211016 Invariant Language Modeling.md
  - 211016 MarkupLM.md
  - 211016 Sharpness-Aware Minimization Improves Language Model Generalization.md
  - 211017 AE-StyleGAN.md
  - 211018 HRFormer.md
  - 211018 Learning in High Dimension Always Amounts to Extrapolation.md
  - 211018 NormFormer.md
  - 211018 Understanding Dimensional Collapse in Contrastive Self-supervised Learning.md
  - 211018 Unsupervised Finetuning.md
  - 211019 CIPS-3D.md
  - 211019 The Cocktail Fork Problem.md
  - 211020 Shaking the foundations.md
  - 211025 DocTr.md
  - 211025 The Efficiency Misnomer.md
  - 211026 Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers.md
  - 211026 Hierarchical Transformers Are More Efficient Language Models.md
  - 211026 NeRV.md
  - 211026 Revisiting Batch Normalization.md
  - 211026 WavLM.md
  - 211027 Neural Analysis and Synthesis.md
  - 211027 Training Verifiers to Solve Math Word Problems.md
  - 211028 Colossal-AI.md
  - 211028 DocScanner.md
  - 211030 Mastering Atari Games with Limited Data.md
  - 211030 Pseudo-Labeling for Massively Multilingual Speech Recognition.md
  - 211031 Efficiently Modeling Long Sequences with Structured State Spaces.md
  - 211101 PP-PicoDet.md
  - 211101 Projected GANs Converge Faster.md
  - 211103 An Empirical Study of Training End-to-End Vision-and-Language Transformers.md
  - 211103 LAION-400M.md
  - 211103 One Pass ImageNet.md
  - 211103 VLMo.md
  - 211111 Masked Autoencoders Are Scalable Vision Learners.md
  - 211118 Swin Transformer V2.md
  - 211122 Benchmarking Detection Transfer Learning with Vision Transformers.md
  - 211122 ExT5.md
  - 211122 Florence.md
  - 211122 L-Verse.md
  - 211122 MetaFormer is Actually What You Need for Vision.md
  - 211122 Neural Fields in Visual Computing and Beyond.md
  - 211123 Dynamic DETR.md
  - 211124 NÜWA.md
  - 211124 Sparse is Enough in Scaling Transformers.md
  - 211129 Blended Diffusion for Text-driven Editing of Natural Images.md
  - 211129 Deblur-NeRF.md
  - 211129 HDR-NeRF.md
  - 211129 Sparse DETR.md
  - 211129 Urban Radiance Fields.md
  - 211130 Diffusion Autoencoders.md
  - 211130 Show Your Work.md
  - 211201 Revisiting the Transferability of Supervised Pretraining.md
  - 211202 Improved Multiscale Vision Transformers for Classification and Detection.md
  - 211202 Masked-attention Mask Transformer for Universal Image Segmentation.md
  - 211210 Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition.md
  - 211210 CityNeRF.md
  - 211210 Deep ViT Features as Dense Visual Descriptors.md
  - 211210 Self-attention Does Not Need $O(n^2)$ Memory.md
  - 211213 GLaM.md
  - 211213 Step-unrolled Denoising Autoencoders for Text Generation.md
  - 211215 Efficient Geometry-aware 3D Generative Adversarial Networks.md
  - 211215 SPTS.md
  - 211215 Value Retrieval with Arbitrary Queries for Form-like Documents.md
  - 211216 Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling.md
  - 211216 GRAM.md
  - 211216 Masked Feature Prediction for Self-Supervised Visual Pre-Training.md
  - 211217 A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation.md
  - 211220 3D-aware Image Synthesis via Learning Structural and Textural Representations.md
  - 211220 Are Large-scale Datasets Necessary for Self-Supervised Pre-training.md
  - 211220 Efficient Large Scale Language Modeling with Mixtures of Experts.md
  - 211220 GLIDE.md
  - 211220 High-Resolution Image Synthesis with Latent Diffusion Models.md
  - 211227 Augmenting Convolutional networks with attention-based aggregation.md
  - 211228 LINDA.md
  - 221125 Solving math word problems with process- and outcome-based feedback.md
  - 221204 Languages You Know Influence Those You Learn.md
  - 221215 Constitutional AI.md
- 2022
- 2023
- arXiv
- reviews
LICENSE
README.md

210607 ViTAE.md

Apr 28, 2022

3aef252 · Apr 28, 2022

https://arxiv.org/abs/2106.03348

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (Yufei Xu, Qiming Zhang, Jing Zhang, Dacheng Tao)