Skip to content

Commit

Permalink
project abstract
Browse files Browse the repository at this point in the history
  • Loading branch information
@JY-Joy
JY-Joy committed Oct 24, 2024
1 parent 44ee013 commit a85ab45
Showing 1 changed file with 56 additions and 38 deletions.
94 changes: 56 additions & 38 deletions docs/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ <h1 class="title is-1 publication-title">InstantIR: Blind Image Restoration with
<div class="container is-max-desktop">
<div class="hero-body">
<script src="https://cdn.commoninja.com/sdk/latest/commonninja.js" defer></script>
<div class="commonninja_component pid-4f58272f-6d2b-4e92-b303-c670441777b9"></div>
<!-- <div class="commonninja_component pid-4f58272f-6d2b-4e92-b303-c670441777b9"></div> -->
<h2 class="subtitle has-text-centered">
InstantIR recovers extrem quality visual details from your image.
</h2>
Expand All @@ -197,13 +197,20 @@ <h2 class="subtitle has-text-centered">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
Handling test-time unknown degradation is the major challenge in Blind Image Restoration (BIR), necessitating high model generalization. An effective strategy is to incorporate prior knowledge, either from human input or generative model.
Handling test-time unknown degradation is the major challenge in Blind Image Restoration (BIR), necessitating high
model generalization. An effective strategy is to incorporate prior knowledge, either from human input or generative model.
</p>
<p>
We introduce Instant-reference Image Restoration (InstantIR), a novel diffusion-based BIR method which dynamically adjusts generation condition during inference. InstantIR first extracts a compact representation of the input via a pre-trained vision encoder. At each generation step, this representation is used to decode current diffusion latent and instantiate a generative prior. The degraded input is then encoded with this reference, providing robust generation condition.
We introduce Instant-reference Image Restoration (InstantIR), a novel diffusion-based BIR method which dynamically adjusts
generation condition during inference. InstantIR first extracts a compact representation of the input via a pre-trained
vision encoder. At each generation step, this representation is used to decode current diffusion latent and instantiate
a generative prior. The degraded input is then encoded with this reference, providing robust generation condition.
</p>
<p>
We observe the variance of generative references fluctuate with degradation intensity, which we further leverage as an indicator for developing a sampling algorithm adaptive to input quality. Extensive experiments demonstrate InstantIR achieves state-of-the-art performance and offering outstanding visual quality. Through modulating generative references with textual description, InstantIR can restore extreme degradation and additionally feature creative restoration.
We observe the variance of generative references fluctuate with degradation intensity, which we further leverage as an indicator
for developing a sampling algorithm adaptive to input quality. Extensive experiments demonstrate InstantIR achieves
state-of-the-art performance and offering outstanding visual quality. Through modulating generative references with
textual description, InstantIR can restore extreme degradation and additionally feature creative restoration.
</p>
</div>
</div>
Expand All @@ -216,18 +223,56 @@ <h2 class="title is-3">Abstract</h2>
<section class="section">
<div class="container is-max-desktop">

<!-- Method overview. -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">Pipelines</h2>

<!-- Interpolating. -->
<h3 class="title is-4">Qualitative comparison</h3>
<div class="content has-text-justified">
<p>
Through leveraging the generative prior, InstantIR.
</p>
</div>
<div class="columns is-vcentered interpolation-panel">
<div class="column is-3 has-text-centered">
<img src="./static/images/interpolate_start.jpg"
class="interpolation-image"
alt="Interpolate start reference image."/>
<p>Start Frame</p>
</div>
<div class="column interpolation-video-column">
<div id="interpolation-image-wrapper">
Loading...
</div>
<input class="slider is-fullwidth is-large is-info"
id="interpolation-slider"
step="1" min="0" max="100" value="0" type="range">
</div>
<div class="column is-3 has-text-centered">
<img src="./static/images/interpolate_end.jpg"
class="interpolation-image"
alt="Interpolation end reference image."/>
<p class="is-bold">End Frame</p>
</div>
</div>
<br/>
<!--/ Interpolating. -->
</div>
</div>
<!--/ Method overview. -->

<!-- Animation. -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">Visual Examples</h2>
<h2 class="title is-3">Restoration Examples</h2>

<!-- Interpolating. -->
<h3 class="title is-4">Interpolating states</h3>
<h3 class="title is-4">Qualitative comparison</h3>
<div class="content has-text-justified">
<p>
We can also animate the scene by interpolating the deformation latent codes of two input
frames. Use the slider here to linearly interpolate between the left frame and the right
frame.
Through leveraging the generative prior, InstantIR.
</p>
</div>
<div class="columns is-vcentered interpolation-panel">
Expand Down Expand Up @@ -263,7 +308,7 @@ <h3 class="title is-4">Interpolating states</h3>
<!-- Visual Effects. -->
<div class="column">
<div class="content">
<h2 class="title is-3">Text-guided Details Enhancement</h2>
<h3 class="title is-3">Text-guided Details Enhancement</h3>
<p>
Using <i>nerfies</i> you can create fun visual effects. This Dolly zoom effect
would be impossible without nerfies since it would require going through a wall.
Expand All @@ -278,7 +323,7 @@ <h2 class="title is-3">Text-guided Details Enhancement</h2>

<!-- Matting. -->
<div class="column">
<h2 class="title is-3">Text-guided Restoration-Editing</h2>
<h3 class="title is-3">Text-guided Restoration-Editing</h3>
<div class="columns is-centered">
<div class="column content">
<p>
Expand All @@ -296,33 +341,6 @@ <h2 class="title is-3">Text-guided Restoration-Editing</h2>
</div>
<!--/ Matting. -->

<!-- Concurrent Work. -->
<!-- <div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">Related Links</h2>
<div class="content has-text-justified">
<p>
There's a lot of excellent work that was introduced around the same time as ours.
</p>
<p>
<a href="https://arxiv.org/abs/2104.09125">Progressive Encoding for Neural Optimization</a> introduces an idea similar to our windowed position encoding for coarse-to-fine optimization.
</p>
<p>
<a href="https://www.albertpumarola.com/research/D-NeRF/index.html">D-NeRF</a> and <a href="https://gvv.mpi-inf.mpg.de/projects/nonrigid_nerf/">NR-NeRF</a>
both use deformation fields to model non-rigid scenes.
</p>
<p>
Some works model videos with a NeRF by directly modulating the density, such as <a href="https://video-nerf.github.io/">Video-NeRF</a>, <a href="https://www.cs.cornell.edu/~zl548/NSFF/">NSFF</a>, and <a href="https://neural-3d-video.github.io/">DyNeRF</a>
</p>
<p>
There are probably many more by the time you are reading this. Check out <a href="https://dellaert.github.io/NeRF/">Frank Dellart's survey on recent NeRF papers</a>, and <a href="https://github.com/yenchenlin/awesome-NeRF">Yen-Chen Lin's curated list of NeRF papers</a>.
</p>
</div>
</div>
</div> -->
<!--/ Concurrent Work. -->

</div>
</section>

Expand Down

0 comments on commit a85ab45

Please sign in to comment.