In image inpainting task, the mechanism extracts complementary features from the word embedding in two paths by reciprocal attention, which is done by comparing the descriptive text and complementary image areas through reciprocal attention.

**Deterministic Policy Gradient**, or **DPG**, is a policy gradient method for reinforcement learning. Instead of the policy function $\pi\left(.\mid{s}\right)$ being modeled as a probability distribution, DPG considers and calculates gradients for a deterministic policy $a = \mu\_{theta}\left(s\right)$.

Text-Guided Neural Image Inpainting

**ConvLSTM** is a type of recurrent neural network for spatio-temporal prediction that has convolutional structures in both the input-to-state and state-to-state transitions. The ConvLSTM determines the future state of a certain cell in the grid by the inputs and past states of its local neighbors. This can easily be achieved by using a [convolution](https://paperswithcode.com/method/convolution) operator in the state-to-state and input-to-state transitions (see Figure). The key equations of ConvLSTM are shown  below, where $∗$ denotes the convolution operator and $\odot$ the Hadamard product:

$$ i\_{t} = \sigma\left(W\_{xi} ∗ X\_{t} + W\_{hi} ∗ H\_{t−1} + W\_{ci} \odot \mathcal{C}\_{t−1} + b\_{i}\right) $$

$$ f\_{t} = \sigma\left(W\_{xf} ∗ X\_{t} + W\_{hf} ∗ H\_{t−1} + W\_{cf} \odot \mathcal{C}\_{t−1} + b\_{f}\right) $$

$$ \mathcal{C}\_{t} = f\_{t} \odot \mathcal{C}\_{t−1} + i\_{t} \odot \text{tanh}\left(W\_{xc} ∗ X\_{t} + W\_{hc} ∗ \mathcal{H}\_{t−1} + b\_{c}\right) $$

$$ o\_{t} = \sigma\left(W\_{xo} ∗ X\_{t} + W\_{ho} ∗ \mathcal{H}\_{t−1} + W\_{co} \odot \mathcal{C}\_{t} + b\_{o}\right) $$

$$ \mathcal{H}\_{t} = o\_{t} \odot \text{tanh}\left(C\_{t}\right) $$

If we view the states as the hidden representations of moving objects, a ConvLSTM with a larger transitional kernel should be able to capture faster motions while one with a smaller kernel can capture slower motions. 

To ensure that the states have the same number of rows and same number of columns as the inputs, padding is needed before applying the convolution operation. Here, padding of the hidden states on the boundary points can be viewed as using the state of the outside world for calculation. Usually, before the first input comes, we initialize all the states of the [LSTM](https://paperswithcode.com/method/lstm) to zero which corresponds to "total ignorance" of the future.

Source	Text-Guided Neural Image Inpainting
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Dual Multimodal Attention?

Viet-Anh on Software