**ShapeConv**, or **Shape-aware Convolutional layer**, is a convolutional layer for processing the depth feature in indoor RGB-D semantic segmentation. The depth feature is firstly decomposed into a shape-component and a base-component, next two learnable weights are introduced to cooperate with them independently, and finally a [convolution](https://paperswithcode.com/method/convolution) is applied on the re-weighted combination of these two components.

ShapeConv

ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation

using transfer learning to transfer knowledge from one building to predict the energy consumption of another building with scarce data

b2b transfer learning

Transfer Learning in Deep Learning Models for Building Load Forecasting: Case of Limited Data

**HardELiSH** is an activation function for neural networks.  The HardELiSH is a multiplication of the [HardSigmoid](https://paperswithcode.com/method/hard-sigmoid) and [ELU](https://paperswithcode.com/method/elu) in the negative part and a multiplication of the Linear and the HardSigmoid in the positive
part:

$$f\left(x\right) = x\max\left(0, \min\left(1, \left(\frac{x+1}{2}\right)\right) \right) \text{ if } x \geq 1$$
$$f\left(x\right) = \left(e^{x}-1\right)\max\left(0, \min\left(1, \left(\frac{x+1}{2}\right)\right)\right) \text{ if } x < 0 $$

Source: [Activation Functions](https://arxiv.org/pdf/1811.03378.pdf)

HardELiSH

The Quest for the Golden Activation Function

**SegFormer** is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based framework for semantic segmentation that unifies Transformers with lightweight [multilayer perceptron](https://paperswithcode.com/method/feedforward-network) (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations.

SegFormer

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

**FAVOR+**, or **Fast Attention Via Positive Orthogonal Random Features**, is an efficient attention mechanism used in the [Performer](https://paperswithcode.com/method/performer) architecture which leverages approaches such as kernel methods and random features approximation for approximating [softmax](https://paperswithcode.com/method/softmax) and Gaussian kernels. 

FAVOR+ works for attention blocks using matrices $\mathbf{A} \in \mathbb{R}^{L×L}$ of the form $\mathbf{A}(i, j) = K(\mathbf{q}\_{i}^{T}, \mathbf{k}\_{j}^{T})$, with $\mathbf{q}\_{i}/\mathbf{k}\_{j}$ standing for the $i^{th}/j^{th}$ query/key row-vector in $\mathbf{Q}/\mathbf{K}$ and kernel $K : \mathbb{R}^{d } × \mathbb{R}^{d} \rightarrow \mathbb{R}\_{+}$ defined for the (usually randomized) mapping: $\phi : \mathbb{R}^{d } → \mathbb{R}^{r}\_{+}$ (for some $r > 0$) as:

$$K(\mathbf{x}, \mathbf{y}) = E[\phi(\mathbf{x})^{T}\phi(\mathbf{y})] $$

We call $\phi(\mathbf{u})$ a random feature map for $\mathbf{u} \in \mathbb{R}^{d}$ . For $\mathbf{Q}^{'}, \mathbf{K}^{'} \in \mathbb{R}^{L \times r}$ with rows given as $\phi(\mathbf{q}\_{i}^{T})^{T}$ and $\phi(\mathbf{k}\_{i}^{T})^{T}$  respectively, this leads directly to the efficient attention mechanism of the form:

$$ \hat{Att\_{\leftrightarrow}}\left(\mathbf{Q}, \mathbf{K}, \mathbf{V}\right) = \hat{\mathbf{D}}^{-1}(\mathbf{Q^{'}}((\mathbf{K^{'}})^{T}\mathbf{V}))$$

where

$$\mathbf{\hat{D}} = \text{diag}(\mathbf{Q^{'}}((\mathbf{K^{'}})\mathbf{1}\_{L})) $$

The above scheme constitutes the [FA](https://paperswithcode.com/method/dfa)-part of the FAVOR+ mechanism. The other parts are achieved by:

- The R part :  The softmax kernel is approximated though trigonometric functions, in the form of a regularized softmax-kernel SMREG, that employs positive random features (PRFs).
- The OR+ part : To reduce the variance of the estimator, so we can use a smaller number of random features, different samples are entangled to be exactly orthogonal using the Gram-Schmidt orthogonalization procedure.

The details are quite technical, so it is recommended you read the paper for further information on these steps.

FAVOR+

Rethinking Attention with Performers

**U2-Net** is a two-level nested U-structure architecture that is designed for salient object detection (SOD).  The architecture allows the network to go deeper, attain high resolution, without significantly increasing the memory and computation cost. This is achieved by a nested U-structure: on the bottom level, with a novel ReSidual U-block (RSU) module, which is able to extract intra-stage multi-scale features without degrading the feature map resolution; on the top level, there is a [U-Net](https://paperswithcode.com/method/u-net) like structure, in which each stage is filled by a RSU block.

U2-Net

U$^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection

**Conditional DBlock** is a residual based block used in the discriminator of the [GAN-TTS](https://paperswithcode.com/method/gan-tts) architecture. They are similar to the [GBlocks](https://paperswithcode.com/method/gblock) used in the generator, but without [batch normalization](https://paperswithcode.com/method/batch-normalization). Unlike the [DBlock](https://paperswithcode.com/method/dblock), the Conditional DBlock adds the embedding of the linguistic features after the first [convolution](https://paperswithcode.com/method/convolution).

Conditional DBlock

High Fidelity Speech Synthesis with Adversarial Networks

Understanding and explaining deep learning models is an imperative task. Towards this, we propose a method that obtains gradient-based certainty estimates that also provide [visual attention](https://paperswithcode.com/method/visual-attention) maps. Particularly, we solve for visual question answering task. We incorporate modern probabilistic deep learning methods that we further improve by using the gradients for these estimates. These have two-fold benefits: a) improvement in obtaining the certainty estimates that correlate better with misclassified samples and b) improved attention maps that provide state-of-the-art results in terms of correlation with human attention regions. The improved attention maps result in consistent improvement for various methods for visual question answering. Therefore, the proposed technique can be thought of as a tool for obtaining improved certainty estimates and explanations for deep learning models.

U-CAM

**Dilated Bottleneck with Projection Block** is an image model block used in the [DetNet](https://paperswithcode.com/method/detnet) convolutional neural network architecture. It employs a bottleneck structure with dilated convolutions to efficiently enlarge the receptive field. It uses a [1x1 convolution](https://paperswithcode.com/method/1x1-convolution) to ensure the spatial size stays fixed.

Dilated Bottleneck with Projection Block

DetNet: A Backbone network for Object Detection

**SqueezeNeXt** is a type of convolutional neural network that uses the [SqueezeNet](https://paperswithcode.com/method/squeezenet) architecture as a baseline, but makes a number of changes. First, a more aggressive channel reduction is used by incorporating a two-stage squeeze module. This significantly reduces the total number of parameters used with the 3×3 convolutions. Secondly, it uses separable 3 × 3 convolutions to further reduce the model size, and removes the additional 1×1 branch after the squeeze module. Thirdly, the network use an element-wise addition skip connection similar to that of [ResNet](https://paperswithcode.com/method/resnet) architecture.

SqueezeNeXt

SqueezeNext: Hardware-Aware Neural Network Design

**DetNASNet** is a convolutional neural network designed to be an object detection backbone and discovered through [DetNAS](https://paperswithcode.com/method/detnas) architecture search. It uses [ShuffleNet V2](https://paperswithcode.com/method/shufflenet-v2) blocks as its basic building block.

DetNASNet

DetNAS: Backbone Search for Object Detection

**Manifold Mixup** is a regularization method that encourages neural networks to predict less confidently on interpolations of hidden representations. It leverages semantic interpolations as an additional training signal, obtaining neural networks with smoother decision boundaries at multiple levels of representation. As a result, neural networks trained with Manifold Mixup learn class-representations with fewer directions of variance.

Consider training a deep neural network $f\left(x\right) = f\_{k}\left(g\_{k}\left(x\right)\right)$, where $g\_{k}$ denotes the part of the neural network mapping the input data to the hidden representation at layer $k$, and $f\_{k}$ denotes the
part mapping such hidden representation to the output $f\left(x\right)$. Training $f$ using Manifold Mixup is performed in five steps:

(1) Select a random layer $k$ from a set of eligible layers $S$ in the neural network. This set may include the input layer $g\_{0}\left(x\right)$.

(2) Process two random data minibatches $\left(x, y\right)$ and $\left(x', y'\right)$ as usual, until reaching layer $k$. This provides us with two intermediate minibatches $\left(g\_{k}\left(x\right), y\right)$ and $\left(g\_{k}\left(x'\right), y'\right)$.

(3) Perform Input [Mixup](https://paperswithcode.com/method/mixup) on these intermediate minibatches. This produces the mixed minibatch:

$$
\left(\tilde{g}\_{k}, \tilde{y}\right) = \left(\text{Mix}\_{\lambda}\left(g\_{k}\left(x\right), g\_{k}\left(x'\right)\right), \text{Mix}\_{\lambda}\left(y, y'\right
)\right),
$$

where $\text{Mix}\_{\lambda}\left(a, b\right) = \lambda \cdot a + \left(1 − \lambda\right) \cdot b$. Here, $\left(y, y'
\right)$ are one-hot labels, and the mixing coefficient
$\lambda \sim \text{Beta}\left(\alpha, \alpha\right)$ as in mixup. For instance, $\alpha = 1.0$ is equivalent to sampling $\lambda \sim U\left(0, 1\right)$.

(4) Continue the forward pass in the network from layer $k$ until the output using the mixed minibatch $\left(\tilde{g}\_{k}, \tilde{y}\right)$.

(5) This output is used to compute the loss value and
gradients that update all the parameters of the neural network.

Manifold Mixup

Manifold Mixup: Better Representations by Interpolating Hidden States

LXMERT is a model for learning vision-and-language cross-modality representations. It consists of a Transformer model that consists three encoders: object relationship encoder, a language encoder, and a cross-modality encoder. The model takes two inputs: image with its related sentence. The images are represented as a sequence of objects, whereas each sentence is represented as sequence of words. By combining the self-attention and cross-attention layers the model is able to generated language representation, image representations, and cross-modality representations from the input. The model is pre-trained with image-sentence pairs via five pre-training tasks: masked language modeling, masked object prediction, cross-modality matching, and image questions answering. These tasks help the model to learn both intra-modality and cross-modality relationships.

LXMERT

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

# WIPA: Wavelet-integrated, Identity Preserving, Adversarial network for Face Super-resolution
Pytorch implementation of WIPA: Super-resolution of very low-resolution face images with a **W**avelet Integrated, **I**dentity **P**reserving, **A**dversarial Network. 
# Paper:
[Super-resolution of very low-resolution face images with a Wavelet Integrated, Identity Preserving, Adversarial Network](https://www.sciencedirect.com/science/article/abs/pii/S0923596522000753?dgcid=coauthor).
You can download the pre-proof version of the article [here](https://drive.google.com/file/d/1GHWiCcScPF1PK4xozoRf-88Rytom-kvl/view?usp=sharing) but  please refer to the origital manuscript for citation.
## Citation
If you find this work useful for your research, please consider citing our paper:
```
@article{DASTMALCHI2022116755,
title = {Super-resolution of very low-resolution face images with a wavelet integrated, identity preserving, adversarial network},
journal = {Signal Processing: Image Communication},
volume = {107},
pages = {116755},
year = {2022},
issn = {0923-5965},
doi = {https://doi.org/10.1016/j.image.2022.116755},
url = {https://www.sciencedirect.com/science/article/pii/S0923596522000753},
author = {Hamidreza Dastmalchi and Hassan Aghaeinia},
keywords = {Super-resolution, Wavelet prediction, Generative Adversarial Networks, Face Hallucination, Identity preserving, Perceptual quality},
```
## Linkdin Profile:
**Hamidreza Dastmalchi linkdin profile:**
https://www.linkedin.com/in/hamidreza-dastmalchi-80bb4574/
## WIPA Algorithm
we present **Wavelet
Prediction blocks** attached to a **Baseline CNN network** to predict wavelet missing details of facial images. The
extracted wavelet coefficients are concatenated with original feature maps in different scales to recover fine
details. Unlike other wavelet-based FH methods, this algorithm exploits the wavelet-enriched feature maps as
complementary information to facilitate the hallucination task. We introduce a **wavelet prediction loss** to push
the network to generate wavelet coefficients. In addition to the wavelet-domain cost function, a combination of
**perceptual**, **adversarial**, and **identity loss** functions has been utilized to achieve low-distortion and perceptually
high-quality images while maintaining identity. The training scheme of the Wavelet-Integrated network with the combination of five loss terms is shown as below:
<p align="center">
  <img width="500" src="./block-diagram/WIPA-Training-Scheme.jpg">
</p>

## Datasets
The [Celebrity dataset](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) used for training the proposed FH algorithm. The database contains more than 200 K different face images under significant pose, illumination, and expression variations. In our experiment, two distinct groups of 20 thousand images are randomly selected from the CelebA dataset as our train and test dataset. In order to test the generalizing capacity of the method, we have further evaluated the performance of the proposed approach on [FW](http://vis-www.cs.umass.edu/lfw/) and [Helen dataset](http://www.ifp.illinois.edu/~vuongle2/helen/) too. All the testing and training images are roughly aligned using similarity transformation with landmarks detected by the well-known MTCNN network. The images are rescaled to the size of 128 × 128. The corresponding LR images are also constructed by down-sampling the HR images using bicubic interpolation. The experiments are accomplished in two different **scaling** factors of 8X and 16X with LR images of size 16 × 16 and 8 × 8, respectively.
 **Before starting to train or test the network**, you must put the training images in the corresponding folders:
- Put training images in “.\data\train” directory.
- Put celeba test images in “.\data\test\celeba” , lfw test images in “.\data\test\lfw” and helen test images in “.\data\test\helen”.

## Pretrained Weights
The pretrained weights can be downloaded [here](https://drive.google.com/drive/folders/18V1kPDHW6F05L0xOOODNHZHO566SA6iC?usp=sharing).

## Code
The codes are consisted of two main files: the **main.py** file for training the network and the **test.py** file for evaluating the algorithm with different metrics like PSNR, SSIM and verification rate.
### Training 
To train the network, simply run this code in Anaconda terminal:
```
>>python main.py
```
We designed different input arguments for controlling the training procedure. Please use --help command to see the available input arguments. 

#### Example: 
For example, to train the wavelet-integrated network through GPU with scale factor of 8, without having pre-trained model coefficients, with learning rate of 5e-5, you can simply run the following code in the terminal:
```
python main.py –scale 8 –wi_net “” –disc_net “” –wavelet_integrated True –lr 0.00005
```

### Testing
for evaluating (testing), simply run the following code in terminal:
```
>>python test.py
```
We have also developed different options as input arguments to control the testing procedure. You can evaluate psnr, ssim, fid score and also verification rate by the “test.py” file. To do this, you have to put the test images in the corresponding folders in data root at first.

#### Example: 
For example, to evaluate the psnr and ssim of a wavelet-integrated pretrained model in scale of 8 and save the super-resolved results in folder of “./results/celeba”, you can write the following code in the command window:
```
>> test.py --wavelet_integrated True --scale 8 --wi_net gen_net_8x --save_flag True --save_folder ./results/celeba --metrics psnr ssim
```
To estimate the fid score, you have to produce the super-resolved test images first. Therefore, if you have not generated the super-resolved images, you have to call –metrics psnr ssim with fid simultaneously. You can also add the acc option to the metrics to evaluate the verification rate of the model:
```
>>python test.py --wavelet_integrated True --scale 8 --wi_net gen_net_8x --save_flag True --save_folder ./results/celeba --metrics psnr ssim fid acc
```
### Demo 
In addition, we have developed a “demo.py” python file to demonstrate the results of some sample images in the “./sample_images/gt” directory. To run the demo file, simply write the following code in the terminal:
```
>>python demo.py
```
By default, the images of “./sample_images/gt” folder will be super-resolved by the wavelet-integrated network by a scale factor of 8 and the results will be saved in the “./sample_images/sr” folder. To change the scaling factor, one must alter not only the –scale option but also the corresponding –wi_net argument to import the relevant pretrained state dictionary.

WIPA

Contextualized Topic Models are based on the Neural-ProdLDA variational autoencoding approach by Srivastava and Sutton (2017). 

This approach trains an encoding neural network to map pre-trained contextualized word embeddings (e.g., [BERT](https://paperswithcode.com/method/bert)) to latent representations. Those latent representations are sampled variationally from a Gaussian distribution $N(\mu, \sigma^2)$ and passed to a decoder network that has to reconstruct the document bag-of-word representation.

Contextualized Topic Models

Cross-lingual Contextualized Topic Models with Zero-shot Learning

A **Neural Turing Machine** is a working memory neural network model. It couples a neural network architecture with external memory resources. The whole architecture is differentiable end-to-end with gradient descent. The models can infer tasks such as copying, sorting and associative recall.

A Neural Turing Machine (NTM) architecture contains two basic components: a neural
network controller and a memory bank. The Figure presents a high-level diagram of the NTM
architecture. Like most neural networks, the controller interacts with the external world via
input and output vectors. Unlike a standard network, it also interacts with a memory matrix
using selective read and write operations. By analogy to the Turing machine we refer to the
network outputs that parameterise these operations as “heads.”

Every component of the architecture is differentiable. This is achieved by defining 'blurry' read and write operations that interact to a greater or lesser degree with all the elements in memory (rather
than addressing a single element, as in a normal Turing machine or digital computer). The
degree of blurriness is determined by an attentional “focus” mechanism that constrains each
read and write operation to interact with a small portion of the memory, while ignoring the
rest. Because interaction with the memory is highly sparse, the NTM is biased towards
storing data without interference. The memory location brought into attentional focus is
determined by specialised outputs emitted by the heads. These outputs define a normalised
weighting over the rows in the memory matrix (referred to as memory “locations”). Each
weighting, one per read or write head, defines the degree to which the head reads or writes
at each location. A head can thereby attend sharply to the memory at a single location or
weakly to the memory at many locations

Neural Turing Machine

Neural Turing Machines

**Submanifold Convolution (SC)** is a spatially sparse [convolution](https://paperswithcode.com/method/convolution) operation used for tasks with sparse data like semantic segmentation of 3D point clouds. An SC convolution computes the set of active sites in the same way as a regular convolution: it looks for the presence of any active sites in its receptive field of size $f^{d}$. If the input has size $l$ then the output will have size $\left(l − f + s\right)/s$. Unlike a regular convolution, an SC convolution discards the ground state for non-active sites by assuming that the input from those sites is zero. For more details see the [paper](https://paperswithcode.com/paper/3d-semantic-segmentation-with-submanifold), or the official code [here](https://github.com/facebookresearch/SparseConvNet).

Submanifold Convolution

3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

**FuseFormer** is a [Transformer](https://paperswithcode.com/method/transformer)-based model designed for video inpainting via fine-grained feature fusion based on novel [Soft Split and Soft Composition](https://paperswithcode.com/method/soft-split-and-soft-composition) operations. The soft split divides feature map into many patches with given overlapping interval while the soft composition stitches them back into a whole feature map where pixels in overlapping regions are summed up. FuseFormer builds soft composition and soft split into its [feedforward network](https://paperswithcode.com/method/feedforward-network) for further enhancing subpatch level feature fusion.

Viet-Anh on Software

AI Glossary

Tech Terminology

ShapeConv

building to building transfer learning