开源软件名称（OpenSource Name）：

开源软件地址(OpenSource Url)：

开源编程语言(OpenSource Language)：

开源软件介绍(OpenSource Introduction)：

Classical ML Equations in LaTeX

A collection of classical ML equations in Latex . Some of them are provided with simple notes and paper link. Hopes to help writings such as papers and blogs.

Better viewed at https://blmoistawinde.github.io/ml_equations_latex/

• Classical ML Equations in LaTeX
  • Model
    • RNNs(LSTM, GRU)
    • Attentional Seq2seq
      • Bahdanau Attention
      • Luong(Dot-Product) Attention
    • Transformer
      • Scaled Dot-Product attention
      • Multi-head attention
    • Generative Adversarial Networks(GAN)
      • Minmax game objective
    • Variational Auto-Encoder(VAE)
      • Reparameterization trick
  • Activations
    • Sigmoid
    • Tanh
    • Softmax
    • Relu
    • Gelu
  • Loss
    • Regression
      • Mean Absolute Error(MAE)
      • Mean Squared Error(MSE)
      • Huber loss
    • Classification
      • Cross Entropy
      • Negative Loglikelihood
      • Hinge loss
      • KL/JS divergence
    • Regularization
      • L1 regularization
      • L2 regularization
  • Metrics
    • Classification
      • Accuracy, Precision, Recall, F1
      • Sensitivity, Specificity and AUC
    • Regression
    • Clustering
      • (Normalized) Mutual Information (NMI)
    • Ranking
      • (Mean) Average Precision(MAP)
    • Similarity/Relevance
      • Cosine
      • Jaccard
      • Pointwise Mutual Information(PMI)
  • Notes
  • Reference

Model

RNNs(LSTM, GRU)

encoder hidden state at time step , with input token embedding

decoder hidden state at time step , with input token embedding

h_t = RNN_{enc}(x_t, h_{t-1})
s_t = RNN_{dec}(y_t, s_{t-1})

The , are usually either

• LSTM (paper: Long short-term memory)

• GRU (paper: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation).

Attentional Seq2seq

The attention weight , the th decoder step over the th encoder step, resulting in context vector

c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j

\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}

e_{ij} = a(s_{i-1}, h_j)

is an specific attention function, which can be

Bahdanau Attention

Paper: Neural Machine Translation by Jointly Learning to Align and Translate

e_{ij} = v^T tanh(W[s_{i-1}; h_j])

Luong(Dot-Product) Attention

Paper: Effective Approaches to Attention-based Neural Machine Translation

If and has same number of dimension.

otherwise

e_{ij} = s_{i-1}^T h_j

e_{ij} = s_{i-1}^T W h_j

Finally, the output is produced by:

s_t = tanh(W[s_{t-1};y_t;c_t])
o_t = softmax(Vs_t)

Transformer

Paper: Attention Is All You Need

Scaled Dot-Product attention

Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V

where is the dimension of the key vector and query vector .

Multi-head attention

where

MultiHead(Q, K, V) = Concat(head_1, ..., head_h)W^O

head_i = Attention(Q W^Q_i, K W^K_i, V W^V_i)

Generative Adversarial Networks(GAN)

Paper: Generative Adversarial Networks

Minmax game objective

\min_{G}\max_{D}\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log{D(x)}] +  \mathbb{E}_{z\sim p_{\text{z}}(z)}[1 - \log{D(G(z))}]

Variational Auto-Encoder(VAE)

Paper: Auto-Encoding Variational Bayes

Reparameterization trick

To produce a latent variable z such that , we sample , than z is produced by

z \sim q_{\mu, \sigma}(z) = \mathcal{N}(\mu, \sigma^2)
\epsilon \sim \mathcal{N}(0,1)
z = \mu + \epsilon \cdot \sigma

Above is for 1-D case. For a multi-dimensional (vector) case we use:

\epsilon \sim \mathcal{N}(0, \textbf{I})
\vec{z} \sim \mathcal{N}(\vec{\mu}, \sigma^2 \textbf{I})

Activations

Sigmoid

Related to Logistic Regression. For single-label/multi-label binary classification.

\sigma(z) = \frac{1} {1 + e^{-z}}

Tanh

tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = \frac{1 - e^{-2x}}{1 + e^{-2x}}

Softmax

For multi-class single label classification.

\sigma(z_i) = \frac{e^{z_{i}}}{\sum_{j=1}^K e^{z_{j}}} \ \ \ for\ i=1,2,\dots,K

Relu

Relu(z) = max(0, z)

Gelu

where is the cumulative distribution function of Gaussian distribution.

Gelu(x) = x\Phi(x)

Loss

Regression

Below and are dimensional vectors, and denotes the value on the th dimension of .

Mean Absolute Error(MAE)

\sum_{i=1}^{D}|x_i-y_i|

Mean Squared Error(MSE)

\sum_{i=1}^{D}(x_i-y_i)^2

Huber loss

It’s less sensitive to outliers than the MSE as it treats error as square only inside an interval.

L_{\delta}=
    \left\{\begin{matrix}
        \frac{1}{2}(y - \hat{y})^{2} & if \left | (y - \hat{y})  \right | < \delta\\
        \delta ((y - \hat{y}) - \frac1 2 \delta) & otherwise
    \end{matrix}\right.

Classification

Cross Entropy

• In binary classification, where the number of classes equals 2, Binary Cross-Entropy(BCE) can be calculated as:

• If (i.e. multiclass classification), we calculate a separate loss for each class label per observation and sum the result.

-{(y\log(p) + (1 - y)\log(1 - p))}

-\sum_{c=1}^My_{o,c}\log(p_{o,c})

M - number of classes

log - the natural log

y - binary indicator (0 or 1) if class label c is the correct classification for observation o

p - predicted probability observation o is of class c

Negative Loglikelihood

Minimizing negative loglikelihood

is equivalent to Maximum Likelihood Estimation(MLE).

Here is a scaler instead of vector. It is the value of the single dimension where the ground truth lies. It is thus equivalent to cross entropy (See wiki).\

NLL(y) = -{\log(p(y))}

\min_{\theta} \sum_y {-\log(p(y;\theta))}

\max_{\theta} \prod_y p(y;\theta)

Hinge loss

Used in Support Vector Machine(SVM).

max(0, 1 - y \cdot \hat{y})

KL/JS divergence

KL(\hat{y} || y) = \sum_{c=1}^{M}\hat{y}_c \log{\frac{\hat{y}_c}{y_c}}

JS(\hat{y} || y) = \frac{1}{2}(KL(y||\frac{y+\hat{y}}{2}) + KL(\hat{y}||\frac{y+\hat{y}}{2}))

Regularization

The below can be any of the above loss.

L1 regularization

A regression model that uses L1 regularization technique is called Lasso Regression.

Loss = Error(Y - \widehat{Y}) + \lambda \sum_1^n |w_i|

L2 regularization

A regression model that uses L1 regularization technique is called Ridge Regression.

Loss = Error(Y - \widehat{Y}) +  \lambda \sum_1^n w_i^{2}

Metrics

Some of them overlaps with loss, like MAE, KL-divergence.

Classification

Accuracy, Precision, Recall, F1

Accuracy = \frac{TP+TN}{TP+TN+FP+FN}
Precision = \frac{TP}{TP+FP}
Recall = \frac{TP}{TP+FN}
F1 = \frac{2*Precision*Recall}{Precision+Recall} = \frac{2*TP}{2*TP+FP+FN}

Sensitivity, Specificity and AUC

Sensitivity = Recall = \frac{TP}{TP+FN}
Specificity = \frac{TN}{FP+TN}

AUC is calculated as the Area Under the (TPR)- (FPR) Curve.

Regression

MAE, MSE, equation above.