numpy - Logistic regression Python implementation

Question

I tried to implement logistic regression only with numpy in Python, but the result is not satisfying. The predictions seems incorrect and loss is not improving so it is probably something wrong with the code. Does anyone know what could fix it? Thank you very much!

Here is algorithm:

import numpy as np


# training data and labels
X = np.concatenate((np.random.normal(0.25, 0.1, 50), np.random.normal(0.75, 0.1, 50)), axis=None)
Y = np.concatenate((np.zeros((50,), dtype=np.int32), np.ones((50,), dtype=np.int32)), axis=None)

def logistic_sigmoid(a):
    return 1 / (1 + np.exp(-a))

# forward pass
def forward_pass(w, x):
    return logistic_sigmoid(w * x)

# gradient computation
def backward_pass(x, y, y_real):
    return np.sum((y - y_real) * x)

# computing loss
def loss(y, y_real):
    return -np.sum(y_real * np.log(y) + (1 - y_real) * np.log(1 - y))

# training
def train():
    w = 0.0
    learning_rate = 0.01
    i = 200
    test_number = 0.3

    for epoch in range(i):
        y = forward_pass(w, X)
        gradient = backward_pass(X, y, Y)
        w = w - learning_rate * gradient

        print(f'epoch {epoch + 1}, x = {test_number}, y = {forward_pass(w, test_number):.3f}, loss = {loss(y, Y):.3f}')


train()

question from:https://stackoverflow.com/questions/66051281/logistic-regression-python-implementation

Answer 1

At first glance you are missing you intercept term (typically called b_0, or bias) and its gradient update. Also in the backward_pass and loss calculations you are not dividing by the amount of data samples.

You can see two examples of how to implement it from scratch here:

1: Example based on Andrew Ng explanations in the Machine Learning course in Coursera

2: Implementation of Jason Brownlee from Machine Learning mastery website