线性回归与逻辑回归详解

线性回归和逻辑回归是机器学习中最基础也是最重要的算法之一。本文将从数学原理到代码实现全面讲解这两种算法。

线性回归

线性回归用于预测连续值输出，其假设函数为：

h_\theta(x) = \theta^T x = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + ... + \theta_n x_n

损失函数

线性回归使用均方误差（Mean Squared Error）作为损失函数：

J(\theta) = \frac{1}{2m} \sum_{i=1}^{m}(h_\theta(x^{(i)}) - y^{(i)})^2

梯度下降

使用梯度下降法更新参数：

\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)

具体更新规则为：

\theta_j := \theta_j - \alpha \frac{1}{m} \sum_{i=1}^{m}(h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)}

Python 实现

python

import numpy as np
class LinearRegression:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.theta = None
    def fit(self, X, y):
        m, n = X.shape
        self.theta = np.zeros(n)
        for _ in range(self.n_iterations):
            gradients = (1/m)  X.T @ (X @ self.theta - y)

            self.theta -= self.learning_rate  gradients
    def predict(self, X):
        return X @ self.theta
示例
X = np.array([[1, 1], [1, 2], [1, 3], [1, 4]])
y = np.array([2, 4, 6, 8])
model = LinearRegression(learning_rate=0.1, n_iterations=1000)
model.fit(X, y)
print(f"参数: {model.theta}")
print(f"预测 x=5: {model.predict([[1, 5]])[0]}")

逻辑回归

逻辑回归用于二分类问题，其假设函数为：

h_\theta(x) = \sigma(\theta^T x) = \frac{1}{1 + e^{-\theta^T x}}

其中 $\sigma(z)$ 是 sigmoid 函数：

\sigma(z) = \frac{1}{1 + e^{-z}}

损失函数

逻辑回归使用对数损失（Log Loss）：

J(\theta) = -\frac{1}{m} \sum_{i=1}^{m}[y^{(i)}\log(h_\theta(x^{(i)})) + (1-y^{(i)})\log(1-h_\theta(x^{(i)}))]

Python 实现

python

import numpy as np
class LogisticRegression:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.theta = None
    def sigmoid(self, z):
        return 1 / (1 + np.exp(-z))
    def fit(self, X, y):
        m, n = X.shape
        self.theta = np.zeros(n)
        for _ in range(self.n_iterations):
            h = self.sigmoid(X @ self.theta)
            gradients = (1/m)  X.T @ (h - y)

            self.theta -= self.learning_rate  gradients
    def predict(self, X):
        probabilities = self.sigmoid(X @ self.theta)
        return (probabilities >= 0.5).astype(int)
示例
X = np.array([[1, 2], [1, 3], [2, 1], [3, 1]])
y = np.array([0, 0, 1, 1])
model = LogisticRegression(learning_rate=0.1, n_iterations=1000)
model.fit(X, y)
print(f"预测: {model.predict(X)}")

总结

| 特性 | 线性回归 | 逻辑回归 |

|------|----------|----------|

| 输出 | 连续值 | 概率值 |

| 应用 | 回归任务 | 分类任务 |

| 激活函数 | 无 | Sigmoid |

| 损失函数 | MSE | Log Loss |