Machine Learning Notes 04

Classification

  For the binary classification, \(y\in {0,1 }\)(Also maybe, \(y\in { 0,1,2,3,\cdots}\), that’s called a multiclass classification problem, we will discuss it later.).
  So, we use a model called Logisitic Regression, and we can see the hypothesis \(h_{\theta}(x)\) should value in the range of 0 and 1. This is to say, \(0\leq h_{\theta}(x)\leq 1\).

Hypothesis Representation

• Logistic Regression:     \(h_{\theta}(x)=g(\theta^{T}x)\), \(g(z)=\frac{1}{1+e^{-z}}\)

• Interpretation of Hypothesis Output. The value of \(h_{\theta}(x)\) equals to the estimated probability that y=1 (on input x, parameterized by \(\theta\) ). This is to say, \(h_{\theta}(x)=P(y=2\mid x ; \theta)\)

• Decision Boundary
  The decision boundaries are like this:

  Emphasis: Decision boundary is the property of hypothesis function, but not the property of training set and its parameters.

Logistic Regression—How to fit the parameters of theta

• Cost Function of Logistic Regression:
  \(Cost(h_{\theta}, y)=-ylog(h_{\theta}(x))-(1-y)log(1-h_{\theta}(x))\)

• Gradient Descent:
  To minimize \(J_{\theta}\)

   Repeat{
      \(\theta_{j}:=\theta_{j}-\alpha\frac{1}{m}\sum_{i=1}^{n}(h_{\theta}(x^{(i)})-y^{(i)})x^{(i)}\)
   }

  And we can see that this algortithm looks identical to linear regression!
  But actually, the hypothesis of them are different.
 Linear Regression: \(h_{\theta}(x)=\theta^{T}x\)
 Logistic Regression: \(h_{\theta}(x)=\frac{1}{1+e^{-\theta^{T}x}}\)

Besides,

“…use a vector rise implementation, so that a vector rise implementation can update all of these until parameters all in one fell swoop.”

Advanced Optimization

• Gradient Descent
• Conjngate Gradient
• BFGS
• L-BFGS
• ……

• Adcantages: No need to manually pick \(\alpha\) ; Often faster than gradient descent;

• Disadvantages: More complex;

Multi-class classification: One-vs-all

• For example, to slove the three-class problem, we can “turn this into three seperate two-class classification problems.”
• On a new input \(x\), to make a prediction, pick the class i that maximizes.