Oct 24, 2003
-------------

- Next topic: neural networks

- Biological motivation and metaphor
        - axon
        - dendrites
        - processes taking place inside a neuron

- Models of artificial neurons
        - weighting different inputs;
        - summing; and
        - thresholding

- What are neurons capable of modeling
        - linear planes in hyperspace
                - e.g., AND in 2D space
                - e.g., OR in 2D space
                - not XOR (why?)

- Realization of classical gates in neurons
        - weights for AND gate
        - weights for OR gate

- Any boolean function can be realized in at most 2 layers
        - use CNF
        - or DNF

- Two types of perceptrons
	- threshold perceptron
		- use sign(x) for thresholding
	- sigmoid squashing perceptron
		- use 1/(1+e^(-x)) for thresholding

- Learning algorithm for a threshold perceptron
        - called an "update rule"

- Learning algorithm for a sigmoidal perceptron
	- similar update rule, with an extra term

- Main characteristics of threshold perceptron update rule
        - backpropagate errors and assign blame
        - importance of learning rate, eta
                - typically small, like 0.1 or 0.01

- Algorithm converges when
        - points are linearly separable

- Another perceptron
        - sigmoid perceptron
        - differentiable, unlike threshold perceptron
        - even converges for XOR (to a "in between" value)