Dr. Mark Humphrys

School of Computing. Dublin City University.

Home      Blog      Teaching      Research      Contact

My big idea: Ancient Brain


CA114      CA170

CA668      CA669      Projects

Infinite weights/thresholds are bad

Infinite thresholds

Say finite weights, and (positive) infinite threshold. Then output 0 no matter what input. Hidden unit always outputs 0. Nothing added to summed input of next layer. Hidden unit might as well not exist.

Say finite weights, minus infinite threshold. Then output 1 no matter what. Every summed input in next layer gets wjk.1 added to it. Might as well scrap wjk and just modify the threshold tk. No advantage in having 2 weights added to form the "threshold" instead of 1. So output 1 no matter what is useless as well.

Conclusion - Infinite threshold (with finite weights) seems to be useless.

Infinite weights

Say finite threshold, one weight is positive infinite, other weights finite. Then if input on that link is positive, output = 1, no matter what threshold is (so long as finite). If input negative, output = 0. Steep threshold at 0.

Say weight negative infinite, still steep threshold at 0, just any negative input leads to 1, any positive to 0.

finite t, infinite w, get steep threshold at 0 only

Is this useful?
Let's say:
xj = w1j I1 + w2j I2 + ... + wnj In
and a single weight wij is infinite, all others finite, and tj finite

Hidden node j does nothing except recognise whether the single input Ii is positive or negative.

It makes no difference what the other inputs are. The links from all the other inputs to hidden node j may as well not exist.

Also this is only of use in wij layer. In the wjk layer, a recogniser for whether yj is positive or not is useless - yj is always positive. It becomes a constant output 0 or 1.

In general, sigmoid is centred on t/n, so infinite weights/thresholds mean centred on 0 or plus or minus infinity

sig(n(x-t)) is centred on t.
For example, sig(5(x-3)) is centred on 3.

sig(nx-t) is centred on t/n.
For example, consider sig(5x-3) = sig(5(x-(3/5)))

Above is centred not on 3, but on 3/5.

sig(nx-t) centred on t/n:

  1. If t infinite, n finite, this is centred on plus or minus infinity, i.e. const output 0 or 1.
  2. As n goes to infinity, t finite, centre goes to zero, no matter what t is.
  3. To centre on non-zero, non-infinite, need t and n finite.

ancientbrain.com      w2mind.org      humphrysfamilytree.com

On the Internet since 1987.

Wikipedia: Sometimes I link to Wikipedia. I have written something In defence of Wikipedia. It is often a useful starting point but you cannot trust it. Linking to it is like linking to a Google search. A starting point, not a destination. I automatically highlight in red all links to Wikipedia and Google search and other possibly-unreliable user-generated content.