Hey, thanks for these. I'll take the evening to read through them. It's the gradient descent calculation that I find to be convoluted. I understand the concept. But I haven't seen a clear example of how that's done. That's what I was trying to nail down with that output neuron example. And reading the encog-java source code wasn't any clearer either.
Thanks very much for the resources. Perhaps these docs will help. Tim Washington Interruptsoftware.ca On Mon, Oct 1, 2012 at 3:43 PM, Jim - FooBar(); <jimpil1...@gmail.com>wrote: > Hmmm...I am not sure I follow... > the error is simply (- target actual). Then you take the square of that > difference (2 reasons for that) and there you have the error per neuron at > the output layer. Then you simply add them up to get the entire network's > error. so far so good... > > the tricky bit comes now when you need to find out how much the error > depends on the output, inputs and weights so you can feed that in to the > gradient descent formula via which the weights will be adjusted. Basically > gradient descent says this: > > "the adjustment of each weight will be the negative of a constant theta > multiplied by the dependence of the previous weight on the error of the > network, which is the derivative of that error with respect to that weight." > > so essentially you're only looking to find the derivative of E (total > error) with respect to Wij , but in order to find that you will need to know > > > - how much the error depends on the output > - how much the output depends on the activation (which depends on the > weights) > > > It is not straight forward to type formulas here but I've put a relevant > document that describes the procedure exactly and the encog book which i > cannot guarantee it goes that deep, in my public dropbox for you. the link > is :https://dl.dropbox.com/u/45723414/for_Tim.tar.gz ...I'll leave it up > until tomorrow... > > hope that helps... > > Jim > > ps: the encog source did not help you? > > > > > On 01/10/12 19:41, Timothy Washington wrote: > > Hey, thanks for responding. See responses inlined. > > > On Mon, Oct 1, 2012 at 4:24 AM, Jim - FooBar(); <jimpil1...@gmail.com>wrote: > >> I've always found this page very good : >> >> http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html >> >> generally, the weight of a connection is adjusted by an amount >> proportional to the product of an error signal δ, on the unit k receiving >> the input and the output of the unit j sending this signal along the >> connection. You use the chain rule to calculate partial derivatives. >> >> also this looks quite detailed with regards to what you're asking: >> >> >> https://docs.google.com/viewer?a=v&q=cache:9ahoWLbap8AJ:www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.3-BackProp.pdf+the+partial+derivative+of+the+error+neural+nets&hl=en&gl=uk&pid=bl&srcid=ADGEEShVr8tHNIryQDGp0jJ2xv6JM40Ja4UFbcr2-QDgU1tK4Di_4mUTVgWsHdjHphbS7MaHmSn8VwB3dEnAQz-w4J-iyF1VJFklvLoDY5fxJbGaHsU0mio7XQYmGom0_cd7aZkUzhq5&sig=AHIEtbTyaLysuBdK7Bn_-cMzE88tqOzlag<https://docs.google.com/viewer?a=v&q=cache:9ahoWLbap8AJ:www.cedar.buffalo.edu/%7Esrihari/CSE574/Chap5/Chap5.3-BackProp.pdf+the+partial+derivative+of+the+error+neural+nets&hl=en&gl=uk&pid=bl&srcid=ADGEEShVr8tHNIryQDGp0jJ2xv6JM40Ja4UFbcr2-QDgU1tK4Di_4mUTVgWsHdjHphbS7MaHmSn8VwB3dEnAQz-w4J-iyF1VJFklvLoDY5fxJbGaHsU0mio7XQYmGom0_cd7aZkUzhq5&sig=AHIEtbTyaLysuBdK7Bn_-cMzE88tqOzlag> >> >> > Looking at page 9 of this document, I take the error derivative > calculation to be: > > Thus required derivative *∂En/∂w ji* is obtained by... > > > 1. Multiplying value of δ ( for the unit at output end of weight ) > 2. by value of z ( for unit at input end of weight ) > > > So take this output neuron and it's input weights. Let's say the total > neural network error is *-0.3963277746392987*. I just multiply that total > error by each weight, to get the bolded, green error values. So what > you're saying is that the error derivative *∂En/∂w ji *( or weight > change) for each input is that same error value, below in green. Is this > the case? > > :output-layer >> >> >> >> ({:calculated-error -1.1139741279964241, >> >> >> :calculated-value 0.9275622253607013, >> >> >> :inputs >> >> >> ({:error *-0.2016795955938916*, >> :calculated 0.48962608882549025, >> >> >> :input-id "583c10bfdbd326ba525bda5d13a0a894b947ffb", >> >> >> :weight 0.5088707087900713, >> >> >> :bias 0} >> >> >> {:error *-0.15359996014735702*, >> :calculated 0.3095962076691644, >> >> >> :input-id "583c10bfdbd326ba525bda5d13a0a894b947ffa", >> >> >> :weight 0.38755790024342773, >> >> >> :bias 0} >> >> >> {:error *-0.11659507401745359* >> :calculated 0.23938733624830652, >> >> >> :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff9", >> >> >> :weight 0.2941885012312543, >> >> >> :bias 0} >> >> >> {:error *-0.2784739949663631*, >> :calculated 0.6681581686752845, >> >> >> :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff8", >> >> >> :weight 0.7026355778870271, >> >> >> :bias 0} >> >> >> {:error *-0.36362550327135884*, >> :calculated 0.8430641676611533, >> >> >> :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff7", >> >> >> :weight 0.9174868039523537, >> >> >> :bias 0}), >> >> >> :id "583c10bfdbd326ba525bda5d13a0a894b947ff6"}) >> >> >> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
