ANN on GPU

There are quite a few publications now on implementing ANN on Graphics Processing Units (GPU) (see for example here, here and a brief review here). There are even a couple of programming libraries available that do this. The great advantage of using GPU are, of course, that GPU are massively parallel while being relatively cheap and ANN are inherently parallel models (this cheapness lends them to being used for other high-performance projects and GPU-based supercomputers are becoming more widely used, for example here and here).

I have yet to see, however, any publications describing constructive neural networks implemented on GPU. I suspect this may be because many constructive algorithms require some steps that are difficult to paralleise, such as a finding the maximum activation in a layer of neurons (which can be done in log2(n) iterations if you compare the values in pairs).

That said, I do see a very bright future for ANN research in using GPU. Definitely something I will be following more closely in the future.