DPRG List  

[DPRG] neural nets fundamental flaw

Subject: [DPRG] neural nets fundamental flaw
From: Chris Jang christopher.jang at yahoo.com
Date: Thu May 29 01:42:35 CDT 2014

> You may be right but as several of the articles about this paper pointed
> out, the researchers have identified the problem in artificial neural
> nets but have not yet determined if biological neural nets share the
> issue.
This research is from a statistical learning (aka ML, machine learning) viewpoint. That is very much part of "big data" approach to problems. A few years ago, Netflix held a competition which came down to a showdown between two teams: BellKor Pragmatic Chaos, The Ensemble. The lesson from this big contest, which attracted a lot of high powered talent worldwide, is that ensembles of very different models are the only way (that we know how to do).

The three main approaches in the Netflix Prize were: 1. factor space models (e.g. latent semantic indexing aka singular value decomposition); 2. neighborhood clustering (e.g. K-means); 3. deep networks (aka neural nets). Hundreds of models were combined as experts in giant ensembles as recommendation systems. Everyone was led basically to the same approach.

Google's interest in deep networks will be in the context of this cultural viewpoint, big data problems at scale and with classifier ensembles. The result is interesting because it violates intuition. My naive take is that it says deep networks (always?) overfit for image classification problems. That isn't necessarily a problem if the neural networks exist in diverse ensembles. The intuition behind the ensemble is that the whole is more than the sum of the parts. Diversity covers weaknesses in the group while the strengths are preserved.

So to be more specific, this kind of deep network image classifier is probably useful for search. The link graph ("PageRank"), centrality in social graphs, keywords, etc will also be part of the system which returns search results. The neural network will never be used in isolation.

My opinion: I don't see any of these statistical techniques as having any direct relationship with biology. These techniques work (sort of...) pretty well. They may be inspired by biology. It is still engineering. They aren't solving the same problems.

More information about the DPRG mailing list