An Introduction to Artificial Intelligence
Pages: 1, 2, 3, 4
Image Classification
Now that we've covered the fundamentals and worked through some of the basics, let's consider a problem with widespread applicability: image classification. This is a problem that the postal service faces every day when routing mail, and neural networks excel at these types of tasks. Given that a zip code contains five (or 9) numbers, the problem becomes a matter of producing a digital output value corresponding to a series of symbols on an image.
To show how neural networks can be used to automate mail processing, let's work through the subproblem of identifying individual numeric codes. We'll train the network with some samples of several different fonts that correspond to handwriting samples, and then test the network with a totally different set of data to see how it performs. After all, for AI to be useful, it must be able to perform well even when faced with new information.
In order to supply and train the network with input values, we'll scale a series of images to a reasonable size and provide each of the pixels in the image to the network as a discrete input value. A very simple file format to use for this task is the PNM format. Starting out with files of other formats, we can use the mogrify command to convert to a PNM file very simply. Type which mogrify in Terminal to determine if you already have mogrify installed on your machine (or try to find it using Spotlight). If you don't get a path back to the command, you'll need to install it with Fink or directly from imagemagick's website.
To use mogrify to produce a 30 by 30 text-based file with 16 colors from a binary JPG file, for example, just type mogrify -format pnm -geometry 30x30 -colors 16 +compress image.jpg. Type mogrify -help for a listing of all of the options. Once this command completes, you can view the contents of the resulting PNM file in a text editor and should see a file with the following or similar PNM format.
P2 width height maxcolorvalue color color color ... ... color color color ...
If you want to view the resulting PNM file as an image and don't already have a capable viewer, the GIMP (also available through Fink) will happily display it for you if you open it via the GIMP's File -> Open command.
Some Guidance
Try starting out with a 30 by 30 image format with 16 colors, and expand from there. You can modify testHarness.java to read the file format and train the network without much effort. Here are three different sets of numeric codes in JPG format that you can mogrify and use. Try using two of the sets for training purposes, and once the network performs well, try testing it with the third set. The number of input units you use is determined by the image dimensions, so for a 30 by 30 image, that's 900 input units (not including the bias). You'll want to use ten output units in an output scheme like that of the 8-3-8 problem. The number of hidden layers and units can vary. You should theoretically be able to use one hidden layer with as few as four units, but go ahead and experiment with other configurations. Also, try varying the learning rate and number of training epochs. For 900 units and a low learning rate, tens of thousands of training epochs is not unreasonable.
I'm going to continue to play with the iSight to learn how we can use it to capture and classify images. In the meantime, check out ALVINN (Autonomous Land Vehicle In a Neural Network)--a project that used neural networks to drive unmanned land vehicles. If you're really hungry, Generation5 also has some good reading on neural networks.
Matthew Russell is a computer scientist from middle Tennessee; and serves Digital Reasoning Systems as the Director of Advanced Technology. Hacking and writing are two activities essential to his renaissance man regimen.
Return to the Mac DevCenter
Showing messages 1 through 15 of 15.
-
Jeff Hawkin's book "On Intelligence" is a must read
2005-10-24 11:05:33 Curt Hibbs |
[Reply | View]
If you are generally interested in the subject of intelligent machines, then *run* (don't walk) to Amazon and pick up a copy of Jeff Hawkin's (of Palm fame) new book "On Intelligence".
You'll only be sorry if you don't! Seriously, its a seminal work destined to change forever the way we think about intelligence, the brain, and intelligent machines.
Audible.com also has it on audiobook so you can listen while you commute.
-
Jeff Hawkin's book "On Intelligence" is a must read
2005-10-24 11:41:37 salamon [Reply | View]
While I agree that anyone interested in computer intelligence (or human intelligence, for that matter) should read this book, I think it remains to be seen how seminal it's going to be. He admits himself that a lot of his ideas have not been tested yet, although unlike many other hypotheses, his at least are testable.
I certainly hope he's right, because if so it should make most AI problems much, much easier once the details are worked out. His basic idea is that the human (and mammalian) cortex uses a single algorithm for all of it's various processing and learning. If that's the case, then once it's been deciphered we should be able to apply it to many different problems with relative ease.
-
Nice article
2005-10-24 10:31:30 salamon [Reply | View]
Neural Nets are the only type of AI I've studied at all (I was a bio major, so it seemed natural), so I'd covered most of what this article did, but I would love to see articles on other types of machine learning, especially if they use software or example code that will work on Mac OS X, like the Java one here.
Thanks!
-
Post Ideas For Future AI Requests HERE
2005-10-23 17:18:49 Matthew Russell |
[Reply | View]
Since there seems to be so much interest in other AI techniques, feel free to drop in specific requests below. If your request is already listen, put it in again, so we'll have a better idea of how to rack and stack the different options that come up.
-
Neural nets aren't all supervised
2005-10-23 16:19:09 TheBoyKen [Reply | View]
It's great to see an article discussing neural nets for people who are new to them; they certainly can be confusing initially. However the article seems to give the impression that neural nets can only work if you present them with the initial training patterns (i.e. "given this input pattern, I want you to give me back this output pattern") - indeed, there are whole swathes of neural architectures based upon unsupervised learning, that is, where you don't have to supply the information that needs to be learned up front. One such example is Kohonen SOMs (Self Organising Maps). Admittedly they're a bit trickier to explain than simple backpropagation nnets, but nonetheless they (and other unsupervised architectures) represent a vast body of nnet research, so I think they'd at least be worth a mention in case your readers get the impression that in order to use any nnet in a problem, you have to supply it a list of dos and don'ts. This isn't the case. -
Neural nets aren't all supervised
2005-11-07 04:06:56 frankberger [Reply | View]
Beyond SOM, you can have unsupervised learning with other algorithms too. An extremely successfull approach was TD-lambda learning invented by Sutton. Thesauro applied this to Backgammon and for the first time there was an AI that played on expert niveau. Other programs adopted that approach and today most people agree that those program outplay any human. On the mac you have two implementations to look at: Bgblitz (www.bgblitz.com used pure TD-Lambda) written in Java (by me BTW) and GNU-BG (www.gnubg.org, uses a mixture of self-learning and supervised learning). -
Neural nets aren't all supervised
2005-10-23 17:15:46 Matthew Russell |
[Reply | View]
Just checked and there's a tiny overview of SOMs on Wikipedia: http://en.wikipedia.org/wiki/Self-organizing_map
But you're right. There are tons of AI learning techniques out there and just about every one of them has about a dozen well known variations of one sort or another. I would again refer the interested reader to Mitchell's Machine Learning for some high-quality treatment of many of the fundamental ones.
-
HINT
2005-10-23 15:40:08 samslaves [Reply | View]
Add a fuzzy generated context free grammar to the neural network; then your iSight is ready to tell if you are in front of the iMac.
-
Great Intro Article
2005-10-22 17:45:38 MrO [Reply | View]
8:3:8 - an "aha" moment for me, as I've been somewhat mystified by neural nets and couldn't connect it back to the bit world. Thanks very much!
I was always fascinated by a little demo that accompanied NeXT boxen, that had a simulated seal learning how to balance a ball on it's nose. The learning was based on a neural net, though I'm now curious how the "success" feedback was represented in the system - perhaps another article in this line?
Thx,
Mark
-
Feedback
2005-10-22 23:22:18 austins [Reply | View]
Usually neural networks are trained by measuring the error in the ouput later and using an algorithm called backpropagation to adjust connection weights to explicitly correct the error. With enough training and a solvable problem, the network will usually converge. Sometimes, though, the network gets "overtrained" and just stores the desired output, so it doesn't generalize to new problems very easily.
Neural networks are still good for some things, but there are now much better algorithms now for face recognition- the eigenfaces decomposition with support vector machines, for example, does a pretty good job. This is still a hot research area, and amazing things are being done. -
Feedback
2005-10-23 06:16:53 Matthew Russell |
[Reply | View]
Yep. Backpropagation is the standard technique for adjusting the weights and is used here as well.
Good point about there being newer cutting edge ways of face detection than neural nets -- you're absolutely right about that. I chose neural nets here because of the general applicability to solve other problems in addition to the one at hand, and for the coolness factor as well. Lots of people hear about neural nets, but not too many people ever stop to figure out what they are.
As for the overtraining: right again, and that's easier to do than you'd think. And another consideration I didn't mention in the main text is that unlike a tool like a decision tree, a neural net is like a black box -- you might get it to learn something, but there's almost never any way to generalize what it's learned. There are plenty of other factors to consider as well.
Really, I just hit the tip of the iceberg with neural nets here -- like anything else, you could write a textbook on the subject. As a good resource, I recommend taking a look at the excellent chapter about them in Tom Mitchell's Machine Learning (the little blue book) if you're hungry for more than what you'll find on the (inter)net. -
Great Intro Article
2005-10-22 21:10:40 Matthew Russell |
[Reply | View]
Without seeing the demo myself, I couldn't tell you for sure -- I wonder if anyone has ported that demo or a similar demo?
Anyway, just remember that conceptually neural nets just take an input and produce an output. The input might be the angle of the seal's nose to the center of the ball, a measure of roll on the ball and whatever else you can think of, and at some point there's a threshold that's going to get crossed. Based on the input and output values, you just quantify the error and then the network basically takes care of the rest internally.
If there's an enthusiastic enough response to this intro to neural nets, I'm sure we'll be able to get you more good stuff on AI -- Baysean lets, decision trees, genetic algorithms, Markov processes, etc. -- it's all good stuff worth knowing about and it's becoming more pervasive in society than you'd think -- your car's transmission, that computer game you love playing, your spam filter, ..., and the list just goes on and on.
If anyone has any specific requests on future AI pieces, feel free to mention them below.
-
Whoa
2005-10-22 07:19:37 snackdog [Reply | View]
Is it my imgination or did this article get really complicated really fast? -
Whoa
2005-10-22 07:28:58 Matthew Russell |
[Reply | View]
Admittedly, it's difficult to provide an overview of neural networks in 2,500 words. But at the same time, you have to realize that AI sometimes isn't so much about "fun" and playing with friendly robots in a lab all day as it is about hard math and complex analysis. Like anything else, however, it starts to get easier the more you study it.
Feel free to wear out the "Post Comment" button if you have any questions about anything. That's what it's there for -- and if you're the kind of person who wants to dig deeper, then I want to help you.
The coding of a simple neural network is enticingly easy. However, the choice of weights can become a problem that rivals the original problem.
Eigenfaces is much more mathematically challenging. You need to know eigenvalues and eigenvectors and in general matrix algebra. But, once you understand the basic concepts, you will find eigenfaces amazingly intuitive.