Leafsnap is a project I've been working on for about 2 years and we finally launched this week! It's a free app for the iPhone (with an iPad version coming next week and an Android version later in the summer) that contains very high-quality images and descriptions of the tree species of the Northeast US. It also has an automatic-recognition feature that uses computer vision to help users identify trees by taking a photo of a leaf.
This project is a joint collaboration between the computer science departments at Columbia University and the University of Maryland, and botanists at the Smithsonian Institution.
Please let me know what you think! Also, I'd be happy to answer any questions about the app or the recognition technology behind it.
We definitely want to take advantage of user contributions in some way, but we're still trying to figure out the best thing to do. Also, the app contains several very high-quality images of each species taken in a particular way by a talented group called FindingSpecies, and to keep the visual integrity of the app, we would want very similar photos for all species that we add. This might be difficult to outsource to the crowd.
It's a similar problem to that faced by, say, open source projects that aim to maintain a certain level of code quality and style matching in contributed source code.
Addendum: the image collections of the Australian National Botanic Gardens and the Australian National Herbarium might be of interest to you down the track:
Question: Any thoughts to partnering with IQEngines to bring their adaptive image recognition engine and database together with yours?
It solves a real problem, maybe you can extend it to mushrooms with HUGE warnings(you don't want people eating something poisonous a computer recognized).
I have this little tree in my yard, I do not know for sure what it is, it seems it is a cherry tree but I'm dubious.
He amazes me by being able to look at a leaf, bark or chunk of wood and tell what it is near instantly. If accurate, this is the sort of thing that will change the teaching process. I will tell him about it.
Another app I am waiting for along the same lines is Shazam for bird song!
Is it open source?
Also needed to cache a bit more aggressively.
1. Segment the leaf from the background. Even though we require users to place the leaf on a white background, this is still a very tough problem because of lighting, shadows, viewpoint, blur, focus, etc. We've tried all published segmentation algorithms and nothing works. We wrote our own, using EM on the HSV colorspace as the basis for it (plus lots of heuristics for leaves). It works okay, but it definitely needs the most work.
2. Extract some features from the contour of the leaf. We use histograms of curvature over scale. The idea is that some leaves have the same rough shape, but differ at a fine scale (e.g., serrated vs. smooth leaves). Others are similar at a fine-scale but look quite different overall. To distinguish all of these cases, we estimate "curvature at a scale" at each point on the contour and build a histogram of curvature values. We then repeat this at coarser and coarser scales and concatenate all histograms together to get a large feature vector.
3. Recognition/matching: We use a simple nearest-neighbors classifier to match the feature vectors to our ground-truth database of labeled leaves. We return the top 20 ranked species or so.
I've often wondered about identifying local trees; I've looked up site of trees, but often leaves seem very similar to several different trees.
Great to have it for other countries, and an edible plant guide.
"Looks like cancer." "Definitely Athlete's Foot." "It's not a tumor!" "Please state the nature of the medical emergency."
Repeat 1000x, pull out the winners, and diagnose!
1. If consistent winners can be identified, I'd expect them to demand much higher then average pay.
2. Everyone of your turks might have to be licensed to practice medicine, or the government might put at least some of you in jail.
I know of at least a few melanoma identification, using software, efforts which died due to regulation and the shocking resistance of doctors.
But I wonder how free apps, good at diagnosing all kind of things, would do? With cellphone CPUs and cameras getting better you could catch quite a few things...
>I know of at least a few melanoma identification, using software, efforts which died due to regulation and the shocking resistance of doctors.
Just make sure that "For entertainment only" is clearly visible on the product label :)
>But I wonder how free apps, good at diagnosing all kind of things, would do? With cellphone CPUs and cameras getting better you could catch quite a few things...
and with portable ECG, EEG, ultrasound devices, DNA chips, etc easy available and connectable ...
I really wish there were more apps like this (and Shazam). Just off of the top of my head, i'd pay a reasonable amount of money to identify: bugs, fish, seashells, animal footprints, mushrooms, and I'm sure a whole lot more.
Being part of a hiking family with two small kids, the questions are endless and even if it were crowdsourced in some way (e.g. please identify this bug for karma), it could be handy learning tool for kids, getting them more excited about science at an early age.
Why isn't it open source?
Ah memories of my 10th grade biology class (bug or leaf collection). The trusty Audubon Guide to North American Trees. It is really shocking that the HGTTG is coming true...
However, the app does let you filter the list of species shown to "New York" or "Washington DC" or "The Northeast".
If the app could recognise more species of elm (English and American seem to be there, but no Wych or Dutch for example), would be a valuable tool for users trying to contribute to the project.
The source for the Django app and PhoneGap-based mobile app will be up on GitHub next week, and the data is publicly available via our CouchOne instance (http://redrobot.couchone.com/_utils/database.html?openelm) – the project isn't live yet so there's only a small amount of data.
Ask one simple question with each pic take "How many trees of this type would you estimate are in your vacinity, that you can see"
Map those trees all over and compare with climate/weather.
The estimates just increase the size of the circle on the map that shows where those trees live.
Eventually you'll get a very clear idea of what trees thrive/live where.