

MIT creates picture-driven programming for the masses  - prat
http://www.networkworld.com/news/2010/012110-sikuli-scripting-language.html?hpg1=bn

======
Maciek416
There's more to this project than is hinted at by the "picture-driven
programming" label. This scripting system is driven not just by pictures, but
by screenshot snippets of the GUI elements that you want to click on /
control. The language uses computer vision techniques to match things on the
screen (and I'm guessing to prevent programs from becoming overly brittle if
minor visual changes happen to the screen). Check out the video on the
project's actual page, it's actually kind of interesting:

<http://sikuli.csail.mit.edu/>

~~~
yannis
I read the paper and frankly speaking, I am very skeptical that this is an
application for the _masses_.

I found this piece interesting:

>The Sikuli Search prototype has a database of 102 popular computer books
covering various operating systems (e.g., Windows XP, MacOS) and applications
(e.g., Photoshop, Office), all represented in PDF4. This database contains
more than 50k screenshots. The three-feature indexing scheme is written in C++
to index these screenshots, using SIFT [11] to extract visual features,
Tesseract5 for OCR, and Ferret6 for indexing the text surrounding the
screenshots. All other server-side functionality, such as accepting queries
and formatting search results, is implemented in Ruby on Rails7 with a SQL
database. On the client side, the interfaces for searching and annotating
screenshots are implemented in Java.

... and the scripting engine is in Python.

Sometimes I wonder if Computer Science wouldn't have been better off with a
constantly evolving _C_ rather than all this mess. Every time I build a web
application I feel as if I am building Frankenstein's body with different
animal parts.

There were two major attempts to bring computers to the _masses_ COBOL and SQL
and they both managed to produce the opposite result rather than the masses
using them a demand was created for very highly paid Programmer jobs!
Electronic Engineers never attempted to teach the _masses_ how to build TV's.
In the late 80's I was teaching laymen things like 'How to Use MSDOS' and
'Introduction to Wordperfect' which paid for the rent but overall was a
depressing experience. I also taught at the same time 'Pascal' and 'C' to
Mechanical Engineers, my conclusion was that some people take to programming -
like some people take to maths, they enjoy it and after an initial push they
will teach themselves the rest and the rest will never touch programming even
if their life depended on it.

In my mind the only successful 'programming for the masses' products are Excel
and partly HTML.

~~~
abdulhaq
I don't believe that COBOL was designed for the masses - after all, this was
1959, when there were so few computers and it was believed by some that
ultimately we would only need one! Also, SQL came out of academia and I don't
believe that was for the masses either. I would say however that BASIC was
intended for the masses, and as such it was successful. A generation in the UK
learnt BASIC on the ZX80, ZX81, Spectrum, BBC Micro etc.

~~~
yannis
Sadly, when CODASYL committee was formed to develop a programming language
ideally suited for business data processing needs they were looking at a
language that 'normal operators' could learn and that it would be similar to a
human language that is why you get Cobol syntax like:

    
    
         ADD YEARS TO AGE
    

A similar motivation was for SQL. We will give you a database and a query
language that anyone can use:

    
    
        SELECT title FROM Book .....
    

During the early days of computing the thought prevailed - pretty much like
now - that it was a matter of 10 years before computers would be able to talk
in natural languages and AI was around the corner.

But, you are right about BASIC that was my first language as well, quickly
dropped for Turbo Pascal which was the sort of the in-thing for a while.

------
wallflower
Wow. This might be perfect for advanced screen scraping.

"GUI Testing Using Computer Vision"

[http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-
chi20...](http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi2010.pdf)

EDIT: It appears it cannot grab the actual text using the Native APIs so you
would have to include a pipeline to OCR black-on-white text

------
physcab
There already is a visual scripting environment--its called LabView and its
terrible. LabView is used mostly by scientists who want to control their
experiments but don't want to learn how to program. It's very effective when
dealing with hardware and you just want to get it working. But if it doesn't
work, you're screwed because its very difficult to debug.

That's actually the problem I forsee with the visual scripting environment.
There is no better single point of instruction than a compiler error or syntax
error highlighting. If you try to circumvent that process to make it more
"accessible" to the public, you're going to keep your visual programmers in
the dark as to what is actually occurring.

~~~
sp332
Did you even read the article? This isn't a visual scripting environment, it's
an intuitive way to use Python to do image processing.

Edit: here's a paper with examples.
[http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-
uist2...](http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-uist2009.pdf)

~~~
physcab
Ah. You're right. I misread it. However, I stand by my points on visual
scripting in general. Even if there is a way to automate certain tasks via GUI
elements, you are placing a great deal of trust on the system to operate as
expected.

I do quite a bit of research in machine learning, and I can say that these
systems are _never_ perfect. Building an abstraction on top of an image
processing technique is interesting, but I really wonder how a system like
that can be debugged if something goes wrong.

------
SandB0x
I thought this might be similar to the incredible Baltie programming language:
<http://en.wikipedia.org/wiki/File:Baltik3-screenshot1.PNG>

(it isn't)

