
Sikuli: Automate Anything You See on Screen - GuiA
http://www.sikuli.org
======
hyuuu
I used this to automate my unemployment benefits, no joke. You have to keep
reporting hours (every week I think), so I just set this up and run it, the
closest thing I get to having a piece of software that makes money #sosad

~~~
fizzbatter
Don't you have to report application/locations/etc too? Or did you actually do
the applications and log them in some type of database (flatfile/etc), and
have this script auto fill in the benis every week?

~~~
SteveNuts
In my state you have a few weeks or a month of grace period before they start
to require resumé and interview skill classes, as well as you reporting on any
applications you filled out.

------
jakozaur
Great project, but it hasn't changed since 2010? Maybe we should add it to the
title?

I was hoping that image based testing would eventually overtake
Selenium/Appium for web/mobile testing. In principle there are several
advantages:

\+ It works the way human works. If your button id has change from #submit to
#submit-application, your UI test should work just fine. However, if your
button became one pixel at top-left corner, test should break. Right now most
frameworks do the opposite.

\+ If it's based on visual, it would be easier to maintain tests. E.g. Your
button change color from light grey to dark grey, would you like to update
your tests.

\+ Tests would be much easier and faster to read and write. Even less
technical folks like current manual QA testing can contribute to automate
testing tools.

~~~
charford
You might be interested in [http://www.sikulix.com/](http://www.sikulix.com/),
which was forked from this project and is more up to date.

~~~
adamhepner
Oh, you beat me to it. But the good news is, that SikuliX maintainer seems to
be actively developing the next major release version now, and last release is
not _that_ old (end of 2015, where the technology don't have to change that
rapidly)

------
eirikb
Made a small bash script to do something similar:
[https://gist.github.com/eirikb/c189a7d8406b2897dad0e86086be1...](https://gist.github.com/eirikb/c189a7d8406b2897dad0e86086be1b23)

~~~
CaptSpify
Nice. Any demos of it in action?

~~~
eirikb
Certainly, it should be pretty easy.

Here is a demo:
[http://i.imgur.com/1BpMKpe.gif](http://i.imgur.com/1BpMKpe.gif) Script is
here:
[https://gist.github.com/eirikb/ac8196beb0b57577a8fc47eb18427...](https://gist.github.com/eirikb/ac8196beb0b57577a8fc47eb18427bd4)
Images are here: [http://imgur.com/a/Uv3NH](http://imgur.com/a/Uv3NH)

What it does: Opens Google translate in Chrome, inserts some Norwegian text,
copies the translated text and inserts it into this comment. To make the
images I used gnome-screenshot, it let me grab areas.

------
patrickk
Another great project in a similar vein is Pulover's Macro Creator, combined
with Autohotkey:

[http://www.macrocreator.com/](http://www.macrocreator.com/)

[http://ahkscript.org/](http://ahkscript.org/)

Sadly Windows only. It's one area where Windows has a leg up on Macs.

~~~
kawera
Keyboard Maestro is a good option for Macs. Fake is another one but only for
automated web browsing. I've been using both for years.

[http://www.keyboardmaestro.com/](http://www.keyboardmaestro.com/)

[http://fakeapp.com/](http://fakeapp.com/)

~~~
patrickk
Thanks for this!

------
Shish2k
FWIW I used sikuli(x) and found it very useful, but I have a chronic allergy
to Java[1], so I've been re-implementing the same API in pure Python -->
[https://github.com/shish/sikulpy](https://github.com/shish/sikulpy)

[1] more specifically, I found sikuli scripts pretty painful to debug, since
neither java debuggers nor python debuggers seem to work quite right.

~~~
holtalanm
This is just plain incorrect.

Yes, the original API was written in java, but Sikuli scripts are written in
Jython. I did an entire automation project using Sikuli written in jython
without any hiccups a couple of years ago.

~~~
Shish2k
> This is just plain incorrect.

Did you somehow manage to read my comment and interpret it as "Sikuli scripts
are written in java"? Because that's not at all what I said (or meant) :P

Sikuli _itself_ was (and still is) written in java - which I find a pain to
work with when I want to modify the internals of the library; and then scripts
are written in jython, which means I don't have access to my cpython-specific
tools and extensions.

------
Contraptor
When I started at my current job (back in 2011) they were using Tevron's
CitraTest product to automate software hosted on citrix. The first task I was
given was to see if we could find an alternative to CitraTest, well Sikuli was
the answer. All of the existing code was written in VB.NET, but with a simple
wrapper class and use of XML-RPC, we dropped CitraTest completely for a free
MIT-Licensed software. It has worked great for years actually.

------
flarg
Sikulix is great and comes with a Python IDE where you drag and drop screen
captures - so a command to click on an icon has the icon itself written into
the code.

Robotic process automation (e.g. Blue Prism) is coming to the fore in
commercial settings - a mixture of API and computer vision (probably a
horrible mix) so there's a market out there for developing something
commercially.

------
arca_vorago
Sikulix is the current fork, and I have used it to surprising effect in all
kinds of situations, really awesome as long as you can ignore some bugs, but
the good news is the dev is super responsive to reports.

------
brazzledazzle
Last time I tried it on OS X the experience was pretty painful but it did a
decent job when all of the pieces fell into place. Has anyone tried both
Sikuli and RobotJS [1] and can compare/contrast them?

[1]
[https://github.com/octalmage/robotjs](https://github.com/octalmage/robotjs)

------
darkFunction
Used Sikuli for some casino fairness testing
[https://pbs.twimg.com/media/B4wlbnhIAAEti1z.jpg:large](https://pbs.twimg.com/media/B4wlbnhIAAEti1z.jpg:large)

~~~
jawshie
I'd love to read more about this. Do you have a writeup somewhere?

------
om42
Used Sikuli for automating systems testing a few years ago (last I heard they
stopped working on it). We had to use a VNC because it was affecting system
performance. Overall really good and glad SikuliX was able to continue
development of it. I used it again a few months ago and there same issues with
the IDE and some other setup/runtime bugs exist that existed a few years ago.
Otherwise it has huge potential for automating QA/systems testing if someone
takes the time and has the resources to do so.

------
kon1
We use Sikuli for automatiom tests on both Windows and Mac. Other than
occasional issues with its image recognition features it works great and
serves us well. We have Jenkins automation jobs spin up VMs in VirtualBox,
load build artifacts on the VM, and run Sikuli tests. All test output gets
pulled back into the Jenkins output and parsed for successes/failures. Makes
it possible for us to test across different OSes use the same Sikuli scripts.

------
brudgers
Related:
[https://news.ycombinator.com/item?id=11859996](https://news.ycombinator.com/item?id=11859996)

------
holtalanm
Great tool. Used it to automate SAP in a project a couple of years ago. Was
using it with a .NET framework though, so it is was a total pain.

That led to me writing Sikuli4NET about a year later.
[https://sourceforge.net/projects/sikuli4net/](https://sourceforge.net/projects/sikuli4net/)

------
projectileboy
I once worked on a project where we needed a tool like this. Sikuli definitely
worked, but for us it only handled relatively simple cases, and it felt a bit
like a science project (at the time, anyway... this was 2010). Good commercial
alternatives at the time included eggPlant and T-Plan.

------
SpaceCadetJones
Reminded me of the SCAR tool many of us used to use back in the day for
botting on Runescape. Will have to look into this for QA automation, as
Selenium isn't always too fun when you've got a hairball of ASP.NET WebForms,
Vanilla JS, and Angular for a code base.

~~~
kaitnieks
Surprised to see SCAR mentioned. I have some connection to it ;) I still use
it now and then to automate some flash games. Helps me to relax.

~~~
SpaceCadetJones
Wow, it really is a small world, I can't believe you replied to my comment.
You're one of the only aliases I still recognize from those days as I was
about 12 at the time. SCAR and the other scriptable clients were a huge spark
for my interest in programming as a kid. Seeing something that could automate
actions in a game I played all day blew my mind and I had to learn how it
worked. Cheers to you for that!

------
leni536
Well, this is certainly more advanced than the xmacro* utilities. Once I had
to use xmacro to do stuff like this, it's a good idea to dedicate a separate X
server in the background though so you don't have to leave your other tasks
while it's running.

------
jzymbaluk
When I was working at a video editing shop, I used SikuliX to automate a lot
of common, repetitive tasks (like inputting chapter names to chapter markers
in Adobe Premiere). It's really great for automating easy repetitive tasks

------
arank
I used Sikuli to automate app downloads from app store. It opens iTunes, goes
to the featured apps page, enters the password if required and downloads 10
new featured apps every day.

------
paulryanrogers
In my experience Sikuli/X was too slow for image search on high definition
desktops.

UIs also tend to change more than keyboard shortcuts.

------
WaxProlix
Also great for automating repetitive/grindy tasks in computer games! (Not a
bot, promise)

------
brunoqc
Anyone knows if this could be used with Go or Rust? like a c lib.

------
jimmcslim
Wonder if an integration with Screenflow or Camtasia is possible?

------
thedeerchaser
I love sikuli! Saved me a lot of time in the past

~~~
eb0la
Fun fact: if you want to click() on something that appears more than once,
Sikuli will choose it at random.

Some time ago I was demoing a script to my boss and the second time I run it,
it clicked on a different place with the same text (and fortunately with the
same destination).

We were shocked because the link wasn't where we expected to be (Ciscosecure
ACS 3.x).

~~~
erikcw
The "region" feature helps protect against this.

