
Frustration project: Automate data entry into PeopleSoft with Selenium - tbensky
https://github.com/tbensky/selenium-peoplesoft
======
hugs
#FunFact: I started the Selenium project. My first job out of college was at
PeopleSoft. I left PeopleSoft and went "in-house" as the PeopleSoft admin at
ThoughtWorks. A few years later, Selenium came out as a side-project for
another internal system we needed. _But_ when we were rolling out Selenium and
dreaming up "world domination plans", one of my mine was to get a copy of
Selenium to ship with every copy of PeopleSoft (supplanting the free copy of
SQA Robot that PeopleSoft shipped by default). Anyway, I never really pursued
that particular "plan", but here we are.

~~~
tbensky
THANK YOU for giving us Selenium. When I first ran my script and saw the
browser pop up start "running on it's own," I was pretty wowed.

------
cranekam
Another +1 for Selenium. I used it to bag tickets for a show I wanted to see.
The seller’s site was terrible and apparently had no protection against ticket
spinners so all 5,000 tickets sold out in minutes. Fortunately their resale
site also had no protection at all so it was easy to write something to reload
the page often and put tickets into my shopping cart as soon as they became
available. When that happened it notified me and handed back control for me to
enter the credit card details (I was too lazy to automate it all). I’m not
proud of this brute force approach but if the reseller’s site wasn’t so awful
to start with it wouldn’t be necessary.

------
unreal6
"Peoplesoft is a curse on humanity"

I would love to know the story of how this wretched piece of software seems to
have embedded itself into nearly every college and university in the United
States.

~~~
xyst
No surprise that it is backed by Oracle. The sales people at Oracle are really
good at getting technology inept people/businesses into multi year contracts.

Once they started citing the list of features in their hundred slide
presentation, the administrator stood no chance.

~~~
tallanvor
PeopleSoft was one of the larger players, at least in education, well before
Oracle bought them. There's really (or at least there was when I was involved
in higher education) only 3 main players at that level. And trust me, each
option you had was bad in its own way.

The problem is that these products started out much smaller, and grew to fill
the needs, although not necessarily as nicely as you might like. So now when
it's time to put out a new bid, there's just the handful of vendors who can
check off all the boxes for what the current system does. And integration with
other systems is always a pain - even today it often includes csv or tab-
delimited files that get sent to ftp sites for batch processing!

------
dfee
There is an entire class of business problems that can be solved using
Selenium.

I don’t love that raw http connections are the universal API, but at least
it’s a tool in our belts when these companies provide lackluster or zero
proper interfaces.

~~~
sqrt17
There's a whole class of solutions beyond Selenium that automates not just web
apps but all apps. It's going under the name of "Robotic Process Automation"
and people really like it because they're not dependent on their software
vendor to get basic integrations done.

------
beatthatflight
Love it. I've built a flight search tool in selenium too, and for isolating
deals. (I've also used selenium for testing). The biggest problem is the speed
and the unexpected (the spinner you experienced, intermittent page load times,
asynchronous requests or when elements are stale as a result). But it is
possible to do it.

Next you can thread the sucker, so you could have 2-4 PS windows open and
enter at a faster speed ;)

~~~
tbensky
Yes, multi-threads of it...that would be scary!

------
jimmyswimmy
I use selenium similarly to drive a similarly awful interface known as
costpoint. I would DEARLY love to know how to use the browser developer tools
to figure out what is actually happening under the hood of websites without
having to read every single Javascript module. The network tab shows you what
is sent and received but it's not easy to figure out how those messages are
created and parsed. It would be nice to learn how to debug and step through
some of that.

If I could learn the underlying api, I could write a tool that would never
change. The ui changes much more frequently than the interface between client
and server code. Besides, it would be much faster this way.

~~~
PetahNZ
If you click on the "initiator" column in the network tab it will jump to the
code that created the request. Can also see stack traces, set breakpoints,
etc.

~~~
mistermann
If anyone happens to know of a particularly good article on this for non-
javascript (but technical) people, please post.

~~~
sk5t
Please clarify--are you looking to understand how browser events work without
learning about the DOM and/or Javascript?

~~~
mistermann
I suppose something along the lines of Chrome debugging for programmers who
aren't javascript developers...undoubtedly there are some fundamentals you
would have to become knowledgeable in, as with any technology.

------
julianeon
I'm always interested in clever use cases for Selenium.

It seems like "automating boring stuff using Selenium" should be a bigger
category than it is - and use cases that can do it profitably are always
fascinating.

~~~
sleet
RPA (Robotic Process Automation) is a big category of tools that can be
summarised as ‘automating boring stuff’. You often see it used for repetitive
data entry, e.g.

1\. Load excel sheet from X:\\...

2\. Login to Y

3\. For each row navigate to ... then enter data

4\. Submit, if there’s a validation error add it to and excel workbook

5\. Email exception report to ...

~~~
halo
What software is generally used for this? Is there anything available for the
average person that doesn’t require spending a small fortune?

~~~
janstice
Uipath and automation anywhere seem to be big players with community/free
starter editions. PWC (who are spearheading the RPA at $work) seem to have
decreed that all their staff are going to use uipath worldwide.

There are a handful of open source RPA tools, but I don’t think there’s a
breakthrough project yet.

------
edoceo
Webdriver is for more than just testing! Very cool demonstration.

My biggest headache was in getting the right selectors for the various form
fields - some services have shifting IDs and classes (ugh xpath)

~~~
bickeringyokel
When you start using xpath that's 20 parents deep and every line of code
scrolls off the page you quickly realize how important it is for your devs to
to write good accessible and unique element IDs.

------
devit
IMHO it's usually better to write a WebExtension instead of using Selenium,
since you have more control over the browser, can intercept and modify network
requests, can access the DOM from JavaScript while being protected from the
page altering the environment, and can ship the code as a browser add-on,
while still being able to communicate with native apps.

Only downside is that you can't drive old IE and mobile browsers except for
Firefox for Android.

------
kfk
Any tools to do this with native applications too not just browsers? We have
been flooded by this RPA companies the past 2 years and I'd love a way to do
this with Python. It is actually a potentially very profitable market since
the impact of automation is very visible (people see cursors do stuff for
them...).

------
stm17
I would love to hear any more you have to say about the genetic algorithms you
use to help with scheduling.

~~~
tbensky
Ah yes, the genetric algorithm (GA). Well, in short, scheduling personnel is a
very visible, policital, and important job. It affects people's lives, and has
to be done farily and correctly. I did my best (by hand) for a few years
(scheduling 50ish professors into 200 class sections), and I knew I was doing
a bad job, but had no choice. It would take 2 weeks to do, because I could
only mentally handle working on such a job for so much in a given day before I
had to set it aside. So I came up with a preference form, that each could fill
out dealing with things like preferred time of the day to teach, daily
schedule compactness, preferred class levels, etc. These preferences could
also be weighted (0..1) by importance to each person. So the GA starts out
with a long, two-part inital random sequence that consists IDs to all
professors followed by IDs to all possible classes. Using crossover and
mutation (the usual GA steps), the order of the professor IDs vs the order of
the classes (prof 0 gets class, prof 1 gets class 1, etc.) are shuffled by the
GA. (There are a lot of strange constraints to this you'd only see in
academia, related to pay, and contractual teaching requirements.) A fitness is
calculated for each professor's latest schedule, by computing a weighted sum
of their preferences, based on the assigned classes pulled from the long GA
sequence. It's incredible how well it works. It is done in under an hour,
people seem happy, everyone's schedule fitness is minimzed, and each's
schedule fitness is explained to them (i.e. you got +3 points because you
wanted an 8am class, and got one). Everyone can also see everyone else's
fitness so it's a "share the pain" kind of resulting schedule. At times when
we go in to tweak things by hand, it is clear how tightly the GA optimized
things, as any human intervention we try usually and obviously makes things
worse. Bottom line, the GA generates the CSV, which Selenium types into PS.
After using the GA, I'm convinced that neither humans (or spreadsheets) should
be invovled in scheduling at all. I also think GAs are also pretty incredible.

~~~
stm17
Fascinating! I am going to try that. I’ve been trying to teach myself SAT
solvers and constraint systems to approach the problem at my junior college.

------
oandrei
Selenium leaves trace in HTTP requests, so its use is quite detectable. The
authors of PeopleSoft can include Selenium detection in their products,
disallowing your automation scripts. Why not use GreaseMonkey instead?

~~~
1024core
Do you have any more information about this trace in HTTP requests? My
understanding was that it was the _browser_ doing the requests; Selenium was
just driving the browser.

~~~
oandrei
Yes, here is info: [https://stackoverflow.com/questions/33225947/can-a-
website-d...](https://stackoverflow.com/questions/33225947/can-a-website-
detect-when-you-are-using-selenium-with-chromedriver)

------
trhway
add AI on top of that which learns what/where/how to enter (or at least
suggests) and you get the modern office automation wave.

------
lifeisstillgood
Ha! I see your crappy UI in PeopleSoft, with hard to automate edge cases, and
I raise you multiple SharePoint instances!

Ante up at your peril in this game :/)

~~~
bash-j
I've managed to automate SharePoint data uploads and downloads using the SOAP
API.

~~~
lifeisstillgood
I have too, and the OP got Peoplesoft to work. But ... my goodness the effort
compared to a proper jason APInis huge and compared to just web scraping HTML.
I mean there is a word file someone just uploaded. That's three clicks

