
How I Hacked My University's Registration System with Python and Twilio - gregorymichael
https://www.twilio.com/blog/2017/06/hacked-my-universitys-registration-system-python-twilio.html
======
makr99
I was in the US and want to immigrate to Canada because of US broken H1B
immigration system ( for me it will take 9 years to get a greencard). Here is
what I did -

The immigration website of Saskatchewan province opens up randomly to apply
for immigration. I missed it many times because there is no indication other
than the "Apply" button being enabled and a small text in their homepage which
says "Applications are now 'open' ". They will close the application intake
when they have reached X number of applicants. So timing is very important.

So I hacked up a script which diffs their home page every 10 minutes for
"open" regex. When there is an "open" keyword in the diff, the python script
calls twilio API to make a phone call to me along with an SMS.

So this script was running in AWS for many weeks and one day I got the call.
Logged in to Saskatchewan's immigration homepage and applied. Now I am in
Canada as a permanent resident. Thanks to Twilio.

edit: add H1B to make clear which type of immigration is broken IMO.

~~~
e12e
Nice story, and great that Twilio worked out for you - but this really is the
sort of thing that email alerts (or IM alerts, or IRC alerts) should also work
fine for.

I would argue that the fact that you find SMS/phone calls a more urgent alert
is a fault with your phone/communication setup - and twilio is an interesting
hack around that.

But there really _should_ be an easy way to make just as much ruckus from a
simple email, based on topic/sender filtering... (hm, maybe there's a nice
side project for an app in there...).

~~~
problems
Yeah, this isn't really a major Twilio win or a hack - it's a basic scraping
technique.

I found a house for myself and a friend in a similar way, reverse engineer
mobile app from MLS, scrape all listings once a day to get new stuff as it
hits the market and you can now be far more selective than the crappy search
on the MLS site allows.

~~~
hbk1966
Yeah you can sends texts to your phone just like sending an email.

~~~
murukesh_s
How to do that?

~~~
KekDemaga
Put the phone number in front of a domain for each carrier, a list is here.

[https://github.com/WilliamFCipriano/FreeSMS/blob/master/data...](https://github.com/WilliamFCipriano/FreeSMS/blob/master/data/gateways.list)

~~~
pbhjpbhj
There are several international lists too, here's a couple:

[https://martinfitzpatrick.name/list-of-email-to-sms-
gateways...](https://martinfitzpatrick.name/list-of-email-to-sms-gateways/)

[https://www.opentextingonline.com/emailtotext.aspx](https://www.opentextingonline.com/emailtotext.aspx)

~~~
icebraining
Do they work? For Portugal it still lists Telecel - which hasn't existed since
'99.

~~~
KekDemaga
My list is accurate as of a year or so ago, or at least they don't bounce back
when you send an email to them.

------
NiceGuy_Ty
I did this for one of my classes as well. But since it was just a one off, I
just put this bash script in a cron job on a free-tier AWS ec2. It just did a
quick and dirty parse of the html, which for my university wasn't behind a
login wall.

    
    
      $seats="$(curl $URL | sed \"140qd\" | sed -nE \"s/<TD CLASS=\\\"$CLASS\\\">|<\\/TD>//gp\")"
      if [ $seats -gt 0 ]; then
          echo "Go register for class $URL" | msmtp -a "default" $EMAIL
      fi

~~~
EE84M3i
Yep. This is what I did. Also, I just used the email address for my phone's
SMS. I'm not sure why you need Twilio for this, seems like over kill to me.

~~~
aw3c2
It's an ad for Twilio, of course Twilio is needed for that.

------
monksy
If you were able to do that prior to taking CS101, you might not need CS101.

~~~
vesrah
If you want to get that piece of paper that some companies require...

~~~
tnecniv
At my school, they would waive requirements if you take a higher level course
in the same track. For example, if I just took CS102 that requires CS101, I
wouldn't have to go back to satisfy the CS101 requirement.

Obviously everywhere is different.

~~~
JoblessWonder
How... are you taking CS102 that requires CS101... without taking CS101? Are
the requirements not actually required?

~~~
deathanatos
IDK about the parent's school, but my at university (RPI), when I was there,
students could simply opt to skip "CS101", and move to "CS102". I think it was
also encouraged (but again, not required) to have taken one of the AP CS tests
if you did so.

I think you could also opt out of "CS102" if you desired. It was on the
student to determine how much they knew. The school just understood that a
number of CS students had been doing this for some time on their own, and had
mastered some of the basics.

I skipped the "CS101" course, which was mostly intro to programming, control
flow, etc. I decided against skipping the "CS102" course; the first half
wasn't that exciting (I knew how linked lists, vectors, hash tables worked)
but trees got me (and some of the more exotic stuff at the end of the course),
so I'm glad I took it.

~~~
JoblessWonder
Interesting. At least when I was at my UC 10 years ago we weren't able to skip
any classes. Maybe they have wisened up and made them optional or have a
system in place to prove them unnecessary. It makes some sense to make sure
everyone has the same level of fundamentals though.

------
ArlenBales
If you're using Python, please don't reinvent the wheel and write your own
scraper. Use Scrapy ([https://scrapy.org/](https://scrapy.org/)). It's one of
the most tested and powerful scrapers out there.

~~~
raybb
For a use case like in the article would it make more sense to use
beautifulsoup or is there an advantage to scrapy?

------
kevsully5
I did a similar thing my freshman year of college, and by junior year I
created an API based around the class registration system and an
iPhone/Android app for it that sends push notifications to students when the
class they want opens up (shameless plug:
[http://www.eaglescribe.com/](http://www.eaglescribe.com/)).

This past semester, the active users was between 1/3-1/2 of the undergrad
students in the school. As you can imagine, popular classes had dozens of
people "subscribed" to receive notifications when it opened, so it became a
race to sign up once the push went out. On the plus side, this gave us a
treasure trove of data on the most popular courses, and we've been in
communication with the school to see if they would be interested in this data.

------
sdca
Twilio is great and all but an SMS or other notification doesn't actually
guarantee you a class that's available to all students.

A class was sniped from me one time while I was still on the confirmation
screen. So unlike OP I fully automated the process.

EDIT: A notification & response is good for solving a captcha if they have
one, if you don't want to outsource that to mechanical turk (or trust their
timing / accuracy).

------
Mister_Snuggles
This is an interesting proof of concept. I'm impressed at how simple the SMS
signup code is.

For many universities though, it won't be this simple. Many hide the open
seats behind a login, which means you need to be a student there to see them.
Many also use student information systems that are a real pain to scrape like
this. Once you get to the point where interacting with the student information
system is done via AJAX requests mediated by a mess of JavaScript, it starts
to push the limits of what you can do with this technique.

When it gets to this point you either have to carefully dissect everything on
the Network tab of your developer tools, or use a real browser. I've used
PhantomJS (basically a headless WebKit) and Selenium WebDriver with great
success on sites that are not amenable to scraping. The neat thing is that you
really only need to use that for the interaction to get to where the
information is. Once you've navigated there you can just have it dump the
rendered page as HTML and parse it using the same techniques shown here.

~~~
saamm
Author here--you're totally right. When we did this for our actual
registration site, we spent hours inspecting requests in a proxy[0] to get to
~194 lines of PHP.

0: We used [https://www.charlesproxy.com/](https://www.charlesproxy.com/)

~~~
Mister_Snuggles
Yeah, that's more what I'd expect to see with a standard student information
system. I can only imagine the amount of work it took - the last time I
encountered a site like that that I wanted to scrape I gave up (this was
before I discovered PhantomJS).

It's really too bad that universities don't foster creativity like this.

------
agentd00nut
Wondering why you didn't also automate the process of actually signing up for
the seat? Getting notified that a seat is open is helpful but automatically
taking the open seat sounds a lot more helpful.

Though at that point you'd have a lot more luck with selenium or some other
web driver.

~~~
0xdeadbeefbabe
Would you rather fail to register for an open seat or fail to find an open
seat? Maybe I'm biased because I made the same choice as an undergrad, I
stopped short of registering for the class automatically.

~~~
XaspR8d
Yeah, automating that next step is making the failure mode much worse.

------
IVDV
Need to be careful about this one. A student at my alma mater was suspended
for doing the same thing:
[https://www.reddit.com/r/ucf/comments/xo5ye/ucf_student_pena...](https://www.reddit.com/r/ucf/comments/xo5ye/ucf_student_penalized_for_writing_program_to)

You have to love the education system...

~~~
jjnoakes
In your example, he was charging for the service, which was against the terms
of use of the web site he was scraping.

So in general, do be sure to not violate a site's terms, but I don't see what
that point has to do with your final comment...

~~~
lostlogin
What a horrible university. If a system is so bad that people will pay for the
improvements, they are doing it wrong.

~~~
jjnoakes
I don't think "people will pay for <thing>" is justification for "any
university that doesn't do <thing> is horrible".

------
kernelmachine
I tried building a similar service at my university when I was in undergrad,
and it led to lots of administrative conflicts, because people using my
service were thought to have an unfair advantage in the registration process.

------
dsfyu404ed
Burp is probably the better tool for the job in most cases since you're
usually just looking to repeat a request until you get the response you're
looking for.

If you just want to automate registration to ensure you get a seat in a course
you should be looking into your school's network topology to minimize latency.

------
nsb1
I'm dating myself, but when I went, it was a standard phone AVR, with touch
tone menus. I used a war dialer to get into all my classes. Ahh, the good ole
days.

~~~
clamprecht
Which wardialer?

------
noonespecial
For the curious, if you just want to one-off this for yourself, you probably
don't need twilio to do it. Most carriers have a special domain that if you
send email with the recipient's phone number, it turns into a text. I've got
Verizon and its as simple as this from the command line:

 _echo 'Text message text' | mail -s 'Subject will display in braces in the
message' -a 'From: Myserver@Mydomain.com' 'number@vtext.com'_

This ended up working faster than Twilio for me. (Less than 10 seconds from
enter key to text on phone).

~~~
rocqua
Is the SMS free? If not, who pays for it?

~~~
MichaelGG
In the US, the recipient pays for texts.

~~~
pbhjpbhj
So we can drain your account sending you texts, how mad is that.

------
tahabi
Some time ago Berkeley switched from Telebears to a new registration system
developed by PeopleSoft/Oracle. Needless to say, it's garbage. I wrote a
couple scripts to pull class data from the API in an effort to help me get
into our heavily oversubscribed CS courses by searching for open sections, but
the system seems to be designed to make using it as painful and worthless as
possible. Luckily this semester the EECS department has put dummy sections in
for all classes so students who are able to register for a lecture don't get
permanently stuck on the waiting list because of a full section.

College administrators who are considering Oracle for your registration needs,
please consider anyone else instead, for everyone's sake.

------
ardacinar
I remember a classmate of mine doing the exact same 'hack' and distributed it
to the whole school (Well, it was posted on school's major Facebook group with
10k-ish members) but it didn't have the SMS bit (It notified you via E-Mail,
AFAIK).

~~~
thaumasiotes
I don't know about you, but for me receiving an email and receiving an SMS are
identical notifications.

------
noer
This is pretty cool. I did something similar with adwords scripts a few years
ago. I was doing facebook advertising a few years ago and the ads featured
products that would occasionally sell out, but there wasn't a way to monitor
them so if something sold out, the ad would go to an empty product page. I
rigged up a system that would take a list of URLs, look for "sold out" on the
content of the page and then send me a text if it found a sold out product. I
used Adwords scripts because it was able to run jobs hourly and all I really
knew at the time was JS. It was a fun little project.

------
teddyc
I used to work in University IT and registration time is the worst. The
systems never handle the load well and everyone freaks out. I think your work
might anger the sysadmins b/c of the increased load.

~~~
tomjen3
Couldn't you just let anybody sign up for $LONG_ENOUGH_PERIOD and then
randomize who gets in the classes? It sounds like a much better solution in
general.

~~~
foepys
My university switched to that for some (mostly optional) classes. This
resulted in a lower attendance rates because as a student you end up
registering for pretty much everything instead of one or two classes you
really want. There was a system in place to fill the empty seats with students
still in the waiting queue by the second session but most students already got
into other classes and just didn't remove themselves from the queues. So the
students got an email that a seat was free for them which they ignored and
because you can only miss two sessions before failing the classes, the seats
stayed mostly empty.

~~~
aqsalose
Okay, so what about this:

Students _rank_ their course preferences in an order and submit it. For each
course, free seats are first allocated to the students who ranked it as their
first choice, then what is left to the students who ranked it as a second
choice, and so on until the round _n_ and there are less seats available than
people who ranked the course as their _n_ th choice: the remaining seats are
allocated by a lot to them. (For a very popular course, they would have to
start with the random selection right away on the round n=1).

However, instead of enrolling the lucky students right away, everyone gets
notified about which classes they have been _tentatively_ admitted, and they
must confirm their attendance before they can take their seat. If they don't
confirm in a timely manner, they lose the opportunity and the free seats are
propagated to students left in the waiting list until someone accepts (again
using random lot).

The one problem I can think of: the process (especially the propagation of
rejected offers in the waiting list) may be too long-winded to be practical.
So maybe skip the confirmation part: there's a set maximum number of courses
you can take, M, and slots are filled until you're enrolled to maximum of M
courses.

But in any case, if the overbooking is enough of a problem so that you are
looking at methods like these, the real problem is that have too many students
compared to the teaching resources. All other attempts can only mitigate the
symptoms of the root cause.

So maybe staff should choose limit the amount of students eligible to take
some very popular courses by an entrance exam, or imposing certain mandatory
minimal GPA requirement in the prerequisite courses. This option should be
especially considered if the attrition rate is high (there's a problem of
students taking much-coveted seats in the class but who fail to show up or
otherwise don't put in enough effort to pass). Usually that would have been
done on the admission level already, though.

~~~
shmolyneaux
You might be interested in bidding systems for enrolling in courses. The
National University of Singapore has one
[http://www.nus.edu.sg/registrar/events/module-
enrolment.html](http://www.nus.edu.sg/registrar/events/module-enrolment.html)

"All students are given an equal amount of points per semester to bid for
modules... The allocation of modules is based on the lowest successful bid
points against the last available quota for the module at the end of each
bidding round. If supply (module quota) exceeds demand (number of bidders) for
a module for any bidding round, the lowest successful bid will be 1 bid point.
If there is a tie in the lowest successful bid points, the outcome will be
based on first-come-first-served. Unsuccessful bidders will be fully refunded.
Any unused bid points after each round will be carried over to the next
bidding round or to the next semester at the end of the registration
exercise."

------
zaidf
I built a similar service when I was in school at UNC. Unfortunately, the IT
dept blocked my server IP and asked me to shut down the service after it got
press (and lots of usage.)

------
pszczurko
There exists something like this for Rutgers University. Anyone can submit a
course to be tracked along with email, and once a course pops up as open, an
email is sent. Text messages are not used since they are not free and this
project is maintained by students, not university. But it can be easily
modified and self-hosted since the source is on Github:
[https://github.com/v/sniper](https://github.com/v/sniper)

------
artursapek
I've done a similar thing with Southwest Airline's check-in. They let people
get seated based on the order in which they check in for their flight. So
right before the 24 hour period began I would run a script that tried to check
me in repeatedly until it succeeded. Usually got really good seats.

EDIT: to those asking for the source, it was just a bash script that I have
probably lost. The GitHub linked in this subthread looks a lot better than
what I had.

~~~
gregorymichael
Greg from Twilio here. I _love_ Southwest[1] and have wanted to do something
like this forever. Would you be up for sharing the source? Can drop me a note
at gb@twilio.com.

[1]: [http://baugues.com/southwest](http://baugues.com/southwest)

~~~
necrodome
I used this last year for ~ 10 flights.
[https://github.com/aortbals/southwest-
checkin](https://github.com/aortbals/southwest-checkin)

~~~
gregorymichael
This is great. And RUBY!! Thank you.

------
mrpippy
I did the same thing at UCSB 10 years ago to constantly poll for seats opening
up in a class. This was back when GOLD was a primitive web interface that was
clearly screen-scraping an IBM AS/400 terminal app. I used python and twill--
put values into forms (no JavaScript and the form IDs never changed), login,
search for the class, and scrape.

~~~
WaxProlix
Same at UCSC, I even considered trying to monetize the script given what an
atrocious system it was.

------
j45
Odd that this is still an issue 20 years after the first time I heard about
this.

This reminds me of the story of beartracks, and how it was shut down by the
University of Alberta to make everyone register by phone .. and eventually was
brought back by the university.

Tried to find a link I had read about the story, but can't seem to find it.

------
ChrisCinelli
[https://github.com/ChrisCinelli/scraperNotifier](https://github.com/ChrisCinelli/scraperNotifier)
\- This can check if regex exist or do not exist. It send an email or a
message on Slack using a hook.

I used successfully with Eventbrite for hackthons sign up.

------
JosephRedfern
I've written a similar thing for the Dell Outlet:
[https://outletmeknow.redfern.me/](https://outletmeknow.redfern.me/). SMS
notification isn't actually working at the moment as I've run out of Twilio
credit, but Email should be fine.

------
eggie5
I remember I made this same thing when I was in undergrad b/c my uni didn't
have waitlists. So people could waitlist on my app. Their priority in the
waitlist is determined by a market.

I remember getting SMS's in the middle of the night alerting me that a seat
opened up!

~~~
specialist
_" Their priority in the waitlist is determined by a market."_

I'm very surprised registration systems aren't (generally) some kind of phased
auction. There's got be some research, experiments somewhere exploring
different strategies.

I worked on student facing software in higher ed. Admins were obsessed with
fairness. Our registration systems had all sorts of arcane rules. Which were
hard to explain, validate, troubleshoot. And probably neither fair or
effective.

Some kind of auction, where blocks of seats were made available to different
populations, progressively over time, would greatly simplify the
implementation and understanding.

For (a greatly simplified) example, GRADUATION REQUIREMENT ABC has 200 seats.
50 seats are released every hour. First hour, seniors have first shot. Second
hour, opens up for juniors and program enrollees. Third hour, sophomores.
Fourth hour, freshmen.

Of course, there are other factors, like multiple sections (same class offered
at different times).

\--

We also talked about predicting demand (capacity planning). A novel solution
there might moot the entire registration stampede. Perhaps a "buying club"
type solution, where students state which classes they'll "buy" and roughly
when. Then registrars form up sections to satisify the largest number of
students. This could reduce the twin connundrums of waitlists and over
capacity.

Anywho. It's an interesting problem, ripe for innovative optimization,
matchmaking, auction/market solutions.

~~~
asr
I'm very surprised prereigstration systems exist at all for college level
lecture courses. Personally, I prefer the system where anyone can sign up for
(almost) any class and then the university hires/reassigns teaching assistants
as needed:

[http://www.thecrimson.com/admissions/article/2013/9/10/cours...](http://www.thecrimson.com/admissions/article/2013/9/10/course-
selection-shopping-week/)

------
soulchild37
I did a similar thing like this back in university, I made it a mobile app
with notification and passed around. It got popular and I received cease and
desist from my university shortly afterwards, now the system has a captcha on
every page haha.

------
jorblumesea
Is there a generic page scraper library or toolset? I can see a ton of demand
here and it seems like something you could use on your day to day. Things like
this actually seem like an outlier, I'd imagine airfare would be a huge one.

~~~
saamm
The nearest thing I can think of is Scrapy
([https://scrapy.org/](https://scrapy.org/)).

For airfare, I think there's already a few players in the "this flight got
cheaper" space (Kayak and Hopper come to mind), but they don't have data on
some airlines (because of the airlines' ToS IIRC).

------
Paul-ish
Someone started charging for this service near the end of my time at
University of Washington. I think the university implemented their own
notification system so that class registration didn't become pay to win.

------
cja
Why not just send yourself an email? Much simpler than using Twilio. Your
smartphone will alert you either way.

------
scottydelta
for sending yourself updates when a bug is encountered or a condition is met,
isn't it better to use something like PushBullet instead of normal text
messages?

------
ptr_void
A lot of things here smells like pure advertisement. hmm...

------
Justin_K
Can we stop calling screen scraping scripts a hack?

------
assafmo
Cool! But I'd use a Telegram bot for free. :-)

------
imron
Now just figure out a way to sell the whole solution back to the university,
and you'll have paid for your tuition fees.

~~~
saagarjha
Sounds like the quickest way to get a call from the university's IT
department…

~~~
imron
It's the IT department he should be selling it to.

It sounds like exactly the sort of service they should be providing.

~~~
saagarjha
Yes, but IT departments at universities are lazy and quick to label incidents
like this as "hacking".

------
anirudt
Probably the word "hacked" made it click baity, atleast in my opinion.
Automation, yes. Not sure there was a vulnerability involved.

~~~
rasmi
It's also a bit risky, as some University administrators may not understand
that he isn't actually being malicious. Automated scraping can be considered a
violation of some IT policies though, so OP should be careful about something
like this.

~~~
ian_d
I'm in higher-ed infosys and we see a number of automated registration sniping
hacks in our monitoring / metrics. We usually don't run it up the chain unless
it's malicious or impacting the service for other students. Otherwise we just
investigate it a little bit and share it on Slack with the group if it's
clever. The only time we've ever reached out to a student is to ask him to
stop sharing it with others (and fix it) because it was performing some really
aggressive polling.

Some other schools in our system are really reactionary, though, and consider
_any_ automation a ToS violation and will freak out.

e: And if you know the URL pattern / platform of your Uni's registration
system, there's probably already a couple of examples on github of a
registration bot.

