

Ask YC: My school wants to shut down my app. How should I respond? - zaidf

Few hours ago, I received this letter from my public uni's CIO: http://www.scribd.com/doc/15099510/i-letter1<p>The site can be seen at http://www.classhunt.com<p>Full story:<p>I launched ClassHunt about a month ago. The campus newspaper covered and ~500 people are using it so far. Many are totally in love with it("ClassHunt is AWESOME!  I give you serious props for creating this site.  I've already told a couple of friends about it and one of them has had the same luck that I have.  You're going to prevent a lot of headaches with this system! :)").<p>The technology behind is in python. It queries the uni's server for every department on campus and parses the course data. At first, I was querying every minute, so ~130 requests/minute. Now I am querying every 15minutes.<p>Few days ago I received an email that my server IP had been blocked. The lady from IT dept was very nice and willing to work with me to find a resolution. For now, we agreed to increase the interval to 15 minutes. Still they could not unblock because they were still having issues with the load and people higher up won't approve.<p>Having been a student here for a few years, what happens is towards the end and beginning of each semester, our course registration system starts crawling. This is because too many folks are checking grades and signing up for classes. It lasts 3-4 days. This is what seems to have happened this week as our school year came to an end. It seems like they are blaming my app for their years of issue with system overload.<p>I am confident it was not my app causing the problems. Nothing is bigger proof than the Chancellor's staff writing to me that they are still having problems even after blocking me.<p>Still, I don't want to get in a verbal war with them and rather just find a solution that works.<p>How should I respond to this letter from the school's CIO/Chancellor for IT?
======
jfarmer
Make a deal with them. It doesn't sound like they object to ClassHunt per se,
rather they don't like the additional load that ClassHunt causes.

So, what do they want? 1\. Reduced load and a more stable class registration
system

What do you want? 1\. For ClassHunt to be used across campus (and not get
banned!)

What can you do to satisfy both these goals? One idea would be to pitch
ClassHunt as a load-reducing service. By acting as a proxy to the class
registration system you can smooth out the load. Have them _actively_ promote
ClassHunt on the class registration page and explain to them how more people
using ClassHunt will result is less load for them, not more.

It will also make the situation more flexible because they only have to deal
with one person (you) rather than relying on the collective behavior of the
whole of campus.

Anyhow, just a thought.

~~~
zaidf
Great ideas! We'll see how receptive they are. From the legalese letter I
received today it doesn't seem to encouraging.

At the same time, the contacts I spoke with at the IT dept. few days ago could
not stop praising the website.

~~~
jfarmer
Like icey said, you know the worst case scenario. They've played their hand.

But reading the letter, their only concrete objection is the fact that you're
using "excessive resources." That's their pain point, so make them see that
your service helps that problem, not makes it worse.

~~~
brodie
I don't know, I could imagine them being somewhat serious about the
license/number of connections violation.

Speaking from direct experience, I wouldn't underestimate the amount of
bureaucracy internally that would come down on this like a ton of bricks. Like
others have mentioned, his best bet is to get someone to champion his cause
internally. It doesn't even necessarily have to be someone in their ITS
department - it could be a faculty member, or even someone in administration.

Hell, you might be able to get a job out of this.

~~~
jfarmer
Why discount the effort before you try? The worst that happens is they don't
buy the pitch, and he's back where he is now.

------
TomOfTTB
Are they at all open to compromise? Where I work we have a school for special
needs kids and I had a problem with getting data from the individual school
districts. So what I did was to write a short program myself that did what I
needed and asked them if they'd agree to run it on their servers. There was
some initial hesitation but after I bought them lunch and used the opportunity
to walk them through the code line by line they were willing to install it.

I'd imagine a similar system could work for you. It's a public database so you
could write something that ran on a client machine (so it doesn't compromise
their security) and since it's internal it wouldn't eat into their bandwidth
(which is the only realistic problem I could see your program causing, one
query every 15 minutes is not going to take that much processor power). Have
it send you short text messages when there's a change and you're set.

------
dmillar
I had this same exact app a few years ago. (dropmessenger.com) It only worked
for the University of Utah, and they did eventually contact me about polling
the server. But, they were pretty cool about it, and even asked if I was
looking for work. :)

The reality is, they can't really make you take it down, but they can take
steps to make your life miserable. Without any reliable API it will be
difficult to scale such an app.

Good work though.

------
chaosmachine
_At first, I was querying every minute, so ~130 requests/minute._

Wow, that's not nice.

~~~
harpastum
It's not the nicest thing in the world, but his university (UNC, as far as I
can tell) has 30,000 students. 130 requests/min is the equivalent of having
0.4% of the students on at one time.

It obviously creates a visible imprint, but it shouldn't be a show-stopper for
the school.

~~~
zaidf
Yeah and like I said, it didn't cause a problem for over 3 weeks. And I am
still not convinced it was the culprit in the last few days.

From what I've experienced and from friends who went here five years ago,
their system goes down/has issues every year during end of semester. This year
they seem to have a scapegoat to blame it all on...except it doesn't help to
admit that they are still having issues even after blocking me.

~~~
jerf
Devil's advocate: Without knowing the details, I would observe that the load
generated by searches can be very different. Searching for all 100-level
Physics classes may be an optimized code path (or DB query) that occurs in a
fraction of a second, where as "Give me all classes in a department starting
with P" might be slow, or "Give me all courses taught by Prof. Bill" might go
down an unindexed code path. They might never have noticed if normal users
only fired it only once per hour, but if you happen to be using more of those
queries than a normal student because it's more convenient for you, then you
might actually be hammering their system without meaning to.

Or, put another way, "queries per second" is a terrible metric for "how much
load I'm subjecting their system to". :)

I can give you one bound you might want to instrument up to check their
contention about load: How quickly they answer. If they answer in 1ms (minus
ping time), then there's a sharp bound on how much of their resources you can
be taking up. (I'm going to assume bandwidth isn't much of a concern here,
based on the numbers.) If you see some queries taking longer, perhaps even
into the seconds, then they may have a point. It could just be general load,
or it could be your thing specifically.

------
cmars232
My recommendation: apologize and drop the issue with IT in a vague and
forgettable manner, use proxies (TOR, open proxies, a cheap VPS), set a normal
browser or web-crawler user-agent -- and please, whatever you do, go _easy_ on
their infrastructure. Realize that schools sometimes (though not always) have
substandard, under-funded IT depts.

Your best bet is to stay under the radar and let your app build momentum with
your users. Don't fight IT, and don't beg permission.

~~~
phugoid
That's what I was going to suggest as well.

From your posts, it seems that the University's IT people like your app, and
there's really no technical problem anymore. So randomize your IP number, and
if anyone wants an explanation you can tell them that you have optimized your
system so that instead of directly requesting all their data, your application
now behaves exactly like a real student accessing their site in a conventional
way.

------
gills
Wow. Schools pay load-based licensing costs for registration systems? There's
a niche just waiting to be demolished by some startup...

I think you have a useful idea here, and you should try to find a way to make
it work. It sounds like your university's IT department thinks so too, but
they have real limitations and good reason to ask you to cease operation
because, unlike your service, theirs is _required_ to be available.

It sounds like without the cooperation of that IT department, your app (at
least at UNC) is going to be a big nothingburger. Do whatever you can to make
them your partners in implementing this service rather than your adversaries.
This could mean collecting data that explains how your service improves their
service; helping them measure and upgrade their current system so the load is
a non issue; integrating your code at a lower level (direct to DB); beta
testing a parallel registration system that will eventually save them lots of
money on licensing.

You have a difficult task with any solution. University policy may prohibit
certain things, such as allowing a student or a student's software to directly
access the real course registration data; or the timeline to have your app
approved to run inside the security perimeter may be too long for you to
bother trying. You may also want to check your school's invention rights
policy before pursuing any tighter integration with their systems.

If this hiccup is not a showstopper for you, just keep building your product.
You can work on displacing whatever is causing the problems at this IT
department, but if that is too far afield from your core (proposed) business,
maybe just move on and try deploying the app against other universities'
registration systems.

------
lleger
I work in the IT department of a major university and this is crazy. I'm also
one of their students. You got a really cool service that solves a major
problems for any large institution. Towards that end, this decision was
probably made by a manager, who doesn't know much. All of the real knowledge
is with the analysts. I'm sure that having an open dialog with them would get
this changed. I doubt they'd keep you out with a little discussion.

------
paul7986
Sell your product to them or set a deal up where you integrate into their
system. It sounds as their is a demand for your service! Take advantage of it
and either be the business man yourself or find someone you know who can sell!

------
pj
Sell them a license to run the software on their servers and query the
database directly to reduce load.

------
codeodor
How often do courses change? Couldn't you just do this once or twice a
semester?

------
nose
Maybe switch to appengine?

~~~
zaidf
Can you explain how that'll help? Thanks!

~~~
natrius
New IP address.

~~~
zaidf
LOL. Yeah that's the worst case option. But I'd rather not get in a cat-and-
mouse game with 'em where I've to find a new IP every hr because they blocked
the current one.

------
seiji
I attempted to automated my course registration process a few years ago. The
registration software had a fixed limit of the number of pages you request
before you get locked out.

When you are locked out, a vague message appears about "attempts exceeded."
Not knowing what to do next, I went to the registration office. They quickly
realized what had happened and threatened me with disciplinary action because
automatically accessing the registration system violated their vague
interpretation of the school conduct guidelines (I also received a free
lecture about how 'spiders' are border line illegal).

After being further threatened with a "last day, last hour" registration
timeslot until the sun burns out, I put on my best "I'm sorry" face and handed
over a letter of apology and the 30 lines of code as they requested. They even
had the courtesy to make snarky remarks about spending more time doing
homework and less time trying to break their systems.

WWW::Mechanize -- so easy it's dangerous.

------
jjs
How often does this info change? Is there any reason why you must poll it so
frequently?

~~~
zaidf
Yes, what I am doing is monitoring EVERY class on campus for its availability,
which changes by the minute.

I can get all the classes by department, there are ~130 depts so ~130 page
requests.

~~~
silentOpen
Convince them to implement push then work in cooperation to publicize your
service?

------
earl
I think you should take it down then ask for the course schedule via FOIA.
They're a public school, no?

~~~
zaidf
I am already using the data under the premise of FOIA. Even if I request it
explicitly, they will need to provide it electronically and will have the same
supposed problems with traffic overload.

~~~
shib71
Except that they would only have to deal with 1 hit/min instead of 130, and
possibly on a static file instead of a one generated on the fly.

~~~
jerf
Where did that 1hit/min come from? Does the FOIA have guidelines about that?

(I ask for information, not as criticism.)

~~~
zaidf
I think what he means is if they make a custom page, they can provide ALL
classes data for ALL depts on one page...where as right now I am having to
query 130 different pages to get class data from 130 departments.

~~~
cflee
In which case wouldn't the traffic overload issue be resolved? 1 request/min
seems a far cry from ~130 requests/min, and quite a reasonable compromise.

