
Unit tests fail when run in Australia - luu
https://github.com/angular/angular.js/issues/5017
======
encoderer
I'll write it up when I finally have this whipped, but we have a unit test
stumper ourselves. In our Django project, the tests have always run slow. Once
we got above 50 tests I finally decided to hunt for the cause. It creates a
test DB from fixtures for each test case but i couldn't understand how that
could possibly take several seconds per test.

Then I was coding on the train home one night and in the tube my hotspot lost
connection. Then I happened to notice... with wifi off the tests ran 10x
quicker. Pull out netstat.... I see http requests to akamai essentially as
each test setUp() is run. It's SSL so I only see a hostname, not the full
path, and they just look like this, but the hostname is dynamic:
a23-34-132-181.deploy.static.akamaitechnologies.com

I spent 30 mins reviewing our code, grepping my code and packages directories,
and stepping thru in a debugger. No luck.

~~~
gred
Could be a remote XSD referenced from a local XML file? I've had that happen a
couple of times...

The worst was having a Map keyed on URLs in Java -- turns out URL.equals() and
URL.hashCode() do DNS lookups... Easiest 50% test speedup ever!

~~~
encoderer
Oh, this is an interesting lead.... I feel pretty certain the request isn't
coming from our Python after stepping-thru the code.

The other suggestions -- using a SSL proxy -- has been my plan of attack.

~~~
ekimekim
Try an strace?

    
    
        strace -e trace=network ./run_my_unit_tests

~~~
mst
Also -f so it follows forking (I always forget that the first time, swear, and
have to re-run strace with it)

------
cpeterso
One of my favorite mysterious test failure is from BeOS folklore:

[http://www.haiku-os.org/legacy-
docs/benewsletter/Issue4-22.h...](http://www.haiku-os.org/legacy-
docs/benewsletter/Issue4-22.html)

A Testing Fairy Tale

Two test engineers were in a crunch. The floppy drive they were currently
testing would work all day while they ran a variety of stress tests, but the
exact same tests would run for only eight hours at night. After a few days of
double-checking the hardware, the testing procedure, and the recording
devices, they decided to stay the night and watch what happened. For eight
hours they stared at the floppy drive and drank espresso. The long dark night
slowly turned into day and the sun shone in the window. The angled sunlight
triggered the write-protection mechanism, which caused a write failure. A new
casing was designed and the problem was solved. Who knew?

~~~
wkonkel
I heard a similar story in the past but with the culprit being a janitor
turning on their vacuum every morning at 6am causing power spikes.

~~~
kibibu
We had a tech support customer who called every few Mondays at 9am because
their keyboard wasn't working.

The cleaner's vacuum cleaner would occasionally pull out the PS2 cord.

~~~
dghf
After the first call (or at latest the second), did they not learn to plug it
back in themselves?

~~~
kibibu
I wish I still had your high estimation of tech support customers.

------
sergiotapia
Somewhat relevant, file this under interesting bug - a story to tell
youngsters when you're a grey and weathered software developer:

[http://web.mit.edu/jemorris/humor/500-miles](http://web.mit.edu/jemorris/humor/500-miles)

~~~
PebblesHD
This has to be one of my all time favourite IT related stories. Nice find!

------
jakejake
Before clicking the link and knowing absolutely nothing about what project
this even is, I knew from the headline that it would be a timezone or DST
issue. I've had the displeasure of dealing with a lot of TZ offset issues in a
time-critical app. They are a always a joy to discover, fix and test.

~~~
deathanatos
I was also thinking this would be related to DST. The "tricky" bit about
Australia is that DST is "backwards", at least compared to those of us in the
northern hemisphere. Because the seasons are reversed from those north of the
equator, DST is also likewise reversed. They spring forward in October, and
fall back in April. It all should make sense when you think about it. (Longer
days are still in the summer, and thus DST is still in the summer.) This is of
course not unique to Australia. (I think they just happen to crop up
probabilistically by having a high population using technology, and are south
of the equator.)

The cool bit, at least to me, is that the mnemonic "spring forward" still
works. Likely, this is only cool to me because the whole thing is unusual.

The other odd bit is that some TZs sit on the "wrong" side of the
international date line, and thus get interesting offsets such as +13 or +14h
from UTC. I think there's some that are -13h too, so the total "length" of day
(that is, how long you could remain on a calendar date) is >48h. Thus, at any
given time, people in their local time are in as many as three different
calendar dates.

There's also a few TZs with half-hour offsets. (e.g., they're something like
4.5 hours off UTC.) There's a 3-hour TZ jump at one point on China's border, I
think, because all of China is on a single timezone, and the nation is quite
wide, east-to-west.

We've not even gotten to leap seconds, or that 2100 won't be a leap year.

And governments regularly muck with this stuff. The set of timezones and their
offsets and DST preferences are constantly changing.

~~~
vacri
There's even a silly bit of Australia that has a TZ on a 45-minute boundary
(UTC+8:45). Eucla is a tiny place in the middle of nowhere, population 86 in
the town, though I don't know how many others might be in that TZ.

[http://en.wikipedia.org/wiki/Eucla,_Western_Australia#Time_z...](http://en.wikipedia.org/wiki/Eucla,_Western_Australia#Time_zone)

~~~
mlakewood
I think the Sillier one is Broken Hill. That although its in NSW is on the SA
Timezone.

~~~
sytringy05
Kununurra have the opposite problem - they are on AWST but are only about 400
kms from Darwin. So the sun comes up at 4am and sets at 4pm.

------
apaprocki
I haven't verified it by any means, but this to me smells like ES5
15.9.1.8[1]. Basically, prior to ES6, Javascript engines did not prescribe the
use of IANA (nee Olson) tzdata to resolve historic dates.

The standard says: "The implementation of ECMAScript should not try to
determine whether the exact time was subject to daylight saving time, but just
whether daylight saving time would have been in effect if the current daylight
saving time algorithm had been used at the time. This avoids complications
such as taking into account the years that the locale observed daylight saving
time year round."

In Spidermonkey, at least, it mapped current years onto prior years where the
current day falls on the same day of the week.

edit: Spidermonkey bug I filed a long time ago:
[https://bugzilla.mozilla.org/show_bug.cgi?id=1029923](https://bugzilla.mozilla.org/show_bug.cgi?id=1029923)

[1]: [https://es5.github.io/#x15.9.1.8](https://es5.github.io/#x15.9.1.8)

------
broodbucket
Daylight Savings is nothing but a frustration nowadays, especially in
Australia where a) our DST does not quite align with the switchover in the US,
but only by a couple of days, and b) out of the four states on the east coast
time zone, one of them does not use DST which makes things more confusing.

On a side note, the title reminds me of a similarly interesting bug in
OpenOffice.org:
[https://bugs.launchpad.net/ubuntu/+source/file/+bug/248619](https://bugs.launchpad.net/ubuntu/+source/file/+bug/248619)

~~~
imron
> Daylight Savings is nothing but a frustration nowadays, especially in
> Australia

Except for you know, when it's still light at 7.30pm.

I like light evenings in the summer and am prepared to accept a little
complexity as a tradeoff.

~~~
MichaelGG
People say this, but what's stopping you from going out at 6:30 instead?

And didn't Australia do some last minute DST nonsense like a Latam country,
versus planning far ahead?

~~~
yen223
I never liked the argument that DST gives you a consistent sunrise and sunset
time, because _it can 't_. DST can't extend the amount of daylight for obvious
reasons.

It's rather like redefining the metre so that everyone is 2 metres tall -
you're adding a lot of complexity for not a lot of consistency.

~~~
imron
Who makes that argument? Daylight savings doesn't give you consistent
sunrise/sunsets.

For example, in Melbourne, Australia, in the winter it will be dark by around
5:30pm. In the summer it gets dark by around 8:30pm. I like having
inconsistent sunrise and sunset times (e.g. extra daylight time in the
evenings in the summer).

Neither do I think daylight savings extends the amount of daylight. What it
does is shift the typical working day so that by the end of it, there are
still a few more hours of daylight left. That's what I like about.

~~~
broodbucket
That's all well and good, however I don't think it comes close to justifying
the complexity.

~~~
imron
Ironically, removing it now would only add to the complexity.

------
rusanu
SQL Server 2005 actually shipped with code that used the local time instead of
UTC time in CREATE CERTIFICATE. Users in Western hemisphere (local time behind
UTC) had no major problems because of it, but users in Eastern hemisphere
(local time ahead of UTC) would create certificates _not yet valid_.
Confusing, to say the least. See [http://rusanu.com/2008/08/25/certificate-
not-yet-valid/](http://rusanu.com/2008/08/25/certificate-not-yet-valid/)

------
pwnna
When I worked at Mozilla, we had an issue where the unittests will always
past, except on very few occasions. The message was not entirely obvious what
the problem was (from as far as I remember) so we ignored it for a bit as it
didn't seem to impede any work.

We eventually realized that the unittest will only fail after work hours, at
which usually no one is around to start tests. It turns out the test will fail
during 6PM - midnight because of an UTC issue.

------
chrstphrhrt
Relevant: [http://t.co/yQb2vSuZDw](http://t.co/yQb2vSuZDw)

~~~
cratermoon
There are countries that have changed which side of the International Date
Line they claim. As recently at 2011 Somoa switched.
[http://en.wikipedia.org/wiki/International_Date_Line](http://en.wikipedia.org/wiki/International_Date_Line)

Now _that_ is the sort of thing that drives programmers nuts.

~~~
shortoncash
I actually had to deal with this exact problem two days ago. I was baffled
because the timezone was being listed as UTC+14, and that places like Kiribati
crossed the date line. UTC+14 didn't exist until the date line was crossed.

I have an app where users are running it while crossing time zones. :-(

------
j_lev
I used to work at a phone company in Australia. DST was the pits. Think a five
minute call where the caller drives across state lines, from a DST-observant
state to a non-observant state. Timestamps come from the base station through
which the client's phone is connected. several weeks later, client is furious
that they have been billed for an hour and five minutes (back when it was
expensive).

Every time we switched to or from DST there were a couple of outliers that got
caught. We'd update the tables to handle the case correctly next time, but
would miss something else, if the change didn't break something else
completely.

~~~
Xophmeister
Why not just normalise all timestamps to UTC?

~~~
lsaferite
I his use case that would work. It won't work for timestamps in the future
though.

------
balabaster
If the problem is indeed network related, I'd fire up Fiddler. You can install
a Fiddler SSL cert locally to decrypt your SSL connections, it'll play man in
the middle quite effectively and should hopefully give you some more intuition
as to whether that's the cause of your problem.

Why do you have unit tests hitting outside services? If you have developed
service clients that need to be unit tested, you should have mock services for
them to run against locally so that your unit tests aren't going down every
time there's a remote service failure. They don't need to be full blown
services, but they should receive the request and respond with the expected
result the service would give when hit by the client.

If your clients are generated by the remote service [i.e. by WSDL], then you
shouldn't theoretically need to test those as they're generated by the third
party - so you should be unit testing against mock clients.

I'd argue that integration tests with real world 3rd party services should be
handled by some reporting system on your build pipeline somewhere and your ops
team should be investigating that failure, you shouldn't need to be running
them locally on your dev machine.

------
geronimogarcia
This reminds me of [http://stackoverflow.com/questions/6841333/why-is-
subtractin...](http://stackoverflow.com/questions/6841333/why-is-subtracting-
these-two-times-in-1927-giving-a-strange-result/6841479#6841479)

------
jlangenauer
We used to have the opposite problem (being based in Australia): certain unit
tests would run fine locally, but then end up failing when run at certain
times of the day on our Travis server which was in the US.

------
bootload
_" Chrome 31.0.1650 (Mac OS X 10.9.0) ngMock TzDate should fake getHours
method FAILED Expected 4 to be 3."_

Hard coded dates not taking into account timezone and daylight saving? Been
there, done that. This week the time was AEST-DS (Daylight saving) and is now
AEST (finished daylight saving) [0]

[0] For example: Melbourne AEST: Australian Eastern Standard Time == GMT
+10Hrs

DS AEST-1 hour when daylight saving is declared. When Daylight saving
finishes, DS is AEST+1 hour or AEST again.

------
junto
This blog post by Jon Skeet is probably the best read when it comes to dealing
with dates, times and timezones: [http://blog.nodatime.org/2011/08/what-wrong-
with-datetime-an...](http://blog.nodatime.org/2011/08/what-wrong-with-
datetime-anyway.html)

------
smileysteve
Had a time issue in a test last week.

Test was that at $1/day, now+1 month would be $31. Wrote the test on the 30th
of March, so found out about it the next day.

If it had been written on the 1st of March, wouldn't have found out about it
for a month.

------
reality_czech
Come on, everyone knows that unit tests run counterclockwise in Australia. How
could you expect it to succeed?

~~~
tempodox
Exactly. And the writing appears upside-down, naturally. After all, gravity
points in the opposite direction as on the other side of the earth; that's why
the Australians are called antipodes :-P

------
miked
Australians and Brazilians: your bits are upside down. Try flipping them over
and then running the tests again.

