
Visa Systems Issues - jpatokal
http://travel.state.gov/content/travel/english/news/technological-systems-issue.html
======
karambahh
Does anyone else think that, all conspiracy theory aside, the information
published is only partly true?

If it's a purely hw issue, say a piece of highly specialized hardware (hw
crypto, piece of old mainframe, etc), it can take a very long time to source
it (although a SPOF is surprising in such a critical infrastructure), but it
does not take 100 people to work on it, 24/7\. It requires a dozen guys making
angry phone calls every 2 hours to some suppliers...

If it really takes 100 people 24/7, mostly likely explanation to me is that
it's software related and they have to rewrite a critical sw in record time.
Causes:

-bugfix (which may be entirely unrelated to secrity)

-emergency migration from one hw vendor to another, for sourcing reason which entails rewrite of a part of the system.

~~~
bigiain
"100 people to work on it, 24/7" sounds like manual data entry to me - who
wants to bet against a loss of a db without usable backups available, and a
bunch of people now hand entering hardcopy forms and data out of emails back
into some critical database?

~~~
eps
... which means that somebody somewhere didn't have backups of some database
server that died.

~~~
pdkl95
I'm guessing a backup _technically_ existed (to satisfy any requirements or
boss's-orders to the "letter of the law"), but nobody bothered giving any
thought to _restoring_ everything from that backup.

Of course, a backup that can't be restored (or nobody knows how to restore) is
more or less equivalent to not having a backup, so this distinction probably
doesn't matter.

// always remember to test your restore plan

~~~
bigiain
Indeed - "usable" backups - they are ones you've tested restoring from
(recently enough to be assured they still work on the latest version of your
system).

I _hope_ they don't have someone saying "we had a backup - it was on a RAID
set!".

It would surprise me less to discover the reported migration from Oracle on
Windows to Oracle on Linux was still partly done, and they were taking solid
reliable useable backups - of the old not-yet-decommissioned windows db
servers...

(For the record, I've made both of those mistakes (and more) in my career...
Fortunately neither represented weeks of 24x7 remedial work by 100s of
people.)

------
jpatokal
Apparently last year's major failure was caused by their Oracle data warehouse
going down hard:

[http://foia.state.gov/_docs/PIA/ConsularConsolidatedDatabase...](http://foia.state.gov/_docs/PIA/ConsularConsolidatedDatabase_CCD.pdf)

 _" The Consular Consolidated Database (CCD) is one of the largest Oracle
based data warehouses in the world that holds current and archived data from
the Consular Affairs (CA) domestic and post databases around the world. As of
December 2009, it contains over 100 million visa cases and 75 million
photographs, utilizing billions of rows of data, and has a current growth rate
of approximately 35 thousand visa cases every day"_

Unclear what's gone wrong this time, but the mention of "biometrics" (like
photographs) makes me suspect it's the same system.

~~~
cm2187
A Visa application system doesn't strike me as an obvious candidate for a
relational database. I would rather store all the information related to a
visa application in a document store with a relational database only used as a
sort of index. I wonder if lots of systems are not built using relational just
out of habit. And then bump into these problems.

~~~
eru
Why not? There should only be at most a few billion rows (since you can't have
more visa applications than humans).

~~~
bigiain
Never bumped into the "beautiful flexibility" of ERD databases - where the app
devs assume all responsibility over things that database designers _really_
should be saying "Hell no!" to?

    
    
      +------------+------------+--------------------------------+
      | asset_id   | asset_name | asset_data                     |
      +------------+------------+--------------------------------+
      | 2147483646 | firstName  | Joseph                         |
      | 2147483647 | lastName   | Bloggs                         |
      |-2147483647 | phoneNumber| +1 415 555 1234                |
      |-2147483646 | email      | joebloggs@gmal.com             |
      +------------+------------+--------------------------------+
    

(I think I still have brain damage from trying to get "too smart" with a
Magento eCommerce site once...)

~~~
duaneb
The flexibility typically comes from transactions and joins, something
document store proponents typically shy away from. And yes, if you give stupid
devs powerful tools they will shoot themselves.

~~~
eru
It's not only flexibility, but also discipline and taking away choices (for
bad modelling) that comes from following the relational normal forms.

------
kalleboo
Was the State Department _still_ running the whole Oracle database cluster on
a single hardware node since last year's issues?
[http://fcw.com/Articles/2014/10/20/State-Department-
database...](http://fcw.com/Articles/2014/10/20/State-Department-database-
crash-1.aspx?Page=all&p=1)

~~~
needusername
10g on Windows 2003, can't make this shit up.

------
jpatokal
This apparently started on June 10th, so down for _12 days_ and counting!

~~~
f00barbazb00
So they're saying _We 're terrible and we don't care that much_. Wow.

If this where a business, they'd be losing customers... but thankfully,
they're a government apparatus that holds people over a barrel and doesn't
have to provide enough decent service to offer enough visas to meet the
demand, hence 12-30 million undocumented immigrants.

~~~
codecamper
What is government but a bunch of provided services? We could replace it with
well written software. No?

~~~
jacalata
Technically, sure (assuming you also build the robot hardware) but we're quite
far from having software that can negotiate treaties, figure out how to
regulate water rights in California, etc...

~~~
codecamper
Well of course I don't mean to completely replace the government.

But how about making it more lean? How about modernizing with the times? How
about making it all much more transparent and accountable? Couldn't open
source software do all of this?

We could start with systems that are used by smaller, poorer countries & grow
from there.

A visa processing system might be a good start.

~~~
pratnala
I completely agree with this. First, we need to make the lawmakers that such a
system is necessary which is understandably, a bigger challenge.

------
Larrikin
I've been trying to get marriage visa things completed since May.
[http://nvc.state.gov/](http://nvc.state.gov/) The payment system was down for
almost the entire month of May. After finally being able to get through step
2, I've been stopped constantly. The entire social security website was down
one night this month. I currently need my tax transcripts from the IRS. The
online system for printing them out is down indefinitely and even the form to
request they be sent by mail was constantly throwing up a technical error has
occurred message.

~~~
shepbook
I know that pain. Took me and my wife almost 5 years, near $2,000 and three
interviews to get her green card. And that was with her already being in the
states when we met and got married.

Keep at it. The feeling when it's finally over is great.

~~~
kalleboo
These stories always fascinate me. At that point why even bother validating
the grounds for the visa - anyone who would put up with a process like that
deserves it!

I immigrated to Japan - a country often held up as an example of xenophobia
and resistance to immigration, and it only took a week for my spouse visa
paperwork to be examined and approved. The only cost was for translations of
some paperwork from home ($40 at the embassy) and then $20 for the residence
card once I was approved.

------
guard-of-terra
On slightly related note, USA doesn't support airport transfers without visa -
which is taken for granted in the rest of the world. Which basically makes
Latin America an island since most flights there are routed via USA, its
gatekeeper.

------
mikhailfranco
FBI site for background checks also has implementation issues:

[http://www.fbi.gov/about-us/cjis/identity-history-summary-
ch...](http://www.fbi.gov/about-us/cjis/identity-history-summary-checks)

So they can record all our conversations in real-time, but it takes 3-4 months
to do a simple query of criminal records.

~~~
Shank
Different departments. The NSA and FBI don't talk to each other; NSA has all
of the technological innovations for monitoring wide spreads of internet
bandwidth.

~~~
us0r
"The NSA and FBI don't talk to each other"

You clearly have not read any of the Snowden docs.

[http://cryptome.org/2013/11/snowden-
tally.htm](http://cryptome.org/2013/11/snowden-tally.htm)

~~~
knowaveragejoe
You're both wrong, in a way. They interact, but that doesn't mean they are
cohesive in utilizing each other's technologies/competencies.

~~~
us0r
[http://cryptome.org/2015/06/cyber-spy-
nyt-15-0604.pdf](http://cryptome.org/2015/06/cyber-spy-nyt-15-0604.pdf)

------
narrator
What's funny is they used to do this stuff with much much slower computer
systems back in the 70s and 80s.

~~~
rodgerd
That was before it became a political imperative to drown the government in a
bathtub.

------
processing
I'm waiting for an advanced parole visa (moved to the US in January) and can't
get a visa in time to travel intentionally to my best friends wedding in the
UK. Applied in Feb. Got told last week they need another 2 months to process
it - I wonder if this is why?

~~~
beagle3
Schedule a personal meeting at your closest USCIS (or whatever the immigration
office is called these days) and explain it to the officer. They can give you
a temporary "paper pass" you can use to travel until paperwork completes if
they find your reason sufficient.

Best friend's wedding might not qualify as a good enough reason, but combined
with the long delay it might - definitely worth your time talking to them.

I know someone who got this kind of pass to visit her sick father.

------
davepage
Took 14 months with Congressional intervention just to get my partner a
immigrant visa -- which is presumably the easiest.

America has been so obsessed with thinking of itself as the "best country in
the world!(TM)" for so long, it has meanwhile regressed into a failed state.

~~~
scintill76
American Exceptionalism, etc. bother me too (an American), but it seems a bit
incongruous to call it a failed state while simultaneously relating the story
of someone desiring to immigrate to it so badly that they'd wait 14 months and
get Congress to intervene. If it's so failed, your partner can leave and I'm
sure someone else will be happy to come.

~~~
davepage
Of course I am a hypocrite. My partner wants to visit the US; I would prefer
to live elsewhere. But, sometimes compromise is needed in relationships.

'failed state' is certainly hyperbole here -- there are large elements of
society which continue to function, despite the problems at the national
government level. It would truly be a failed state if, for example, the
national government's performance level were propagated to the whole society.

------
DyslexicAtheist
> There is no evidence the problem is cyber-security related.

have they fixed their data-leak into archive.org yet? (LOL)
[http://blog.valbonne-
consulting.com/2015/05/20/misconfigurat...](http://blog.valbonne-
consulting.com/2015/05/20/misconfiguration-of-state-gov-website-exposes-pdf-
files/)

------
Nano2rad
Government holding biometric data is unsafe for citizens. There will always be
problems, security of data and security of citizens.

~~~
klipt
Every visitor to the US is fingerprinted these days.

~~~
fleitz
When did they start doing this?

I arrived two weeks ago, no finger print required.

~~~
mahyarm
[http://www.immihelp.com/visas/usvisit.html](http://www.immihelp.com/visas/usvisit.html)

If your canadian, probably not.

~~~
liquidmetal
They probably already have you fingerprints.

~~~
mahyarm
No they do not. I've never been fingerprinted as a Canadian until I got a work
visa for the USA. For many Canadians who visit the usa for tourism or similar
will never have their fingerprints in a canadian or us database under current
laws.

------
anindha
US visa processing is back online (same link).

"The Bureau of Consular Affairs reports that the database responsible for
handling biometric clearances has been rebuilt and is being tested. 39 posts,
representing more than two-thirds of our normal capacity, are now online and
issuing visas. We are working to restore full biometric data processing."

------
zuma
Just to add to the conspiracy theory, wonder whether this is related:
[http://www.reuters.com/article/2015/06/08/us-g7-summit-
obama...](http://www.reuters.com/article/2015/06/08/us-g7-summit-obama-
cyberattack-idUSKBN0OO1RA20150608)

------
acyacy
It is natural for issues for happen like this time to time. If it's not
technology, something else. Australia's visa offices (DIBP) are on industrial
action at the moment.

~~~
jpatokal
So, what's the last time a Fortune 500 company's IT systems failed so hard
they couldn't do anything for _two weeks_?

~~~
acyacy
On a technical level, it's possible, on a practical level it's not. You can't
get 100% uptime due to the unforeseen problems that will hit once in a while.

If the downtime was anticipated it can always be averted. Even a Fortune 500
can expect some downtime over a period large enough.

Take Sony earlier this year. The US government itself the year before.

------
friism
I'm glad that they're at least prioritizing H2A agricultural workers.

------
istvan__
Such a muppet show this entire situation there.

~~~
f00barbazb00
Statler and Waldorf are especially hilarious.

~~~
istvan__
I wish others would also appreciate a little bit of humor on HN, but
everything like that gets downvoted all the time.

------
f00barbazb00
Makes me glad that I _accidentally_ put my RFID-enabled passport in the
microwave, typed in 1 second and hit start.

~~~
acyacy
It must fun be waiting in the non-RFID queue at immigration.

------
malkia
On 18 June 2015 the Lufthansa site looked like this
%templateSomethingVar/somethingVar1% and lots of variables like that (not sure
what template system were they using). Now it's okay.

But Polish airlines is down too (unrelated systems I guess)... but could this
be something coordinated?

Ah... probably not :)

~~~
saryant
Lufty's site has always had bugs like that.

------
contingencies
HA-clustered microservices: 1

Dated megaliths: 0

(Edit: Sheesh, what's with all the downvotes? Hardware failures are a route-
aroundable thing that should never cause downtime... as long as you don't
adopt an outdated "here sits the one source of truth, and its name is [fat
server manually configured]" psychology. HA clustering is at the point where
it's a generic, drop-in thing for arbitrary services. If you fail to recognize
the above, go do some learning.)

