
Paypal has not been sending payment notifications to merchants - emilepetrone
http://discuss.tindie.com/t/disbursements-through-us-bank-accounts-or-debit-cards/599
======
thenduks
You can make an API call to check the status of a transaction very easily.

When my users are redirected back to my site (thanks page, or similar), I
check if their transaction is completed, if not, I kick off an every-five-
seconds check while asking the user to hold on while we talk with paypal. I
will eventually fail after some number of checks of course, but this means
PayPal can stop sending IPNs and everything will just keep going along just
fine.

If the user might not end up back on your site for some reason, run a cronjob
that tries to verify transactions created in the past day/hour/whatever.

An issue like this doesn't have to, and really shouldn't, cripple your
business.

~~~
jarcoal
Agreed. It doesn't take a lot of working with PayPal to know the IPN system is
fragile at best.

PaymentDetails is mandatory for any adaptive payments integration.

[https://developer.paypal.com/docs/classic/api/adaptive-
payme...](https://developer.paypal.com/docs/classic/api/adaptive-
payments/PaymentDetails_API_Operation/)

------
contingencies
If you've ever sat down and looked at Paypal's APIs properly, for instance
with a view towards a proper risk analysis and/or from-scratch integration,
then this should not surprise you. I did this for the first time perhaps ten
years ago, and here's what I found.

Firstly, the degree of marketing bullshit that you have to push through just
to get to the truth of the APIs is _staggering_.

Secondly, they perform backflips to avoid telling the truth about their
integration options, which is to say that they're all fundamentally insecure
for real time digital goods and services _unless_ you implement IPN plus
polling plus duplicate detection / additional round-trip validation calls, ie.
you can't trust an IPN notification. The degree of complexity (order state
tracking, IPN state tracking and duplicate detection, retry support, running
your own IPN-receiving server) and latency requested here of client businesses
is immense.

Thirdly, their idea of international support is pathetic. It seems that
they've basically duplicated their entire business process to other countries,
translated it, and then assumed that everyone in that country requires only
one human language in all of their interactions: documentation, support,
emails, etc.

Finally, as is widely known throughout the industry they have a shocking
reputation for the arbitrary suspension and seizure of accounts and assets,
with little to no recourse for those affected.

I am not surprised that Stripe has taken off. Unfortunately, that's still
lipstick on a pig: fundamentally, the settlement, risk, transparency and
government interaction model of credit card networks means they are unsuitable
for an increasingly large volume of business around the world.

~~~
bkirkbri
Agreed. Over the past 9 years I've had to implement double, triple and end-of-
the-month checking on PayPal to be sure that we have some reconciled idea of
reality with respect to payments.

What's worse, their system essentially rewrites history over time. This means
that rather than keeping a ledger of payment events, they change the dates and
amounts depending of payment status / reversal /etc. Querying later in the
week and the data is different. For the same charge.

I ended up keeping an event log of everything we ever get from them (IPN, API
polling, reports) and synthesizing that into a current snapshot at query time.
The code is riddled with comments saying, essentially, "this charge, then
return, then recharge and failure means that the payment is OK".

------
erikcw
Not sure if this is related to this particular case. From previous experience,
PayPal will silently stop delivering IPNs if your webhook URL ever returns a
non-200 status code for an extended period of time.

It took us a fair amount of time to figure out that this was the cause of our
issue -- the PayPal UI didn't indicate any problems and PayPal business
customer support was unable to see the source of the problem either.

~~~
jcoby
IPNs are annoyingly difficult to use.

Some accounts simply will not send them until an IPN URL is set, even for
transactions that include the notify_url in the request. You cannot even see
the IPN history until you enter an IPN URL, even if it was used through the
notify_url param.

IPNs that include UTF8 characters will not validate by default in most PayPal
accounts. PayPal defaults to Win1252, translates the UTF8 content to Win1252,
tells you it's Win1252 and then seems to validate the IPN against the original
UTF8 data. I have never been able to validate a Win1252 IPN that had UTF8
content. I've tried iconv, stripping bits, and plain substitutions (ü -> u).
Nothing seems to work. This seems to be getting better though.

For about 2 months last year they forgot to include a field that had to be
added in order to verify the IPN. And you had to iterate through a few
different values to see which one was correct. So we got a rash of support
tickets asking why our customers were suddenly getting notifications of issues
verifying PayPal purchases and were having to manually approve orders. I
submitted a ticket after a day of dealing with the issue and got a response
about 2 weeks after they fixed the issue. There was no indication of the
problem on their status site.

The IPN verification service randomly returns INVALID for valid data or simply
doesn't work. Sometimes IPNs are delayed.

Dealing with the PayPal APIs are far, far harder than their sales sites would
have you believe.

~~~
yannk
Here is my solution to the encoding issues, let me know if it works for you
(assuming your environment let you access the body of the request without
parsing it, or already knows how to parse it correctly).

[http://blog.cyberion.net/2014/08/parsing-paypal-ipn-
requests...](http://blog.cyberion.net/2014/08/parsing-paypal-ipn-requests-
with-flaskwerkzeug.html)

~~~
jcoby
Thanks, but that's a wholly separate issue from what I was talking about.

When you have UTF8 data in the IPN, PayPal translates it to win1252 in the
IPN. But it verifies the IPN against the original UTF8 data. The only way to
fix it is to go into the profile and change the IPN format to use UTF8. There
is no programmatic way to set the encoding for IPNs to be sent in and there
are no workarounds that I am aware of.

If you never see international names or addresses with accents and diacritics
you won't run into the problem.

~~~
yannk
That's not my experience, validating the data has always worked by just
spitting it back out to them without any alterations to the data.

------
driverdan
Here's an idea, stop using shitty services like PayPal. Today it's IPNs
failing, tomorrow they're locking your account and holding your money for 180+
days with no justification.

~~~
ejr
As much as people complain about them, I still haven't found a service that
matches their reach or ease. Let's face it, PP is a de-facto bank without any
of the regulations, requirements or safeguards.

~~~
BorisMelnik
ease of use is still kind of key for me. I pay a lot of people in Europe /
Asia and if it weren't for paypal I'd literally have to make a trip to my
supermarket Western Union twice a week.

~~~
ejr
Yes, exactly. When I tell people where they need to send funds, their first
question is "do you have a PayPal account?" Western Union, Moneygram and such
are all highly inconvenient for both parties.

And services like Stripe aren't an option for me due to their restrictions on
use as well
[https://stripe.com/us/prohibited_businesses](https://stripe.com/us/prohibited_businesses)

My work covers multiple items on that list. Compare that to PayPal's
[https://www.paypal.com/webapps/mpp/ua/useragreement-
full#9](https://www.paypal.com/webapps/mpp/ua/useragreement-full#9)

Edit: I should mention, of Strip's list of "prohibited businesses", nothing I
do is illegal or unethical. Just in case I gave that impression.

~~~
bmelton
Hell, I had no idea that Stripe's prohibited businesses list was so ...
massive.

The list runs the gamut from "risky" to "value judgement", and I can't help
but wonder how all those items ended up on the same list.

~~~
eli
Some are clearly legal liabilities. The rest almost certainly have high
fraud/chargeback rates. Can you imagine how many people must dispute what they
paid to a psychic

~~~
mietek
What about "48\. Telecommunications equipment and telephone sales"?

~~~
eli
Yes. A quick google reveals:

> _Telecom – The telecommunications industry is much more proficient with
> technology than it used to be. However, once services have been provided,
> recipients may dispute them. Card-not-present transactions are a mainstay
> for telecom companies._ [http://www.aliantpayments.com/who-we-service/high-
> risk-proce...](http://www.aliantpayments.com/who-we-service/high-risk-
> processing-merchants/)

I would imagine that things like VoIP service are targeted by criminals who
constantly need new, anonymous phone numbers for other scams.

Stripe's real business is measuring and assessing risk. The moving dollars
around is the easy part.

~~~
mietek
I was thinking racks of Alcatel-Lucent and stacks of Sagem, but perhaps you
have a point.

------
jay-saint
We have been experiencing sudden weirdness with out PayPal Pro payment
processing integration with our Magento cart over this time period. I have
spent hours looking over logs and on support call with PayPal integration
support. You would think that since PayPal and Magento are both part of eBays
X.commerce that they would work together well.

I received a lot of "hmmm that's weird" and "well i've never seen the API do
that" comments from the integration support specialist at PayPal."

What is happening for us is that Paypal is simultaneously sending a
authorization and then a second later a capture. the results are weird some
transactions process and other get a duplicate invoice it error.

------
myddryn
This has always been a problem with Adaptive Payments, just not always so
severe. Paypal's Express Checkout product has the best solution to this in
that a payment cannot be completed without an explicit API call by your
application. This makes the payment confirmation IPN redundant.

It is a more complex integration, but our product has been much more stable
since we made the transition to Express Checkout.

This does not however prevent other problems with IPN delivery on refunds and
other post order activity. Paypal's IPN infrastructure is pretty weak in
general, very poorly documented and completely untestable.

~~~
lucaspiller
The issue here is that Express Checkout isn't available in all countries. When
I was integrating PayPal for the startup I was working at last year we had to
use Adaptive Payments because of this. I don't mean some small South American
county nobody has heard of, the business was based in Ireland.

As other commenters pointed out we didn't have much choice of providers
because of our business model (hotel booking). Stripe and Braintree didn't
want anything to do with us, so we went with PayPal to start with. I think
they are now using RBS WorldPay, but even that had it's issues...

------
emilepetrone
This is a mission critical bug for any other startups that depend upon the
Adaptive Payment API. That may only be a subset of Paypal merchants, but for
those that do - you need to know this ASAP.

------
csomar
I don't remember accurately the documentation URL but there was a mention (I
did lots of PayPal API development) that IPN cannot be always trusted and you
should instead query the PayPal API (in a cron job fashion) to check if the
transaction went through.

There are many reasons why an IPN will fail. Like having your firewall
settings messed up, network issues on your or PayPal's side, PayPal actually
sending an IPN but your infrastructure not making use of its information...

------
WatchDog
If your business process relies completely on a callback that can fail for
plenty of different reasons, then your business process is broken.

~~~
LocalPCGuy
Except that part of PayPal's business model is saying that you can depend on
that callback. This isn't on the business here, it's on PayPal.

Now, I'd agree if you said that PayPal isn't suitable for a business to use,
it should be relegated to the realm of individuals and small businesses who
don't know better, but that's different than what you seem to be saying.

~~~
csomar
Can you link to the part of PayPal's that mention you can depend on IPN?

------
pbreit
I'd be very surprised if IPN was down for several days, much less 9. Can
anyone confirm such a thing?

IPN hasn't gotten much love in years but still pretty cool that it was (one
of) the original webhooks back in 2001 and has worked decently for the past 13
years. In fact, an integration from 2001 should continue to work OK.

~~~
jtheory
It's certainly not _generally_ down for everyone. I've not seen any oddities
in IPN for a few years, and can confirm notifications have gone through over
the past few days.

It's important to be careful about jumping to drastic conclusions when
debugging a connection like this.

E.g., from the OP:

> I called Paypal Technical Support to verify exactly what we were seeing.
> Sure enough, they had received other calls today about this exact issue.
> Other vendors were not receiving IPNs for their orders. The ticket for this
> "critical bug" was created today.

> The problem with that fact is that our customer's order was from 9 days ago.
> For at least 9 days (we are going through all of our partial orders now to
> see the full extent of this bug), Paypal was not sending IPNs to merchants
> and did not know.

This doesn't actually follow logically. The tech support rep mentioned getting
other calls about IPNs not going through does _not_ mean NO other IPNs were
going through. Likewise, if Tindie has actually not gotten any IPNs for 9
days, that doesn't indicate anything about the other millions of PayPal
merchants.

This reminds me a bit of customers who call in a panic that "your entire
website is broken!", and after a bit of discussion it becomes clear that their
internet connection is down.

The same stream of events could be caused by lots of things, including minor
configuration changes on Tindie's end.

~~~
r1ch
IPNs were unaffected for us. This probably only affected a small segment of
customers.

------
d_runs_far
I organize a bunch of trail running events, our registration page has both a
stripe and paypal option; 85% use stripe, but the other 15% I don't want to
give up on. Oh, and no IPN glitches here in the last 2 weeks (or ever actually
- knock on wood).

------
abritishguy
a) Stop using paypal (or at least make it an option but recommend stripe or
similar)

b) IPN is clearly going to be flakey, there are several reasons why IPNs might
not be delivered and they are not all paypal issues. In the paypal docs it
says that if your webhook doesn't return 200 to some IPNs then it will stop
sending them. For my new stuff I don't use paypal at all (stripe is just too
good) but when I did I had a cron job running every 5 minutes that checked
every transaction that had been created but no IPN had come through for. Over
the cause of a couple of years I caught several transactions where the IPN had
been lost.

------
kawsper
What is up with that blog title?

> Disbursements through US Bank Accounts or Debit Cards

Is it only related to US bank accounts and debit cards? Or is it an IPN issue
affecting all card?

~~~
emilepetrone
Title is Tindie specific... I wrote it for our audience but the news I thought
HNers would want to know

------
Xorlev
This explains a lot... my IPN handler had been awfully quiet, but thankfully
it was only used for automatic thanks.

------
mgkimsal
had this problem for years, along with google checkout's callback process.
google was worse. they seemed to get to a point where if/when they'd ping, if
the request took longer than a few seconds to reply, it would stop, then never
retry.

~~~
jcoby
Google Checkout was awful to work with. It was hard for the vendor to setup
and use. It was hard to program against. It was hard for the customer to use.
It was hard to test. And of course it had Google's famous lack of support.

Speaking of support, we qualified for the integration reward ($1500 or so;
maybe more). We tried to claim the reward for over a year. Every time they'd
say "yeah you totally qualify!" and we'd ask how to claim the reward and then
silence. Of course every email response took 3-7 days. A few months later we'd
go through the same process. Still waiting on that check to arrive.

We supported it for about a year and eventually gave up. Payment notifications
could take up to a minute to arrive. You couldn't specify the tax (or shipping
from what I remember) for an order in any sane way. For a while it was over
50% of our support tickets while being used by less than 10% of our customers
and less than 1% of all transactions.

I was so happy to delete it from our codebase. And we won't support any
payment processor they have to offer anytime soon.

~~~
mgkimsal
For me it was.... either instant or 15 minutes. The 15 minutes was 'security
check' stuff, apparently. But I was selling digital downloads. I'd just have
to approve them instantly - no one expected they'd have to wait 15 minutes to
get approved, even when I tried explaining it. So... I had some bad charges
come back, but the download was already done - no way to protect myself. GC
was pretty bad.

------
emilepetrone
BTW this post was #7 on HN, and then the HN gods flagged it for banishment.
Not sure why...

~~~
dang
It set off the ring detector. It looks like a false positive to me, so we'll
turn that off.

We also changed the title (from "If you use Paypal, you should know they
aren't sending IPNs") to make it more neutral.

~~~
emilepetrone
Ring detector?

~~~
gknoy
A voting ring is when you submit something, and then tell a few dozen of your
"friends" to vote for your submissions (and you agree to do the same for
them). HN has heuristics which detect this sort of behavior, and penalizes the
ranking of an article that looks like it's being voted up by a voting ring.

(Disclaimer: I haven't read the HN code, and have no idea how they actually
detect such things.)

------
kevin_thibedeau
Not to worry. Just the NSA deploying a MITM attack. The bugs will be ironed
out soon and your transactions will proceed smoothly as before.

