
Reversing Safeway's private APIs to automate coupon collection - jonluca
https://blog.jonlu.ca/posts/safeway
======
Someone1234
I enjoyed the article and would recommend it.

Aside: At the end of the article they have a "Bonus: Speeding things up"
section where they automate adding 300~ coupons via 300 HTTP connections in 5
seconds (instead of 60~ seconds).

In my _opinion_ if you're going to automate stuff like this, you should do so
with the goal of minimizing disruption (and frankly, detection). They run the
script automatically at midnight in the background via cron, so why was going
slower problematic? 300 requests in a span of 5 seconds seems much more likely
to trigger an IDS[0], get flagged as unusual in the logs, or similar than 300
over 60 seconds. Particularly at midnight.

I'd be trying to look at human as possible and not set off automated security
systems. Heck you could add a randomized delay (e.g. 1~2 seconds) between
requests and it would still be completed inside of 10 minutes. Plus then
nobody can reasonably accuse you of trying to "DoS" them/violate the CFAA.

[0]
[https://en.wikipedia.org/wiki/Intrusion_detection_system](https://en.wikipedia.org/wiki/Intrusion_detection_system)

~~~
jonluca
I agree - I actually built the thread pool stuff for work while I was testing,
since I had the code from a previous project.

If I was more worried about Safeway catching on I'd probably do something as
you suggested (at the very least I'd add user agent headers and the other
cookies expected from a real session, as right now it's trivial to detect my
requests).

Sneaker bots do this exceedingly well - it's a constant cat and mouse game to
make the requests look as human as possible, very interesting space right now.

~~~
penagwin
I highly recommend not making more then say 5req/s - and preferably do it at
like 2am.

These projects are fun IMO, but it's best to not hammer anyone's servers if it
isn't necessary.

~~~
Scoundreller
But at 2AM your traffic will create a more noticeable spike...

It doesn’t blend in as well...

~~~
penagwin
In my experience, unless they have excellent monitoring and a team looking to
ban you they won't notice. It's half courtesy, and the other half is because
it's not likely to affect their business operations (by slamming the servers
when they're busy) they're less likely to notice or care.

------
withinrafael
I did something similar with Safeway's API back in the day. But rather than
chew up their API with unsupported usage--CFAA case territory--I now just log
into Safeway's website and issue a one-liner on the coupon page:

$(".grid-coupon-clip-button button").click();

~~~
mlrtime
what does this do exactly?

~~~
guessmyname
> _$( ".grid-coupon-clip-button button").click(); _

> _what does this do exactly?_

The command allows you to click all buttons in the page at once:

• The dollar sign is an alias for jQuery [1];

• The text between double quotes is a CSS selector;

• Select every DOM element with a _“grid-coupon-clip-button”_ and _“button”_
CSS class;

• The thing at the end is a JavaScript function call which triggers an
“onclick” event [2];

[1] [https://jquery.com/](https://jquery.com/)

[2] [https://api.jquery.com/click/](https://api.jquery.com/click/)

~~~
altec3
This is a great little write up, but the third bullet point should be:

\- Select every button with a parent element that has the css class “grid-
coupon-clip-button”

------
cjslep
This reminds me of a very similar case a decade ago I never figured out.

Anyone remember Coupon Guy from 4chan in the 2009/2010 timeframe? You could
make your own "Buy 1 Get 1 Free" or variant (for both N, "buy 1 get 10", or
for kind of coupon, like "buy 1 get half off") Tv, Xbox 360, near anything
etc. For Walmart, Best Buy, other chains etc. The tools were passed around
stegonographically in the instruction images. Since coupons weren't properly
accounted for until they hit a place in Texas, whole threads full of Anons
over the course of weeks fabricating working coupons.

Until they stopped working, and of course rumors of "the FBI" apparently
grabbing the guy.

I never did figure out what happened tech wise under the hood there.

~~~
icelancer
>> The tools were passed around stegonographically in the instruction images

Whoa, really? I remember the coupons flying around 4chan of course. Had no
idea the tools were in the images.

~~~
yuubi
It's easy to make a file that both a valid image and a valid zip file, because
zip readers start from the end-of-central-directory at EOF and most other
things start from a header at the start of the file. Many such images included
a winrar icon.

There might have also been some steganographic images, but I know for sure I
saw several of the image+archive kind around then.

------
atonse
This is the kind of stuff where I tell people that coding can be a real
practical skill (that gives you an unfair advantage these days).

I did something similar (read-only) for Home Depot Truck Rentals. To check if
the truck was available at my local store, each time, you had to put in your
zip code, and click a couple of times. Once I found that was an API, I rebuilt
the call in Postman and kept hitting that endpoint until a truck was
available.

That way I could check really fast.

The twist: None of it mattered because their data itself didn't update
accurately (I saw one in the parking lot and they had one available and never
updated their site). :)

~~~
rainyMammoth
Exactly this.

I did the same thing to find available camping spots in Hawaii because they
are super difficult to get. Wrote a script that would query their "API" every
5 minutes and alert me if a spot became available anywhere.

~~~
nyquist
Have you considered whether doing this might be...wrong?

Presumably those campsites are permitted by some government agency (NPS, BLM,
the state of Hawaii, etc.), and presumably that agency designed permitting
system with the assumption that _people_ with _limited time and attention_
would be vying for the permits by having to visit the site themselves to get
one.

This encodes a particular definition of fairness: that those who register
early, or are very motivated, or simply those with a lot of free time to
refresh the site, will get permits.

I can also whip up a quick script to replace refreshing an unprotected HTTP
API with a notification email. Does that make me more deserving of the camping
spot?

~~~
kelnos
> _presumably that agency designed permitting system with the assumption that
> people with limited time and attention would be vying for the permits by
> having to visit the site themselves to get one._

That's quite a very large assumption that I don't think we can accept as fact.

> _that those who register early, or are very motivated, or simply those with
> a lot of free time to refresh the site, will get permits._

Perhaps someone who writes a script would count as "very motivated"?

> _Does that make me more deserving of the camping spot?_

"Deserving" has nothing to do with this in the first place. The only way that
works is if you define "those who register early", "motivated", "a lot of free
time" as "deserving". I could maybe see an argument for the first two classes
of people as being "deserving", but I don't think you can justify "I have a
lot of free time" as a reason for deserving anything, really.

Doing first-come, first-served based on availability and the random
possibility of a cancellation isn't ever going to be a "fair" system. This
sort of system is put into place because it requires very little coordination
and work on the part of the agency that maintains the reservations. Holding
that up as some sort of standard for fairness, and suggesting that anyone who
thinks outside the process is _wrong_... is a little much.

~~~
nyquist
I'll bite.

I think you're right the agency probably didn't sit down, write down
definition of fairness, then design a permitting system around it. They
probably implemented the cheapest/easiest digital analog they could find to a
traditional fax-in/walk-up first-come/first-served permitting system.

However, the intentionality of the implementers was not central to my
argument.

The system was created by (probably) non-technical people under a certain set
of assumptions: namely, that this digital first-come first-served system would
function approximately like the old paper one, but with fewer dead trees and
toil. The old one was rate-limited by having to call an office and probably
talk to a human, and the assumption that if you call every 5 minutes that
human will probably get annoyed with you and stop answering your calls. The
new one is rate-limited by the assumption that most campers simply can't spend
all day refreshing a website.

The traditional first-come first-served system isn't intrinsically "ethical"
or "fair" for some classes of people (as you've astutely pointed out), at best
it's a crude approximation of some version of fairness. While crude, it was
established by a democratically-elected government tasked with allocating a
shared resource. "People who can automate HTTP API calls" and the nearby
"people who can hire people to automate HTTP API calls" (as has actually
happened with some outdoor permits) were almost certainly _not_ in the groups
of people the government was seeking to advantage by choosing this system, and
I think most engineers are smart enough to be able to intuit that.

So the root of my comment was this: GP is using special knowledge they have
(and probably worked hard for) to extract more of a public good than the
public really intended to have access to.

* Is that fair to everyone who doesn't have GP's knowledge? Do people like the GP deserve more camping spots than others? This is a public resource, not sneakers, so fairness is important. * If everyone with programming knowledge acted the way GP acts, would that maximize the public good? * If everyone with programming knowledge acted the way GP acts, would the system even function at all?

My answers are basically: * No. Everyone who wants deserves an equal chance at
the spots. If there's more demand than spots, it's the government's job to
decide. Random programmers on the internet intentionally subverting the
government's intentions is _wrong regardless fairness (or lack thereof) of the
original system)._ It would nearly-minimize the public good. Only programmers
and people who can hire them would get popular camping spots. This is a real
problem is popular outdoors areas around tech hubs. There's a reason NPS will
only accept old-fashioned faxes provably _not_ sent from a free online relay
for the most popular Sierras routes in CA (e.g. the JMT and much of Yosemite),
and it's not because they want to rock like it's the 80's or because the
government is backwards. It's because assholes tried to spam the process with
automation. I think some (like Half Dome) were migrated to a new lottery
system on outdoor.gov this year. * The system would completely collapse, and
most smart programmers could predict that. The government would have to spend
more money on servers just to serve bots pinging the registration system
constantly, or it would crash. Even if they did that, the people who gots
slots would be a vanishingly small subset of the population (programmers) or
people who can hire them. Likely, a grey market for "scalped" permits would
arise.

~~~
rlucas
If you want to see a world governed by a non-human understanding of fairness
and one in which a highly technical cohort was entrusted to make these sorts
of judgment calls, look no further than the domain industry in the early
2000s.

Specifically, consider Snapnames -- a company born out of the notion that
snapping up a domain name coming up for renewal was something that was
legitimately fairly awarded to the automated process that was fastest.

------
bredren
Another tip: (area code) 867-5309 works for Safeway club card discounts in
most area codes while piping tracking of your purchases to essentially null.

Yes, Jenny’s number.

~~~
lozaning
At risk of ya'll taking "my" safeway gas rewards, this is almost always good
for whatever the maximum discount off per gallon is at any safeway that also
has a gas station.

~~~
taurath
Its amazing to me the degree to which people will go to save 3% or less on gas
once a month. If you fill up 4 times a month you're saving about $4, and all
it costs is a profile of everything you buy attached to your phone number.

~~~
organsnyder
Given that they're probably already tracking everything via my credit card
number (whether I sign up for anything or not), I don't feel like much privacy
is being given up.

Though as a rule I don't bother with most of these programs, mainly due to the
inconvenience factor.

I've lost hope of maintaining my privacy through any actions I can take
individually. A legislative solution is required.

~~~
billh
Correct me if I'm wrong, but if you're using a credit card the bank is only
getting a receipt that shows the sum total of the transaction and not an
itemized receipt. The club cards are consuming and tracking the data of the
entire transaction.

~~~
rovr138
The club card acts as unique id to track you. The credit swiped at the machine
has a unique id that they can use to track you.

The club card works across multiple credit cards and cash. That’s the main
difference. And you might possibly track 2 people who control the household
(wife gets one, husband gets one)

~~~
fingerlocks
Using Apple Pay would circumvent the CC tracking because it only transmits a
one-use number.

~~~
jacobwil
Unfortunately (from some perspectives), this isn’t true. Each instance of a
card in Apple Pay has a long term Device Account Number (DAN) that functions
the same as a physical card’s number (though there is no link between the DAN
and actual card number from the merchant’s perspective). The same DAN is used
across every transaction. The only way to get a new one (which is pretty easy,
relatively speaking) is to unload the card from Apple Pay then re-enroll.

~~~
fingerlocks
Was not aware, thanks for the clarification.

------
throwaway_bad
Is there a private API for a particular store's prices?

It seems like coupled with couponing, you can build a decent price tracker
that can tell you if you're actually getting a good deal. (like
[https://camelcamelcamel.com/](https://camelcamelcamel.com/) for amazon,
[https://steamdb.info/sales/](https://steamdb.info/sales/) for steam games)

Otherwise I've noticed that many(though not all) coupons are for items which
recently had their base price increased to make the coupon seem like a better
deal than you're actually saving.

------
dfxm12
When I see access to private API's like this, I wonder how judges would
interpret these actions as they relate to the CFAA. By accessing "private"
API's like this, are you knowingly accessing a computer without authorization?
Are you exceeding your authorized access?

The fact that Aaron's Law never went through has disturbed me...

~~~
dewey
I know you are just playing devil's advocate but why "without authorization"?
You are using your account and your cookies so it should be fair game?

~~~
maerF0x0
+1 you're only doing things you're already authorized to do, just with a
different client.

~~~
LocalH
Many sites view that as "unauthorized" because they like that top-down control

~~~
taftster
It's not any individual site's view that matters to me. It's the view of the
courts that's the most concerning/pressing.

------
huangc10
On a slightly unrelated note, I was trying to figure out why this guy is using
a .ca domain when I realized, his name is Jon Luca. At first I thought his
name was Jon Lu (and therefore asian). Good use of domain name.

~~~
Mister_Snuggles
I'm in Canada and shop at Sobeys, which owns the Safeway brand up here,
thought there might be some relevance to me thanks to the .ca domain name.
Unfortunately, there was no relevance but it was still an interesting read.

On a side note, Safeway and Sobeys in Canada don't have a loyalty program,
instead they piggy back off of Air Miles. All of the special offers available
just amount to bonus Air Miles, so they're not actually that worthwhile (IMO).

~~~
huangc10
What a blast from the past. I am actually Canadian but have been living in the
Bay Area for over 10 years so I've already forgotten that you can collect Air
Miles through Safeway (which was why I clicked on this article in the first
place thinking of the .ca)

I don't think Air Miles are completely useless. I do remember redeeming air
miles at least once in the past for a one-way trip somewhere...

~~~
Mister_Snuggles
They're not completely useless, but I do find that I don't seem to get much
benefit from them.

For example, I needed to rent two cars on a trip last year. The first car I
was able to rent through Air Miles, but I had to pay the taxes and any fees
beyond the base price myself so it didn't feel like much of a deal. The second
car required more Air Miles than I had, so I had to pay for the whole thing
myself, I couldn't do part on Air Miles and part cash.

I ended up renting the second car through Costco and felt like I got a better
deal overall than I did with the first car.

Maybe rental cars aren't the best way to use them.

------
chipperyman573
Haha, I did something similar a while ago - I just invoked a click() on every
"add coupon" button on their website, I didn't want to reverse engineer their
APIs - way easier but also it requires a desktop browser with the page open.

I'm not sure if using this API is any different but a few months ago Safeway
made a change that only lets you have 20(?) coupons at any time. After you add
more it kicks off your oldest one. Which sounds like plenty but if you're
adding every single coupon you're gonna get a ton that you have no desire for
($1 off diapers when you don't have a kid etc).

It worked for a while, though!

~~~
tenaciousDaniel
You can use Selenium to do the same thing in a headless browser, no need for a
desktop browser with a page open.

[https://www.seleniumhq.org/](https://www.seleniumhq.org/)

~~~
bdcravens
In which case you want to probably use their headless Docker containers.
[https://github.com/SeleniumHQ/docker-
selenium](https://github.com/SeleniumHQ/docker-selenium)

You can also use Puppeteer for this purpose (using headless Chrome)
[https://github.com/GoogleChrome/puppeteer](https://github.com/GoogleChrome/puppeteer)

------
ipsum2
Curious, why implement your own threadpool? Python has two built in already:
concurrent.futures.ThreadPoolExecutor and multiprocessing.pool.threadpool.

------
cbhl
For a while I actually tried doing something like this -- the coupon page
included jQuery, so a quick one-liner self-XSS clicking the right elements
would iterate through all the pagination, and then you could call the click
handler on all the Add buttons (which share a CSS class for styling).

The problem came at checkout -- whenever I typed in my phone number (to apply
the coupons from the loyalty account), the point-of-sale system would hang for
tens of seconds while loading and then trying to apply all those coupons.

------
jedberg
This is cool but I'm not sure the purpose. I use the Safeway app every time I
go to Safeway. I scan all the offers and just add the ones I want, which
builds a shopping list for me.

This gives me a list of things to buy but more importantly _I know what 's on
sale_. If I just added all the coupons

I'd still have to scan the list of things to find what I want to buy, but then
I'd have to track them elsewhere, because the built in list would be useless.

Safeway actually made a decent app that helps me shop faster. This feels like
it would ruin that.

~~~
floatingatoll
Coupons have two purposes, which seem contrary, but are essential to
understand how coupons are a form of advertising:

1) Saves you money (versus the retail price of the item)

2) Makes you more likely to buy the product on the coupon.

Your use of coupons adheres to both value #1 (saves you money, because you
know what's on sale) and value #2 (advertises products on sale, increasing
your chances of buying them). This is the core advertising model of coupons
and why they've been popular forever.

This automated use of coupons saves you money _if_ you happen to purchase an
item for which a coupon exists, _without_ requiring you to do any extra work
to save that money — but also _without_ having the desired advertising outcome
of making you more likely to purchase that item.

This compares well to adblocking. Internet advertising on websites:

1) Saves you money (versus subscribers-only paywalls)

2) Makes you more likely to buy the product on the advertisement

And in that analogy, ad-blockers are directly equivalent to coupon auto-
adders: they allow you to save money _without_ having the desired advertising
outcome of making you more likely to purchase that item, in a fully-automated
manner that doesn't require you to exert any effort doing so.

Coupons are a precursor form of advertising ("pay-per-clip" :) where you are
paid money to view the advertisement, and are more likely to commit to buying
the product when you 'clip' the coupon — it's a marketing psych thing. They
also have perfect tracking, since retailers provide coupons to manufacturers
along with purchase date and location.

Coupon autoclippers break that agreement, such that you're 'paid' for being
influenced by the advertisement without ever having been influenced. The
coupons are no longer valid for tracking the effectiveness of advertising A/B
tests in different markets (your Safeway account's zip code is surely part of
that data). They are no longer proof that you viewed an ad at all.

You're not wrong that this app would ruin how you shop quickly, but you're
also shopping quickly _using_ a list of products that were predetermined by
Safeway and/or other marketing divisions to be of maximal interest to _them_
for you to purchase. As long as you're okay with that, coupon clipping is an
excellent approach. For others, autoclippers would minimize the price paid
_without_ changing their purchasing methods (which may be paper-based, brand-
focused, or random-chaotic)

~~~
ergothus
> ("pay-per-clip" :)

You have a well-written and accurate comment, but I'm overwhelmingly
distracted by the combination of the adorable "pay-per-clip" and the historied
debate about how to handle an emoticon smiley at the end of a parenthetical.
(conclusion: you're doing it wrong and are a terrible person!....but, "pay-
per-clip"...tee-hee)

~~~
floatingatoll
Thank you, I do my best :).

------
davecardwell
I had written a similar program for US supermarket Publix, however I used it
as an exercise to learn puppeteer[0] rather than using their private API
directly.

Reading this inspired me to finally release it here:
[https://github.com/davecardwell/publix-coupon-
clipper](https://github.com/davecardwell/publix-coupon-clipper)

[0]
[https://github.com/GoogleChrome/puppeteer](https://github.com/GoogleChrome/puppeteer)

------
brownbat
> This brought what used to be a painstaking process of clipping coupons from
> magazines to a slightly less painful process of having to click “Add” on all
> the coupons every week after logging onto their site.

This is an insane trend in supermarket usability. You want to offer low
prices, but only to people who go through a practice round of online shopping
before doing it in person?

Why are we doing this to ourselves?

~~~
moftz
They wouldn't make as much money if everyone clipped the coupon. However, most
of the time, the coupons aren't really special. The loyalty club member price
is already shown on the shelf anyway. Most "coupons" are more of an
advertisement for the old folks who still do that. The incentive for the
actually good coupons is to get you to come into the store and buy other stuff
alongside the on-sale item. If I only have a few key items on my shopping
list, I might be more inclined to buy more if the coupon app shows buy one get
one half off for something. Or I might buy a different brand than I normally
would because there is a coupon only on the app.

~~~
brownbat
> They wouldn't make as much money if everyone clipped the coupon.

I get on a basic level it's a form of price discrimination, it just seems
unfathomable that this bizarre skeuomorph, totally dependent on a series of
random historical developments, would actually happen to line up with the way
to extract the most return from your customers and/or give them the best
experience.

But hey, I found the perpetual fake "50% off" sales at JC Penney's crazy
making, and when they reversed that for more honest pricing they almost went
out of business. I know I'm just shaking my fist at irrationality for no
reason.

It still bothers me though.

------
crooked-v
This seems like prime territory for a Mac menu bar / Windows system tray app
that runs this process once a day.

~~~
markovbot
Why would it need a visible UI component? Just put it in a cron job

~~~
crooked-v
Ordinary people don't use cron jobs.

~~~
barbellguy97
This is not made for ordinary people I would say

~~~
crooked-v
Ordinary people would absolutely use an app that lives in the menu bar/system
tray, has a Safeway login page and a "run on startup" option, and once a day
does magic stuff in the background that adds all the available online coupons.

~~~
SamuelAdams
Right but now Safeway will update something that will cause this to break, and
now people who use this super handy app will expect the maintainer to update
it - even if it was just a one-off project that the original developer had no
interest in maintaining.

So now people will complain and your inbox is flooded with "please fix this"
because people feel entitled when all you really wanted to do was just try
some cool thing.

------
tracker1
Nice article... I'd have probably just used node+puppeteer for this.

------
orf
Great article. Not sure why the author re-implements a thread pool when there
is not one, but two in the standard library (futures and
multiprocessing.pool.ThreadPool).

------
sleighboy
It needs some tweaking to work now, but here is what I was using (in Python):

[https://gist.github.com/danielatdattrixdotcom/bd6a05e3d8c499...](https://gist.github.com/danielatdattrixdotcom/bd6a05e3d8c4994f6994d71527397c5d)

Executed via cron every day at a time before I would make a potential grocery
run meant the coupons were already waiting on my account when I went to the
store.

------
greyfox
Anyone know if the dev lurks here? I'm wondering if there's a way to fork his
code or add Kroger functionality to it. If im not mistaken Kroger is a safeway
acquisition that is a popular grocer in the south/southwest of the USA. this
would be very helpful for those of us without safeways.

~~~
roland00
Kroger is the second largest supermarket in the US (after Walmart) that has
many subsidiaries grocery chains that they have acquired in its 100+ years of
existence. Pretty much Kroger bought up other "family" grocery chains and then
keeps those chains local names such as Dillons, Frys, King Soopers, City
Market, Gerbes, Kwik Mart, etc.

Safeway by contrast is part of the Albertson and Albertsons and its
Subsidiaries is the 3rd largest Supermarket chain.

------
mgiampapa
Next time you are at the store look at the circular. There is a barcode you
can scan with their app that adds everything to you app that week.

~~~
natch
That doesn’t fix the information overload which the OP’s solution solves.

------
mistrial9
digression - if anyone has evidence that Safeway Inc gave back to the postgres
project, it would be great to hear about it (in public)!

btw- how many people here have contributed directly to PostgreSQL in any way..
"food for thought"

------
samstave
Can I just subscribe to the coupons? Please. I give you my email, and using
this - email the coupons to me in a pdf that I can print out? (Or is this a
stupid request)?

