
Amazon AppFlow - beninada
https://aws.amazon.com/appflow/
======
zmmmmm
I really wonder about all these services that eat into the layers of
implementation that organisations usually do in house, but at the same time
quick form a complex web of lock in that prevents you ever leaving Amazon. It
seems to me that these services are very seductive but there are very few
cases where it's truly in an organisation's long term interest to adopt them.
At best, use them as a quick interim solution to bootstrap what you are
building and then use that as your requirements for an open source or at least
vendor-neutral equivalent.

Curious what the HN crowd thinks about this issue?

~~~
dataminded
Why do you worry about vendor lock-in?

If you get to a point where you are locked-in it's because you spent years
building into a solution that was your most expedient and renumerative option.
At some future point in time, a future you or the person who replaces you may
want to reconsider a re-platform. Let them worry about that.

I worry about development working on glue code/solve problems to the detriment
of their businesses actual needs.

~~~
hobofan
Vendor lock-in is just another form of technical debt and should be treated as
such.

Sure right now it's your easiest option, but in the future the vendor might be
the only one offering a compatible version of CoolNewFeatureXY at prices that
make your business uncompetitive to the other businesses that have the freedom
to choose between vendors.

~~~
throwaway894345
Why do you think it’s technical debt? Because you might have to pivot away
because it might not support your requirements in the future? Then just pay
that cost in the future if it ever comes. Building everything in-house seems
like premature optimization—you’re obsessing over a case that may never come,
and you’re very likely not going to build these web service components as well
as amazon for the cost because Amazon has many customers paying for these
services such that each customer pays only a tiny fraction of the operating
cost—if you roll your own, you have to pay the whole cost (roughly) for every
web service you build and no one is paying your company to build web services,
so you have to build that cost into the price of your product/service, so your
product/service becomes dramatically more expensive (because it has to cover
the cost to develop and maintain many full web service component
implementations, which are each many times more expensive than the
corresponding AWS service). If in the future AWS starts charging each customer
the full operating cost for a given web service (which is very unlikely to
happen given their business model) then you can spend time pivoting to your
own solution, content in the knowledge that you saved many years of operating
costs because you didn’t have to maintain your own service.

~~~
hobofan
I am not sure why you (and every other person that replied) assumes that I am
suggesting an in-house solution when I explicitly wrote "freedom to choose
between vendors".

> Because you might have to pivot away because it might not support your
> requirements in the future? Then just pay that cost in the future if it ever
> comes.

That's pretty much the textbook definition of debt.

~~~
throwaway894345
> I am not sure why you (and every other person that replied) assumes that I
> am suggesting an in-house solution when I explicitly wrote "freedom to
> choose between vendors".

I used "build in house" as a simple example of a broader principle--if there
is a "freedom to choose" option that magically lets you migrate from one
platform to another at zero cost while imposing no overhead over the AWS
solution, then absolutely you should do it. But generally these solutions
require a significant up front investment and rarely actually deliver on their
promise of abstracting away the underlying platform (now you're wed to
$ABSTRACTION on AWS, for example), but that's another story.

> That's pretty much the textbook definition of debt.

Nonsense. That's not debt, it's deciding to wait to pay for something in cash
until you know you want to buy it. It's not debt to wait to spend $30K (cash)
on a car until you're sure you want to buy it. The "tech debt" analogy is
intended to convey interest--you know you want to change course but you do the
expedient thing now knowing that you're going to continually bump into it
(each bump is "interest") until you can pay it off properly. This is the
opposite--if you build it in house or use a dodgy "platform-agnostic"
abstraction, you're going to be paying interest on something you will very
likely never even need!

------
degenerate
The full list of SaaS integrations:
[https://aws.amazon.com/appflow/integrations/](https://aws.amazon.com/appflow/integrations/)

Amplitude, DataDog, Dynatrace, Google Analytics, Infor Nexus, Marketo,
Salesforce, ServiceNow, Singular, Slack, Snowflake, Trend Micro, Veeva,
Zendesk.

I feel like a notably missing product from this list is Jira. As horrible as
it is, lots of companies use Jira for tasking so it should have been supported
out the gate.

~~~
hanniabu
What are some good Jira alternatives? We use Trello, but it's limited and
plugins are needed for everything.

~~~
masonhensley
Answer really depends on the size & culture of the organization.

Clubhouse.io, Asana & PivotalTracker are nice, but a little opinionated (can
be a good or bad thing depending on who you are :) ).

I think where Jira really rubs people the wrong way is it's speed,
particularly after someone customized it with a byzantine workflow or with
dozens of views/reports and no primary source of truth for the team to use.
Imagine a PM looking at one report and the dev's thinking another is "the
one." A fresh Jira (Cloud verion on corp-name.atlassian.net) with out of the
box Kanban board should work for most.

There are some industry vertical plays out there that focus on Agency or
Freelancing work.

~~~
jsjohnst
> but a little opinionated

That’s exactly the problem with Jira (and honestly Bugzilla in the same way a
decade ago), it tries to please everyone and thus succeeds with no one. Trying
to do everything typically ends up in bloat/complication/slowness, which seems
to happen disproportionately for bug tracking and cms systems.

------
lilfermat
This is an interesting addition to AWS! I wonder how is this better than
stitch or fivetran? It’s not immediately clear to me. Also I am doing the back
the envelope math. It seems like app flow cost more than our current usage of
stitch or even fivetran to our redshift warehouse and doesn’t even include
some of our bigger other sources we use (MySQL, stripe). I am gonna do more
analysis on this and see if this can be a potential cost saver for us in the
longer. It’s exciting to see more competition in this space none the less.

~~~
dataminded
The days/weeks/months/years it takes to get IT security in a highly regulated
industry to let stitch/fivetran have access to my data.

This is one of a handful of very significant deficiencies in AWS's
data/analytics portfolio. They haven't had good tools for getting data into
AWS from the various places that profit centers store it. I hope they very
aggressively pursue extending the number of supported integrations and finish
building the product.

This is a huge step forward IMO.

~~~
SkyPuncher
I'm not seeing any compliance indications of this tool and it's not listed
under BAA (HIPAA) eligible services: [https://aws.amazon.com/compliance/hipaa-
eligible-services-re...](https://aws.amazon.com/compliance/hipaa-eligible-
services-reference/)

Hopefully, that can change.

------
all_usernames
"send logs, metrics, and dashboards from Datadog into an Amazon S3 bucket to
create monthly reports"

This is an odd use case given that Datadog can already store logs in your S3
buckets, and Datadog integrates with AWS Cloudwatch to pull in AWS metrics!

------
jedberg
> Hydrate data lakes

The power of metaphor right there. Although I've never seen that phrase
before, I know exactly what it means.

~~~
scottlamb
Does "hydrate" add some nuance here that "fill" doesn't? If so, I'm missing
it.

~~~
jedberg
An empty lake can be filled with lots of stuff, like trash. But if you hydrate
it then you are specifically filling it with water.

Also it just sounds cooler.

~~~
scottlamb
I specifically meant nuance relating to the product as well as actual lakes.
If there's none, the word choice (and maybe the whole metaphor) is a
distraction.

Also, as I mentioned in another comment, this phrase doesn't work for the
metaphor. Dehydrating and rehydrating a lake isn't a good idea; it'd kill the
ecosystem, and it'd be tremendously expensive. I'd rather stick to dehydrating
and rehydrating something else, like fruit. Here, this looks similar to the
fruit soup my in-laws make:
[https://www.cheaprecipeblog.com/2015/11/norwegian-sweet-
soup...](https://www.cheaprecipeblog.com/2015/11/norwegian-sweet-soup-sot-
suppe/)

------
hans0l074
We are trying to provide a service based on CRM (Salesforce) data to our
customers in China. Salesforce doesn't exist (yet...) in CN, and wondering if
something like this could be useful in moving needed data to AWS China. Then I
realized that Redshift is not your typical OLTP db and not suitable for
regular querying from an application. Curious to hear HNs thoughts.

EDIT: Someone mentioned Heroku Connect in one of the comments. We do use this
to archive certain objects (> 75 million records) and I have thought about
replicating this data to AWS China (Postgres). Could be one option but some of
our data is real time (e.g. customer orders), not sure what the "freshness" of
the data would be.

~~~
unixhero
You should probably post an ASK HN about this.

------
enahs-sf
Really curious how well the salesforce integration works. Heroku has a product
that does this (Heroku Connect), but it doesn't work well with large tables
and you have to pay them an obscene amount of money to even properly test it
(which might have something to do with them being owned by Salesforce).

So many companies with tons of data trapped in Salesforce's expensive walled
garden and no way to get it out. Dumping it to SQL is the first step to
freedom.

~~~
borisjabes
Couldn't agree more. This is obviously a shameless plug but we built a product
(getcensus.com) specifically focused on this problem of getting data to/from
Salesforce & data warehouses (not just Redshift).

Heroku Connect is pretty powerful but not only is it super expensive as you
point out but also forces you into creating an extra layer in the stack by
creating a Heroku Postgres DB. The secret advantage of Connect is that since
it's owned by Salesforce it bypasses Salesforce API quotas but that's a whole
other can of worms :-)

------
nickk314
I'd love to see how this works under the hood.

I've built a few projects that provide real-time reporting across sass
applications but it always seems like an uphill battle. Especially when your
only interface with the sass is a hypertext API.

Can a data source change while you're completing a multi-part query? Are
records hard deleted? If so how do you detect removed records to maintain
deltas? How do you maintain deltas throughout lossy pipelines that group or
filter?

In the end, it always feels like it'd be easier to build one monolith to
replace all the systems and report on that instead...

------
jtwaleson
For a minute I thought this was the release of the "AWS for Everyone" product
with a new name, but this is something very different.

Interesting product though! I think everyone that's using SaaS solutions has
nightmare scenarios about providers screwing up or going out of business, or
even seeing versioning of data at the providers. Exporting in a standardized
way to S3 is great for insights in changes as well as for backups.

Of course when you starting building triggers that react to changes this
creates a huge lock-in to AWS, but that's a cost a lot of people are willing
to pay.

I would love to be able to do this with a load of different SaaS solutions
that we use (gitlab.com, support tools, even google docs) (I'm in a regulated
industry and this would solve some issues for us). Curious to see how this
will expand and if custom integrations will appear soon.

Any AWS marketing people that are reading, there's a typo: route cause -> root
cause.

------
basch
How does this compare to

Azure Data Factory [https://azure.microsoft.com/en-us/services/data-
factory/](https://azure.microsoft.com/en-us/services/data-factory/)

and

Microsoft Power Automate -
[https://flow.microsoft.com/](https://flow.microsoft.com/)

------
masonhensley
By now - the amount of time teams have spent doing this sort of work from
scratch has to have surpassed the engineering hours taken to get to the moon.

~~~
toomuchtodo
As long as there is revenue to be had doing it, folks will continue to do it,
reinvent it, and so on. It’s a billion dollar market, at least. Not everyone
is going to be a developer, nor should they have to be (it’s unreasonable to
assume everyone has the aptitude). Everyone is trying to be the Excel of the
business process automation market.

Conversely, who knew there was so much money to be had in performing ETL on
API payloads!

~~~
masonhensley
Totally - I've had to architect & build similar work in regulated environments
with custom raw data from clients... yuck.

There are definitely healthy businesses in different market sectors that have
popped up to do DataA <=> DataB

------
mikkom
Kind of interesting offer when you think what kind of implications it has
related to this:

[https://www.wsj.com/articles/amazon-scooped-up-data-from-
its...](https://www.wsj.com/articles/amazon-scooped-up-data-from-its-own-
sellers-to-launch-competing-products-11587650015)

------
ozten
I can finally hydrate my data lakes.

------
propter_hoc
Related, I've been trying to figure out what the best way to populate a
Redshift database from regularly updated SFTP sources is and I can't quite
figure it out. AWS Glue? A function in Lambda? This?

AWS presents so many options that it's a bit overwhelming.

~~~
fraserharris
We, Fivetran, have an SFTP connector. You can be sync'ing data with 10 minutes
of effort. Our value is in (nearly) zero configuration, automatic schema
migration, and automatic failure recovery. We take the pain out of Extract-
Load, and push Transformation into the warehouse.
[https://fivetran.com/docs/files/sftp](https://fivetran.com/docs/files/sftp)

------
rob-olmos
> Automate data back ups

This has always killed me about SaaS-es that don't provide any decent method
for backups (and less so: restores), especially for data modifications (eg,
accidental deletions) that there's no ability to rollback.

------
pjmlp
Also known as BPEL, Business Process Execution Language and BizTalk.

Love how these things keep getting re-invented.

------
rgharris
Ionic Appflow is a very different platform but it is interesting to have
products with the exact same name (except the capitalization of "F") in the
software/platform world.

[https://ionicframework.com/appflow](https://ionicframework.com/appflow)

~~~
rvz
One of the worst things in naming products is when a big business
unintentionally clashes with your product name and nearly forces you to
rename/tweak it.

As they are in the same category (Cloud technology), I won't be entirely
surprised if Ionic renames their 'AppFlow' name to something else. (Unless
they're feeling lucky in winning a trademark lawsuit)

------
grwthckrmstr
Noob here with a question - is this in any way a competitor product to
Segment?

~~~
borisjabes
Yes and no. It's primarily focused on centralizing data into a data warehouse
/ data lake rather than sending data out to other apps. Segment also pushes
events into a warehouse but most of its functionality is about forwarding
events to 3rd party apps. That said, you can use tools like getcensus.com to
move data/events from an AWS-populated data warehouse back to other apps
(disclosure: I'm one of the founders of this service).

------
reffaelwallen
Is this a Segment type thing?

~~~
lbotos
No, more like [https://fivetran.com/](https://fivetran.com/) and others like
it. (unless Segment has recently pivoted.)

~~~
mrwnmonm
I am confused, Fivetran obviously has more sources than Segment, but in the
end, they both are pipelines, right?

------
howmayiannoyyou
Integromat Parabola CloudHQ Zapier Apache Nifi ... and literally dozens more.

Very, very crowded space.

------
seibelj
This seems like a Zapier killer, makes sense.

~~~
tjbiddle
"Killer" \- No.

90% of the market for Zapier is people who have no technical knowledge. Those
people won't be jumping on AWS any time soon.

Obviously pulling the 90% out of my ass, but I'm heavy Zapier user for my
e-commerce business. I have (Or had - Haven't renewed as e-com has replaced my
DevOps income) multiple AWS certs and I'd still much prefer Zapier over this.
Especially when you can run custom code on Zapier for more complex cases
anyway.

~~~
Axsuul
Can I ask what sort of tasks you have setup for your e-commerce business?

~~~
tjbiddle
\- If a new voicemail comes in, see if it's an existing customer and send them
an email we'll be in touch if we do \- Create new ZenDesk users for any
Shopify customer \- Send emails when positive product reviews come in asking
if they would mind sending a Facebook review \- If a negative product review
comes in, ask what we can do to make things better \- New order comes in: Will
create draft emails to all necessary vendors I need to contact, pre-filled
with shipping details. \- New order comes in: Separate order cost by vendor,
grab taxes, grab shipping, grab discounts, put it all in an airtable database
and create invoicing and purchase order records to attach it to. \- Slack
notifications \- Pull PDFs from Google drive, parse with DocParser, upload
into Xero accounting.

I'm sure there's way more I can do as well. I have to find the balance between
automating and just telling an employee to do it But automation leads to less
mistakes

