Hacker News new | past | comments | ask | show | jobs | submit login
You probably don't need AI/ML. You can make do with well written SQL scripts (threadreaderapp.com)
1075 points by passenger 11 months ago | hide | past | web | favorite | 322 comments

My startup was approached by a corporate VC that wanted to make a strategic investment. Based on the attendee list from our meeting, which included very high up folks from the company, I felt good going in. They expressed interest in our technology that makes reading on screen easier [1], but they were surprised to learn that we didn't use machine learning to accomplish this.

I indicated that it was actually quite effective without ML, and that it was easier to explain to users this way. They kept prodding around on the ML stuff, and how we might be able to use ML to accomplish roughly the same thing.

A week later they said that they were no longer interested because, although they liked what our tech was able to accomplish, it didn't fit with their investment thesis — which was all about ML.

My wife asked me why I didn't just make some stuff up and say we could do v2 using ML. Perhaps she was right.

1: http://www.beelinereader.com/individual

update: in response to feedback below, I edited the link to point to a page with relevant content instead of our generic landing page. Lesson learned!

That is just how larger corporations work.

They probably decided they wanted to be part of that future market about AI and therefore want to invest in AI/ML startups. Someone sold your company internally as being a nice fit for the AI portfolio, but when you answered the question about ML with "No" the door was shut.

From someone who makes large trade deals as his daily business, I learned "Never say 'No'" (during a negotiation). In most cases, it is better to be diplomatic. So might have been better off with something like:

'So far we had decent results without ML, but we are constantly evaluating options to improve our technology, which includes optimizations based on machine learning.'

That way, some might hear '[...] includes optimizations based on machine learning.', while you just said 'we do not use machine learning' ;-)

You should never mislead a potential investor. Why would you want an investor who is misaligned with your core technology/strategic direction?

Well, nobody wants a misaligned investor. But I had the impression the negotiations already failed, because one buzzword could not be satisfied. My comment was meant more as advice to overcome such problematic situations during negotiations, than to mislead someone about what you are actually doing.

I remember every dishwasher and washing machine had "fuzzy logic" in the 90s (at least according to stickers). Just because it was the AI/ML/Deep Learning/Big Data/IoT of the time.

No, fuzzy logic won't make your washing machine better. Yes, you can implement some stuff in fuzzy logic.

I don't remember this from the 90's very well, so I read the wikipedia page. I saw the following diagram and wondered how many washing machines and dishwasher's considered themselves to use fuzzy logic because they simply kept track of the temperature instead of just being in hot / cold states: https://en.wikipedia.org/wiki/Fuzzy_logic#/media/File:Fuzzy_....

We wouldn't have this problem if civilization didn't continue to give so much money to the ID-10(t) class of investors.

Ideally, people that do nothing but tick off boxes in their mental buzzword checklists should not even be allowed to attend the pitches, and thereby waste other people's time by forcing them to hack around the unrepentant ignorance. But they that have the gold, make the rules.


How is this misleading?

This is a quaint sentiment, but marketing, sales, PR is all about misleading and creating unnecessary want. This is a major tenant of running a business and living under capitalism. There is also a moral argument for doing what you need to do in order to survive the difficulties of the market in order to survive it and change it into something more akin to your honest attitude about it.

I think you'll find that a lot of stem people tend to lean Kantian in their ethics. This implies that misleading people for any reason -- even "for their own good" -- is frowned upon.

I am not sure the Categorical Imperative rules out misleading people for their own good. If a win win result is highly likely the categorical imperative says it’s OK if you do it because if the whole world did it, things would become better!

No, it's still using people as a means to an end. It's even wrong to lie to the murderer at the door. Kant himself addressed this in an essay titled: On The Supposed Right To Lie From Altruistic Motives[1].

[1] https://www.unc.edu/courses/2009spring/plcy/240/001/Kant.pdf

Being Kantian doesn't mean being correct. It's not that hard to disagree with Kant's extreme hypothetical.

Yes I am aware, however the categorical imperative by itself does not seem to rule out such lying.

“Act only according to that maxim whereby you can at the same time will that it should become a universal law.” — Immanuel Kant, Grounding for the Metaphysics of Morals

But it's even better in the long run if inefficient companies consistently lose to the better ones, since it will lead to better allocation of capital.

He did say marketing, sales and PR but your point still stands.

I was referring to marketing, sales, and PR as activities moreso than the people who work in those fields. I can't speak for all stem people, of course, but I, personally, have a strong aversion to those activities.

"There is also a moral argument for doing what you need to do in order to survive the difficulties of the market in order to survive it and change it into something more akin to your honest attitude about it."

There's really not, but it might make you feel better to think there is.

How the stock market cargo cult works.

We have already seen this with things like offshoring.

A few big name companies did the numbers and found they could save some on moving their production somewhere else.

Then came a rush of companies that made big that they had offshored who knows what, simply because that was what the stock market expected of a would be forward looking company.

The basic problem is that these decisions are not done based on what is good for the company long term, but what is good for the stock price short term. And in large part because CEOs are paid in stock options, and can be ousted by activist stock owners.

I went to your website, and I can't figure out what your product is or does. I had to click in to individuals before I figured out what it does.

Why not put "BeeLine Reader makes reading on-screen easier, faster, and more enjoyable. We use a simple cognitive trick — an eye-guiding color gradient — to pull your eyes from one line to the next." on your front page?

Maybe you guys have already done some kind of testing and found out that the current layout is optimal?

Appreciate the feedback. We recently redesigned the website to offer silos, and one of the goals was to make the landing page not offer too many specifics that would make some audiences bounce. But perhaps your suggestion is just what we need to do. Thanks for taking the time to share!

One lesson learned: I should have linked to the /individual page instead of the generic landing page in my comment. Updating link now...

The lesson you should have learned is to make the generic landing page actually describe the what the product attempts to acheive! You have just had a load of inquisitive wanting-to-be-impressed HNers descend on your page and nobody knew what the thing was about! :)

Publishers, educators etc also need to understand what the value proposition is for their users.

I also think the page ought to be written in longer sentences. The current silo buttons aren't even making it clear that they explain more.

So, suggestions:

You just have to take the few sentences from your Individuals page "Reading on-screen can be tough on your eyes, especially if you have to do it all day long. We’re here to help. BeeLine Reader makes reading on-screen easier, faster, and more enjoyable." and put that on your landing page instead of "A New Way to Read."

Now change the "(choose)" subheading to "perspectives:", put it lower nearer the buttons instead of near the description, and perhaps left-align it.

Change the buttons from meaningless colours with labels to something descriptive like "for you the user :smile:", "for the publisher $" and "for the educator :graduation-hat-icon:".

Voila! good luck!

Super helpful, thanks!

It's good to keep market-specific language to market-specific silos, but you need to put some little (milquetoast, plain-info) bit up front. Otherwise you turn off every market.

Totally agree. Would you say that GP's suggested language is sufficiently plain? I would worry that it would turn off publishing partners, who want to hear things like "increase audience engagement" not "read easier".

Maybe something generic like "improve focus while reading", which could appeal to individuals and publishing partners? Appreciate your taking the time to share feedback — this sort of stuff is not something we're skilled/experienced at.

I don't really have a good feel for any of those customers, so... ^_^

I agree with the comment above. I wanted to ask you what your product does, then I thought "why should I ask you when I can check the website out". Then I checked the website and I didn't find a description.

I think it will be very helpful to have a simple description in your site. First impression matters.

p.s. I like the idea of your product.

I disagree with everyone else!

The silos are good, you just need to make it much more obvious that the user needs to select one of the silos in order to get at more information. It's not particularly obvious that your buttons are buttons. I would also recommend that the content be full height / width so that on desktop at least the user doesn't scroll at all. The stock photo is probably unnecessary for the main page. Also you don't really need left/right margins since there isn't really text on the page, and that's what large margins are good for (but I'm assuming you know that since you made this app!)

I bounced as soon as I couldn’t find pricing information.

Good point — this is something we need to address. Our most popular products (browser plugin and iOS app) both have free trials so that people can see how much they like to read with BeeLine. The iOS app actually has very significant free functionality, so if you just want to use it as a reading list (like Pocket/Instapaper) you can do so for free via the Share Extension, which populates an offline reading list.

Things get more complicated, as the browser plugins are subscription ($2/mo or $22/yr) and the iOS app is one-time IAPs (mostly a few bucks, but more for the Kindle feature). I'm not sure how to succinctly communicate this, but you and other commenters make clear that we need to have some info here.

Thanks for taking the time to share your thoughts!

Thanks for the follow up. What if I have a website and I want all my pages rendered in Beeline? I immediately noticed being able to read faster and more comfortably. I view Beeline as a conversion optimization tool.

We have a JS widget that can be dropped into websites. I'd be interested to see if we can increase your conversion rate or other key metrics.

CNET did a study [1] that showed BeeLine readers were much more likely to finish an article, but it would be great to take things to the next level and see how that translates into purchases or other conversions.

Shoot me an email! contact@[domain]

1: the CNET study is discussed at the bottom of this article http://www.theatlantic.com/technology/archive/2016/05/a-bett...

PS While your wife is pretty awesome thinking on her feet, I don’t see the case for ML providing meaningful optimization for Beeline. Maybe you dodged a bullet getting a pass from a VC with unrealistic expectations.

I have nothing to do with the website/product in question, but could you explain a bit more why you would "bounce" (assuming bounce means you would lose interest) if you didn't immediately see pricing information?

For me, I almost always bounce if I don't immediately see pricing information as it is an important signal with respect to whether or not this is something that's targeted towards me and/or whether or not they think I'll get sticker shock if they show it to me right away. In either case, the lack of readily available pricing makes me negatively inclined to spend more time looking into a product where I don't know if I'd ever consider paying whatever price they don't want to tell me.

It doesn't need to be a lot of detail, or precise. Even just a "from $..." or "plans starting at ..." would be enough.

I agree. There have been a lot of times I’ve been interested in something that seems to be exactly what I’m looking for but when I manage to find pricing information it’s starting at $990/mo. And now I’ve wasted all that time on a product very obviously not designed to help me remember to pick up bread at the store.

I appreciate that Citus Data is right out in the open that entry level is $800/mo. I know straight up I can’t afford it. But they also tell me the size of their minimum instances, which also tells me my product hasn’t yet scaled to the level to need their service. And by the time it does, $800/mo will be acceptable.

Ah, so were you talking about the partner pricing, for integrating into a site/app? I can see how it would be helpful to at least have some prices shown for that, to show the ballpark. For a casual website owner who just wants to make the copy easier to read, the pricing is de minimis (just enough to make sure that folks are serious about integrating and not wasting our time with a bunch of back-and-forth). Shoot me an email if you want to talk further!

I’m planning on it. Thanks for all your awesome interaction on this thread.

* So I just randomly came to your website because I slacked off and should actually do work.

* I know that I will forget I even clicked on that link. The page has 3.5 seconds to convince me to stay longer.

* I'm mildly intrigued and form, in my head, a price I'd be willing to pay for the product.

* I scan the navbar for a pricing link to see if my expectation matches reality. I find nothing.

* The site obviously doesn't value my time, so why should I stay?

It's disrespectful.

Two reasons. First, the tech spoke for itself and I was sold, if I could afford it. Don’t insert the friction of having to exchange an email and suck my information out of me to find out the cost. That goes against every theory and study about conversion, dating back to before the web. They have a saying, when you’ve made the sale, shut your fucking mouth. And as my example demonstrates, sales leak out of that funnel simply because people can’t be bothered.

Second, the other saying is if you have to ask how much it costs, you can’t afford it. So by concealing the price, the company is telling me the cost is too high and they know it. They are telegraphing that part of their sales process requires a salesperson to coerce the money out of me.

Interesting point about involving a salesperson — it came across that you would have to negotiate with a literal person before purchasing? I sort of always figure that it's just a matter of an upgrade notice that displays a price and that's that.

Also, note that we don't actually require (or even ask for) email addresses or anything personal when you install the browser plugin. You just click install, try it for 2 weeks, and then buy if you want. The standard pricing is $2/mo, with a discount for annual purchase.

I totally appreciate the feedback and will see how we can better communicate our pricing to alleviate concerns like this.

You'd already updated the link above when I first looked at the site, but having a look at the home page, I didn't immediately notice that the Individual / Educational / Partners text were links, so I'd have probably been confused too!

I liked your reading challenge idea! Very interesting, even though apparently the results indicated that I didn't read any faster.

Yeah, as some here have pointed out, the topics are aimed a bit toward younger/less technical audiences, so it's perhaps not like your typical reading material.

I should also note that reading ease is an important (but difficult to measure) piece of this as well. So if it felt better for you, then it might be worth trying out the browser plugin and then trying the reading challenge again. There could be a little bit of a learning curve, as your eyes adjust to how quickly you can jump from line to line.

Agree with db1's feedback. I almost immediately closed the page after not finding the pitch.

If your code has even a single magical constant (I guess it likely has, if only because your code has to figure out whether a period is sentence-ending), you can and, now that it is cheap and easy, probably should use machine learning (a moniker that includes such algorithms as gradient descent) to optimize that constant.

Also, if you ever have considered two code variants where one is a bit better in situation A and the other in situation B, you can use machine learning to combine the two methods, (e.g. using a random forest (https://en.wikipedia.org/wiki/Random_forest)

You don't need AI to find values for free parameters. Unless you consider all of statistics to be a subfield of AI.

I personally consider all of modern AI (i.e. statistical pattern matching) to be a subfield of statistics. Mind you, I'm probably just a grumpy old man.

No, you're being fairly accurate.

Why use the term "AI" when the GP specifically used the less ambiguous Machine Learning?

Machine Learning = statistics + linear algebra + computer science, mostly.

Naive Bayes and Graphical Models are pure statistics, but they are mostly used for toy problems. Machine Learning scales these approaches to high dimensionality problems, and tasks where data is abundant.

Funny, I would consider many ML usages to be toy problems. Coming up with funny nonsensical Shakespeare plays or generating random Pantone colors or Pokémon names isn’t something you hear from Bayesian systems. That’s the domain of ML. Recommending related products is again simple statistical analysis. And I can generate a prediction based on past performance real easy with some SQL and a couple of “if” statements.

Just because Bayesian isn’t over-hyped doesn’t mean it doesn’t solve real problems. Not everything can be solved with ML, unless the problem is getting more investor money.

There are some methods that I don’t see used outside of the machine learning community, but there appears to be a fair degree of overlap in actual methods used between classical stats and machine learning.

Now, the biggest difference that I can see is that, within machine learning contexts, people are more concerned with the quality of the predictions than the interpretation of the independent variables. That’s not hard and fast, but it seems to be a common thread.

And for some problems, making a good prediction is really the right thing. In other cases, understanding the mechanisms you might use to effect an outcome is more important. Both are valid uses of statistical methods, depending on the problem.

I think within the realm of classical statistics, Bayesian methods’ super power is in being able to generate results that are much easier to communicate to lay people. Also Bayesian methods are nice if you want to do sensitivity analysis in a principled up front kind of way.

But I could imagine using those methods in an ML context even if they aren’t the current darlings of the methodological pantheon.

I agree with all of this, except are you really implementing bayesian models in SQL? I'm not sure that is what the author is advocating in the first place. There are tools for bayesian everything, I much prefer using those over hacking together some illegible SQL script.

No, it’s just SQL statements pulling data into the if...then in your preferred language. I’ve never done actual logic with SQL, not even sure if it’s possible.

Each DB engine has it's own incompatible imperative stored procedure language.

Having worked with Oracle's version, and seen entire systems buult out of that, I dread it.

If someone had something as basic as logistic regression or k-means clustering in a production app they are already ahead of the vast majority of ML practitioners

Gods, maybe I should stop telling people that I don’t have much in the way of a machine learning background.

Naive Bayes is machine learning. A very simple machine learning method, but machine learning.

Also, the argument that it's pure statistics doesn't really fly - support vector machines, random forests or deep learning multilayer perceptrons are just as much statistics as probabilistic graphical models.

> Naive Bayes and Graphical Models are pure statistics, but they are mostly used for toy problems.

I think they just feel "toy" because they've been used with great accuracy for so long.

Complex problems are rarely solved by a single approach. The harder the problem the more likely a suite will be used, and often naive bayes will be part of that in some capacity.

Can you give one example where Naive Bayes has been used with great accuracy for so long?

Naive Bayes is a common baseline solution because it's very simple to implement and very fast to run; however, pretty much any other ML method gives better accuracy than Naive Bayes. Sometimes the accuracy improvement is small and doesn't justify the highly increased computing cost, though, so Naive Bayes becomes the appropriate method to use, but it's not because of its accuracy.

Aren't Bayesian models essentially statistics + linear algebra? It sounds like you're characterizing ML as something beyond Bayes, which isn't really the case. I don't know what you mean by "toy" problems... that might be kind of true for Naive Bayes, but a Bayesian statistical model can be built in a non-naive fashion and provide useful multi-dimensional probabilities. It's really a kind of underrated technique, though I suppose it's because you can't simply `import Bayes from 'bayes'` to build a non-naive model.

Bayesian models are also advantageous in that they are much easier to describe to stakeholders. Neural networks, on the other hand, are highly difficult(if not impossible in many cases) to describe how they function.

AI and ML have different goals, which sometimes overlap. The goal of ML is to discover relationships in data that can only be (easily) observed by a machine. That usually serves as decision support data for a human decision maker or another more conventional algorithm. This is usually based on an entire data set at once. Accuracy and precision are valued over quickness and robustness.

AI is about substituting human decision makers or other conventional computer decision algorithms. AI is used for situational decision making. It needs to be quick (like the conventional algorithms) while also recognizing the nuance and multi-dimemaional nature of a lot of decisions. Robustness and quickness are often valued more than precision.

IOW the GP is only using ML for everything because it is a buzzword.

ML is one of subdomains in the AI field, so instead of "sometimes overlap" it's technically correct to state that ML is a strict subset of AI.

What you use as description for AI tempts me to use the "use it as a buzzword" angle back at you here - it is a stereotypical description of some use cases of AI approaches; but there are others - [machine] learning, knowledge representation, planning and scheduling, reasoning (both formal logic reasoning and also reasoning under uncertainty e.g. Bayesian approaches), intelligent agent representation, etc, are all parts of the AI field.

This last line seems unnecessarily rude. GP seems to understand well the two ML methods mentioned.

> Machine Learning scales these approaches to high dimensionality problems, and tasks where data is abundant.

Machine learning is the use of computers to make decisions (or classifications) based on data without human intervention.

Great product idea! I'm glad people find it useful.


> corporate VC that wanted to make a strategic investment

They want to put money in your company to make sure they don't have to compete with you or compete with someone else who might buy you down the line.

> They kept prodding around on the ML stuff, and how we might be able to use ML to accomplish roughly the same thing.

They're looking for secret sauce, barriers, friction. Something patentable. Something to keep others out and keep them ahead. Perhaps something that they don't think they can do themselves or maybe rip off when they realize it's useful. Remember Flux?[0]

They're investment thesis makes sense from this perspective: they're thinking like classic, amoral businesses.

Also note that if your product is simple and popular enough, people will likely make free clones. You'll need to be a good steward to keep your value.

Best of luck!


A bit of feedback: you need to be more transparent and up front about what how and why you're using analytics, and let people opt-in. I was a bit disconcerted to see you using Google Analytics in your extension without informing me.

[0]: https://justgetflux.com/news/2016/01/14/apple.html

> you need to be more transparent and up front about what how and why you're using analytics

That's a new one to me. Is this Facebook changing the landscape right now, or have you been expecting this for a while? Do you have sites in mind that warn about GA?

Pretty much all websites use GA or something like it, and in my experience it's extremely rare to be warned about it. It always goes in the privacy policy, which you should be able to find. But I'm not sure I've ever seen an advance warning that analytics was taking place. I suppose it's assumed, but in any case appears to be normal and acceptable to not warn people that logging and searching of those logs exists.

Cookies are a different story, since the EU passed legislation requiring notice of their presence.

> They're looking for secret sauce, barriers, friction. Something patentable.

While that may be true, I'd suggest the VC's are just looking for a later exit when a greater fool buys them out. They're betting on the ML investment market getting hotter, and providing you meet the criteria of being a growing startup (by some metric) nominally in the ML field and that field attracts more investment dollars later - even whether or not you succeed as a bottom line business, or whether your business success comes from ML or not - doesn't necessarily matter.

Appreciate the feedback! Regarding analytics, I think this is some A/B testing, which I don't think is even active in the current version (though still in the sources).

More importantly, we don't tie GA or any other usage analytics to individual users. We basically just use it to see where the extension is used and where people are blacklisting sites. We use this data to make the extension run better on more sites. We don't monetize user data in any way. You're right that we should make this point more salient. In our Privacy Policy, we do describe how to opt out by blocking GA, but we should put this somewhere more prominent.

Thanks for the feedback!

> we don't tie GA or any other usage analytics to individual users

You may not identify individual users but Google probably do...

> we should put this somewhere more prominent

You should make it opt in by default. Or stop doing it!

> You should make it opt in by default. Or stop doing it!

What? Can you post some examples of sites that already do this? Everybody has analytics on their landing pages, and I don't think I've ever personally seen an opt-in.

What you can find are thousands of examples of site policy pages that say something like: "by choosing to visit our site, you are sharing information with us that we log for the purposes of improving our customer experience. By using our service, you agree to this logging. If you don't want to be logged, then please don't use our service."

IANAL, but have worked on GDPR a fair bit.

Google Analytics is already GDPR compliant, as long as:

* you are not pushing any unencrypted customer data to them in the clear

* you're careful regards who has access to your Google Analytics data

You can (and most businesses will) argue that you need to know how your website is used. It's a completely standard practice that doesn't put user privacy at risk. Moreover, Google has already offered an opt-out to Google Analytics globally, and you would be covered by that. As such, you don't need them to opt-in.

This isn't about law it is about ethics.

I wouldn't like it if someone followed me around meatspace with a clipboard all day. I don't like it online either.

That's not the best analogy - it's more like someone has two chairs in their waiting room and they note which chair is more popular among people waiting, so they can replace the worse chair with a copy of the better one.

Except that this business saves money by allowing a representative of a third party to take the notes. The third party then notes down who sits in the chairs, follows them into the appointment and follows their client to the next business and do the same there.

We're totally open to making this opt-in. I'm curious how we should gather usage data, which some of our nonprofit funders require us to report for their impact questionnaire. How can we know how much the tools are being used if it's opt-in? Or are there other systems (other than Google) that raise fewer concerns?

I really appreciate all the candid advice in this thread and hope that we — and others in similar situations — can use it to improve our products.

A few other meandering thoughts, you'll need to be the one to figure out what makes sense.

- You're already not getting data from people with ad-blockers, consider also respecting Do Not Track[0] - Be transparent about what you track and how it's used - The data collected from opt-in users might end up being enough to build convincing enough reports for impact questionnaires? - Your privacy policy seems confused: "We think the best way to ensure that your personal information stays safe is to never collect any in the first place" then "we only use Google Analytics, which gathers information about our site users and generates aggregate reports to help us figure out who is using our site and extensions, and in what ways"

[0]: https://developer.mozilla.org/en-US/docs/Web/API/Navigator/d...

I posted that link in slack and it's twitter:* meta content is really strange:

    Parade of Fans for Houston’s Funeral

    NEWARK - The guest list and parade of limousines with celebrities emerging from them seemed more suited to a red carpet event in Hollywood or New York than than a gritty stretch of Sussex Avenue near the former site of the James M. Baxter Terrace public housing project here.
I found an example [1] with the same content so I guess you just haven't updated it.

1: https://gist.github.com/blairanderson/85cc961295fd03a6c4b3

Wow, how embarrassing. Thanks for pointing this out. I have no idea how we didn't catch it sooner.

On a related note, the link to twitter:image is also on wrong domain.

Often times the output of ML/AI will result in writing code for an expert system.

The system doesn't actually use ML or AI in realtime. ML just validates the correct conditionals and decisions are used in the software.

Most ML models can be distilled down using the Pareto principle anyway, and the 80/20 rule written into the code.

A politically correct and high-integrity story to tell investors might be "We use machine learning regression analysis to validate our expert systems are operating on optimal statistical models."

Awesome product! Been using it for a bit, and does indeed help.

Just a quick bug report - it does the wrong thing on right-to-left text. e.g. on this article [1], it highlights the beginning (right side) of one line and the end (left side) of the next line in the same color, which doesn't actually reflect the flow of reading. There's also a weird effect on LTR text embedded in the RTL (on my setup, the "a" in "axios" on the third line of the body is bright red, while the rest is complete black).

EDIT: [1]http://www.maariv.co.il/news/world/Article-633899

Can you share the link? I thought we fixed the right-to-left issue a long time ago. Feel free to share screenshots or more info with contact@[domain].

Ah! Sorry, forgot to paste the link in. It's here: http://www.maariv.co.il/news/world/Article-633899

Sending a screenshot by e-mail now.

Can someone explain the dynamic that’s going on here? I somehow can believe that VCs are so stupid to be purely hype driven, so what is pushing them to make this their investment focus?

Do they just jump on every bandwagon in the hope that one of them might pan out?

VCs raise money the same way startups do, only with more regulation. For that bucket of money they likely presented a thesis for how they would used to invest in the ML/AI wave. Investors have rights if the money isn’t used they way they were told it would be.

VCs raise money from fund managers (sovereign wealth funds for example) who also present a thesis to their stakeholders for how the money will be managed.

Depends on the timing but I think VC focus on ML is a good indicator of where smart money thinks the money is going to be made. But there are a lot of people that catch a trend in the tail in, I don’t think we’ve gotten anywhere close to that for ML investments.

You make a good point about a VC's responsibility to stick to the plan they sold their LPs on. This could explain a lot of group-think and myopia in the industry.

But in this particular case, it was a corporate VC, investing their own money.

That's even worse for you case, I suppose. In case of a usual VC funds it's about the returns, but corporate VCs also look for some potential synergy. They probably were looking for some ML startups that they could utilize later.

For corporate VCs, s/LPs/board/

They put a lot of study into hyped categories, so they at least understand the businesses they're investing in. They can evaluate the caliber of talent, understand the depth of your moat, etc. Whereas with businesses outside of the hype (or outside of their specific specialty) they're not confident in their ability to even evaluate the potential of an investment.

(I'm working on a startup right now that's in a similar position - non-hyped B2B segment. VCs are all "hey, look, we don't understand X market very well, maybe go to Y VC?" And Y VC is all "hmm, you're only tangentially related to Z market that we actually understand, so you'll need to get to Series A-sized revenues before we'll put in a seed-size investment.")

They invest in future potential. What you can do now is nearly irrelevant besides showing vision and competency. The next big thing isn't going to come from a couple of clever sql scripts.

Neither will the next big failure.

What has AI really given us so far? Amazon / Youtube recommendations and some self driving cars that have crashed in unexpected ways. (Have they managed to debug that yet?)

From a VC perspective failure has a maximum downside of whatever you invested. Success has a limitless upside. That's why 9 out of 10 companies failing isn't an issue.

VCs are very stupid most of the time. They don't understand even how ML works orh blockchain or any new trend.

But their manager told them ML, ML, so they need to invest in ML. This is not a joke.

The thing is we're at a stage where companies hoover up every bit and byte of data they can get hold of on every customer and many non-customers because... well it's gotta be valuable some how right?

But it's so much damn data so loosely coupled to a company's actual product that trying to get any actionable intelligence out of it is basically impossible. Enter ML. "Just" chuck your data at a NN use A/B tests to train and hope the company ends up with higher revenues, if they do: claim the 'data scientists' are definitely a profit center; if they don't: claim you need more data.

Until Exxon-Mobile stops storing metrics on how many goldfish I own in a given month, cranking out ML related companies seems like a good bet for VCs because it's not going to get any easier to turn progressively more esoteric data points into money.

Yes, due diligence is rare. This is portfolio theory in action, investing in many losing bets because the 1% will win big and create the returns.

Disqualify the easy "no", then everything else is a "maybe". Invest and move on.

The VCs is this case were probably looking for some tech that had a higher barrier to entry.

Whether it is true or not in this case they may have perceived that if it was based on a ML approach it might be harder for the competition to replicate if it was successful.

Its not just every VC, its a good percentage,maybe the majority of developers out there.

Well, they probably want engineers with ML experience, and companies who already have a foot in that door when an opportunity arises for an actual profit maker.

You may have pitched a perfectly fine product to somebody in the market for a perfectly fine acquihire.

I did your test. Interesting idea. Feedback: I got "It looks like BeeLine didn’t improve your reading speed this time through." which I think needs another pass from your copy editors.

Interesting idea overall. I've been using spritz a lot recently and I'm liking it more and more though it doesn't seem to have progressed from when I discovered it a few years ago.

Re the investors blindly wanting ML - we have this problem too (and it's definitely a problem - I can see this in spite of having a masters in ML). On the bright side, some investors - often the better ones - have said things like "I was interested the moment you didn't use ML or blockchain in your pitch", so stay true to your vision :)

General comment on colour highlighting -

I read the updated /individual page and found it very... difficult.

The colours led me to speed up and slow down at an uncomfortable rate, to the extent I had to re-read it three times. I typically receive 200-1000 emails per day which all need to be read (yay!) plus at least 30k words per day out of email (business stuff but excluding newspapers, books, etc).

I speed-read about 6-10 words per flick of the eyes (for a short document), which is about 50 words per second. Do slow down to ponder: careful phrasing, a needingly precise written document, some graphics, and this is not for detriment.

I found the colour-coding very difficult, however.

I'm curious to know if you tried it in the more subtle color schemes. OTOH, if you're a speed reader who uses other techniques, this might not fit your reading patterns.

Similar experience here. I suspect it's not a good option for speed readers.

Also, there are issues with red-screen sleep promoting systems like f.lux.

Wow! Would you mind sharing some practical advice on how to actually train to read this fast?

After writing this, I realise you may need to speed read it!

My first company provided a speed reading training course for the best part of a day.

It wasn't the best targeted course for most people there. We mainly read legal and policy documents which by their nature are terse, and the method of speed reading taught traded-off retention against speed against accuracy. It was more of a concept, which you can go off with an develop yourself. So here's the idea -

1. Start with a text that is quite verbose but not overly so. A code specification or Brett Easton Ellis novel is a no. A light news article should be fine or non-technical book should be fine.

2. Recognise that when you read, you don't read a word from letter start to finish, you basically look in the middle of it and the word is recognised. If you have dyslexia, this technically probably doesn't work; but if you have dyslexia, you've probably invested countless painful hours to something a lot of us take for granted and hats off to that.

You might not notice, but you probably read multiple-words at a time already. Commonly paired words are often read together, like 'a few'. YMMV. Harness that.

3. Mentally split a page of text into narrow columns, perhaps the width of 2 medium length words. Flick your eyes across a line of text. Only focus on the middle of the mental column and absorb the words, like a snapshot (no need to blink).

This is where accuracy starts to degrade. If the text consists of obscure words or lots of figures, accuracy can hurt at the start. Grammar patterns are important too - as long as the text is written by a native or very close near native speaker, anticipation and auto-ordering in this mini-snapshot can be hard or impossible; likewise unfamiliar or complex sentence structures from a native speaker.

4. Now, start to speed up over a paragraph. This is where retention starts to hurt. You certainly don't want to be wording-out words, as this will slow you down and defeat the purpose. But when you've read a block of text, have you been able to follow the meaning at the end.

Again, using well written, non-overly terse training text is important.

5. You should be tired by now, having done this for several hours. Our trainer then suggested changing the 'snapshot' from one line, to two lines, so you're essentially reading the start of the second line before finishing the end of the first line. I can do this but your text needs to be super non-terse for this to work at the start, and even then it's weird. If you can cover a single line pretty fast in less than a second certainly, then this starts to become feasible.


Went on a training course. Was pessimistic as were my colleagues at the start. Persisted afterwards by trying a few minutes per day. Became a habit, especially when seeing articles or documents that I needed to read but didn't need to scrutinise. Important to choose the right tool for the right job. I'd not use this technique to read a document where I'm not familiar with the content-scope; I wouldn't use this technique to read a contract I'm signing, or where specifications/requirements are being very clearly outlined. I do use it to read things like bloomberg.com articles, or PMBOK Guide :)

Hope a little useful, and the above can set you on some kind of direction to learn more. I highly recommend a half-day instructor-led course to kick you off. I imagine there's a college near you which has something.

You should definitely have made something up.

Up front, many investors only seek buzzwords before they shift focus to numbers. Be a sales(wo)man, and feed them the ones they want. Later on, once you end up making them money, they're not going to care much about what technologies are involved.

I wish you the best of luck with your startup in the future.

(P.S. I had a FOUC on your webpage. The icons in the "want more" section were missing, and haphazardly popping in. Might want to fix that.)

This seems pretty cool. I found a good difference reading your education product. It was easy on eyes.

I read a lot of blogs. Thought you could add a wp plugin if it doesn't exist.

Also you must have a ml/ai roadmap for tracking user interactions, etc for possibility of further personalization. Leaving aside the buzz of ai/ml, I think every business has opportunity to apply ml/ai. It doesn't happen often but when it does, its gold for any business

It's true, we could use AI or ML to do stuff not related to the core product. But these investors thought that we should be using ML in the coloring algorithm itself. Even if this made sense to do (that is, if it improved reading ability for people compared with our current product), it would be much harder to explain.

Right now, it's easy to say it's a color gradient that wraps from one line to the next, guiding your eyes. I don't know how I insert "machine learning" into that sentence somewhere and not sound like I'm trying to trend-surf.

If you can apply any kind of approach that automatically feeds measurements of user success back in to an algorithm to automatically adjust e.g. the colors used for the gradient based on measuring what works best, you can justify saying you're using machine learning.

I think people are too often too quick to assume that machine learning necessarily implies deep learning and neural networks, when often it can reasonably be reduced to simple statistical approaches.

First time I've heard of it, but that app is actually pretty cool. I can feel it dragging my eyeballs RTL :-)

Thank you for your work on Beeline. I've been using it for several months and it definitely helps when diving into large Wikipedia articles, research papers, blog posts, etc. I found that it facilitates my reaching flow quickly when investigating large problems.

I'm curious to know where you heard about BeeLine a few months ago. We haven't had press in a long time, and most HNers who know about us found out from our Show HN way back in 2013.

I tried to track down what it was, and I think I stumbled across the name while looking at the ReadMe! Android app. It was pay to use, but I was eager to try it. Thankfully I found your /pasteboard tool and it helped me dominate wiki articles, whitepapers, research papers, etc. I was sold because of the pasteboard and moved to paying for the desktop chrome extension

So, I went to the page, clicked the button for the Firefox extension, and was greeted with this page: https://s.mort.coffee/d/img/scr-4494758.png

You should probably link to the add-on page (https://addons.mozilla.org/en-US/firefox/addon/beelinereader...) instead of directly to the .xpi file. I would never accept to install a plug-in from a random link even if it hadn't been for the dialog's scary language.

Thanks for the feedback. We tried to make it as easy as possible, but I can see that it comes across as more legit if the install comes from the Firefox Store, which requires vetting.

Wow this is amazing! I don't understand the VCs input about ML, why and how would it help in this use case?

I think they wanted us to do something with NLP and ML, to make it syntactically-based. I happen to have studied linguistics and know how complex this would be — and of course it would have to be custom-built for every language. The current version, which just uses line position, works fine and is language-independent.

Another explanation: they didn't actually know how they wanted us to us ML — they just knew they wanted us to use it.

I do ML/NLP, and recently have been doing color related things.

I think you are underestimating how useful NLP could be in your application.

I tried out one of your test cases, and I did find it useful. However, I think that blending colors based on entity recognition along with your line based system could focus attention on important parts.

Have a look at [1] (sorry for the long URL) and imagine the colors blending with yours, so entities were slightly brighter than the rest of the text.

[1] https://explosion.ai/demos/displacy-ent?text=John%20Fitzgera...

I don't disagree that NLP could be powerful, and I have thought about ways to make it syntactically aware.

But honestly the biggest barrier we face in adoption among licensees (platforms that would integrate our tech) is that they are simply uncomfortable with text that has colors in it. It's not about how much it helps any metrics (the things that NLP could improve) — it's just that most folks don't want to be early-adopters of crazy-looking tech.

Though I should note we do have some great licensees, especially in the education, impact, and accessibility markets.

Regardless, I'd love to chat with you further — please shoot me an email (contact@[domain]) if you're up for chatting about how we could deploy NLP as we develop.

So.. the VCs weren't wrong, and it is really UI/UX issues stopping you? And those same UI/UX issues are actually things your primary business has to overcome anyway?

It sounds to me like you should talk to those investors again ;)

I'll drop you an email.

That sounds like a surefire way to annoy and distract the reader. Imagine if every entity was in bold, in a piece of text. It wouldn't draw your attention to important information, it would just disorient you completely. "Why is 'John' in bold? I already know this sentence is about him!".

Yes, I think I agree. But OTOH I thought different colours for different lines of text would be annoying and distracting, but that’s what this app does, and apparently it works.

So I guess test it and variations?

Is there a white paper or a clinical study to back up the claims on bee reader? I have read numerous studies claiming the exact opposite especially on a light projecting surface. I personally found the colour changing on each line to be extremely distracting and straining on my eyes. Furthermore, this seems like a bad joke for someone who is colour blind. Finally, VCs asking for ML are really asking how your technology fits in to the general future. With a script that reads line length and changes text between 3 colours, let’s be honest, I could have made beeReader when I was 10. Perhaps the VCs were initially wow’d by how profound you believed your product to be but then were underwhelmed by the tech (I’m guessing that’s why the individual page is burried). And if any publishers are in talks with you about using this in printed media (welcome to imaginary land) just keep nodding to what ever they say take their money and run.

Yep, there have been several studies done, ranging from media to education to optometric.

• CNET found that readers were 35% more likely to finish reading an article if they were reading with BeeLine turned on. See more on the CNET study in this article in The Atlantic [1].

• An optometric study using eye-tracking glasses found that BeeLine reduced the number of regressions and/or skipped lines for the vast majority of participants.

• Various educator-led studies (some more rigorous than others, and none as informal as the "case studies" that are unfortunately common in edtech) have found strong gains in reading fluency and/or comprehension.

I totally appreciate your skepticism, which is especially reasonable for someone who does not personally see a benefit from the tech.

The response from doctors who work in vision has been very positive, and our tools are recommended by doctors at Stanford Med School and UC Berkeley School of Optometry.

We even got featured by the American Optometric Association. The AOA committee that evaluated our technology said that this was the first time they'd ever had a unanimous vote in favor of anything. Apparently to them, it was quite obvious that this has beneficial effects (without having run any formal studies on it themselves).

To your point about colorblindness, I would note that I regularly come across people with red/green colorblindness who love/use our product. You can change the colors to be whatever you want, or use it in grayscale.

We are talking with publishers, but mostly on the digital side. Thanks for the input and questions!

1: http://www.theatlantic.com/technology/archive/2016/05/a-bett...

Why would anyone think you need ML or AI to make letters on a page fit a color gradient. Are these investors crazy? I can't even think of an application for that technology in bee line reader.

I have been playing around with the reader, and it doesn't seem to play well with mathjax-enabled pages.. Is it a known issue?

Not before now — thanks for flagging! A couple questions: what do you mean by not play well? Does it undercolor (not color the text you want to have colored), overcolor (accidentally color text that should not be colored), or something else? Thanks for sharing this feedback. As another HNer pointed out via email, we need to have a contact/help link in the Settings panel so that we can gather more feedback like this.

The coloring is correct, but the chrome extension 'undoes' the equations rendered on the mathjax-enabled pages. In other words, without the beelinereader extension I can see the equations, with it I see only the latex source code. You should be able to check this on any Springer article.

OK thanks for describing this in more detail. Sounds like we just need to skip these nodes when coloring. Look out for an update in the next month or so, and in the meantime perhaps you can blacklist auto-run (via Settings) on the Springer domain.

>I indicated that it was actually quite effective without ML, and that it was easier to explain to users this way. They kept prodding around on the ML stuff, and how we might be able to use ML to accomplish roughly the same thing.

Just call what you do ML.

IBM, for one, never shied from calling any old BS "AI" and branding it Watson.

Just rename your company BeeLine Reader Blockchain and you'll be fine.

Seizure/Nausea warning!

What the article describes is called an "expert system" and is what AI in the enterprise used to look like.

Basically, you try to capture the instinct of a great salesmen by formalizing it into computer logic.

Often that's done with rules like in the article.

It works good, but has its limits. The finer reasoning of human judgement are often not expressable, people don't know why they made that decision. Making it hard to capture. And human also have their limits. Too many variables, too much noise, too much data and they won't make the best prediction/decision.

That's when ML shines. Instead of trying to encode an expert's intuition, instead you let the machine develop its own intuition, itself becoming an expert through training.

The downside is it now similarly becomes challenging to formalize the machine's intuition. Why it made a given choice is no longer easily apparent.

I do think expert systems still have value. Especially when you lack the dataset to train a machine expert.

"itself becoming an expert through training"

One issue this runs into: the machine becomes an expert at something that matches what it was exposed to during training. The research on adversarial examples shows that the thing being learned != what the expert learns, frequently to an extreme degree.

And anyone working on ML/AI who doesn't understand overfitting is selling snake oil.

Yes, these are two main approaches to AI, with expert systems being the "old" AI and ML being the "modern" AI. In the old approach, you knew exactly what you wanted to achieve, and very often you had a relatively clear idea how to do it; the challenge was to construct the solution manually. In the new approach you are not so much concerned with how to do it as you train on some dataset anyway. The results are less predictable though, you might even not understand why certain things happen. The number of mistakes/false positives can be relatively high.

Still, with these methods, I think we're very far from real artificial intelligence, where you can really learn something new on the basis of something old. I believe until we have a profound understanding of what it means "to understand", such an advancement in AI won't be possible.

I think we do have a good-enough understanding of "understanding" to characterize what is required. However this is to be found in the textbooks of neuroscience, not in the machine manufacture manuals of a silicon lithography plant.

There isn't as much mystery as there seems. When a neuroscientist asks "how does a animal perceive the world?" the answer is reasonably methodologically obvious.

When a computer scientist asks, "how does a machine perceive the world?" the mystery arises only because it doesn't.

There is a huge difference between perception and understanding.


> Marcus also observes “a bias in the field which is to assume that everything is learnt.” Marcus makes the argument that human beings do not learn everything by ‘trial-and-error’ – that part of our knowledge is innate, learnt over evolution – leading him to suggest that “we need more innateness if we are going to build intelligent agents … It’s not learning trial-by-trial in the way that our contemporary machines are.”

Well, not as much as you think.

Perception is structured by concepts which are learnt. Perception is the prototypical mechanism of understanding.

Insofar as we talking about "abstractions" we are just using language to model the conceptual structure of perception; and then higher order concepts to model the structure of these "perceptual" concepts.

We tend to think of understanding as being equivalent to the linguistic act of accounting for a higher-order concept, when this is really only something humans can do -- and then only with tremendous difficulty needing quite significant methodological assistance from others, etc.

This is the mistake Socrates makes when he asks for "definitions", as-if the dog doesn't understand the geography of where he lives only because he has no linguistic model of the concepts he's using. Or as-if the general has no knowledge of justice because he has no precise linguistic model of it.

When Potter Steward said in the pornography case, " I know it when I see it " he was literally correct. The concept "pornography" is part of seeing pornography, it is knowledge which structures your perception. The lack of perfect linguistic model for that is neither here nor there, and a bit of a blind alley for AI (and socrates).

Animals, including humans, understand a great deal more than any one person can model linguistically. It is this which structures our perception and allows even trivial engagements with our environment.

> When Potter Steward said in the pornography case, " I know it when I see it " he was literally correct.

Here you refer to categorization, not understanding. From the article I refer to:

> “Deep learning is good at certain aspects of perception, particularly categorization, but perception is more than categorization and cognition or intelligence is more than just perception. There are many things that go into intelligence … And what we have made real progress on is perception, but the rest of it, we still haven’t made that much progress collectively in the field.”

Understanding, real understanding, the "aha moment", is the very basis of new discoveries and real evolution. What we see now are systems quite good at categorizing, pattern recognition in a given set once prepared correctly and so on. There is no relation whatsoever with real intelligence, as in being able to make a new discovery based on analytical and synthetic skills.

There are quite a few voices in the community saying the opposite and maintaining that mimicking intelligent action is practically the same as the intelligent action itself, but there are quite obvious limits to what can be achieved in this way.

> The downside is it now similarly becomes challenging to formalize the machine's intuition

Tooting my own horn here: I'm taking up this challenge professionally. There are new methods being discovered to extract interpretations for humans out of a model[1][2][3]; and I'll gladly work and provide advice and implementations on this, as an independent contractor.

Having an interpretable/explicable ML pipeline helps in:

* Knowing what drives your customers' buying patterns

* Knowing where to look for signals closer to the root, helping go from correlation to causation

* Orienting your marketing messages. You don't engage users the same way if they're a repeat buyer or a "hot now" lead

* Concentrating on "would buy our product but won't because of X", which is where the market share battle lies; as opposed to "definitely would buy" and "definitely wouldn't buy", which are cases where you have little influence on.

* Listing buying patterns exhaustively, enabling you to discover low noise niches

* Providing auditable explanations of the decisions the model has taken, if it has been required legally

Here is my email: pro@benoit.paris

[1] https://distill.pub/2018/building-blocks/

[2] https://www.darpa.mil/program/explainable-artificial-intelli...

[3] http://interpretable.ml/

Newbie question: would you not design ML systems to be able to explain themselves? Alike: hey, query planner, what's your query plan?

This is often called "interpretable machine learning" and it's an active area of research. By default, ML systems just make predictions; they don't explain anything. And it's not obvious how to get explanations from them.

There is something close to what you're describing:

1) You train a complex ML model (say, deep neural net) to get the job done, solve your problem with high accuracy.

2) When you want to explain one of your model's results, you train a simpler more explainable model (say, linear regression) in the neighborhood of that point, such that it is locally similar to the more complex model. The task is too complex to be explained by a linear regression model over all possible inputs, but it often is simple enough around a specific point.

Is there any guarantee that every concept which a machine can learn can be represented in present-day human language in a manner compact enough to use?

One worry is that a machine might learn to do something we can't represent efficiently. I mean, people write whole libraries of books on ethics, so how can you expect a machine to punch out one or two lines on its own?

There's no upper limit on how long a proof to a given problem must be in any fixed language. And if you have to invent new language, people won't be able to read it anymore.

You could, there are many approaches to design interpretable/explainable systems (Here https://distill.pub/2018/building-blocks/ is a nice and recent review of some of these problems with illustrations), but there are serious tradeoffs so people usually choose not to do so.

The first issue is that, in general, explaining a decision is a harder task than making a decision, just as reviewing and understanding someone's decision takes more effort than it took to make it. This means that adding explainability and interpretability to a system that achieves some result will take more effort, time and money than building that system itself - you'll triple your budget and get no better actual results, just a better feeling about them. This also limits the effort invested in tools and methods for this; there's large demand and funding for making stuff work or making stuff a bit better, so many people are getting paid to work on that, and not nearly so much on interpretability.

The second issue is that best performing systems take "everything" into account and thus become too complex for a human to understand. A system can be explainable, interpretable and accessible, but if making a correct decision inherently relies on complicated interrelations between many factors, then it will be difficult (or at least time-consuming) for humans to understand anyway. If you limit the complexity of the system to something that can be easily explained, then you're severely limiting its power, and you get poor results.

For your query planner, the analogy is the scenario where we could build a query planner that is capable of optimizing beyond the commonly used basic blocks in query plans, instead outputting optimized machine code to perform particular query better than the standard blocks, achieving improved performance by merging and interleaving operations. The resulting output would be much harder to interpret, since no "block-level" query plan can be equivalent; and requiring the operations to be neatly separable in a limited number of blocks each with separate human-understandable meaning would mean restricting the system and inevitably getting a worse result.

The same applies to humans, by the way - being able to explain all the factors why you believe your decision is likely to be correct takes more knowledge, insight and effort than simply making that decision. It's not like humans can explain why exactly they see (or don't see) an animal in picture like http://www.naute.com/images/cow.gif , consciously understanding that in a structured way is very, very hard. If you would require a human to explain the criteria they'll use before they get to make a decision (as opposed to post-decision rationalization that often has no relation to why the decision was made, and has more relation to making up a socially acceptable plausible justification why you did what you did), then you get poor, simplified criteria that can't reflect the complexity of the problem; you could get their "rules of thumb" or their major factors, but explaining why exactly it is so is generally a mentoring task that takes months (if not more) of effort both from the teacher and the student.

AlphaZero system plays Go and Chess better than any human. It could be modified to describe in detail why it chose this move over the other, but would you (or any other human) have the mental capacity to truly understand it, no matter how well it's explained? The distill.pub article illustrates lots of ways how an image classifier can explain why it made one choice or another - but is it actually useful for tasks other than for R&D to debug or improve the model? It seems like a nice-to-have feature that would be liked if it's not to expensive, but I'd bet that for most models the company wouldn't actually use the explainability feature (again, outside of its builders debugging or improving it) because the whole point of ML is that people get removed out of the loop and don't look at the decisions.

Companies have a large problem of having their data tucked away or inaccessible to the stakeholders. When people talk about AI / ML what they actually need is their data cleaned to the point where they can communicate to their stakeholders. Also, all of the companies who sell AI / ML as consultants are really good already at cleaning data.

When companies actually hire data scientists what they typically do is clean data for a few months to a year . Then they interpret the data by probably being able to perform linear regression. At that point the data is in a state where it can be easily understood by those stakeholders and then they have created value. Whether or not the linear regression or whatever model has been learned may mean something. But, at the end of the day you need to tell stakeholders how they can create value and guess what SQL and Bash will do 90% of the job.

100%. What this all boils down to in 2018 is data. Data acquisition, data cleansing, data filing, etc. But what else is there, now? You're going to create a new, novel ANN network? Cool, can't wait to peruse your paper. Maybe someday it's considered groundbreaking. Right now it's not solving immediate problems. Immediate problems were solved with 1970s ideas + 2018's infrastructure.

A word against hasty generalisation - just because you don't interact with it on a daily basis doesn't mean it isn't being worked on elsewhere.

There is an increasing number of ML-first companies out there. They are solving problems like object detection, portfolio strategy optimization, medical diagnosis, etc. Things that are 100% not possible to do with just SQL and Bash, and much less expensive than to have humans run those tasks. In those fields, these companies will outperform their competition on average.

I think it's smart for investors to look for in knowledge and talent in the companies they look into. If there's AI/ML talent in the team, their value for acquisition (acqui-hire) is higher.

If you want to solve problems for your existing company, you certainly don't need AI/ML. You can tackle many, many low-hanging fruits with just SQL and Bash.

If you want to build the kind of company that will be around in 30 years, you need to have an ML-first mindset, and you probably need to start now.

Maybe we could all just agree that AI and ML have their use cases, but they are not necesarily a good fit to any and all computing problems indiscriminately?

My point is less that everything can be done with SQL, but that most of what people are doing is based on things possible 10, 20, 40 years ago. This leaves ml engineers dealing more with software modeling than ml modeling.

I think the 10% is exActly what these AI first companies should be targeting .

Yes, and there are some exceptional papers from the 1960's as well.

Agreed. The advertising agency I previously worked at would take the first month and only work on how data is collected and stored. After that first month they would begin addressing goals, plans, etc. About a year later is when true value would be realized because data that previously never existed could be analyzed.

Few things were more enjoyable for me than getting a new client, imagining what analysis I'd like to do, figuring out what data would be necessary, and then implementing the system to make it reality.

how did you convince them to stick on for a year when most clients want to see some results within 2 months or they give up?

according to your schedule they’d ask what I’ve been doing and if I tell them I’ve just been “collecting data” that translates to them as “I haven’t done anything in 2 months”. If I say “keep paying me and you’ll see results in a year” that translates to “I haven’t done anything in 2 months but I want you to continue to pay me for another 10”.

What do you do to qualify a client? How do you know your engagement with them won’t just waste everyone’s time when they quit halfway through and then damage your reputation?

Bigger clients (and intrepid smaller clients) are willing to commit to longer term plans. The key is to be clear about what it will take and what it will return. If it's the first time they have even thought about using a large dataset the ROI could be high, and it's a long term strategy to start collecting and using data anyway.

Bigger clients budgeting can also be conducive to longer term planning as well. In year one they can write it off as an R&D investment, year two it can start to make returns.

Clients who have been burned by less capable companies are also comforted by very well defined plans, so they are happy to hear you have actually accounted for the issues that likely "came out of nowhere" last time.

Companies with budgets are willing to think in terms of quarters or years. You’ve laid out the plan for them and they’ve agreed to it. The money is allocated in an annual budget or obtained with a purchase order, so it’s not going to be suddenly reallocated unless there’s a big problem. Experienced clients wouldn’t move to another agency to do analytics “faster” because they would know they’d just have to ramp up the relationship and the systems again.

well then in that case in many regards working with companies that have quarterly / annual budgets is easier than working with clients who do not. But it appears to be harder to make the sale if you don’t already have a legendary track record that warrants their one-year-commitment.

Making B2B sales is hard, that's unavoidable. Expect to spend literally as much manpower and budget on sales as you do on engineering.

Also, you need a proven track record - if you don't have previous clients, have a working proof of concept for something similar; which can lead you into signing them up for a 3-6 month pilot project, followed by a real contract or two and maintenance.

This also leads nicely to the pricing - this can not be done cheap, since if you sell, say, 20 man months of engineering to a customer, it's not enough to get paid for 20 man months of engineering they do; the contract also needs to pay for 5 man months of sales (including engineer time) it took to arrange their contract, the 15 man months of sales you spent on other, failed leads to get this one contract (I mean, most sales will fail), and for a part of 20 man months of R&D it took to develop the various proof of concepts and demonstrations that were absolutely necessary to start getting contracts but not paid by anyone.

No, analytics management is something many companies feel they need to pay for since the marketing team wants to show good numbers, they need to justify ad spending, web development etc. You do need a good relationship with a client to sell more than piecemeal work, but you don’t need to be “legendary” to get enough clients.

The other answers are good - For my experience specifically it was at a very very small agency, so the client chose us specifically for this purpose and knowingly agreed to it.

The first few months aren't just us working silently. It was always a very intensive process, I'd be on the phone with the client a few times a day to work through issues. They also get valuable tidbits of data early on. Most clients had ideas about how they expected data to look (70% of my revenue comes from product x) but often times the reality is different and they see immediate value.

Clients sign up for a contract that auto-renews every three months and they have to give 30 days notice to terminate. At a small agency you have the luxury of denying clients all the time too that you don't want to work with.

Even worse than stakeholders not having data is stakeholders having relatively raw data and building “reports” themselves with Excel that don’t actually mean anything. Sometimes this is just done to provide justification for something someone wants to do, but over time these things get built up and entrenched and the business is actually driving decisions off of completely meaningless metrics.

So while it may be tempting to simply hand data over to the business because “that’s not my job”, that can totally come back to bite you in the ass when your product roadmap is being dictated by ill conceived Excel formulas.

It gets even worse when they want an outcome that they themselves only believe instead of being impartial.

This is a very good point. Analysis can become quite trivial when you have clean data. With messy data you may need 3+ orders of magnitude of data which is often not possible. Unfortunately VC pays a higher premium for data scientist than data janitor.

Pretty much this. However I think that you can actually get pretty far with cleaning a surprisingly small subset of data and just quickly tossing together a prototype. Talking to people helps a lot. "Ok so you have this and that data...how helpful would it be if you could predict the price of item X that you buy 10% better"...well ok it would be worth roughly Y$/year. Let's build a small prototype and see how well we can actually predict it. At this point you quite often find out that "oh we more or less predict it via one person who makes the buying decisions by experience" and it might take some convincing to actually compare the two.

I'd summarize that the "big three" of ML/DL are all nontechnical

1) Finding the data/cleaning the data

2) Is there actually some reference you can benchmark against

3) It's not really all that hard and you might need less data than you think to show enough business value (broadly speaking a pretrained net + some usecase layers is good enough to get started for most business cases)

#3 is interesting. I tend to be quite honest but I've seen a lot of mumbo-jumbo hand waving "oh it's very tough" which of course makes sense if you want to sell shiny new stuff. Reminds me a lot of ERP sales where I've seen basically SQL+a little webfrontend pitched as rocket science :D

Where you can learn to become a data scientist? Linear regression and similar things? At what field you need to study? B.S. on Computer Science?

I thoroughly enjoyed this post, so maybe I'm biased.

I think AI is extremely overhyped and under performant. In fact, I think a major strength of AI is founded in the technical ignorance of certain project managers or decision makers. The type of person who doesn't appreciate the simplistic elegance of sql+bash/cron for simple tasks is the person who will bite a pitch for AI customer retention strategy. Customers are people. Business is people. You don't need a rack of gpus to understand why sending someone an email who has a saved cart is a good idea. It's common sense. It doesn't matter if we can force machines through trillions of operations to vaguely capture a customer pattern of a guy at a console can write it by hand in five minutes.

(not always, I know, I work in finance so a lot of my business IS machines and not people, but you catch my drift)

I'm pro-AI research, and anti-AI hype train. They're computers. They're objects. They're not us yet. Consider the magnitude of the AI research market, which is tens of billions, and compare that to what they are actually capable of doing relative to human performance.


Maybe HN skews my perception on what the public tech enthusiast's perception on AI is...

I agree completely, at one of my previous employers, the CEO of the company sent out an email, with a list of links, and said everyone should learn AI/ML, and it would be important for the future products. And he gave a number of examples of potential features that AI could achieve.

When I looked at it, every feature shown, could be more reliably delivered and have a better customer experience through deterministic behaviour.

So I agree, I think AI has made certain technologies way better, but I see it as a tool, and like any tool, it sometimes applies to the situation and sometimes doesn't.

I almost took a "big data" data scientist job about a year ago with a local company.

After talking to a number of their engineers, it became quite clear to me that instead of a data scientist, they just badly needed a DBA / someone with ownership and a complete vision of the data structure.

They had no foreign keys, poorly 'designed' indexes, and tons of redundant tables with no rhyme or reason to them.

They'd organically grown their database with hardly any review. They did not have big data, they just had a big mess. And wanted someone else to clean it up.

I'll bet they found they got 10 times more applicants for a data scientist role as opposed to a dba/sql/data engineer.

But what's to point? You will get a lot of applicants from non-cs backgrounds. Not only will they be unprepared for the job but also most likely very unhappy with their tasks.

The point is that you can get paid more by calling yourself a data scientist than a DBA, at the moment.

Maybe they should have a data warehouse (which is typically denormalized) instead of a database (which should be normalized)?

They had a data accessibility / source of truth issue. You need to normalize before you denormalize or else you're just spreading the mess into the warehouse.

This could've been a valid criticism of people that use ML where it's not appropriate, but it ended up being a bit of an irrational rant, and a dishonest one too:

> I mean, why send a letter with breast pumps to a man that just bought a pair of sneakers? It doesn't even make sense. Typical open rate for most marketing emails is anywhere between 7 - 10%. But when we do our work well, we saw close to 25 - 30%.

How do you know what items are compatible to each other? Why only recommend sneakers to somebody with sneakers, instead of also recommending sport clothing?

Oh, I guess you could build some type of topology of all your shopping items. But what about recommending soccer balls to people that bought soccer shoes? You could also add that to your database, but now you also need a heuristic to score item similarity: `category_matches * 10 + subcategory_matches * 5 + color_matches * 2 + ...`

This is the whole point of ML. People have been building rule-based systems built on "domain expertise" for ages, only to find that they are limited and cannot compete with simple algorithms fed with enough data.

But, that might be in the realm of SQL too. Find out what items were frequently bought with the item that this customer bought, and send them as recommendations.. Rule-based does not always mean that a user is sitting down writing that tennis balls and tennis shoes are related items. Don't you think?

Such a simple system would recommend many items that are frequently bought by everyone (like bread, toilet paper, batteries). You would have to weight the items in some fancy way to get useful recommendations... And I have just described the introducing slides of a applied machine learning university lecture.

This article and thread gives the impression that it’s dominated by people who don’t even have a rudimentary understanding of ML. If they think SQL is a replacement for ML, I’m really not sure what they’re doing in this field. ML is for making sense of data in a large number of ways beyond “hey let’s query a database for some trivial information”.

True. I don't think the OP or I deny that. The premise is that some practitioners use ML as an overkill for things that are simple enough to be solved by SQL.

Expert systems are brittle and don't generalise well outside the data on which were created.

You know, counting items and dividing by total number is a kind of machine learned model, too. In technical terms it is "Computing the maximum likelihood estimate (MLE) for the PMF of a random variable taking finitely many values."

But it's a poor man's model. That's why in order to solve complex problems we use stuff like neural nets and gradient boosting, and in unsupervised learning, matrix factorisation.

Well, we are in the peak of a wave of hype about AI/ML, maybe even just past that peak. Many fundamental technological advancements in the field of AI/ML have sort of coalesced together at the current time to form a strong feature set that can be more broadly applied by a wider audience, not just those hardcore computer scientists who invented the technology.

I've been in the thick of this previously, facing a complex rules-based engine that did most of its incredible feats in the fraud detection domain using a number of really complicated SQL queries. At the same time, I've used the results of such queries combined together with machine learning and predictive analytics, giving you the best of both worlds. Both have strengths and weaknesses.

These are tools in the toolbox, and I think the adage "try to use the best tool for the job" still applies. Sometimes, you use the tool you have and you know, and all the more power to you if you can get the job done using that tool. If you are a master of that tool (i.e. SQL in this case), you can often push its capabilities very, very far.

That said, I think the best thing to do right now is try to separate the signal from the noise regarding AI/ML and find what really works and what does not. Then find how these new tools can either complement or replace previous approaches. I think they work together quite nicely - and we see that sometimes, for example, with AI/ML tools integrated close to SQL engines.

AI/ML has a place, and so does SQL. I will say, though, that I for one don't want to be caught on the side of the discussion where I don't learn enough about what is possible with AI/ML, and then get left behind. I think many of my colleagues and professionals in the field and here on YC feel similarly.

Actually, I think even non-technical people feel the same way - the fear of being replaced by AI/ML is higher than ever.

So, keep applying SQL and get that low-hanging fruit. But make sure to learn the new stuff too, and add it to your toolbox.

imho, AI and ML are being applied to problems for which they are not the best solution, primarily out of ignorance by the practitioners, or by their desire to hop on the AI/ML train. I expect machine learning to steal some attention from data science, but in name only as at the end of the day it is all based on statistics.

>I will say, though, that I for one don't want to be caught on the side of the discussion where I don't learn enough about what is possible with AI/ML, and then get left behind.

I understand that feeling. On the other hand, aren't there a lot of trains you can miss? Do you try to guess which trains are the critical ones (and can you, really?) or do you try to keep up with all of them?

AI/machine learning, quantum, block chain...the list goes on. It seems like the best you can do is not be blindingly ignorant of any of the technologies and their potential...I think few people can really "keep up" unless they have the luck (or misfortune?) to have a job like "Quantum Crypto AI Researcher".

I disagree that this is the peak of AI/ML. Companies are desperately looking for PhDs in AI that can fulfill their business needs. I think we're about to see a lot more applications of AI/ML.

Companies are also looking for "blockchain developers to fulfill their business needs" :)

The original poster specified the peak of a hype wave. I think we are due for a bit of a collapse where the more fanciful claims fall apart but that seems healthier to me because it’d avoid distracting from the solid real applications.

But if you say "I'm going to use a bunch of shell scripts to parse logs" you are boring. If you say "I am going to use groundbreaking ML/AI technologies to transform big data into customer retention solutions", you are a visionary.

providing insights to your log data faster then ever before with machine learning big data

The biggest threat of AI is not its ability of taking jobs or exterminating mankind, but the amount of distraction it creates.

When politicians say they improved the economy like 30%, nobody buys into that. It's an overly exaggerated misleading political talk. But when some tech gurus talk about how AI improved their profit 30% or something, everyone seems to hop on. It's an effective marketing, for sure, but this is a worrisome trend. The root cause of this is I think the lack of proper understanding of fundamentals (and intellectual sloppiness). AI will continue to plague us on this front, and I'm still not sure if the net gain is going to beat all the distractions it created.

Yeah but non technical people who don't know what they are doing but for some reason have money to spend just know they want you to use machine learning for everything.

One time at work I wrote a simple web app with a search box (just doing an sql query, nothing fancy). One of the "higher ups" was impressed and decided to flex their knowledge, pointing to the search box saying "and this uses nlp". It was a damn sql query on a full text field.

As someone who sells both of these services, I can only add that it depends, and if you have a good dataset, it's trivial to write either one.

But once you start having to account for noise or seasonality or autoregression or dynamic weights or non linear kernel spaces, pure SQL really starts to fall down on the job.


OP gave a few examples for Ecommerce where SQL will do fine. Can you give a few where ML will do something otherwise impossible or harder with SQL?

Product Recommendations. Trending Items (Top items being sold this week as opposed to last week, while filtering out items that are generally popular.) Much easier with Elasticsearch than SQL https://www.elastic.co/blog/significant-terms-aggregation

That's not really ML.

If I were ever to make ML bingo, "That's not really ML" would definitely be on it.

I see, so just how everything became "big data" a few years ago, now everything is ML.

"Managing big data with MySQL" - the syllabus mentions nothing of clustering or sharing. Ten years ago that was just "using a database". I am getting to old for the faddish nature of this industry.


As a classic example, how about music or movie recommendations? You have a giant set of attributes about songs and a binary classification (thumbs up/thumbs down) from millions of listeners. You can use this this to train a random forest or neural net (or other algorithms) to make song recommendations to someone based on that person's pattern of thumbs up/down. I'd guess most recommendation systems do something along these lines, though I don't really know.

I'm not sure how I'd do this with SQL. That certainly doesn't mean you couldn't take a SQL approach to the problem of "how do I make recommendations based on this training set and this particular individual's attributes" and do something creative and useful. But I do think you can, in this situation, get something out of ML that would be difficult (ok, I'll say impossible) to get with a pure SQL approach.

To be clear, I love SQL. I still get a little flash of anger when I remember my arguments with "architects" who announced to a room full of non-technical people that I was pretty much a dinosaur unwilling to move away from an obsolete technology when they advocated no-sql approaches to data that was deeply relational. And I'm sure that there are plenty of people who think "need answer from data, must use ML" and end up somehow trying to train a neural net to perform a WHERE clause. Having seen the hype cycle several times, I am absolutely certain this has occurred in boardrooms/open office cubicle farms all across the you ess aye and beyond.

But keep in mind, even though reports of SQL's death were remarkably exaggerated, "nosql" approaches often do make the most sense. Plenty of good ideas and technologies came from nosql, and many of the people who created them and advocated them continued to use (and argue for) SQL and relational databases depending on the problem. A graph database is a vastly superior approach to some types of network problems than standard SQL, and not helpful for others. When the hype cycle around ML dies down (and some people start writing articles that ML is "dead" or whatever), ML will continue to be used effectively in all kinds of places, as it is now.

I didn't say that it was SQL, I said that its not ML. Just aggregation / scoring as far as I can tell.

It was clear that you were only saying that it wasn't ML.

I looked up a bit on elasticsearch's text classification, and it's interesting. There's a overview of "traditional" ML (similar to the song review classification I mentioned) and how elastic search differs.


"The MLT query is a very important query for text mining. How does it work? It can process arbitrary text, extract the top n keywords relative to the actual "model" and run a boolean match query with those keywords. This query is often used to gather similar documents."

Yeah, looks like aggregation and scoring. It'd be interesting to see if/where this outperformed various ML algorithms (accuracy as well as performance).

Well as I said it takes extremely intricate and not all that well performing SQL to perform signal processing and latent trend analysis, as it's not really designed with matrix multiplication or iterative transformation. So anything with noise, seasonality, ARIMA etc shouldn't be done in SQL, it should be done in a proper ML tool, because 99% of the time your developer will miss something and you'll improperly assign value to something that is in fact no pattern at all.

You can't recreate the Netflix Prize in SQL, you can't really do NLP, you can't do pretty much any clustering or unsupervised learning. Again, "can't" meaning "can't do it easily or in a sustainable manner (i.e. shouldn't.)"

And if we add in AI, I don't need to explain that SQL has no abilities to generate new content - it can't write a new song or headline, understand context and provide feedback, or drive a car.

Now what OP is arguing is that you can get a lot of lift with SQL and basic exploratory data analysis and I totally agree. But ML can take you further out into the speculative realm and can work on a much less explicit and prescriptive model than SQL.

Basically in SQL if you don't write it (whatever it is, a pattern, an anomaly, a correlation, a context) in your query you will never learn a single thing about it.

Feature extraction is the most obvious.

I give you a problem e.g. "tell me what factors are influencing a customer's likelihood to leave a bad review". And with ML you can actually produce a list of sorted factors with weighted percentages. You simply can't do that with SQL.

Also remember that SQL is for databases so you often can't do anything algorithmic e.g. K-means clustering, linear regression, random forest, decision trees etc. Where as ML encompasses non-databases as well e.g. from a Kafka stream of numbers predict the next number.

Forecasting and prediction. Specific example: given a high quality dataset, forecasting sales is a good use case for machine learning. Of course, you could do this with SQL as well, but again, given a good dataset and a complex domain like sales, some relatively simple models will be hard to beat with a 10 line SQL query.

Sorry, I disagree with you. Let's take forecasting. This is a _very_ hard topic and being applied correctly using actuarial methods it will beat ML. To give you some example, you cannot take sales data for IPhone X and predict its future sales. Reason is simple - market will change, IPhone 11 will come out cannibalizing IPhone X sales. So, you need to have 2 more variables into play. One is the lifetime of a product (approximated) and second is, you need to "put in one basket" several products. Having this, you can predict your sales for some specific product group, no magic ML needed. However this requires you to define "similar products". This is hard because currently it is done semi-automatically using human labor. In fact ML is actually the best one to automagically categorize products into groups, however I have not seen any implementations. To summarize, ML do has value, however not where everybody is pointing out.

Give me all the time frames when there was statistically significant bump in X.

Anytime you need to make sense of unstructured data like images or text and drive an automated business process off it.

Getting funded


staying in business

controls and sensing.

While I see the author’s point, I fail to understand what any if this has to do with SQL. The problem ML solves isn’t querying databases, it’s making decisions. If a human came up with the idea “let’s lookup people X and send them email Y” and it works, great. But a human made that decision, and SQL is just a tool for making it happen. If you want to take the human out of the loop, SQL won’t save you.

I don't think you understand author's point.

His point is highlighted in the first tweet, in which the author appears to be specifically annoyed by the potential founders and investors that can't understand that ML isn't a good solution for all of the problems.

He then goes on and gives an example of such problem by explaining a shopping cart that doesn't actually need ML, but just some old-fashioned SQL. He doesn't claim that SQL is a solution to all ML problems, just this one.

(I'm not the parent commenter who you replied to, but I think I understood what his/her point was).

Taking the shopping cart example: "In a former life, I used to write SQL to extract customer of the week. Basically, select from orders table where basket size is the biggest."

The author decided that 'customer of the week' will be selected by 'biggest basket size'. Not by 'biggest $ amount spent', 'fastest time from add-to-cart to checkout' (and numerous other attributes or combination of them). This decision (the "best attribute") was taken by a human, leaving a field open where a combination of attributes could've resulted in overall better business outcome (how much did 99% of these retained customers shop for, in $ value over lifetime?, etc)

This is possibly what the parent commenter is hinting at - this human decision leaves a lot of optimization scope, where ML could have helped.

A huge part of the "data revolutions" that we've seen in the last few decades really has nothing to do with data and everything to do with process.

Data Warehouses changed the way people and companies do data. They expose all kinds of things that were never available before. It was magic!

No. It wasn't. Not that Data Warehouses are bad or ineffective. But it's a lot like the problem you face when you observing something changes it. The work you have to go through to build a real data warehouse is that you have to get disparate parts of an organization to codify process. Data warehouses don't model data. They model processes.

The mere fact of forcing the company to pin the process is often more beneficial than the warehouse itself.

The same thing goes for ML and AI. The only way to extract features is for them to actually exist. And that means the data needs to exist in a certain form, and there's a human process that leads to that. Absent that, it's pretty useless.

I cut my teeth on SQL, and it's a big part of my professional career. I think it's great. It's one of my favorite languages, and it does a lot that maybe a lot of people don't know about.

But this title and the content are really pretty garbage. Anyone who thinks that good SQL can do what good AI/ML can do is really misunderstanding both.

Hi, I'm the guy that wrote the tweets. Let me know if you have any other questions. I'm happy to answer any question.

eh. I built a lead gen system at a fortune 1000. The heuristic SQL version brought in 10M a year. The random forest version brings in 100M a year. It saw things we didn't

can you elaborate?

Built a lead generation system. Looks at searches on our site and picks out the best people for our sales team to call.

Set up a bunch of rules created by the sales team. Tweaked it over months. Made money

Then used real sales data tied back to search history and built a machine learning model. It found new patterns that the sales team hadn't thought of, and performs much better

"Set this as a CRON that fires at 2AM everyday, period with less activity and traffic. People wake up to emails reminding them about their abandoned carts"

Hah I wondered why I got so many notifications in the middle of the night. Now I know that it's from people who think they're helping - not realising that it actually sours my opinion on their company/product.

You get a notification beep from emails? How about just turning that off? If anyone really needs to reach you they can just call you, no?

I get a vibrate notification. And you are right that there are steps I can take to stop this if this became truly disruptive. However it is a mere annoyance, but annoying enough that I think "seriously, fuck off!" if it happens which presumably is NOT the sort of engagement they are looking for :-)

I mean turn it off fully at night, actually. Notifications are a psychological hack–our brains get a little endorphin rush off seeing or hearing them, which makes us want more. They don't serve any actually useful purpose, though. You could stand to not know about new emails, Twitter and Facebook replies, etc., until the morning. If it's an emergency anyone you care about can already reach you by phone.

App notifications are just programming you to be psychologically dependent on them. You can break free of the programming.

Or in the middle of the day for me, compounding the reputational damage by not thinking globally.

What time would be better?

It depends? It's clear that the time chosen is important, it will influence the likelihood that the recipient will go back to your store or not. For example, if they get the notification while on their way to work, they're likely to set it aside because they won't open the shopping cart while driving; you want to "hit them" sometime when they're sitting with their device of choice and would have a spare minute to do the things you want them to do. That time wouldn't be the same for all people.

And the choice to do this properly would depend on your volumes. If you're a small shop, you just pick one time - definitely not 2 AM, but, say, 11:30 AM (so office workers can do their thing in a lunchbreak) or 8 PM when they're likely to be home; it depends on your target audience. If you have a distributed client base, you'd want to take time zones into account. And if you're large enough so that small changes in this result make enough money to worry about it seriously, you might even do some ML to pick the optimal reminder time for each customer; e.g. training a predictor on what factors will influence the 'desired action' chance in a "multi-armed bandit" approach to explore the options initially and then start using the ones that work best. That's obviously overkill for most companies, but for large online retailers that would be a natural choice, it all depends on scale.

6am or later? Even if they played it safe with the usual business hours of 9-6 in the users locale (since companies generally know your location) it’d be fine. I'd be relatively forgiving for a US company who doesn't really know where I live sending me stuff at these times - they've no way to know what is "sensible". However I was getting pestered by Vodafone in my country at 2am nearly daily for a while

This is why I set "do not disturb/alarms only" when I set my alarm before going to bed.

The writer seems to describe very basic data mining in some cases, which in itself is a form of AI/ML, but then other examples have no relevance to needing to use AI/ML at all.

If their data is already clean enough for SQL queries to work reliably and they are familiar with the SQL syntax, why not look into things such as DMX in MSSQL to make predictions on what these customers are likely to want to buy. This solves the whole marketing breast pumps to a man who bought sneakers scenario, while it also providing more personalized recommendations.

If your current technique is to send an email about sneakers to recent sneaker purchasers, do you really thing they are in the market for another pair?

Sure, it might not make sense to implement a deep learning neural network just to send something like a semi-personal marketing email but their are so many varying levels of AI/ML that seem to get ignored in favor of the flavor of the month Tensorflow/IBM Watson/Whatever else. Quite frankly, the whole thing just comes across as a very closed minded rant from someone who isn't interested in exploring what new technologies are capable of.

Good points but really a false dichotomy. The purpose of AI and machine learning is to find patterns in data that aren't simple heuristics like this.

The problem with SQL is that eventually you will end up with thousands of SQL scripts. Have you ever tried to debug a 100k SQL? It’s a nightmare. Some of the scripts used to be simple, but got too complicated due to new requirements like this article doesn’t mention how he would deal with multiple time zones, currencies, different type of customers, multiple promotions for repeat customers and etc.

Are you suggesting that AI/ML makes those new requirements go away? Or that managing those requirements becomes easier because AI/ML software figures it out?

Not that I disagree with you, but does machine learning solve any of those problems?

SQL is annoying to debug, ML is impossible to debug.

I don't get this article at all. The author does not really back up their argument with any examples of ML. What in the world does common marketing practice & seemingly basic SQL queries have to do with AI/ML? What am I missing here? To me, this just sounds like a "Get off my lawn" type of rant. "Why do we need the newfangled AI when we still have good ole' SQL & bash!(waving fist in the air)"

On the other hand comments are talking about hiring data scientists for months if not a year or more (yikes!) To clean data & wait for it ... perform linear regression. To me this sounds like a great application of machine learning. Couldn't someone train some models to clean the data, then do one of the things ML does best, linear regression, in a fraction of the time the human data scientists could do it in?

Data cleaning is reeeally messy. It requires a lot of training data to get a system that does even a bad job of it automatically. So you still end up needing a lot of clean data, and getting the system to that point probably isn’t worth it if you’re not one of the biggest tech companies (you can spend time creating a system to clean the data or just clean the data). But your other points are spot on. This article is garbage.

I see what you're saying about cleaning the data. This is something I'm very unfamiliar with, so good to know!

Yeah I think the article is garbage too, makes me wonder why it gained so much traction? The argument/ topic are not developed at all.

I guess the point they were going for is there are people who want to use ML because it's 'trendy' or something, and simpler solutions would suffice. I could see that being true, but this article is BAD. I hate seeing low quality articles get rewarded.

Who on earth are these people describing ?

I've never heard of anyone hiring expensive Data Scientists, spinning up Spark/H2O clusters, building a data lake, doing a database offload to S3/HDFS all for a "select from orders table where basket size is the biggest" query.

AI/ML doesn't even work like this. It's simply not designed for giving 100% accurate answers to highly structured queries.

Anecdotally[1] speaking, you're correct but you're missing the message. You're correct that no one should hire expensive data scientists for this. But what happens is that there's no marketing against these sorts of pragmatic best practices, so it never comes on the radar of business executives making decisions. Instead they're inundated with ML/AI/Data Science pitches and mentions everywhere. And so when they go to invest in improvements like this, they reach towards the buzzwords they know instead of the solutions they're not aware of.

What ends up happening is effectively the Data Science engagement becomes 90% data cleaning, a handful of SQL statements that should have existed beforehand but never did because the data infrastructure wasn't there, and possibly a veneer of ML/AI just to say it was used. Clients come out happy (sometimes), despite overpaying for what was a much more basic engagement than they think it was, and they go on preaching to their business exec friends the virtue of ML/AI and the cycle continues.

[1] I built up a Business Intelligence/Analytics team at my last job, and currently work for a marketing agency managing digital analytics for Fortune 50 clients. Lots of exposure to analytics in lots of varying environments, and I've seen firsthand how ML engagements get pitched and results get presented.

I also own a consultancy that's the anti-version of this phenomenon, offering digital analytics management and support services. 50% of my work involves being a knowledgeable resource for marketing and business execs to lean on to cut through the bullshit. With most of the rest being basic Google Analytics/Google Tag Manager management, CrazyEgg, and drip marketing campaigns. All of which seems like AI-level magic for clients when done correctly.

Interesting comment, makes me wonder if people just generally underperform. You hire an "AI" team with multiple PhDs, you get work that a DBA could've done. You hire a DBA, you get work that a PHP intern could've done. You hire a PHP intern, you get work a kid could've done with Excel formulas :P

It's less about over/under-performance and more with incentive alignment and political perception, and organizational maturity.

It's really, really hard to get executive budgetary approval for a foundational data audit/cleaning project (comprehensive data cataloging, data cleaning, source auditing and validation, etc). Doing so implicitly admits that you weren't doing that before, and now you have to pay gobs of money to fix it. The larger the company, the more infeasible it is to push this through because of the breadth of technical/analytical debt that has accrued and the price tag associated with the project, combined with the perception of incompetency (i.e. it's an expensive project that's fixing a problem you as the executive shouldn't have let happen to begin with).

Whereas an ML/AI project/initiative/push is a net new capability that you're spearheading, and it's easier to get the political traction to spend money on net new things, especially buzzwordy net new things that the firm can use to be viewed as cutting edge. The fact that you're rolling up the cost of a complete data management audit to be able to even do the ML/AI project is a minor bullet-point that doesn't matter. Executive expectation is that anything ML/AI/new-age-techy is going to be astronomically expensive anyway, so it doesn't get noticed that they're paying a premium on labor to do it as a combined project rather than as two separate projects.

Effectively, the foundational work that's needed to support ML work is also the work that's needed to do basic SQL-based analytic work, but it's way easier to get that budgetary line approved in a flashy ML project, even if you're paying a premium by having the ML/AI firm do the foundational work instead of the specialist work.

Plus, spearheading ML/AI initiatives make for better resume points than "data management initiatives". So there's little reason for anyone in this process to attempt to change anything, unless you happen to materially benefit from a firm's profit. For this reason, my main consulting clients are bootstrapped firms that actually care about being pragmatic over being trendsetters.

Note: This is a huge generalization, and doesn't apply universally. But it's far more common than you would expect, especially as you veer away from the type of companies that pop up on HN towards more traditional industries.

> Who on earth are these people describing ?

I was asked about my thoughts on AI/ML at work, I said it didn't really apply to us. I was told "but with ML we can figure out when deliveries are happening and scale the machines before the deliveries happen based on the peak traffic times". I tried to explain that we could so all that from SQL and looking at our data. We have all the data we just need to formulate it into something that makes sense to predict which times of day, days of week, for each region, where we have more traffic then use that data to pre-scale. I was shot down to "you clearly do not understand ML and should go read up on it".

The catch it the “and looking at our data” part. ML is basically a collection of thorough ways to look at your data, understand the patterns and infer what that means for the future.

In your example, you should absolutely start by cleaning your data up and run some basic SQL aggregations and plotting volume over time. So you look that that and notice (1) volume is increasing over time, (2) some holidays bump a few days ahead but drive very low volume day of (3) weekends are higher, but the effect isn’t pronounced the whole year and (4) summer is better for you than winter except for the Christmas season. Now: it’s two days before Halloween, what’s our anticipated sales volume?

If you baked all those observations into an ARIMA model, it’s trivial to crank out a forecast with quantifiable accuracy. If you just have lines on a graph, it’s hard to pin down all the independent effects and recombine them for arbitrary scenarios.

Haha, I wanted to have a bar graph which were by day of week:

Bar 1: Last years deliveries.

Bar 2: Predicted this years deliveries (based on % of increase from other markets if no previous years, or percentage of growth from last 2 years)

Bar 3: Actual deliveries.

Then another graph:

Bar 1: Average processing time from previous year.

Bar 2: Average processing time for current year.

I'm terrible with graphs and such tho, I can get all the data, I really suck at displaying that data tho.

(and also show the holidays in that region, India has a lot!)

You're absolutely on the right track. There are well-established statistics tools that formalize your intuition and carry it forward to its logical conclusion.

> "by day of week"

This means you have a time series with daily resolution (one observation per day) and you expect Weekly Seasonality to matter. Model this as 7-period lag in your daily series.

> based on % of increase from other markets if no previous years

There are multiple markets, each with their own time series? Congratulations, you have Panel Data [0]. Do some regions have similar trends? Need to account for that correlation, maybe try a Mixed Model [1]

> percentage of growth from last 2 years

So there's a trend component (constant growth over time). Easy enough to fit the I term of an ARIMA model for this. You'll need to do some custom work to integrate this with your cross-regional correlations though.

> Average processing time

You'll want to model this at least as well as you're modelling the demand. That means seasonal effects, correlations between factories / warehouses, etc. PS any time you're dealing with a two-day weekend you'd better use at least 3 years of data in case major holidays happened to fall on a Saturday and Sunday, blindsiding you when it shows up on Monday this year.

> show the holidays

You'll definitely need to put together a calendar of major holidays for each region. These models will calculate the effect for each holiday. Specifically, the effect of the holiday AFTER accounting for the day of week, overall growth, time of year (season), and region. You might even get nifty charts like [2]


Anyway that's the basics. You can take graduate math courses in just this kind of modelling. Easier - you can contract a decent statistician for a couple weeks to build the model for you. They'll be delighted that you can produce SQL queries with the relevant data, and their models will help you get a lot more value out of those queries.


Resources 4 U

Generic time series reference:

[0] https://en.wikipedia.org/wiki/Panel_data

[1] https://en.wikipedia.org/wiki/Mixed_model

[2] https://assets.digitalocean.com/articles/eng_python/prophet/...

[3] https://en.wikipedia.org/wiki/Autoregressive_integrated_movi...

R tutorials:

[4] https://www.datascience.com/blog/introduction-to-forecasting...

[5] https://cran.r-project.org/web/views/TimeSeries.html (Particularly the "Forecasting and Univariate Modeling" section for your problem)

Python tutorials:

[6] https://machinelearningmastery.com/arima-for-time-series-for...

[7] http://www.seanabu.com/2016/03/22/time-series-seasonal-ARIMA...

[8] https://www.digitalocean.com/community/tutorials/a-guide-to-...

> We have all the data we just need to formulate it into something that makes sense to predict which times of day, days of week, for each region, where we have more traffic then use that data to pre-scale.

But then.. you would be doing ML ;)

Don't love your tone, but I agree. I have been working in what was called predictive analytics for 16 years.

I've done tons of projects, for tons of companies, and this sort of refrain from people is pretty common when they don't have experience in the field. They think of it like some fad that doesn't make a lick of sense outside of a C-level discussion.

But the reality is, predictive analytics is extremely powerful. One of the last projects I did was to save Trains from derailing. Another was to improve crop yield of a farming company by using satellite imagery to determine when a field was most needed to be harvested. Tons of other examples.

To even explain the particular use cases would take quite awhile because they are domain specific issues. Cron isn't solving these problems.

What the person is really saying is, I don't have experience in these topics, what can be so hard about them?

The same style for people who arm chair sports, or politics, or programming, or any other topic. It all seems easy when you don't know the details.

Didn't you use expert systems 10 years ago for those? Rules running of SQL queries, which you crafted yourself or through the help of a domain expert?

I think its fare to claim that companies who skipped that process might want to consider it first, as a cheaper way to start with predictive analytics.

But, I'm not sure, I am actually intrigued, what was the techniques used before ML in that field otherwise?

custom c++, pulling data from a db, if that's the SQL rules. Data scientists, plus domain experts.

These people are describing 99% of the Fortune 500 companies who have no idea what AI means other than hiring a team of data scientists that will hopefully solve all of their problems in the name of technology.

I've worked for half a dozen Fortune 500 sized companies within Data Science teams.

Nobody, repeat nobody is spending tens of millions on Data Science programs for answers to problems that a Data Analyst could already do.

If you know names please be specific but what you are saying is a bit ridiculous.

A far greater percentage know what it is, and use it than you are giving credit for.

Man, I am really curious what position you hold that you know the AI strategy for 99% of Fortune 500 companies. Those same Fortune 500 companies would pay you a lot of money for this level of insight into their competitors.

Wow, at least one other person shared my opinion. Clickbait article/post title. Also, the use-case for simple sales coupons mentioned in his article are not even close to the kind of things a company like Amazon does AI on in the recommendation process of WHAT product to advertise to each customer (as mentioned by another poster, I would not expect to see a discount for breast pumps when I just bought something for myself-male-such as men's deodorant).

People who don’t know what AI actually is and buying it anyways. I’ve actually seen this first hand. The developers/data scientists involved simply did what they were asked even though it didn’t make much sense (we had tried and failed to explain why this was a waste too many times and got nowhere)

Although to be fair, this outcome was still an improvement. At least with using machine learning unnecessarily the data actually meant something and wasn’t just arbitrary excel numerology.

Nobody knows what AI is though.

Even Data Scientists couldn't tell you if it just means neutral networks or if it include ML techniques. There are technologies like AutoML which automate feature engineering but is that ML or AI. Not sure.

So I am not concerned whether people know AI/ML or not. What I have issue with is people thinking that you can 90% of AI/ML using SQL. Which makes no sense.

Maybe they're describing a case where the systems have been well designed so they already have the data they need in a useful format and don't need to do any of that bullshit to make it usable.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact