
Show HN: Visual SQL - thingsilearned
https://chartio.com/blog/why-we-made-sql-visual-and-how-we-finally-did-it/
======
trts
I've seen a lot of products similar to this, and it looks like one of the
nicer ones. But the use case in the video and screenshots make me immediately
think it wouldn't be viable in any of the organizations I've written SQL for
in the past 10 years.

At least 95% of the challenge in querying data doesn't come from the
difficulty of writing SQL statements IMO, but from the complexity/brokenness
of the data itself. Especially with the move to unstructured data streams,
most data warehouses seem to have become pretty bad when it comes to
extracting truth out of the information within. Many business users know some
SQL but blanch at creating coherent reports from what's available to them.
What BI people are paid for seems to be having knowledge about the problems
particular to a domain.

If your data is relatively clean and follows a good model, this would be a
great way to help someone join, group, and aggregate their data without
knowing SQL. I think a lot of people would use Tableau for this if it didn't
also have a steep learning curve / high expense. However as soon as you get
into data where you're writing case statements, coalescing null fields,
matching on different data types, decoding, partitioning over streaming data
etc., it doesn't help someone without knowledge of the caveats within the data
sources themselves. Show me someone who doesn't know SQL using this to produce
insight out of compromised data and I will be impressed.

~~~
mulmen
Spot on. My favorite way to summarize this is:

SQL is easy. Data is hard.

If you aren't investing the work up front to make your data useful then no
amount of tooling or magic beans will make writing meaningful SQL easy. If
your reports (SQL) are hard to write it's because you have bad data.

Having said that making data exploration easier is always a worthwhile
exercise. The more business people can self-serve their questions the better.
It falls to data people to make that possible not just with tools but with the
data itself.

~~~
hcarvalhoalves
100x yes.

Questions to be made on data are always an afterthought, after the systems to
gather and produce this data have already been designed.

The idea that it’s ok to just give a BI tool to business people later, and not
involve them at the beginning to inform the system and data model design, is
maybe why we’re in the tar pit.

A really excellent process/tool would help business figure out what kind of
questions they need answered, and work backwards to the data model and
implementation.

~~~
koreth1
Sometimes it's not for lack of trying, though. I've done a decent amount of
data model design in my career and as I got more experienced, I learned to
make a point of asking the business people what specific questions they'd need
the data to answer. Very often I got only very vague answers that were of
little to no use in informing my design. I've seen this repeatedly at a
variety of companies over the years. The business people are often winging it
as much as the engineers are.

------
thingsilearned
Dave, founder of Chartio here. We're so excited to launch what we call Visual
SQL today. It's been a lot of work based on customer feedback and extensive
prototyping and user testing. If you'd rather skip the story attached here you
can also check out our product walkthrough video or give it a spin yourself
here:

[https://chartio.com/product/visual-sql/](https://chartio.com/product/visual-
sql/)

~~~
russell_h
Congratulation on the launch!

I sit in a monthly "metrics review" meeting where we mostly speculate about
why the dashboards are broken, and what the data team will have to do to fix
it.

If our TPM was able to self-service this through an intuitive interface it
would be a massive productivity win.

~~~
BenS
Same here! Seeing how useful tableau (a far less powerful tool) has been for
our company, it's clear that there is a big market for empowering everyone to
dig for data-driven insights.

------
mstade
Looks nice, but like others here we need something that can be self hosted or
even better – run offline.

Somewhat related: does anyone know of any good components/libraries that can
be embedded in a web page to let users interactively build SQL queries of
reasonable complexity? By that I mean something that not only lets you pick
columns and filter and maybe group things, but also do joins on other tables
as well.

Edit: just to clarify, it doesn't have to be be visual only. In fact,
something that lets you edit a query interactively either by visual components
or actual SQL code is perfect, especially if it's bidirectional so that edits
in the code also show in the UI and vice versa.

~~~
hermitcrab
If you want an offline, no code, visual data transformation tool, please check
out our new product:
[https://www.easydatatransform.com/](https://www.easydatatransform.com/)

~~~
mstade
Thank you, I'll take a look!

Just want to clarify – it doesn't have to be no code. In fact, a bidirectional
workflow where you can edit the query either by writing SQL or clicking
buttons is ideal.

------
aerovistae
It would reallllly be nice if you set up a web demo. I looked all over trying
to find the link that would let me just TRY this marvelous thing you're
telling me about, but it's all behind sign-ups and accounts and I just want to
_see the product_. Not in a video-- I mean play with it.

Can you make a demo session with dummy data for people with play with?

~~~
thingsilearned
Ah yeah, you do have to signup to try it out (though it does start you off
with demo data). We wanted to offer a non-login version but it makes things
more complicated for a company that obviously needs to prioritize security
highly.

This video is maybe the closest you can get without trying yourself -
[https://www.youtube.com/watch?v=YBXMTipHGfQ](https://www.youtube.com/watch?v=YBXMTipHGfQ)

~~~
eterm
Is the security story why you prevent signups from gmail.com also?

~~~
thingsilearned
No, sorry that's a marketing/sales team decision. I'll talk with them about
it.

~~~
aloisdg
Why would they want to ban them?

~~~
nujabe
They want users to signup with their company email. I think a few companies in
the enterprise software space do something similar.

------
ken
As someone who used to work on ETL systems, from a distance, this looks
basically like Tableau to me. (My first thought was actually "oh, neat,
someone made an open-source Tableau clone!", but it's not open-source, is it?)
You should describe how it's different.

If I'm going to pay money for a _hosted_ application for serious work, I'm
going to tend to prefer an established service that I'm confident will be
around for a while. I'm sure you've got something good here but I'm not sure
what it is.

~~~
thingsilearned
We do compete very directly with Tableau and are often used along side it as
well. In the article I explain a lot of what's different about it from other
BI products like Tableau. The main things are - it's way more flexible as a
visual language, and through a lot of prototypes and user tests has shown 80%
of business users can write significant queries on their first try.

As for an established service - Chartio's been around for almost 10 years,
we're profitable, and are the main data interface behind some really great
companies and brands
[https://chartio.com/customers/](https://chartio.com/customers/)

------
mhink

        Over the years, we’ve found that even power users who know they’ll eventually go into SQL mode prefer to start in the visual mode, as the grouping, date formatting, and joins are all done automatically.
    

This is huge, right here. I can use and understand SQL well enough, but the
syntax annoys me _constantly_. Avoiding that would be a huge boon.

~~~
thingsilearned
Yeah - changing date formats, double typing things you're grouping and
ordering by, remembering the oddities of each dialect, typing out full join
paths - it's a nice experience to have that done automatically.

Right now the SQL we write is very proper - with quotes around all of the
column names and table names listed before each column name. It's not what a
human would write. I'd love to one day make it a little more human so that the
switching into SQL mode will feel even better. It'll be a fun project.

~~~
infinite8s
What do you think of Looker's idea of turning SQL into a higher level analytic
DSL (which can be written/implemented by data engineers)? This DSL then makes
it easy for end users to do a more complicated analytics.

One issue I've had with pure SQL for analytic purposes is that a user can't
build a library of higher-order functions that can be parameterized (imagine a
window function to do a cumulative returns across various time periods,
applied to different underlying source columns), so its difficult to build up
a common business domain language without taking the typical approach of
constructing SQL through glueing string fragments together typical of most
programming language SQL libraries.

------
wayneftw
All of my other SQL tools are running right on my machine. I hope there will
at least a self-host version of this (or something just like it) available.

It would be amazing to have something like this embedded in Azure Data Studio.

~~~
thingsilearned
Ah, unfortunately there isn't. We're 100% cloud hosted - though we call
ourselves a Hybrid cloud solution because through our reverse SSH tunnel
connections (where your servers SSH into ours) we also work well with on-prem
data.

If security is your main reason for wanting self-hosting you may be interested
to know we're also SOC2, HIPAA, and GDPR compliant.

~~~
ulkesh
That's great you're compliant with those! For HIPAA, do you enter into a BAA
with prospective customers who deal in HIPAA data (PHI)?

EDIT>> The implication being that you may not be HIPAA compliant if you don't
enter into a BAA. IANAL, so I defer to the experts, but I figured I'd ask the
original question anyway. If you are above board on all of this, I will
definitely take this tool to my boss.

~~~
ecpreston
yep, Chartio does enter BAAs with customers to deal with HIPAA data! I worked
with several healthcare customers and lawyers here to make that happen. Happy
to connect and share more.

~~~
ulkesh
Awesome, thanks for the info! I will gladly forward this up the chain to my
boss, the first chance I get. I think this tool would be great for us, just
had to be certain about that. I would love a self-hosting option, but I
understand that your core business model might not allow for such an option.

------
ulkesh
Can we self-host, please? Is there a standalone app? We deal in PHI (HIPAA)
and we cannot have a tool like this have any access to our data store unless
it's private and secure.

~~~
yingw787
I think there are health-specific AWS clouds; are they strictly client-side,
or is there a managed instance you can deploy from AWS marketplace?

[https://aws.amazon.com/health/healthcare-
compliance/?nc=sn&l...](https://aws.amazon.com/health/healthcare-
compliance/?nc=sn&loc=3&dn=2)

~~~
ulkesh
We already have a live product using servers from a host that specializes in
HIPAA. I'm not asking about how to host HIPAA-related data. I'm asking
specifically about being able to use this tool. I'm not going to hand over
credentials to our database to an untrusted party (meaning, not our company).
So the hope is that we can either self-host this tool, or use it as a desktop
application.

------
davidjhall
>Please enter your business email address. This form does not accept addresses
from yahoo.com.

But my business address, right now, is @ yahoo.com

Or, I don't want to give you my real business address yet because I don't want
spam at that account.

Or...

~~~
minikomi
... you might work at yahoo?

~~~
bmn__
[https://news.ycombinator.com/item?id=18395589](https://news.ycombinator.com/item?id=18395589)

Then you would have a mail account @yahoo-inc.com

------
duxup
That is super cool.

But as a sort of tangent / related thing:

As a visual learner I don't feel like I usually have problems visualizing SQL
(that doesn't mean I wouldn't have use for this product, quite the opposite).

On the other hand I personally have been playing with Firebase and OH MAN do I
have problems visualizing what is going on with document type databases, I
start writing code and think of the DB and my brain starts sputter like
there's sand in the gears ;)

~~~
thingsilearned
To be clear - this isn't about making visualizations from SQL results (though
that does happen). This was us making a visual query language. It's a very
flexible interface that writes SQL for you - and has been proven to be
intuitive enough for 80% of business users, so they can write SQL now too.

The interface right now only works for structured data sources and my brain
also starts to sputter when I think about making a visual language for
anything not structured :).

------
navs
Awesome app. I'm keen to give this a go. I'm currently running my data
visualisations using python and plotly. Flexible but tedious sometimes.

------
samokhvalov
Looks good!

Worth noting, the idea of "visual SQL" appeared long, long ago, it was called
"Query by Example"
[https://en.wikipedia.org/wiki/Query_by_Example](https://en.wikipedia.org/wiki/Query_by_Example).

Congratulation on the launch and thank you for having PostgreSQL support.
(Side-question: any reasons why you have mirrored Postgres logo?)

~~~
thingsilearned
Thanks! Hadn't noticed that the icon was mirrored, thanks for pointing it out!

------
mjirv
I like the user-research-first approach here a lot. Very different than many
other BI tools in a refreshing way. In the past I've hesitated to consider
Chartio for my organization because most people don't know SQL, but I'm
excited to give it a try now.

I also love how everything is a table. Looker does this, and it's great
(though Chartio's implementation looks better). Tableau does not, which makes
certain things a headache.

I do wonder how well this will scale to large datasets and columnar databases,
though. Showing a preview of the tables is really cool and useful, but it is
not what data warehouses like Redshift are designed for. The preview probably
wouldn't load in time for my users to see it unless they just sit there
waiting for it.

~~~
thingsilearned
Thanks, the user-research-first part was definitely enlightening and important
to do. Our first many iterations were very wrong.

As for large scale datasets - we do cache those sample tables and we only grab
the first 10 rows. Also, BigQuery actually has a great API for fetching a set
of sample data that we utilize heavily. They made it because of exactly the
potential concerns you outlined around columnar stores.

~~~
mjirv
Oh neat, I didn't know that about BigQuery. Thanks for the tip!

------
ta1234567890
Looks like a modern version of Microsoft Access :)

~~~
phkahler
I didn't see the visual joins that you can do in MS Access, though I just
looked at the first page. If it's anywhere near that useful this product is
way overdue.

~~~
thingsilearned
Yup - if you grab columns from multiple different connected tables (or
schemas) the join will happen automatically for you and can be adjusted to
different paths. This is actually a really hard problem, especially since 70%
of datasources these days don't have foreign key information and we have to
detect that automatically.

We also have the ability to with our visual language join data from multiple
different queries/datasources. You can see how that happens here
[https://chartio.com/docs/visual-sql/merge-
queries/](https://chartio.com/docs/visual-sql/merge-queries/)

------
simjue
Congratulations, looks like a really nice product! As many people here ask for
a self-hosted solution, Apache Drill and SuperSet might be an alternative.

~~~
thingsilearned
Yes we do require a work email to signup as our target audiences are
businesses. We've discussed launching a public version which I'd love to have
sometime soon.

------
methyl
The pricing is a bit too high – if I'm reading correctly it starts at
$200/month. It's hard to justify such a spending for a startup where utility
of the tool is unknown and 14 days trial is not enough to evaluate that. It
makes sense to charge that much power users, but to get involved I'd love to
see a cheaper plan.

~~~
dwild
I'm not them, but a price too high can be a way to make sure to get only a few
client at first. They clearly like to test stuff, and it's much easier to do
it with a few client, than thousands of them.

We also often forget the indirect cost of supporting a client, at my current
job our software is great for small companies, but we found out that under 50
employees, it was just not worth it to support them.

You can also always lower a price, nobody will complains for a lower price ;),
but making it higher without justification is much harder.

So all that to say, it can makes sense to start with a price way too high,

------
punnerud
Is there a self hosting solution an engineer can test for free? I love
Metabase, but this solutions seems worth a try. The comparison with AWS/Google
Cloud on being SaaS is not viable because there is almost always (?) an
engineer testing out the software at the workplace, that’s why it have to be
free to test and OnPrem.

------
krat0sprakhar
I have some data in BigQuery (for some personal project) for which I'd love to
use Chartio and the visual SQL builder. The sign up page just shows company
login - is the product not intended for standalone consumers?

------
gklitt
An awesome research project in this area:
[https://people.csail.mit.edu/ebakke/sieuferd/](https://people.csail.mit.edu/ebakke/sieuferd/)

------
exabrial
If you have a solid data model and team or person that is charge of preventing
people from implementing bad ideas, it becomes trivial for people to do data
discovery on their own. The problem is many orgs have pushed far to the
opposite direction, unstructured, undocumented, unspecified data structure.
While this is better than "no data" is puts tremendous burden on the user to
interpret it later. As the old rule says: crap in crap out

------
johnthescott
"sql easy, data is hard". perfectly written.

why even use sql if you are simply looking for patterns in static data? why
not use an app dedicated to analytics, of which many exist? sql comes into
play when data needs to be shared, and those types of apps need coders who
would not struggle with sql. i simply do not buy the argument that sql is
difficult to learn for competent coders.

------
oarabbus_
I would honestly much rather have a powerful IDE where I write my own sql and
my query output is visualized than any of this UI stuff.

~~~
thingsilearned
Yeah we're somewhat targeting a different audience here but I also think this
will end up being a much better experience even for the power users than an
IDE.

I believe most data exploration just shouldn't be SQL/text based. It's just
faster to do it visually if there aren't extra steps added and the interface
is good/flexible enough.

Even as developers we spend most of our time in a GUI environment vs the
command line. It's not just that it's easier/better for business users - it's
also just better for everyone.

~~~
nojvek
Really like what you’ve done Dave. I’m working on a very similar idea but a
different niche (turning databases into a cms + simple charts) at
[https://boomadmin.com](https://boomadmin.com)

Totally the future is hybrid. Quickly press some buttons to get to a starting
SQL and then go complex from there on if needed.

My principle is that “simple things should be simple, complex things are
possible”

------
graphememes
I'm not going to lie, this really hurts my brain to try and wrap around, it's
a lot of cognitive overload in one place.

~~~
thingsilearned
Ha, sorry about that! Are you talking about the interface, the idea, or the
content?!

~~~
graphememes
The interface, it's very dense

------
stiray
Just a detail. Have you seen Microsoft Access query builder? Please do check
it, especially how you create joins. As far as I have seen tools for building
queries none came close. Maybe sqlyog.

[https://www.youtube.com/watch?v=pcjXQqKWWNw](https://www.youtube.com/watch?v=pcjXQqKWWNw)

------
didip
If the alerting story is amazing, then this thing would be very compelling.

Nowadays people use an amalgamation of random tools: Grafana, Redash,
Superset, Tableau, etc. But they all have rudimentary alerting or none at all.

Self hosting is another feature that is important. Enterprise customers simply
won't buy if it's SaaS only.

------
pbreit
Looks interesting. Alas, no support for VPNing into DB.

Like what mode.com is doing with its "bridge":
[https://mode.com/help/articles/how-mode-
connects/#bridge](https://mode.com/help/articles/how-mode-connects/#bridge)

~~~
thingsilearned
We do have a reverse SSH tunnel connection [https://chartio.com/docs/data-
sources/tunnel-connection/](https://chartio.com/docs/data-sources/tunnel-
connection/)

~~~
pbreit
If I understand, it involves changes to the DB server? That would be a show-
stopper for me.

~~~
MechanicalTwerk
Haven't used Chartio in a while but when I did the reverse ssh tunnel was just
a standard reverse ssh that you could deploy on any server in your vpn. It
doesn't have to be on the actual db server, it could be on its own server
acting as a bridge between Chartio and your db. Also, you could use whatever
ssh client you wanted - you didn't have to install anything from Chartio in
your vpn.

~~~
pbreit
This made me think id need to do something on the DB server which I cannot:

“On your database server, run the following to install autossh”

------
tmpz22
It's always a tug of war between "power-user features" and "simplified-
interfaces for common users". It seems like Chartio is using visual interfaces
to bridge this very gap and it seems like a very polished outcome.

I think the elephant in the room though is that this is a technical solution
to a non-technical problem. You can't teach all users to be highly proficient
in your app through a perfect interface - because the perfect interface does
not exist. You teach them through traditional training methods and continuous
training investments and infrastructure.

Going through their landing pages it seems like Chartio is offering support
and training packages, so there's definitely nothing wrong with them
supplementing that with better visualization tools.

In general however I just wish more companies would invest in the training
layer earlier and not try to bail themselves out with the perfect redesign -
_cough_ reddit _cough_.

~~~
thingsilearned
Totally - it's not a solution to everything. We can though do as much as
possible making the UI intuitive, flexible and fast. Those gains are not just
for business users but for everyone.

Every product needs some education (excel is still of the most popular online
courses) and we knew ours would be no exception which is why we've also last
year launched DataSchool - our free online community driven courses on data

[https://dataschool.com](https://dataschool.com)

------
vvern
How does this product relate to
[https://www.sigmacomputing.com/](https://www.sigmacomputing.com/)? It
superficially feels very similar.

~~~
thingsilearned
Sigma and Chartio have similar missions but a very different product approach
and functionality. Sigma also tries to bring the influence of spreadsheets
into the query building/data exploration phase.

We do this though as a very flexible visual language, where you can create a
pipeline of actions including merging queries from multiple different datasets
and then doing some post query computation.

------
longtermd
Postgres' pgAdmin software is still a huge pain for many operations,
especially for creating functions and views. I believe they could learn a lot
from this.

~~~
grenoire
To me, pgAdmin seemed like a very light header on the administrative (hah) end
of pg. Checking which instances are running, and seeing if the databases are
healthy etc.

------
BLanen
I made this in an internship except that I get the metadata from Entity
Framework and those subqueries but those woudn't have been hard to implement.

------
rco8786
Seems reasonably interesting, but I can’t get past the claim that they made 21
figma boards every day, including weekends, for 3 months.

------
stephen82
For some reason, it reminded me of Kirix Strata [1]...well, it feels like it
in a way.

    
    
        https://www.kirix.com/

------
solarkraft
The page stopped my media playback (which happens when another piece of media
is played). Why?

~~~
thingsilearned
No idea. Thanks for reporting. We'll look into it

~~~
shrew
I think it's because your looped demos are autoplaying videos. Modern browsers
can control media playback, and presumably these autoplaying videos are
hijacking the currently played media for some people?

As an aside, I was actually looking for a comment about the opposite issue: I
have a browser which blocks autoplaying videos, it took me a while to realise
that what I thought were static images, were in fact videos without playback
controls (and a little bit longer to work out how to re-enable playback
controls!).

Either way, once I got them playing, it looked like a slick experience,
congrats on the launch!

------
figital
I typed in a basic text query to "count reps per account". Then I clicked the
visualize button ... no re-visualization. This is an extremely basic feature
in every visual query tool the past 20 years (at least the ones worth paying
for). You would be limiting yourself to their canned charts and "demo magic".

------
techolic
For some reason I read 'Viral SQL', these days...

------
xrd
Reminds me a lot of DabbleDB, acquired by Twitter long ago.

~~~
ptman
airtable seems to be dabbledb du jour

------
larodi
No visual has beaten SQL for the last 40 years

~~~
bmn__
It's not a competition where one solution has to beat the other. Querying
through a textual language and through a visual interface complement each
other.

I consider a product to be a success when it combines the best features of
both and makes them available to the user so that he may organise his work
flow according to the own preferred style.

~~~
larodi
That is true, you are right. What I'm saying is that eventually things that
need be SQL are either SQL... or sometimes R. The language is more expressive
and it takes less time to 'select something from something where conditions
group by something', rather than drag and drop it. and is the reason why SQL
is so fundamental to all areas of information technology and particularly when
data is involved.

besides, SQL is DATALOG which is PROLOG which is to 'program logic', which
gets as close as it can to 'wiring' logic. many attempts to visualize this
process has failed so far, because... they are more complex and more specific.

SQL takes like 30 chars and dozen keywords to master. any visual paradigm -
takes more.

------
ComodoHacker
Signup to try — no, thanks.

~~~
boublepop
Seems like a fair trade, you get to try out for free as long as you let them
know that you are interesting in trying it out. They are aiming for business
users who don’t mind dropping an e-mail for a quick eval.

