

Q – A Data Language - miloshadzic
http://www.q-lang.io

======
koloron
That's at least the third programming language named Q:

[https://en.wikipedia.org/wiki/Q_(programming_language_from_K...](https://en.wikipedia.org/wiki/Q_\(programming_language_from_Kx_Systems\))

[https://en.wikipedia.org/wiki/Q_(equational_programming_lang...](https://en.wikipedia.org/wiki/Q_\(equational_programming_language\))

~~~
Kurtz79
Also, naming a programming language with a single letter is not very search-
friendly.

~~~
baldfat
As someone learning and using R I agree 100%

~~~
itsnotvalid
With q.js I could feel the same as well.

------
lkrubner
I am surprised that EDN is not making more progress as a data interchange
format, especially now that it has fantastic validators and coercion
libraries, such as Prismatic's Schema:

[https://github.com/prismatic/schema](https://github.com/prismatic/schema)

This is, by far, the most intelligent data exchange system I have ever seen.
It offers a richer language of types than what is offered by JSON, it is as
readable as YAML, it is far more concise than XML. And yet it seems unable to
break out of the world of Clojure. But it deserves to be taken seriously as a
data interchange language.

~~~
jared314
EDN [1] has one flaw. It is not JSON. I will just repost the response [2] I
received when I mentioned wanting EDN support in PostgreSQL.

> So the same argument as (say) YAML, Lua tables or TNetStrings. Or, if you
> include binary representations, stuff like Thrift, XDR and ASN.1. You can
> embed a JVM in PostgreSQL and write the functions yourself (or use PL/Scheme
> I suppose), but I would be basically amazed if anyone decides to do it for
> you. JSON gets the nod because it's understood by billions of systems. Other
> formats are going to struggle.

While I do not agree with his position, it is shared by many people.

[1] [https://github.com/edn-format/edn](https://github.com/edn-format/edn)

[2]
[https://news.ycombinator.com/item?id=5548404](https://news.ycombinator.com/item?id=5548404)

~~~
dj-wonk
Yes, JSON is "understood" in the sense that it is easy to parse, yes. That
doesn't mean actually working with the results is any easier. (1) Without a
schema, doing so can actually be harder. (2) JSON leaves out many useful data
types and it is not extensible. So, perhaps JSON has "succeeded" by lowering
expectations of data interchange. :)

------
skrebbel
I really like this. In general, a well-supported non-crappy schema language
for modern data exchange formats is long overdue (e.g. JSON Schema is horribly
un-human-friendly). The fact that they aim higher and allow for a much more
detailed set of validations is great, IMO.

I hope that this becomes a success and gets implementations in many
programming languages.

~~~
nevi-me
> I really like this. In general, a well-supported non-crappy schema language
> for modern data exchange formats is long overdue (e.g. JSON Schema is
> horribly un-human-friendly).

I think I'm a bit confused, is Q the actual schema or a validator of the
schema? I can also validate a JSON document using JavaScript and respond with
a true or false based on whether it meets certain criteria. So I think that
there's something amiss with your statement above, specifically that "JSON
Schema is horribly un-human-friendly". Doesn't Q take JSON and let the user
know if it's correct or incorrect? WOuldn't JavaScript, or any other language
at that, do the same?

~~~
rpedela
JSON Schema refers to [http://json-schema.org/](http://json-schema.org/)

I don't personally understand what is so "un-human-friendly" about it. It is
easy to use and understand in my opinion.

~~~
nevi-me
I know what JSON is :)

~~~
bjeanes
JSON Schema != JSON.

------
anonu
I got excited at first - I thought they were referring to the Q language from
KDB. This reminds me quite a bit of Google's protocol buffers.

~~~
aet
Same here. I'm interested in figuring out q/kdb, but it seems a bit opaque.

~~~
cangencer
As someone who had to teach myself kdb/Q recently, I can recommend the Q for
Mortals book:
[http://code.kx.com/wiki/JB:QforMortals2/contents](http://code.kx.com/wiki/JB:QforMortals2/contents)

~~~
jd-uk
This is a bit of a _plug_ but anyone interested in learning about kdb/Q can
take a look here: [http://www.timestored.com](http://www.timestored.com)

They have some interactive, online kdb/Q tutorials with programming questions
and answers to try out the language and get a feel for it.

There is also an IDE for day-to-day users of kdb/Q with lots of convenient
features

------
mercurial
A "data language", in this context, is a language for representing and
validating data. I'm not clear on why it's called "Q", or why it calls
marshalling/unmarshalling undressing/dressing. The language itself looks
alright, it's dependently typed. It doesn't say anything about the speed of
the existing JS/Ruby implementations.

~~~
hgh
Perhaps the name is based off the R language (which came from S), a major
data/stats language. Data representation comes before analysis, hence Q?

Renaming marshaling, couldn't tell you.

------
tel
It's strange to me how dependent Q becomes on its host language. Instead of
defining its own types in order to sit between various serializations and host
systems, Q seems to simply augment the Host with a few extra types.

~~~
bokchoi
I agree. This is something that XMLSchema got both right and wrong. The
XMLSchema basic datatypes [0] are well defined and for the most part fit
nicely in host languages. However, they really screwed up with the complex
types making it difficult to map onto programming languages for multiple
reasons (sequence/choice can't be easily modeled in most programming languages
without intermediate types, extension versus restriction, derivation versus
substitution groups.) For a good write up of the design philosophy behind
XMLBeans, see:

[http://davidbau.com/archives/2003/11/14/the_design_of_xmlbea...](http://davidbau.com/archives/2003/11/14/the_design_of_xmlbeans_part_1.html)

[0] [http://www.w3.org/TR/xmlschema-2/](http://www.w3.org/TR/xmlschema-2/)

------
islon
In Clojure you can use schema
([https://github.com/Prismatic/schema](https://github.com/Prismatic/schema))
which does the same thing without needing a separate language.

~~~
blambeau
The point here is to _have_ a separate language. Data interoperability is
about having multiple languages, not having only one to rule the world.

~~~
emidln
I've done (limited) schema validation in Python. Schema exports reasonably
well to EDN (it is just clojure after all). One can read EDN pretty well. I
don't have a full-blown Clojure interpreter that would be necessary for
arbitrary Schema, but I can handle the core types as well in Python (or random
other language) as I could in Clojure/Clojurescript.

------
hipsters_unite
This is the second project named 'Q' that's been on HN in the last week or so.
All cool ideas, but some Googling first by all parties would probably ease
confusion.[0]

[0][https://news.ycombinator.com/item?id=7290655](https://news.ycombinator.com/item?id=7290655)

------
tinco
It's interesting, but is it powerful enough? There's a great talk by Zed Shaw
about implementing authorization code. I think the same issue holds here.

A language might work for simple and more complex type constraints, but at
some point the constraints might become so weird that you really need a full
blown turing complete language to define them.

And that point might be sooner than you'd like. Why not instead of using
expressions in this constrained language, use something like Haskell?

It's got the super powerful typesystem, and you're certain that it will be
able to express _any_ constraint you throw at it.

That said, this language does look nice, and I haven't really tried to find
out if there's any obviously important constraints you wouldn't be able to
build in it.

~~~
Bjoern
This one?

[http://vimeo.com/2723800](http://vimeo.com/2723800)

~~~
tinco
Yes that's the one :)

------
bjourne
Given that you use a powerful rdbms, such as postgres, you don't need anything
more than sql constraints. For example, the constraint to check the
temperature would be something like CHECK(temp between 33.0 and 45.0). To
check that no two user accounts share the same alias, a simple
UNIQUE(user_alias) will do. More complex constraints can be enforced using
plpgsql.

Then name your constraints. Then when the sql layer throws them, it's trivial
to remap them to user readable error messages. It works especially well in
newer postgres versions where the error reporting is much more detailed and
the messages easily machine parsable.

~~~
nicksardo
SQL constraints are a given; always use them. However, it's best practice to
also validate your data before it gets to the database for several reasons.

A) Depending on a database check constraint failure is analogous to using a
try/catch when a pre-emptive check would work; it's dirty.

B) You're putting unnecessary load on the hardest system to scale.

C) Users/systems will get instant feedback with a client-side validation.
Depending on the data, this could be an enormous bandwidth savings.

~~~
spacemanmatt
I am not sure I would say "best practice" but I would rank it as "often
useful." I'll explain.

I disagree with your characterization of using exceptions when a check
condition behave identically. It's not dirty, it's exception oriented.

I disagree with your characterization of the database as the hardest system to
scale. What's hard is fixing a performance problem in the database after an
organization has effectively put the schema in a cast by developing 30
apps/scripts directly coupled to the core of the (broken!) schema.

I DO however agree that client-side feedback is important. Validation on the
client might include a lot of data used for prompting, or perform validation
_only_ the user-interactive application can do. This exceeds the role of the
data constraints you might implement in your database for obvious reasons.

I'm not sure Q does more than duplicate the constraints I would put in my
database to start with. There's a lot more to collecting input than enforcing
closed-form rules.

~~~
nicksardo
I disagree with most of what you said.

"exception oriented" programming is dirty. It's especially filthy when your
expecting exceptions from a database for these reasons

a) Returning errors to the user is asking for a slipup handling the different
exceptions, thus returning more info than desired. Cue the injection hacks!

b) Depending on the DBMS and its configuration, an upsert destined to fail may
block reads and other writes from executing!

I had a feeling somebody would gripe about duplicated validation rules; it's a
problem that needs an elogant solution on your own stack.

------
snorkel
I like it because you can practically use this to define a grammar for parsing
a programming language, because:

1\. Each data type can inline express its own value constraints (not just type
constraints, but rather value constraints) which is key to source code parsing

2\. With subclasssing you can combine data types into higher level objects
that inherit all of the lower level value constraints. This is similar to how
a recursive lexer drills down from a high level statement into the different
parts of each statement.

3\. The data type value constraints support sequences, alternatives, and
unions, most of what a lexer needs to parse a language based on a grammar spec

With these ingredients you can actually parse code, and thereby write a code
interpreter (or parser at least). You begin with a grammar rule (a high level
data type) that defines a statement, and a statement is defined as a list of
alternatives forms of a statement, and each statement form is defined as a
sequence of keywords and supported value types, and so on down the language
primitives.

... but I'm not sure if supports recursion, and thereby recursive descent
parsing. Can you say something like "Sum = [Sum, Operand, Sum]" If not then
you're limited to immediate values only and no nested expressions.

------
nevi-me
Am I missing something fundamental here? I think in the home page Q shouldn't
be compared to just JSON as they are in principle two different things. If I
were to write a 'data language' or to extend an existing language to cater for
data, wouldn't I write my validators in that said language? If I am correct,
then I was expecting Q to be compared to the closest other 'data language',
not be compared with a document/schema.

For example, if I was doing it in JS, I'd say: var validDoc = true; var Temp =
function (t) {return t >= 33.5 && t <= 45.0};

then when I get the doc, I loop through my conditions, as such:

validDoc = Temp(doc.temperature);

// reject if !validDoc

From here, the benefit of Q becomes that it's much easier to do validations,
instead of writing them from scratch, considering that my example above only
returns a true || false, as it doesn't have the error handling to let me know
what went wrong and where.

One could also achieve some of what Q is doing by using an ORM if said data is
going to a database/document store.

So, HN, am I missing something here?

------
Bjoern
Interesting... here is a tool, which is also going twoards data oriented
approach.

"Drake - a kind of make for data"

[http://blog.factual.com/introducing-drake-a-kind-of-make-
for...](http://blog.factual.com/introducing-drake-a-kind-of-make-for-data)

------
jdc0589
I can't count the number of times I have wanted to set up re-usable model
definition and validation, but haven't been able to find a good portable
library that isn't part of something much larger I don't need. This definitely
addresses a space in need of more options.

My answer was modlr
([https://github.com/jdc0589/modlr](https://github.com/jdc0589/modlr)), which
I have been ignoring, but Q looks like it could do well. I'm not super pumped
about the name though, conflicts with super popular promise library lots of
people already use.

~~~
blambeau
I'm not very good at naming things. I'm aware of the name clash with many
other tools and would really like to find another name. If you have any better
idea, just let me know.

~~~
rkrzr
How about _Scheme_! ...er, no wait... Or _Grammeme_ , the well-known
combination of grammar and meme
([https://en.wikipedia.org/wiki/Grammeme](https://en.wikipedia.org/wiki/Grammeme))?

------
bane
Oh dear, this might conflict with a SQL variant we cooked up at my old company
called Q-Lang. Where you could write a SQL statement, leave bits of the WHERE
clause "blank" and it would generate a GUI interface with drop downs and
calendars and everything to let people fill in those bits without too much
effort. Nothing terribly fancy, but it let you go from a working example SQL
statement to an integrated query interface in the app in about 10 minutes.

It had lots of limitations, but most of the time it absolutely annihilated
some of our competition's multi-month integration engagements.

------
dj-wonk
Q is trying to do both data and validation, which are very different problem
spaces. I think designing for both involves serious tradeoffs for the
individual parts.

------
elwell
Finally a new language that does something remotely novel. I like the blur
between data and code here; (very) vaguely lispy.

------
mrcactu5
where is all the theory behind q-lang coming from? are there theoretical
principles behind this new schema language?

