Wow, I thought I would resurrect my HN account for this.
I wrote this article a little over a decade ago, when I was still in college, and I knew almost nothing about the world. (I still don't, but I didn't then, either) It's fun to see that it still resonates with people despite the naivety and the slight sense of "I am headed for success" that makes it hard for me to re-read.
My initial reflection on the article was that I had fallen off the "do things, tell people" wagon a little bit, having let go of my twitter account and with it much of the easiest path to doing things and telling people. I still work in tech but I haven't created anything I'm particularly excited about in a good while, so I felt that I was failing my own ideal.
Then, of course, I remembered that I hiked all of Ireland's long-distance trails (https://toughsoles.ie) and have a reasonably successful, if niche, youtube channel (https://youtube.com/toughsoles). Funny that my mindset, especially regarding this article, is so tech-focused that I automatically discount something I've been doing for five years.
All that said, I'm starting to look for new opportunities in the larger-than-startup space. Shoot me an email at carl@flax.ie and let's chat.
This comment of years can be categorized under "Do things, tell people" rule. haha.
You wrote and you're telling people about your life after it. It was a good read. Hope you get good leads from here!
Seriously though, I like your advice, I am someone who thinks the same way. I have artistic daughters that I have tried to instill this in since they were young. Publish or perish is another way I put it.
Or it will manifest itself as, "Got a good idea for a novel or video game? Cool, write it. Because you better believe that nearly everyone you meet every day also has a great idea for a novel or video game (or movie, etc.). The difference between Ray Bradbury and everyone else is that Bradbury sat in front of a typewriter every day." (It's possible too of course that Ray had some pretty good ideas though that not all of us had.)
That also points to the sad reality that I have found (now in my 50's BTW, no college undergrad). So many people are not motivated to "do",
I think that first came home to me when I was writing shareware games and sending floppy disks of the source code to people that would write me (send me a check).
My thought was that, having been handed compilable sources, each one of these people would have read the code, gutted it, refactored it, added their own art/sounds and started putting out their own games. That it was just the "how to" they were missing.
Not so as you might (now) imagine. Taking that step to create really is a difficult one for a lot of people. Further, going beyond "demo" and adding scoring, game completion, configurable controls, high scores, about box.... Very few put in that extra (and frankly tedious) effort.
Thank you for the original article! Reading it last night gave me a jolt of motivation with the project I'm working on :) Have built it and now time to talk to people!
Yes, we always heard people tell us that Ireland was an amazing place for hiking, but being Irish ourselves, we didn't quite see it that way - until we walked it. It's really spectacular landscape, although some access issues can mar the experience a little bit. It's certainly a lovely place to go regardless!
This is actually very good advice from personal experience.
The best career results (promotions) I have seen came from people who built a prototype of some sort then evangelized it, even if it wasn’t necessarily a complete product, often just a good prototype.
First example: This guy took the Intel Edison and built a pretty cool project and brought it to our maker faire events. He ended up in conversation with the CEO of some company and ended up landing a $500K exec job. Just from building a skateboard hack and trying to tell people about it.
Second: I knew a guy who built this neat prototype of a compute service and then did a keynote talk at a Meetup. It didn’t really work as a product but the website was convincing. It was enough to “get bought” by a major tech co and land him a CTO job.
Another time I saw a guy leave a FAANG and build a compelling IoT type project in three months. Again, he ended up with a CTO level job after a few months.
My take away is if you want an executive job, build something somewhat cool, even as a prototype, and go try to sell it for real. Even if you don’t raise any money or make it work as a business it is often way better than submitting resumes.
There is something much more powerful about creating something that generates a lot of engagement and conversation, even if it is only half baked but looks decent and your storytelling is compelling.
Yeah, building stuff is generally the best way to get noticed. A warning sign should be added however that there's a hard survival-bias here.
On top of that, it's idyllic to imagine their tech skills were the only reason these three people got the great job. Did their class and heritage line up at all?
>what is there to be gained by attributing their success to all the forces that they can't control.
Supposing for a moment that there is truth to what the parent comment says, we would gain a better knowledge of how the world actually works instead of perpetuating old myths.
I believe there is definitely truth to it. I’ve seen first hand how Stanford/MIT or FAANG on your resume can give you a leg up in the hiring committee over many people without the pedigree who turn out to be far better engineers.
That said I agree with GP, there is not much to be gained from focusing on systemic biases. Just because a narrative is truthy does not make it definitive; the map is not the territory and all that. As an individual focus on what you can control and you have the possibility of achieving outcomes unique to you.
As a thought experiment, how far down the rabbit hole would you go?
What were the factors involved in ones outcomes?
- Was it his "class"?
- Skin color?
- Family connections
- Country?
- Education?
- Genes?
- Moon's gravitational pull?
- A butterfly flapping's it's wings?
Not saying it's not relevant, i'm saying it's not practical. If there's any lesson to be learned from the story, it's definitely not from the X factors that cannot be controlled.
The only outcome of such discussions would be to complain.
Speaking of perpetuating myths, people attributing the success of other people to anything else other than competence would be up there.
It's demonstrably not true that we have no control over societal prejudices, nor is it true that attributing success to anything other than competence is a myth. Moreover, comparing the effects of bias to the flapping of a buttery's wings is laughably dishonest.
If you wish to ignore these things, that is entirely your prerogative, but your thesis isn't convincing.
Bit hammered but I'll try explain. Being creative and pushing that energy is great, and will put you into a better bracket than your peers. However, that bracket is still going to be made of thousands of people, so it's worth understanding that we are not measured strictly by our technical ability. We're judged by how we smell, how we look, our fashion, our projection and others personal interpretation of our trustfullness.
I didn't mean to say, don't try unless you're a certain race. I tried to say, understand the factors that work in your favour and the ones that don't.
I used to work at one of those body shop contracting places. A few years ago, I heard that the boss managed to sell the company, presenting themselves as developing an in-house blockchain product.
I asked around some ex-colleagues and as far as anyone knew, all they had were some fancy PowerPoint decks and a very rough implementation in PHP.
After getting bought, everyone still continued doing the same old body work jobs, except now for the parent company’s other subsidiaries. The website still mentions blockchain.
I lol'ed, because around 2017 is what I called the "sprinkle in some blockchain" era. There was so much FOMO from investors that blockchain was "the next big thing", that an easy way to literally double or triple your valuation was to "sprinkle in some blockchain".
It wasn't like the investors were really even being scammed, because nobody could succinctly articulate why blockchain was better for these particular use cases in the first place. All I ever heard was some marketing speak that sounded like it was from a markov chain generator, but it apparently had the effect of hypnotizing investors so they'd add a few zeros to the end of their checks.
At least where I'm from FinTech is harder, since there's a lot of regulatory hurdles that you are going to have to convince your audience that you've cleared.
Non-tech people don't know what it takes to make a blockchain viable, but they do know that you have to have licenses to operate a FinTech company.
This reads a bit like that LinkedIn inspiration porn from yesterday[1], honestly. These people may have other qualifications not listed here but working at a company where this is the path to being CTO sounds a bit like my nightmare :)
Implicit in this argument is that you need to build the right thing and talk to the right people.
I've been to industry events where I've seen people give "career limiting" presentations on something awful they've built. And I've seen great people stuck in a corner with the worst kind of industry fraudster.
Sure, you've got to kiss a lot of frogs to meet your prince. But you also have to remmeber that you might end up kissing a poisonous one in the meantime.
People who don’t “build and show” in their free time do not necessarily have harder time getting hired, nor are they necessarily worse as hires.
Some people have circumstances or attitude where they prefer to get paid to build things. One can find oneself in demand by being good at networking, a capable and creative engineer, able to work autonomously, a good communicator and a pleasure to work with. I know some people like that.
If we are talking about valuable hires landing jobs, relevant qualities seem completely orthogonal to the inclination of building things as a hobby—if anything, the latter may imply a degree of unsustainability and/or tendency to flaunt responsibilities if it results in neglect of one’s personal life or previous job.
Thank you for this. I often feel poorly about myself because I've created so little (especially since leaving school), but between 13 and 31 I didn't have financial stability or basic safety, and even now I'm a disabled person who is the sole breadwinner and caretaker for my also disabled sister, who can't work (I have MS, she has bipolar [which obviously takes some mental energy to handle]).
I've just been so tired trying to feed + shelter myself and escape abuse that my building urge was completely extinguished until the last year.
I always beat myself up for it, but it turns out that I just couldn't get into the flow state needed to build things when I was constantly stressed-out, hungry, cold, and scared.
Not to take away from people building things - I love seeing what people come up with! It's just a pleasure to know that not everybody will consider those decades as me being lazy.
> I've met plenty of people who have built nothing, but are good enough at talking to people that they can get jobs.
Sure. I don't think the author would claim that this approach is the only way to get noticed and/or a job. I certainly won't. But this one is cool, because at the very least you did something and you honed other important skills along the way.
> As for your last point, how can someone tell if they're talking to the right person?
What I meant and could have made clearer was: As the ones building and showing, it's entirely our choice to be honest about it. When it comes to spotting someone who is not, having more experience in doing and showing also helps on that front.
That's not how survivorship bias works. Some things really do increase the chances of survival. You might as well dismiss life jackets as survivorship bias, because hey, what about all the people who were wearing life jackets and drowned anyway?
It's only survivorship bias if you're talking about an unrelated factor that doesn't have any influence on the outcome.
If I get GP's argument, a more appropriate analogy would be that some life jackets are filled with lead and you can't necessarily tell them apart. That is to say, presenting yourself to the external world carries an inherent risk as well as the potential for reward.
But as with leaden life jackets, you can tell them apart. The subset of projects and people that are poison pills can be discerned, so "build the right thing" is more like "don't build the wrong thing, and keep building" where the wrong thing can be avoided the majority of the time.
Which is basically the process of gradient descent, aka learning!
Took me way too long to realise this, at least first two failed companies were as a result of over indexing on product. We do not have perfect market intelligence and inexperienced entrepreneurs most common mistake is relying on how good their work / product / service is. It's a business killer as not enough customers will ever find it.
It actually makes more sense to build distribution first, so that your product (however good or bad it is) gets an opportunity for as much exposure as possible. Somebody will buy, if enough people see it
I think you're giving a big key that many of us as just developers/engineers miss some times, and it's the distribution part. For us, as we haven't trained in that area, it's hard to find a solution.
How would you say that's the best strategy to build this distribution?
'build distribution' = this really is audience / channel building. And how you do it really depends on who is going to be buying your stuff. For example, I currently sell to recruiters (strange I know!). So my distro is a build a powerful presence on LinkedIn (where recruiters hang out), run newsletters, do podcasts / livestream, build community around recruiting content.
This is full time for me though, so the above approach may not be suitable for everyone. The idea of building audience though, is relevant no matter what you do
not just developers. I know loads of people who have such great product - coaching services, graphic design, you name it. But they retreat into doing what they love (the product) and don't realise that you got to sell it - and that needs distribution. I only learned this through repeated failure, fortunate enough to have the economic resilience to bounce back
Distribution explains Meta's and Google's powress (in the consumer market), that's literally unmatched right now.
Distribution ("Install Chrome" prompts on the Google homepage) is how Chrome rose to dominance, and distribution is why Google pays Apple billions to be the default search engine on iOS.
The other 3Ps (product, price, positioning) [0] of the marketing mix matter too (more so for consumer-grade products than enterprise where sales is takes precedence), but of course distribution (place) is king.
A couple decades ago I made a Tarot deck. https://egypt.urnash.com/tarot/ Sadly you can’t get a copy right now, I really need to get a reprint happening. It’s got spot gloss on the cards so that’s kind of complicated to do.
Then I went on to draw a comic book about a robot lady dragged out of reality by her ex-boyfriend. https://egypt.urnash.com/rita/ It managed to get cover quotes from Phil Foglio, Charlie Stross, and Peter Watts. You can buy a copy of the printed collection if it tickles your fancy.
Now I’m working on a space opera comic. https://egypt.urnash.com/parallax/ It’s still in progress, you can’t buy a copy. But you could support me on Patreon if you have a lot of money from your software job.
> A couple decades ago I made a Tarot deck. https://egypt.urnash.com/tarot/ Sadly you can’t get a copy right now, I really need to get a reprint happening. It’s got spot gloss on the cards so that’s kind of complicated to do.
I don't know if it is the blocker but makeplayingcards.com has a `high gloss` finish and does print on demand.
Doing a full gloss is pretty easy. Doing spot gloss, where only spots of the print has another material (typically a gloss material) requires special printers or has to be done by hand, as I understand it (I'm not an expert).
Finding places that does it can be tricky. I remember going down this road a few times in the past, too.
It's also commonly known as "spot varnish" or "spot UV coating" and it's a very commonly available finish from offset printers, similar to foil stamping. It's just another layer in the separations.
Special printers and another step or three in the printing process, yeah. It’s a cool effect but it’s a thing you have to go to Serious Printers and set up a large run to do, no print-on-demand shop is ever gonna do this.
It’s been a a long road but I’m finally at a point where I can definitively sat that this advice is some of the best you can give anyone in or out of tech.
Writing about software I’m building, errors I run into, and situations I get into (and often stumble through) has been huge for my own development and my career (https://vadosware.io).
Doing this has led me to habitually bite off more than I can chew which seems to be the only way to really grow:
- A managed services provider for smaller clouds/infrastructure providers called NimbusWS (https://nimbusws.com), which came out of all my blogging about Hetzner and kubernetes.
- LoginWithHN.com from how often I write about cool stuff on HN that was built for HN
- A salary sharing site for Accountants (https://nomorepizzaparties.com), which came about due to trying and investigating Baserow and NocoDB.
Outside of tech there’s awesome collectives like MoonMusiq (https://moonmusiq.com/), a collaboration of musicians that I was made aware of via HN actually -- its obviously an outlet for people to do these two things, in that order. Make cool shit, share it.
This is good advice for techies, who are usually good at the “do things” and but not the “tell people”.
The same principle applies to a startup/side-project. At supabase we call this “ship and shout”, which is about as simple as it sounds (Ant wrote about our full “process” here: https://supabase.com/blog/2021/11/26/supabase-how-we-launch )
The other one that took us some getting-used-to was being repetitive on platforms with a short “half-life”, like Twitter. It made me uncomfortable at the start since the general advice is that “reposting is annoying”. But any given tweet probably reaches only 1% of the audience i thought it would, so not many see the repost. Sometimes we will talk/tweet about something we shipped months ago and we still have developers commenting that they just learned about it from the tweet.
> techies, who are usually good at the “do things” ...
But since they don't tell people, they often end up building useless shit. I've experienced this a few times when I spent weeks or even months working on a feature, then showing it to customers, and realising that they don't get it, or that it doesn't fix their problems, etc. I try not to get too attached to my ideas, so if something doesn't work, I throw it away and try a different approach.
So I think doing things and telling people after the fact when you are done is not going to lead to success. You need to tell people early, so you can adjust what you're doing to make sure it's actually something interesting.
I just want to add there are a lot of "business people" who are good with coming up with ideas but then those ideas are useless.
So it is not only techies.
One of company owners for example did not understood market our company is in and mandated that we built in Stripe integration. Well it was useless burning of money and he did not read "do things that don't scale", where our market worked you have high touch sales and send invoices and no one ever signed for payments, I removed the code last year and we are growing with customer base.
I'm usually very wary of this kind of post, narrative building and all that, but I like this. And I feel like the honesty at the end of the authors success / path leaves room for the reader to put it all in context.
> If you you don't have any marketable skills, learn some. It's the future.
> Then make something that you can talk about. Make something cool.
> Next, find events where the people you want to work with are.
> Then get a drink into you (or don't) and talk to them about it.
I might add a Henry Rollins quote at the end of "(then) say yes to everything but make it work for you."
Then maybe one final step of, get super lucky and keep trying until you cant anymore.
> Then maybe one final step of, get super lucky and keep trying until you cant anymore.
I think one of the keys here is that you can manufacture a certain amount of luck. Perhaps you've heard of the micromort[0] a fascinating statistical tool many countries use to determine how they spend money in healthcare. The general idea is to measure all activities in their chance to add/remove the probability of death even by very tiny amounts, this is applicable to luck.
When you go to a some event that's a networking opportunity, chances are nothing will come of it. You might go to a dozen with no good outcome but your lifetime chances of a good outcome go up once you decide to try to network, it's conceptually useful to think of these events as giving you a few points of "microluck", attending while following the author's advice even more.
I'm a marine biologist, often when we fish it is in some difficult circumstances due to species or habitat of interest. A quote passed down from one of my academic great grandparents (that was probably in a discussion about the futility of it all) is that you definitely aren't going to catch any fucking fish if you aren't fishing.
Making (software) stuff seems easy. I do post about it on social media, but honestly it gets 0 traction since I have almost no audience.
> If you you don't have any marketable skills, learn some.
This seems like "rest of the owl" where it leaves out a ton of critical steps. Become good at marketing is a lucrative career if you are good at it. So excuse me for being jaded when "learn marketing" seems as helpful to me as "sell profitable services". Sure, but how?
The post recommends Khan Academy, Wikipedia, and Code Academy. Only Wikipedia has info on marketing, but reading it is not the same as learning how to do marketing.
The filter is the skill of self-directed learning. It is a meta skill that can be built, which then increase opportunities dramatically.
When I see the jib "they skipped a bunch of steps", it tells me that the person who received the advice needs to work on creating their personal learning system. Ideally, you feed your system a topic or idea, and it walks you through the exploration and work of understanding and applying that knowledge. It's not necessarily a piece of software. It can be a checklist, a journal, doodling on whitebords, blogging, whatever. Ultimately, all human knowledge is accessible through language. Whether that is true or not, if you act from that frame, there's nothing you can't learn with the right support and systems in place.
In a very meta twist of fate, I just heard about this article today on a podcast, and then realized Aaron (from the podcast) is the OP! If anyone wants to get some advice on how to use Twitter like a human being I recommend a listen. It was a great convo: https://share.transistor.fm/s/c854b56e
In the spirit of the post, a thing I did last year is build a minimal video editor (https://getrecut.com) focused almost exclusively on cutting out silence. And then lately I’ve been building a cross-platform version of it with Rust and Electron.
‘Round here a lot of people dislike Electron, and I kinda count myself among that crowd, so it’s been a fun challenge to build an app like this that defies the idea that Electron apps have to be slow. Turns out you can get a lot of performance out of it if you write most of the heavy lifting in native code, pay attention to the algorithms, make things cache-friendly, keep an eye on the profiling and optimize as you go. So far the Electron app is on par with or faster than the native Mac app, which is exciting to see! Hopefully I’ll have something out in the next month or two.
Same for me. Find a niche, make blog posts introducing technical topics, make blog posts explaining your work. As long as you can afford to wait for a couple good contracts a year you’ll do fine. Textbooks are apparently even better but take a lot more work.
Similarly here, word of mouth and people finding github repositories or posts on issues/forums where i typically post a solution to an obscure question. (Maybe helps having a consistent username)
But its really dented any marketing/sales skills, and now I wish I spent more time honing those (as I want to make some of my own things)
Fine. I'm starting Severus, which actually deals with "contacts, business cards, email addresses", but with some interesting privacy mechanisms. I haven't quite launched yet, but I've seen this topic come up frequently on HN, and I never talk about it. I need to get out of my comfort zone and talk about it.
Okay, here you go. I have an amazing day job. But I always wanted to build some software utility that others found value in. So I shipped a forum platform. Because I can’t bring myself to share this with my friends and professionals network, it has no users: https://discoflip.com/
Plenty of people have rich and supportive parents. Very few of them become quite so successful. You can buy a lot of things but that levels of success isn’t one of them.
Don't diminish other people's hard work and talent. It's not a good look, especially not in people with the sort of career outlook the average HN-reader has.
What about all the talented and hard working people we are not speaking about, merely because they did not have rich and supportive parents? Why is it so popular to act as if this does not make a difference? Economic status is the elephant in the room when it comes to discrimination. It does not feel good to acknowledge this, in particular if one personally is higher up the ladder. Nonetheless it’s the truth.
Which success is not in part due to outside circumstances?
If it's not your parents' wealth, it's your upbringing, or your genes, or the people you happen to know, or where you happen to live, or the time at which you come up with an idea.
Thought this was a wonderful article. Thanks for sharing.
I figured to get in the spirit of sharing I'd share a little passion project I've been working on for sometime now:
https://github.com/thebigG/Tasker
It's an app that allows you to accurately track your commitments via hardware hooks(audio, mouse and keyboard). The UI can definitely use some work, but figured some people might find it as useful as I do.
I tried building a break enforcing app once: Locks the screen. Asks you what goal you want to accomplish and how much time you need. Then unlocks the screen for the requested duration. I stopped using it because of the very issues you mentioned. Pure time based solutions just aren’t practical.
I use pomodoro for this. Writing software doesnt always require you typing at the keyboard. Most of the time I sketch the solution/design using pencil and paper
I tend to obsess with detail[*] and used to be quite self-conscious about my work.
I am yet to find a small project I regretted sharing with people.
You spend weeks working on something that never leaves a GitHub repo. Then, 3 years later, you see someone else, implementing a similar idea and sharing it. "this could've been me, if I wasn't such a coward", "my project was better because of x and y", or: "I'll never get anything done in my life" are the usual types of rumination that kick in. I know will resonate with some of you.
2. have a place where I keep intentionally shitty, unfinished drawings(important meeting notes): https://potato.horse
The point of 1) is to build things that just give me joy, they don't have to solve a particular problem.
The point of 2) is to get more comfortable with things that are messy, broken. It feel more honest when I put unfinished doodles there. I don't care if people think they're ugly. (spoiler: people like my shitty drawings more than anything I try to make look perfect)
With 1 and 2 it's just easier for me to get things out there, talk to people, learn that besides some parts of HN or Twitter, people are more likely to be excited about whatever the thing you're doing is, instead of judging you. I wish I had this mindset 20 years ago.
----
[*] which is somewhat ironic as I used to train people in prototyping and love hackathons
There’s probably a lot more to it, though. How we tell people is fairly important. I like the author’s self-effacement.
Some people are so good at "telling people," that they don’t need to actually do anything (but they still work just as hard).
I do both, but I’m not so good at self-promotion. I suspect that people feel that my "tell people" is empty boasting. I'm not a fan of the "humblebrag." If I say something, it's a fact; usually accompanied by links to resources that prove it. I won't make claims that I'm not ready to prove, but I will also mention what I can do.
I’m often told that I’m lying, or inflating, by someone that never even followed the link that I provided in my statement (that proved what I stated). Getting that from prospective employers, was infuriating, and a big factor in my deciding to stop looking for work. I don't like being called a liar.
I have three Twitter accounts (a personal one, and two corporate ones, for each of my companies), but I seldom use them. Social media is hard work, and I prefer to budget my hard work for the "do things." Pretty much all my social media interaction happens right here.
I don’t especially care, anymore. It’s bemusing to have someone hit me with an insult[0], where they could have avoided embarrassing themselves, simply by clicking a link. It’s amazing that we are so focused on attacking others, that we jump at every opportunity to do so. Not sure what it buys us. There was a post, yesterday, about the "tone" of HN. I haven’t been here long enough to know whether or not it has declined, but this joint is Sesame Street, compared to a lot of other venues.
Doing things is the easy part. Getting anyone to notice is the hard part.
I've been pushing https://concise-encoding.org/ for awhile now, and it's a LOT harder to attract people than it has been with my other projects.
Probably the nature of it is part of the issue. After all, who gets excited over a data format? The big security scandals recently have helped somewhat since people are starting to take security more seriously, and this format is all about security. But still, it's fighting against inertia, and the going is slow...
The first question I have when I see a new generic data encoding -- a question not answered anywhere on your site's front page -- is what advantages it has over the current incumbent (Protocol Buffers).
I can write text-formatted Protobuf in any text editor, convert it to efficiently encoded binary, and parse it in dozens of languages with both first- and third-party libraries. Protobuf also has a JSON encoding for use in browsers, an ecosystem of tools such as linters and formatters, and the continued support of a large commercial entity. For use cases where Protobuf is too slow, there is Cap'n Proto and Flatbuffers.
Why would I switch to your format, which seems to have extremely limited language support (Go only?), is significantly more complex (look at all those built-in data types!) and is supported by a single person?
I'm very interested in this topic, having authored several such tools.
Some remarks after a first glance at the README, it's very possible I missed some obvious answers and I apologize in advance:
- You ask for a review on your site but that just links to the github project page. Have you done a show-hn by the way?
- I believe schema representation and encoding should be two separate things; In other words, a good data manipulation tool should support several schema formats and several encoding formats
- I personalty prefer the types to be a little less opinionated (for instance, there are many legitimate definitions for a "date" so I do not want the data layer to favor one over the others, although I reckon having some support for dates is convenient) - having a type for markup is particularly suspicious.
- You do not mention extensibility in the features list, which is a very important topic when a data format is used for RPCs in a distributed system, for instance
- Are strings UTF-8? UTF-8 only?
- Why do you need specific types for edges and nodes?
- No type for records/structures apart from the top-level one??
- No sum types? How do you handle nulls?
- Lists are of heterogeneous types? Can we have a type for "lists of some type"
- How come comments ended up being types?
- The front-page should display a comparison in encoding and decoding speed and encoded size compared to the major contenders (which are protobuf and json, I guess)
- It should also display the corresponding go code for each presented schema examples
> - You ask for a review on your site but that just links to the github project page. Have you done a show-hn by the way?
I'm asking for a review of the specifications, which is what the main github page is for. There's also a reference implementation, but that's in a seprate repo. I've done a show HN in the past. It might be time for a fresh one since it's been a couple of years.
> - I believe schema representation and encoding should be two separate things; In other words, a good data manipulation tool should support several schema formats and several encoding formats
The schema representation will not be tied to the encoding. I'm leaning towards maybe https://cuelang.org/ as the officially endorsed schema format, but there's nothing stopping someone from using something else.
> - I personalty prefer the types to be a little less opinionated (for instance, there are many legitimate definitions for a "date" so I do not want the data layer to favor one over the others, although I reckon having some support for dates is convenient)
I put a date in there because it's such a fundamental data type that everyone wants one, and if it's not specified in an opinionated manner, everyone comes up with their own (likely incompatible) interpretation, which is what I want to avoid. The idea is that a user of the format shouldn't have to worry about HOW to encode their data unless they're using exotic types.
> - having a type for markup is particularly suspicious.
I'm still not 100% decided on whether markup will stay or go. I've already tried and dropped dozens of other types already so it might go before I release...
> - Are strings UTF-8? UTF-8 only?
Yes, UTF-8 only.
> - Why do you need specific types for edges and nodes?
Because nodes can't represent weighted or other complex or non-directed graphs, and edges are by their nature too bulky for representing trees.
> - No type for records/structures apart from the top-level one??
Not sure what you mean? structures are represented using the map type and a schema.
> - No sum types? How do you handle nulls?
Null is allowed. You can pass {"result" = 500} or {"result" = null} or {"result" = [1 2 3 4]} if you want. Sum types would be enforced by the schema.
> - Lists are of heterogeneous types? Can we have a type for "lists of some type"
This would be the job of the schema. There are typed arrays for primitive types like int and float since that's a fairly common use case and would be too bulky otherwise.
> - How come comments ended up being types?
Comments are "types" in the sense of what types of data a document can physically contain. They aren't "types" in the sense of actual data to be passed to applications (although an application could listen for it - for example a CTE reformatter or sanitizer).
> - The front-page should display a comparison in encoding and decoding speed and encoded size compared to the major contenders (which are protobuf and json, I guess)
Encoding and decoding speed would depend on the implementation. This is just the specification.
> - It should also display the corresponding go code for each presented schema examples
Not sure how useful that would be since every single example would be cbe.Marshal(myobject, stream) and myobject = ce.Unmarshal(mytype, stream)
I skimmed your site and didn't really get it. Maybe that's what everyone else does. In case it helps, here are the thoughts that occurred to me, in the order they occurred.
0. To me "data format" makes me think of things like PNG, or the DWARF debug symbols format, or the MPEG transport stream, not general purpose text formats like JSON and XML (I'm not a web person).
1. A secure data format. OK, weird. I thought it was always the programs/libraries that dealt with the data formats that were guilty of the security bugs, not the format itself.
2. All the bullet points under simple and efficient are already true for all the data formats I know/use/care about. So already I've dismissed your project as interesting - ie the "simple" and "efficient" are already solved and I don't believe "secure" is a real problem. But I'm aware I might be jumping to conclusions, so I keep reading.
3. I skipped straight to "Security - Protecting your data". I thought the sentence explaining why security matters was superfluous - your audience already knows why security matters.
4. "The existing ad-hoc data formats are too loosely defined to be secure, and can't be fixed because they're not versioned". OK, this looks like the meat. I click the link.
5. "There are many vectors that attackers could take advantage of when they control the data your system is receiving, the most common of which are induced data loss, field omission, key collisions, and exploitation of algorithmic complexity". If the data is from an attacker, data loss and field omission sound like good things - I don't want their data or fields because they are an attacker.
6. '"change user" command with a group of admin\U+D800'. I'm still confused. It still sounds like the "admin\U+D800" string is processed by my program. Your data format doesn't know whether that is a valid string or not. Telling your data format is no easier than telling my program. Oh! Maybe it is. Because you only need to specify it in the data format, not in every program/library that implements the format. Is this the point of the system?
Hmm ok I'm definitely not communicating properly what it is then...
I'm not sure what nomenclature I could use besides "data format" to describe what this is, any more than one could describe JSON as anything other than a data format...
The security aspect is where I'm having the most trouble communicating, because it's just not a well known issue. The problem comes from the different ways that codecs deal with the data they receive (for example Java Codec A vs Golang codec B vs Python codec C vs Python codec D, etc). If the spec isn't strict enough, a conformant codec could behave in ways that are insecure when coupled with other systems.
Security really does start at the specification, in that the spec has to define what a codec can and cannot do. For example:
"A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character".
A rule like this would prevent the admin\U+D800 vulnerability because a codec is not allowed to behave this way (truncating or replacing). With JSON, this isn't specified, so systems where the same data is parsed by multiple subsystems are vulnerable.
Specifically regarding the admin\U+D800 problem:
Any sufficiently complex system will have multiple subsytems.
So say you have a user-facing system that accepts a "new account" request in JSON. The attacker passes in "admin\U+D800" as the group (meaning the 5 character "admin" plus the unicode character D800, which is invalid).
Your account creator's JSON decoder ignores bad characters, so when it checks the group vs its blacklist, it finds that "admin\U+D800" != "admin". Validation passed, it sends the new user info to the account storage subsystem (as a JSON message).
Your account storage subsystem's JSON decoder silently strips bad characters, so your account subsystem only sees "admin", not "admin\U+D800".
Now this new user is an admin, thanks to a privilege escalation vulnerability brought on by a lax JSON spec (all codecs involved were conformant).
If the JSON spec had specified that "A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character", this kind of vulnerability could not happen with a conformant codec.
> I'm not sure what nomenclature I could use besides "data format"
Yeah, I don't know either. I see that the Wikipedia page for JSON describes it as a "data interchange format". Maybe that helps. I'd probably side-step the problem by having your home page start with "Like Protocol Buffers but more secure", followed immediately by the example you just gave me.
Either way, the responses here give me a lot to go on!
I guess I need to revamp what is emphasized and in what order. It's going to have to hammer on security all the way. Everything else is incidental and a "nice-to-have".
First, the need for tight and precise specs, not allowing optional or under-defined behaviors or too many features, and what goes wrong if a data format doesn't do that (I'll see if I can simplify the admin example or maybe do a $0 cost purchase exploit example).
Next, versioned documents and why that's important: Without it you're limited in how you can update the format to deal with emerging threats as they come along. Otherwise you get deprecations, loss of code space, and the possibility to get permanently stuck with an unfixable problem.
Then talk about fundamental type support and why we need so many: If you don't do that, everyone has to make their own encodings for common types (like dates, media etc), which won't be compatible or as carefully thought out, opening up security holes again.
The trick is how to make security sound sexy enough for people to take notice...
1) It wasn't obvious what you are offering until I scrolled down to the examples. You stress that you offer something secure before you show what it is.
2) Why is JSON not good enough? I can also encode trees in JSON. What's the advantage of your format?
Both are built on Flutter/Nim/Postgres. I actually also want to build an NLP project too, which I started on, but it's such a huge endeavor that I can't figure out an MVP which won't take 10 years! Plus I can't do too much at once.
Nim on the back-end. The UI is defined in Nim (back-end) and sent to Flutter for rendering.
I'm trying to figure out what to do with this part, because it could be useful for others. I'm thinking part Open Source and part commercial, maybe what Qt does. It could work with any back-end, e.g. Python, in theory.
If you want to be notified of when I do release something, please email me (see my HN profile).
I'm building a tool called MicroKeys. It's a macro program, for Windows right now. It uses MicroPython as the script engine to let you register hot keys that do things. It's very much a work in progress right now.
I'm writing it to fill a very specific niche I have, but if it's useful to others, I'd love to hear feedback on what it could do to be better to help it come to fruition.
"Tests done since 1933 show that people who talk about their intentions are less likely to make them happen.
Announcing your plans to others satisfies your self-identity just enough that you’re less motivated to do the hard work needed.
In 1933, W. Mahler found that if a person announced the solution to a problem, and was acknowledged by others, it was now in the brain as a “social reality”, even if the solution hadn’t actually been achieved.
NYU psychology professor Peter Gollwitzer has been studying this since his 1982 book “Symbolic Self-Completion” (pdf article here) — and recently published results of new tests in a research article, “When Intentions Go Public: Does Social Reality Widen the Intention-Behavior Gap?”
Four different tests of 63 people found that those who kept their intentions private were more likely to achieve them than those who made them public and were acknowledged by others."
I don't think this is complete. I think the product also has to fill a need, or an imagined need. An ancient quote from Thoreau, along a similar vein:
“Not long since, a strolling Indian went to sell baskets at the house of a well-known lawyer in my neighborhood. “Do you wish to buy any baskets?” he asked. “No, we do not want any,” was the reply. “What!” exclaimed the Indian as he went out the gate, “do you mean to starve us?” Having seen his industrious white neighbors so well off—that the lawyer had only to weave arguments, and, by some magic, wealth and standing followed—he had said to himself: I will go into business; I will weave baskets; it is a thing which I can do. Thinking that when he had made the baskets he would have done his part, and then it would be the white man’s to buy them. He had not discovered that it was necessary for him to make it worth the other’s while to buy them, or at least make him think that it was so, or to make something else which it would be worth his while to buy."
I think about this a lot. I have a good sense of how much time I spend working on things (say, editing photos and videos) but can often lose sight of how often evidence of that work is seen by others.
I might film a location, spend ages editing content from that location and upload it somewhere, but until I share that location, it doesn't count for much. It won't lead to sales, win new clients or build my profile. I often find myself going through the process and sharing the link with a friend or one client, and sitting back satisfied as though I'm finished. Have to then remind myself that I reached all of one person when I need to reach hundreds.
I need a dashboard tracking visitation to projects on the web, how often I talk about them on Twitter, or post examples on Instagram, etc.
Here's the thing about doing things and telling people.. if the people ghost you then it can seriously take you off balance and if what you did is even slightly hard to explain then they will ghost you.
It takes a lot of patience to do both the things and the telling to people but I think the latter actually takes more patience than the former even though it seems to be the other way around.
I can’t agree more… I graduated from an average university with average GPA, so I decided to do some personal projects to make my portfolio stand out. I always wanted to be a mobile app developer so I picked up Flutter and made some apps using it, most recently I have made a Hacker News client - Hacki: https://github.com/Livinglist/Hacki
I wonder how one generates interesting ideas to work on. Maybe I should look up product ideation as I am not really great when it comes creative idea generation.
Don't worry about that, perhaps ever, but at least not now. Just do something you want, perhaps fix a bug or whatever. And write about it. Get in the habit -- it will make you feel better.
Then you will discover that there are other people who are also interested in X. Also you'll discover that what you consider obvious info isn't obvious to others -- and they are smart people too.
I used to be shocked by some posts that made the front page: "that's just trivial stuff everybody knows". That was pure snobbery on my part: I only knew that stuff because I'd been exposed to it decades ago (and there's of course tons I don't know, and never will). And the posts that made it to the front page were good explanations -- and many started with "I didn't know this at all so I looked into it and this is what I learned"
You'll be amazed what support and interest you get from simply showing up.
And don't bother to try to have a readership in the instagram-influencer zone. Better to have a group of people who are interested because they are interested in things you also are.
It's eye opening once you realize that most interesting ideas are simply iterations or slight improvements on existing ones. Simply releasing a similar product with one additional feature, or at a cheaper price point can excite the right people. In my experience most ideation comes from building a product. Spend enough time in a product space or market and you will naturally come across "interesting" ideas. I put interesting in quotes, because when you are in that space, the idea of iterating on a product may seem obvious or mundane, but if you can execute, and convince the customer, it's anything but.
That’s a spot-on short article. Last year I came to similar conclusions. You need to build something. Small, big, funny, serious, relevant, stupid, doesn’t matter. Something that will make you learn new stuff (by building it).
I work in an industry (sports analytics) where most people now doing it professionally started out tweeting terrible charts based on scraped data. It's certainly still a very rich pipeline of talent even now. That said, be aware of the biases this introduces into your hiring pipeline - many people don't have time to do all the things, nor the platform to be heard by the people.
This is down to earth advice that matches my own experience. Here is a workflow app intend for work-from-home "gig economy" workers and contractors intended to be as easy to use as a search engine.
Still experimenting. https://workflow-magic-svelte.vercel.app/
This is very similar to my advice to people just starting out in whatever industry they picked to work: say yes more.
The unbelievable amount of opportunities I see people say no to put of insecurity in their abilities, out of social pressure, out of image concerns, etc… is absolutely baffling to me.
Say yes more. Be smart, keep your wits about you, but say yes more.
This advice is good for the subset of people who are realists. For the optimist, it's akin to saying 'do what you love and you won't work a day in your life.' The optimist will be happy to "make things" no one needs which have no practical value.
Got me inspired to bring my tiny tech blog portion of my site back. It's been a while, but you made me realize that it would be good to reflect on my work again. Really enjoyed this post.
Oh, man, Game Closure! I keep seeing their tiny little office space in the back of a Korean hot-pot restaurant just up the street. It's some sort of... game engine middleware company?
and sometimes the things you think aren't interesting, are interesting to other folks. I stumbled into the vidalia onion industry, and randomly wrote an essay on it, and wound up discovering other folks liked the story. HN is a blessing in this way - it helps expose these fringe stories that'd never see the light of day in normal print/web publications.
Yes! I've got a website with pages on a variety of subjects, and the most visited pages are on the most obscure topics, that I wouldn't have imagined anyone would ever be interested in. I guess those pages appear in a google search. If I was starting over, I'd just write on super-obscure topics. The more obscure the topic, the less "competition", the more often someone reads it.
I think I've been bad at the telling people side of thing.
In part because I've frequently built extremely esoteric projects, the sort of thing that's made even some technically minded people squint and go "huh, what?", real Terry Davis-tier stuff. Although these were mostly to scratch my own itches.
Even now when I'm building stuff that's more widely appreciable and aimed at the enrichment of lives, it still feels unnatural to promote my work.
I wrote this article a little over a decade ago, when I was still in college, and I knew almost nothing about the world. (I still don't, but I didn't then, either) It's fun to see that it still resonates with people despite the naivety and the slight sense of "I am headed for success" that makes it hard for me to re-read.
My initial reflection on the article was that I had fallen off the "do things, tell people" wagon a little bit, having let go of my twitter account and with it much of the easiest path to doing things and telling people. I still work in tech but I haven't created anything I'm particularly excited about in a good while, so I felt that I was failing my own ideal.
Then, of course, I remembered that I hiked all of Ireland's long-distance trails (https://toughsoles.ie) and have a reasonably successful, if niche, youtube channel (https://youtube.com/toughsoles). Funny that my mindset, especially regarding this article, is so tech-focused that I automatically discount something I've been doing for five years.
All that said, I'm starting to look for new opportunities in the larger-than-startup space. Shoot me an email at carl@flax.ie and let's chat.