Hacker News new | past | comments | ask | show | jobs | submit | iblaine's comments login

One of the best gifts you can ever gift yourself.


Is crypto or the pastor more to blame?

TL;DR; do the easy things fist, in this case it was to fix bad SQL

Given the options to optimize SQL, move read operations to replicas, shard data or go towards micro services, optimizing SQL is the easy choice.

Actually, I disagree. The "TL;DR:" in the article is "first outgrow, then upgrade". In today's software development practice, efficiency is second class citizen, because moving fast and breaking things is the way to keep the momentum and be hip.

However, sometimes everyone needs to chill and sharpen the tool they have at hand. It might prove much more capable than first anticipated. Or you may be holding the tool wrong to a degree.

A walking bike with a wheel in the front and 6 Theo Jansen type legs in the back


The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst.

SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.

Agreed. The job title of Data Engineer gets abused. Some DE's are Analysts, others are Backend SWEs. Their goals are aligned, to create value out of data, and their methods may differ greatly.

A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.

Same as “game developer”

But that’s not how markets work.

Also a job title doesn’t really say much. On all of these 4 roles there’s people doing trivial work, as there’s people doing very deep technical work. Same with pushing the envelope on tech.

I would certainly prefer it if that were the case, but it largely is not, in my experience.

It seems like your experience doesn't overlap with data engineering, in that case.

Ok. I think this is just going to end up being "no true scotsman". I'll say "in many companies, data engineers are more doing BI data preparation and presentation than building distributed processing infrastructure or whatever and they are compensated more like analysts than general software engineers" and you'll say "well that's not data engineering".

My argument is not an "ought" argument, it's an "is". I agree with you that data engineering "ought" to be, if anything, a more difficult specialized practice of software engineering, because you need all the software engineering skills and then also specialized data skills. But I'm saying that what it often "is", is more like a specialization of BI and analyst roles.

I think the OP is a pretty good demonstration of this. If "all you need is SQL", does that sound more like a specialization of a software engineer skillset or an analyst skillset? I think the latter... And sure you can say, "well the article is wrong, if all you need is SQL, that's not data engineering", but we're just back to "no true scotsman"; I believe it is common to see the role this way.

for a moment there I thought you said "what makes data engineers is excel"

do not speaks its name

I saw the kickstarted version of this at a maker fairy and was blown away. The product was great for what it did.

Idea for a new linter; throw a warning if it sees extreme non-intuitive uses of Python. Given the trajectory of Python improvements over the past few years, this linter may not be far off.

It’s monumentally frustrating to be forced to give estimates too soon in the planning process. The compromise, I hope, is to end up in a place where most of the problem is known and some is unknown, and you get there by breaking down the original problem. Leave it up to the “buyer” to digest your best estimate into their planning process.

Same thing happened at LinkedIn after the Microsoft acquisition. LinkedIn was using gmail, Google was quick to let the g suite contract die, so the timeline to move from gmail to Outlook was accelerated. Engineers at LinkedIn were livid about being forced to move off gmail.

> Engineers at LinkedIn were livid about being forced to move off gmail.

Is that really worth being livid over? Outlook is fine. I left gmail a few years ago and haven't looked back. I survived.

If I were building a tool to test SQL, then I'd try to load the SQL into a dataframe, then test it by mocking the tables and the output. This is a tough problem to solve. If testing is important, possibly move away from SQL and towards ORMs.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact