>Is there a reason for this? I would think this was a good chance to show off your ability to write succinct/standardized documentation w/ sections that anyone else would be glad to read/see.
I threw the docs in the README for the submission, but for HN readers they want to see the problem first. The submission didn't even include the specs.
>Yeah I can see this, but I think "automated tests" are quite big in peoples' minds. Feels like it would be a checkbox on the item.
Yeah you're completely right here. I'm gonna go with it next time.
>I disagree here -- table stakes for a good API is integration and E2E testing -- and they're the easiest to do (no need to manipulate a browser window, etc).
Yeah but many companies don't do integration testing as the infra on this is huge and writing tests is more complicated. It's certainly overkill for take home. I would say you're wrong here. Completely. This company was not expecting integration tests as part of the project at all.
I've worked for start ups most of my career, and I'm applying to start ups. It's mostly the huge companies that have the resources to spend the effort to have full coverage on that.
Integration tests are Not easy, btw. Infrastructure is not easy to emulate completely, how would I emulate google big query or say aws iot on my local machine? Can't, most likely integration tests involve actual infra, combined with meta code that manipulates docker containers.
>Sorry INPUT is what I meant to write there -- what happens if you get a -1 ?
You get a 500. Which is not bad. But you're right a 400 is better here.
>Right, but this is what I mean -- when I read that I wondered if it was complete, and it wasn't. That's an unnecessary ding to take.
I feel this is such a minor thing. I'm sure most people would agree if you dinged that you'd be the one going overboard, not the candidate.
>This is going to build objects dynamically every execution right? That's what I thought could be avoided. Less about lists vs generators, more about doing things that look like they allocate so much.
You're going to build objects dynamically every execution Anyway. This saves you the extra intermediary step of not having to save it to an extra list.
numbers = [i*2 for i in range(500)] #allocated list in memory: 500*sizeof(int) (big)
for i in numbers:
print(i + 1)
numbers = (i*2 for i in range(500)) #allocated generator in memory: sizeof(*generator) (small)
for i in numbers:
print(i + 1)
Look at the above. Same concept. Generators are drop in replacements for actual state. State and functions are the isomorphic (ie the same thing). Building a generator is cheaper then building a list. You should read SICP about this topic of how functions and data are the same.
>Ah OK, but this isn't what they asked for, right? If we want to be really pedantic, your endpoint returns a 3xx for every single error.
It's a reasonable assumption that all users assume /xx/xx equals /xx/xx/. It's also reasonable for a client to handle a 302 redirect. The specs didn't specify the exact definition on either of these things.
>The measurement endpoint will be called far more (100x, 1000x, ...) than the stats endpoint -- it's the hot path.
So? logN is super fast. You have 100 objects binary or whatever indexed insert redis uses gets there in at most 4 or 5 jumps. So even if it's the hot path redis can take it.
NLogN is slow enough that if N is large there is noticeable slow downs EVEN when it's a single call. Recall that redis is single threaded sync. NlogN can block it completely.
Either way this is all opinion here right? Ideally the results are returned unsorted. the client is the best place to sort this, but that's not the requirement.
>Also, note that sorting is zero cost, but retrieving the data will still be in N time including network round trips, which will likely dwarf any time you spend sorting in local memory. Even the conversion to JSON will dwarf the time you spend sorting for large N.
Of course the act of reading and writing data will cost N regardless. But your reasoning is incorrect. Look at that picture again: https://i.stack.imgur.com/osGBT.jpg. O(NlogN) will dwarf O(N) when N is large enough. It will even dwarf IO round trip time and has the potential of blocking redis completely on processing the sort.
>Also, the point of memory usage being potentially large is still relevant. My instinct is that when I see something returning a possibly very large piece of data, setting up for streaming is the way to go. Doing a scan you have the possibility to use far less memory for an individual request, and you could actually stream responses back in extreme cases, so you can avoid holding the entire result set in memory at all.
Sure but look at the specs. It's asking for http requests. It's not a streaming protocol here. I mean at this point if you were evaluating my code for a job you'd be going to far already as your going off the rails completely.
>The point about error handling still stands with respect to reporting though -- you generally want to see error handling in code, rather than just leaving it up to the runtime to fail the task and carry on.
I mean I could build a full logger and error handler framework here. I'd put custom handling in a decorator and have the error handling invisible in the http route handlers. Flask already does this, but it just returns 500 on all exceptions. That is the most elegant way imo. But again this is a bit outside of the scope of a takehome, given the framework handles a lot it and the extra effort required.
>If you want to get really fancy, you can get creative with how data is stored on the redis side and get the size to N.
For takehomes and in general people prefer readable code. Only optimize if it's needed. I mean I can write the whole thing in assembly code for ultimate performance. I can def see some nitpicker dinging me for optimized but unreadable code.
>The things you include/don't include by instinct I think is valuable signal.
Not to be insulting here. But I think your instincts are pretty off here. There's a limited amount of time assigned to this project. All the basics were included with what I did in the time allotted.
When you have good instincts you know when to apply automated testing, when to use error handling. These aren't things you should do all the time. Your attitude is something I see in many younger engineers. This attitude of rule following and best practices is just adhered to without thinking it all through properly. For the scope of the project. most of what you wrote here isn't needed.
You're also not testing what's traditionally tested in interviews. They test intelligence, depth knowledge and ability to design and solve problems. You're testing the ability to follow written rules. I mean following these things is trivial. It doesn't take much brain power to write error handling or unit tests. Perhaps you're just testing for someone that agrees with your programming philosophy. I mean these things can be followed regardless
Your mention of integration tests is telling. I mean the code for integration tests can rival the size of the code in the project itself with the a minor benefit in safety . It's not just take home assignments but many many many companies loaded with senior engineers skip over automated integration tests and keep those tasks in todo lists for the longest time. There's a huge engineering cost to building out testing infra to the maximum ideal, and many companies just don't bother. It's mostly the bigger well established companies that have the resources to do this.
>I'd disagree that it's worse than algo interviews, because algos are rarely the most important thing in a professional code base, especially one with that's about web services and not something more demanding.
It is worse in my opinion because it's biased. What if your opinion was more correct then my opinion but I judged you based off of my world view? Would that be right? No it wouldn't. Programmers are diverse and the best organizations have diversity in opinions and expertise.
All in all I mostly disagree but I think your right on one thing: At least one of those job candidate evaluators that are looking at my code likely had a similar attitude as you do and I should bias my code towards that more. Likely won't write integration tests though, hard no on that one.
Also good catch on the /test/-1/. That's definitely a legit ding imo.
> Yeah but many companies don't do integration testing as the infra on this is huge and writing tests is more complicated. It's certainly overkill for take home. I would say you're wrong here. Completely. This company was not expecting integration tests as part of the project at all.
>
> I've worked for start ups most of my career, and I'm applying to start ups. It's mostly the huge companies that have the resources to spend the effort to have full coverage on that.
>
> Integration tests are Not easy, btw. Infrastructure is not easy to emulate completely, how would I emulate google big query or say aws iot on my local machine? Can't, most likely integration tests involve actual infra, combined with meta code that manipulates docker containers.
I think I can agree with this -- it's true that most companies don't do it, but personally just spinning the thing up and throwing it a web request usually has rails in most languages these days. I spend more time in the NodeJS ecosystem, so I have things like supertest (https://www.npmjs.com/package/supertest) so maybe I'm spoiled.
> I feel this is such a minor thing. I'm sure most people would agree if you dinged that you'd be the one going overboard, not the candidate.
Yeah this is pretty reasonable -- opting in to typing at all is more a plus than a negative, on balance.
> You're going to build objects dynamically every execution Anyway. This saves you the extra intermediary step of not having to save it to an extra list.
It seems like I wasn't clear about how it's not about the lists/generators, so here's an explicit instance:
If your argument is that the lambda gets optimized out, or that the benchmarked difference is insignificant (it very well could be!), then I could understand that.
> It's a reasonable assumption that all users assume /xx/xx equals /xx/xx/. It's also reasonable for a client to handle a 302 redirect. The specs didn't specify the exact definition on either of these things.
I'd argue the difference is so significant that it warranted a note in the documentation on the semantics, and if you google things like "trailing slash" it's been a thorn in peoples' sides for a long time.
Specs did specify test cases -- and none of them had a trailing slash.
I think we're really in the weeds here (in any reasonable working environment, this isn't a big deal), but it's wasteful to have every request become 2.
> So? logN is super fast. You have 100 objects binary or whatever indexed insert redis uses gets there in at most 4 or 5 jumps. So even if it's the hot path redis can take it.
>
> NLogN is slow enough that if N is large there is noticeable slow downs EVEN when it's a single call. Recall that redis is single threaded sync. NlogN can block it completely.
>
> Either way this is all opinion here right? Ideally the results are returned unsorted. the client is the best place to sort this, but that's not the requirement.
These are good points. Thinking about it though:
- logN is not faster than O(1)
- NLogN is in the API, not redis -- redis experiences N while the scan is going
You're right that NLogN would certainly have a chance of blocking redis completely much more than N would, but I think in both cases redis experiences O(N) behavior for the stats endpoint, not NLogN.
> Of course the act of reading and writing data will cost N regardless. But your reasoning is incorrect. Look at that picture again: https://i.stack.imgur.com/osGBT.jpg. O(NlogN) will dwarf O(N) when N is large enough. It will even dwarf IO round trip time and has the potential of blocking redis completely on processing the sort.
Ahh, see the point about N/NLogN -- it's not Redis that experiences NlogN, it's the API. Redis just happily feeds all the values down, and they're aggregated at the python level.
Now, as far as the python level goes, I'm arguing that NLogN for in-memory number is going to be tiny compared to doing network requests and serializing JSON.
> Sure but look at the specs. It's asking for http requests...
That's reasonable -- the assumption is basically that the return fits in one request. I do think not trying to download the whole world at the same time is a legitimate concern, but it's not necessarily a pressing one since it wasn't mentioned.
> I mean I could build a full logger and error handler framework here...
Ah see I would expect you to pull one in -- just like you know of poetry, I'd be impressed by seeing more well-built creature comfort tools like that. Knowing good off-the-shelf tooling is a plus, IMO.
I agree with you that this is probably not what they expected (it's only a 4 hour take home!) but just wanted to make the point.
> For takehomes and in general people prefer readable code. Only optimize if it's needed. ...
Yeah that's true, too much optimization would definitely be bad -- but my point wasn't that you should optimize, it was that the approach I was talking about could be optimized (so that would be my answer to the NLogN versus N questions).
I think my version is the least surprising one -- no one has to know about pipeline or worry about atomicity. Just an O(1) operation to redis, like most people would expect to see.
> Not to be insulting here. But I think your instincts are pretty off here. There's a limited amount of time assigned to this project. All the basics were included with what I did in the time allotted.
>
I think maybe I wasn't clear -- your instinct is what's on display. What you choose to include/use/not use is valuable signal (and that's the point of the take home).
> When you have good instincts you know when to apply automated testing, when to use error handling. These aren't things you should do all the time...
Well, if making sure to include automated testing for APIs and error handling is junior, I look forward to staying a junior engineer for my whole life. I don't think I'd submit a take home coding test without tests.
I'm absolutely not dogmatic about it (and I have the repos to prove I don't always write tests :), but I certainly am not proud to show anyone untested code -- it is enshrined in my mind as a measure of quality. If I'm putting my best foot forward, the code will have tests, especially when it's languages with weak type systems.
If the prompt was "write this like you're at a startup that has no money and no time", then sure. But even then -- technical debt forces rewrites at startups all the time, and the often the effect is worse when people don't write tests. Most of the time, you move slow so you can move fast.
> You're also not testing what's traditionally tested in interviews. They test intelligence, depth knowledge and ability to design and solve problems. You're testing the ability to follow written rules. I mean following these things is trivial. It doesn't take much brain power to write error handling or unit tests. Perhaps you're just testing for someone that agrees with your programming philosophy. I mean these things can be followed regardless
>
I think this is the difference between take homes and whiteboard algo tests. I think the point of the take home is to see what you will do, on a realistic project that you were given full control over. That's what makes it better than the algo tests IMO -- it's more realistic and you have full control.
Everyone knows they should write tests... But do you actually? Because one thing is definitely harder/takes more discipline than the other.
> Your mention of integration tests is telling. I mean the code for integration tests can rival the size of the code in the project itself with the a minor benefit in safety . It's not just take home assignments but many many many companies loaded with senior engineers skip over automated integration tests and keep those tasks in todo lists for the longest time. There's a huge engineering cost to building out testing infra to the maximum ideal, and many companies just don't bother. It's mostly the bigger well established companies that have the resources to do this.
Yes, but this is a small API -- you literally have to write a test that hits the server once. There are libs for doing this with flask, there is documentation showing you how. It's not rocket science, and it's crucial to catching bugs down the road.
> It is worse in my opinion because it's biased. What if your opinion was more correct then my opinion but I judged you based off of my world view? Would that be right? No it wouldn't. Programmers are diverse and the best organizations have diversity in opinions and expertise.
>
> All in all I mostly disagree but I think your right on one thing: At least one of those job candidate evaluators that are looking at my code likely had a similar attitude as you do and I should bias my code towards that more. Likely won't write integration tests though, hard no on that one.
>
> Also good catch on the /test/-1/. That's definitely a legit ding imo.
Just about everything with humans in the loop is biased -- that's the world we live in. I don't know if they tried to run your test, but it could have failed with a 3xx and then they disqualified you right there, or bad input or whatever.
But yeah unfortunately it's hard to learn anything from this other than giving opinions on what rang bells for me.
Would be awesome if they gave you feedback.
Overall the code is reasonable and probably works for the normal cases -- it is a mystery why they wouldn't give feedback.
>Well, if making sure to include automated testing for APIs and error handling is junior, I look forward to staying a junior engineer for my whole life. I don't think I'd submit a take home coding test without tests.
Error handling is already done by the framework. If you want explicit handles that's just busy work. Most python crashes involve traceback sent to stderr telling you what error occured and returns a 500. which is exactly the kind of handling I want. I'd just be replicating the framework code with the same logic everywhere if I did what you mentioned.
I'll give you credit for the unit tests. There's only 2 unit testable functions in the entire code and it's all in utils.py. Practically speaking only the random path generator is worth any sort of automation testing, and the http request callee is io, so not unit testable. Integration tests are raw overkill on takehomes. Manipulating docker compose in test code is just overboard.
Are you currently a junior? If you are then the insight you should eventually achieve is that unit tests for web application servers is not as important as you currently think.
This is because web servers are primarily io based applications. There's overall not much pure logic you can abstract to unit tests.
Pure logic rarely has errors as it doesn't deal with state. You will find that unit tests become a lot of extra busy work. If your pure logic minimizes mutations and you properly use typing and use functional code your error rate will drop to the point where you mostly write unit tests for other people. Writing it for your own code is essentially writing thousands of lines only to find it didn't catch a single error. It's still good to have unit tests but the cost benefit analysis becomes out of whack the majority of the time.
Did you know Edward Dijkstra was famously against testing? He promoted a different sort of art that would guarantee your code worked 100 percent of the time for all cases. Say I had a unit test: assert(add(100, 1) 101). That only covers one test case out of trillions of possible errors. His method would cover the entire spectrum of possible cases without a single test.
Basically a good coder should be hybridized in his approach. Code in a way that uses Dijkstra's philosophy such that the need for tests are minimized then write the tests you need without excessive focus on it because you should know that your tests are only covering a fraction of possible test cases anyway.
Not all engineers discover this method or even take the time to internalize and utilize it. Thus their code often ends up being error prone enough to justify unit testing or test driven development. Many super senior engineers don't even know about this. The staff position is more of a political title that has to do with leadership and architectural planning.
Anyway, unique to backend engineering... Integration tests are where most errors occur. It's where state is mutated and where race conditions, dead lock and where most compute logic is directed towards. It's 100x more important to focus on testing at the integration level.
The problem here though is, like I said, the infra around integration tests is huge. So often it doesn't get done and you see a bunch of companies just end up using zero integration tests and some small set of unit tests that are mostly inconsequential.
>but I certainly am not proud to show anyone untested code -- it is enshrined in my mind as a measure of quality
Deshrine this if you can. There are many techniques outside of this single methodology of testing for writing good code.
For example, have you heard of Idris? Idris is a language where you can use the type system to prove the program is correct. Test cases only cover correctness for a specific case. For Idris you can write a sorting algorithm and literally prove it to be correct for all cases without writing a single test.
I'm not a complete adherent to total proof based methodologies. As mentioned previously I follow a hybrid approach such that my need for testing is minimized. Remember Enshrining any concept blinds you from alternative paths.
I view excessive unit tests as a crutch. Any programmer who actually relies on unit tests during programming to catch a lot of his own errors actually imo has a flawed model inside his head on how to compose logic while minimizing errors.
The experience of a web application programmer with a good model of logic composition in his head is that the unit tests should be passing asserts the overwhelming majority of the time.
If you are consistently using it to pick out hidden logic errors in your code it sort of works from a practical standpoint but you're not on the ideal path.
Unit tests are good, don't get me wrong but Enshrining the concept and relying on it, is a crutch.
>Everyone knows they should write tests... But do you actually? Because one thing is definitely harder/takes more discipline than the other.
They only do it if the entire team does it. Otherwise the majority of people don't do it in the face of looming deadlines. Unit tests are usually baked into dev culture, if standards require it they'll do it. Thus logically if unit testing is required it's not really that important to measure this aspect during the interview as they will write it regardless of whether or not they do it in the interview.
> Error handling is already done by the framework. ...
You literally had a bug in your code that would have been caught with testing, and it did not have to do with the logic but with input sanitization, which has to do with your integration with what the framework provided you.
I am not advocating for unit tests, I was advocating for any tests at all, in the code sample. I already said that the highest value tests in my mind are E2E tests. The ones that make sure your code does what it says it does, in as black-box a way as possible.
Also, I want to note that the logic you're applying works best with languages with strong type systems, not Python.
I don't know what your point was on not testing is (were you trying to allude to property testing/fuzzing?), but here in the real world, tests are good for testing edge cases and preventing regressions.
I am only comfortable skipping tests completely in languages with strong, expressive type-systems -- Typescript, Rust and Haskell. Even then, I still make sure to write regression tests and think of what can go wrong.
Again, YOU think that the difficulty to writing integration tests is huge, and that's because you don't do it a lot. To me that's a liability, not something I want to adopt -- the faster I can write tests and prevent regressions, the better code I am able to write, the easier it is to integrate my code and have confidence that changes don't break existing functionality.
> Deshrine this if you can. There are many techniques outside of this single methodology of testing for writing good code.
Absolutely not -- no matter how strong your type system is, it will bend and break during contact with the real world. E2E tests, regression tests and the like are crucial for good engineering.
> For example, have you heard of Idris? Idris is a language where you can use the type system to prove the program is correct. Test cases only cover correctness for a specific case. For Idris you can write a sorting algorithm and literally prove it to be correct for all cases without writing a single test.
>
> I'm not a complete adherent to total proof based methodologies. As mentioned previously I follow a hybrid approach such that my need for testing is minimized. Remember Enshrining any concept blinds you from alternative paths.
>
> I view excessive unit tests as a crutch. Any programmer who actually relies on unit tests during programming to catch a lot of his own errors actually imo has a flawed model inside his head on how to compose logic while minimizing errors.
>
> The experience of a web application programmer with a good model of logic composition in his head is that the unit tests should be passing asserts the overwhelming majority of the time.
>
> If you are consistently using it to pick out hidden logic errors in your code it sort of works from a practical standpoint but you're not on the ideal path.
>
> Unit tests are good, don't get me wrong but Enshrining the concept and relying on it, is a crutch.
You seem to be battling one heck of a strawman here. I have not said that you should write excessive unit tests. I said that you should have at least any tests, and that I find the most value in E2E tests.
Also, Idris is basically Haskell w/ dependent types, but what you're saying about it is full of half-truths. You cannot use the type system to prove a program is correct, you use type systems to prove a program is sound. Correctness is problem dependent, input dependent, and many other things that are situation-dependent.
With a strong type system, you can avoid unit tests, but only if you use that type system properly (ex. using a Natural instead of an Integer where it is appropriate).
Regardless of whether you have a strong type system or not, tests (of all kinds, but especially E2E and regression) are valuable and have their place.
> They only do it if the entire team does it. Otherwise the majority of people don't do it in the face of looming deadlines. Unit tests are usually baked into dev culture, if standards require it they'll do it. Thus logically if unit testing is required it's not really that important to measure this aspect during the interview as they will write it regardless of whether or not they do it in the interview.
I think it's a matter of culture. Are you the person that makes time to write these/makes it part of your plan, or not?
>You literally had a bug in your code that would have been caught with testing, and it did not have to do with the logic but with input sanitization, which has to do with your integration with what the framework provided you.
No this bug wouldn't have been caught. I'd have to write the test for it.
I did do integration tests. I just did them manually. If I wrote all the manual tests and made them automated this "error" would still be there. It's a matter of luck whether I manually test or write the integration test that finds this error.
It's also technically not huge bug. The server returns a 500 error instead of a 400 error.
>I am not advocating for unit tests, I was advocating for any tests at all, in the code sample. I already said that the highest value tests in my mind are E2E tests. The ones that make sure your code does what it says it does, in as black-box a way as possible.
Ok you got this. You're right on this. I should put in some testing here. even if it's minimal.
>Also, I want to note that the logic you're applying works best with languages with strong type systems, not Python.
This is where you are wrong. Python has what I call an opt-in type system meaning it's optional and it can be as strong as you want it to be. You choose the external type checker like mypy and those type checkers can be as powerful as you want.
As you've seen with my JSON type, python allows for recursive types and sum types. Which makes it more versatile than golang which can't do either even though it has a "strong" type system. All the bells and whistles for a modern type system are there and more.
In fact I believe python may even exceed typescript in power. I believe type script won't do exhaustive checks on sum types.
>I am only comfortable skipping tests completely in languages with strong, expressive type-systems -- Typescript, Rust and Haskell. Even then, I still make sure to write regression tests and think of what can go wrong.
So modern python matches these language in type level power, it's just optional so it's not as pervasive as typescript but the option is there and when utilized it's equal if not better in power. But proof based coding isn't solely just on using types to prove out your code. I just used idris as an example of an alternative path. The reality is Idris is a bit too extreme because coding those types takes too much effort.
Basically i change my coding style to both minimize errors and allow me to build a minimal proof in my head. I use coding style, proofs in my head, and a strong/versatile type system all in conjunction to minimize error rate to low enough levels that tests are less important.
>Again, YOU think that the difficulty to writing integration tests is huge, and that's because you don't do it a lot. To me that's a liability, not something I want to adopt -- the faster I can write tests and prevent regressions, the better code I am able to write, the easier it is to integrate my code and have confidence that changes don't break existing functionality.
You can't do integration tests without using infrastructure and spinning up processes. Let me put it to you plainly. It's not ME who thinks it's difficult. It's the majority of the developer population knows it's much more difficult then unit tests. That's why such a segregation even exists. The fact that we categorize the tests into integration tests and unit tests speaks to huge differences between the two in difficulty.
You may think integration tests are easy, but that's just your own opinion. This is different from the overall general opinion which is more inline with what I'm saying.
>Absolutely not -- no matter how strong your type system is, it will bend and break during contact with the real world. E2E tests, regression tests and the like are crucial for good engineering.
Sure I can agree with your philosophy in spirit. Practically speaking though there's the effort involved in this that makes it so that many companies just don't do it.
Maybe it's changed recently, I've moved from backend web to embedded systems to arm devices and GPU programming so I've been away from web stuff for about 3 years. Generally no docker or as much infra in this universe. If docker-compose and mocking infra locally is universal on everyones local machine now, then yeah you could be right but from what I'm seeing... I don't think so.
Unless we have metrics I'm still going to go with the fact that in general it's not easy and most companies don't do it.
>You seem to be battling one heck of a strawman here. I have not said that you should write excessive unit tests. I said that you should have at least any tests, and that I find the most value in E2E tests.
Alright. I get what you're saying. Write tests, integration tests are the most important thing here. Unit tests are basically more or less practically useless for this takehome. And specific for this takehome I could've written integration tests here by writing some hack python script to call docker-compose. I agree.
Where we don't agree is the difficulty level of writing these tests and the general pervasiveness of integration or E2E tests in the industry overall. We can just leave it at that, without metrics the argument goes in circles. We've taken it as far as we could.
>Also, Idris is basically Haskell w/ dependent types, but what you're saying about it is full of half-truths. You cannot use the type system to prove a program is correct, you use type systems to prove a program is sound. Correctness is problem dependent, input dependent, and many other things that are situation-dependent.
No this is truth. You're just misinformed.
Dependent types are enough to prove a program is correct given a set of specifications. Of course not everything can be proven. But most things can. You can't prove the halting problem for example, but you can prove specs like determining if a list is sorted: https://dafoster.net/articles/2015/02/27/proof-terms-in-idri...
Theorem proving is the title. or aka proving anything mathematical or aka proving mathematical programs meet mathematical specs. Dependent typing is one way to go about this. There are other ways. The intended design of idris is similar to TLA+ the intention is to allow the user to prove the program correct to a spec and avoid unit tests via proof.
>With a strong type system, you can avoid unit tests, but only if you use that type system properly (ex. using a Natural instead of an Integer where it is appropriate).
How is using a nat more safe than an int? A nat just makes it more general.
I'm not 100% in agreement here. Let me add some nuance to my opinion. Unless the type system is more powerful then what you have in say haskell, it can only reduce the need of unit tests. You can't completely avoid it, but it can be reduced to the point where practically speaking you don't always need it. Certainly not for this takehome.
With something like idris you can, technically speaking, eliminate the need for unit tests all together. But this is a technical thing. In practice it's really hard to achieve this due to how hard it is to write these proofs. But the technical level of safety is operating at the highest level here.
>I think it's a matter of culture. Are you the person that makes time to write these/makes it part of your plan, or not?
Depends on the situation, the timelines, the brittleness of of the project and the technology. Highly, highly dependent on all sorts of critical variables. I'll tell you this: I don't enshrine the concept of testing. It's always a case by case basis.
If i had unlimited time though, I think your philosophy would be correct. Always write tests. But with limited time I artfully choose between a myriad of techniques (including choosing what to test) to meet certain timelines in the best way possible. This sometimes means not writing tests.
Also sometimes laziness is a factor, I'll be honest. If I'm confident my toy project or take home is correct I just won't write unit tests even if I have unlimited time. I think what I learned from this overall thread and from you is that I should always write a bunch of tests for the take home.
Hey I know this thread got a little heated. I want you know that I appreciate your viewpoints even though I disagree with a lot of points. I think heated but respectful debates are healthy and I definitely learned something here. Conversations where people are overly polite I feel like I learn nothing because people rather be polite then honest. So appreciate all the time you took to reply. I may be clashing hard against your points but there's nothing but respect here. You are clearly experienced.
> No this bug wouldn't have been caught. I'd have to write the test for it. ...
Ah OK, so this is where I think that we're talking past each other. When I say "testing", and the reason I focus on making them automated is that when I sit down to write tests, if I'm really up to it I get myself in a mood where I'm thinking of what can go wrong.
Doing that is what I'm suggesting might have made the difference.
> This is where you are wrong. Python has what I call an opt-in type system meaning it's optional and it can be as strong as you want it to be. You choose the external type checker like mypy and those type checkers can be as powerful as you want. ...
>
> The python type system can even do Haskell/Rust level exhaustive matching on sum types: https://tech.preferred.jp/en/blog/python-exhaustive-union-ma...
>
So this comes back to me not writing much python -- but I have used typing + mypy briefly a while ago, but I'd need to see some claims it is nowhere near as powerful as Typescript/Rust/Haskell.
I'm not a python + mypy regular user though, so I don't know where it's limits are, but particularly with the haskell comparison, it does not support GADTs (https://github.com/python/mypy/issues/8252). Also, I know it doesn't support Rust's affine type system (ownership/borrows).
It's true that mypy could in theory do this, but it does not, and I do not expect it to anytime soon (backing for the other projects is more sound anyway).
I personally believe NodeJS is the best of the "scripting" languages, but that's a whole 'nother discussion.
> So modern python matches these language in type level power, it's just optional so it's not as pervasive as typescript but the option is there and when utilized it's equal if not better in power. But proof based coding isn't solely just on using types to prove out your code. I just used idris as an example of an alternative path. The reality is Idris is a bit too extreme because coding those types takes too much effort.
>
> Basically i change my coding style to both minimize errors and allow me to build a minimal proof in my head. I use coding style, proofs in my head, and a strong/versatile type system all in conjunction to minimize error rate to low enough levels that tests are less important.
I agree on being flexible, I like using Haskell/Rust because of this as well. I certainly disagree that python matches any of those others in type level power though.
> You can't do integration tests without using infrastructure and spinning up processes. Let me put it to you plainly. It's not ME who thinks it's difficult. It's the majority of the developer population knows it's much more difficult then unit tests. That's why such a segregation even exists. The fact that we categorize the tests into integration tests and unit tests speaks to huge differences between the two in difficulty.
>
> You may think integration tests are easy, but that's just your own opinion. This is different from the overall general opinion which is more inline with what I'm saying.
Regardless of what other devs thing of it, it's worth doing IMO. Here's the project I linked before
> Dependent types are enough to prove a program is correct given a set of specifications. Of course not everything can be proven. But most things can.
Disagree with this. For example, memory leaks. If I define "correct" to be "doesn't leak memory", Haskell (for example -- don't know what the memory is like in Idris) is going to have a hard time.
There are things that are hard/impossible to encode into dependent types.
> Theorem proving is the title. or aka proving anything mathematical or aka proving mathematical programs meet mathematical specs. Dependent typing is one way to go about this. There are other ways. The intended design of idris is similar to TLA+ the intention is to allow the user to prove the program correct to a spec and avoid unit tests via proof.
Theorem proving != engineering, and this is why Haskell/Idris/ML languages at large never hit mainstream properly. If you try to write the kind of types that could even attempt to get correctness absolutely right, less than 0.1% of programmers would be able to use or work on your codebase.
I don't think TLA+ is relevant here because it's mostly used for testing distributed/concurrent systems in practice, and is not a PL so-to-speak.
> How is using a nat more safe than an int? A nat just makes it more general.
Naturals (& peano numbers) are the categorical example of dependent types that are useful -- in Haskell, natural numbers are the set of positive integers including zero. It makes the -1 case impossible (i.e. fail to parse @ the program boundary), which is the point.
> With something like idris you can, technically speaking, eliminate the need for unit tests all together. But this is a technical thing. In practice it's really hard to achieve this due to how hard it is to write these proofs. But the technical level of safety is operating at the highest level here.
I think we're agreeing, but you just said two very different things -- most things (especially important ones) cannot be easily solved at the type level.
How do you type-check a redis connection at compile time? or a database like postgres at runtime? You can't, really -- because they are inherently dynamic things.
Now in the case of unit tests, I'd argue that a good type system is one of the only things that can prevent you from having to write lots of unit tests. I'm pretty sure we agree on this!
> I'll tell you this: I don't enshrine the concept of testing. It's always a case by case basis.
Yeah this is where we disagree -- I think this case called for it, you disagree.
We agree that sometimes tests are not necessary or warranted.
> Hey I know this thread got a little heated. I want you know that I appreciate your viewpoints even though I disagree with a lot of points. I think heated but respectful debates are healthy and I definitely learned something here. Conversations where people are overly polite I feel like I learn nothing because people rather be polite then honest. So appreciate all the time you took to reply. I may be clashing hard against your points but there's nothing but respect here. You are clearly experienced.
No problem, always glad to have a debate, and being able to have them on HN is why I still come to this site!
We can disagree, and I benefit from learning where your disagreements are (I don't there are actually many, I think our stances on testing and types are very similar outside of me just not liking in python in any meaningful way).
>Ah OK, so this is where I think that we're talking past each other. When I say "testing", and the reason I focus on making them automated is that when I sit down to write tests, if I'm really up to it I get myself in a mood where I'm thinking of what can go wrong.
>Doing that is what I'm suggesting might have made the difference.
What? I get myself in the mood for manual tests too.
We're not talking past each other. You can't just quote "testing" and embed extra meaning into that after the fact. There's no way I would know what you're talking about.
>I'm not a python + mypy regular user though, so I don't know where it's limits are, but particularly with the haskell comparison, it does not support GADTs (https://github.com/python/mypy/issues/8252). Also, I know it doesn't support Rust's affine type system (ownership/borrows).
Nah when it comes to the lang extensions almost nothing comes close. I'm just talking about regular haskell ADTs.
The affine stuff isn't relevant as python uses a GC. What does it even mean to do borrow checking when nothing owns a variable in python?
It's quite obvious I'm just referring to the ADTs here.
Python supports regular ADTs better than typescript. So it's better imo. But that's just my initial opinion, I don't use ts extensively so it might be better.
>I personally believe NodeJS is the best of the "scripting" languages, but that's a whole 'nother discussion.
If You mean JS. then no, JS is horrible. Typescript has an argument here, I think that's what you mean.
>I agree on being flexible, I like using Haskell/Rust because of this as well. I certainly disagree that python matches any of those others in type level power though.
It matches it in terms of ADTs and where relevant.
>Theorem proving != engineering, and this is why Haskell/Idris/ML languages at large never hit mainstream properly. If you try to write the kind of types that could even attempt to get correctness absolutely right, less than 0.1% of programmers would be able to use or work on your codebase.
So? I never made this claim. I just said there are other methodologies to employ other then strict fanatical loyalty to testing. I simply used Idris as an example of an alternate path. In fact I specifically STATED that it would be hard to use proofs with Idris.
>I don't think TLA+ is relevant here because it's mostly used for testing distributed/concurrent systems in practice, and is not a PL so-to-speak.
The point was using proof to write correct programs. So it's completely relevant as TLA+ is used for that.
>Naturals (& peano numbers) are the categorical example of dependent types that are useful -- in Haskell, natural numbers are the set of positive integers including zero. It makes the -1 case impossible (i.e. fail to parse @ the program boundary), which is the point.
ah my mistake I had a brain fart and thought you were referring to Haskells Num or something along those lines. Yes if by Nat you mean Haskell Nat or unsigned int, then that makes sense.
>I think we're agreeing, but you just said two very different things -- most things (especially important ones) cannot be easily solved at the type level.
I never said two different things. I think you just made some assumptions and responded to some paragraphs without reading far enough. Like your "Theorem proving != engineering," comment. If you read further you'd realize I already know about what you're saying.
My overall point on this front is aspects of proving can help type checking and FP and other techniques reduce error rate by a large margin.
>How do you type-check a redis connection at compile time? or a database like postgres at runtime? You can't, really -- because they are inherently dynamic things.
I'm not sure why you're telling me this. Did I make a claim that type checking can work on a redis connection?
Do you remember when I said integration tests are more important then unit tests? And unit tests are mostly practically trivial? Well type checking is sort of involved with that. I mean these are points I've been making throughout our conversation suddenly your like "How do you type check a redis connection?" as if I haven't made any of those points. I made a point and you made the same point.
Anyway we agree here.
>Yeah this is where we disagree -- I think this case called for it, you disagree.
Yes. To get into more detail here. The point is the infra cost of integration tests we disagree on that, I don't think it's worth it all the time I think you believe it's worth it in more cases.
For the take home project I still don't think it's worth it. But I concede I'm wrong because the Reviewer is looking for tests and I need to cater to what the reviewer is looking for.
>We agree that sometimes tests are not necessary or warranted.
Well you told me you have testing enshrined. So I was disagreeing with that. I think you were just making a statement about the importance of testing. In actuality we're more in agreement about the nuance of testing.
>No problem, always glad to have a debate, and being able to have them on HN is why I still come to this site!
Sometimes these conversations generate some heat on HN. I actually don't mind the heat, keeps people brutally honest.
> What? I get myself in the mood for manual tests too.
> We're not talking past each other. You can't just quote "testing" and embed extra meaning into that after the fact. There's no way I would know what you're talking about.
Writing tests generally means... thinking about tests to write? If you think that thinking of edge cases is not a part of writing tests, then I don't know what to tell you.
I won't say any more on the subject.
> Nah when it comes to the lang extensions almost nothing comes close. I'm just talking about regular haskell ADTs.
>
> The affine stuff isn't relevant as python uses a GC. What does it even mean to do borrow checking when nothing owns a variable in python?
>
> It's quite obvious I'm just referring to the ADTs here.
>
> Python supports regular ADTs better than typescript. So it's better imo. But that's just my initial opinion, I don't use ts extensively so it might be better.
We're not considering language extensions (which are built in BTW, on par with using the stdlib), but we're considering completely separate ecosystem plugins that do type checking? Nonsense.
Tracking usage of variables is useful, whether you use GC or not -- it's a matter of ability in the type system. My point is that you cannot specify in your code that a value should be used "at most once", which is what affine types afford you.
I will not say more on this -- the point is absurd from the beginning, that mypy is as advanced as the literal PhD marsh that is Haskell or innovation that was Rust's borrow system for decades.
If you think mypy is better than Typescript, then we have nothing to talk about -- it's just unlikely for me to gain anything from that discussion.
> If You mean JS. then no, JS is horrible. Typescript has an argument here, I think that's what you mean.
No, I mean JS, and in particular NodeJS as an execution platform, because it has no GIL, can do threads, async io is a first class concept, were flexible enough to get used to transpilation (which lets something like Typescript exist).
> ah my mistake I had a brain fart and thought you were referring to Haskells Num or something along those lines. Yes if by Nat you mean Haskell Nat or unsigned int, then that makes sense.
The type is called Natural, and I wrote Natural.
> I'm not sure why you're telling me this. Did I make a claim that type checking can work on a redis connection?
You said that it can prove a system to be "correct". Unfortunately I can't know what you meant by "correct", but the type system will not help you with many of the practical issues that are most important when writing code.
That's where good engineering comes in.
This will be my last on this discussion, was good!
>We're not considering language extensions (which are built in BTW, on par with using the stdlib), but we're considering completely separate ecosystem plugins that do type checking? Nonsense.
It's not nonsense, the language extensions should not be included because the functionality in Haskell is so far and above what ANY type checker typically does. Clearly.
You're being utterly too pedantic here. When I say python types match haskell in power I am obviously not touching upon GADT or RankN. You're getting into the weeds.
It has nothing to do with extensions being "on par with stdlib"
>Tracking usage of variables is useful, whether you use GC or not -- it's a matter of ability in the type system. My point is that you cannot specify in your code that a value should be used "at most once", which is what affine types afford you.
Yeah but not strictly necessary for python which has a GC, but strictly necessary for Rust which has ownership.
>I will not say more on this
Yeah don't bother. Sort of rude. But whatever.
>No, I mean JS, and in particular NodeJS as an execution platform, because it has no GIL, can do threads, async io is a first class concept, were flexible enough to get used to transpilation (which lets something like Typescript exist).
It can't do threads. I just looked it up. Worker Threads are processes that use some form of IPC. Literally a whole new v8 engine. I was right. Also JS is a horrible language. It's crazy you use haskell and you're ok with things like undefined which basically is a nonsense instance that can flow from one end of your code to another.
But this has nothing to do with anything does it? I thought you were referring to a more general type Num.
>You said that it can prove a system to be "correct". Unfortunately I can't know what you meant by "correct", but the type system will not help you with many of the practical issues that are most important when writing code.
And I wrote extensively on what that means and you read it so you know what I'm talking about. There's no need to get pedantic and argue on pedantic points that are obviously contrary to the obvious meaning of what I'm saying.
>That's where good engineering comes in.
Yep, but do you have a point? Why make this statement?
>This will be my last on this discussion, was good!
I don't think you think it was good. I think you're annoyed. That's why you want to cut it off.
>If your argument is that the lambda gets optimized out, or that the benchmarked difference is insignificant (it very well could be!), then I could understand that.
In interpreted languages I believe the thing that gets saved to mem is a pointer to the code itself. So no inherit difference here in terms of allocation. For closures reference counting is increased for the referenced variables but this is not a closure.
Still this is a minor thing. Scoped functions are used for structural purposes performance benefits of using or not using them are negligible.
>logN is not faster than O(1) - NLogN is in the API, not redis
My point is log(n) is comparable to o(1). It's that fast.
NlogN is not comparable to O(n). In fact if you look at that pic NlogN is more comparable to O(n^2).
You definitely don't want NlogN on the application server whether its nodejs or python. That will block the python or node thread completely if n is large.
It's preferable to use SORT in redis then in python or node because redis is highly optimized for this sort of thing (punintended) as it's written in C. Even if its not in redis, in general for backend web development you want to move as much compute towards the database and off the webserver. The webserver is all about io and data transfer. The database is where compute and processing occurs. Keep sorting off web servers and leave it all in the database. You're a nodejs dev? This is more important for node given that the default programming model is single threaded.
Overall it's just better to use sets with scores because logN is blazingly fast. It's so fast that for databases it's often preferential to use logN tree based indexing over hashed indexing even when hashed indexing is appropriate.
Yes hashed indexing are O(1) average case insert and read but the memory savings of tree based indexing overshadows the negligible logN cost.
>I think my version is the least surprising one -- no one has to know about pipeline or worry about atomicity. Just an O(1) operation to redis, like most people would expect to see.
Two things here.
1st. Sorting on the web API is definitively wrong. This is especially true in nodejs where the mantra is always non blocking code.
Again the overall mantra for backend web is for compute to happen via the database.
2nd. Pipeline should 100 percent be known. Atomic operations are 100 percent required across shared storage and it's expected to occur all the time. Database transactions are fundamental and not unexpected. Pipeline should be extremely common. This is not an obscure operation. I commented about it in my code in case the reviewer was someone like you who isn't familiar with how popular atomic database transactions are, but this operation is common it is not a obscure trick.
Additionally pipeline has nothing to do with your or my own implementation. It's there to make atomic a state mutation and retrieval of that state.
I add one to a score then I retrieve the score and I want to make sure no one changes the score in between the retrieval and the add one. This is needed regardless of what internal data structure is used in redis.
>I don't know if they tried to run your test, but it could have failed with a 3xx
That'd be stupid in my opinion. 302 redirect is something I expected. Maybe I should have stated that explicitly in the docs.
>IMO -- it's more realistic and you have full control.
Unlikely. Even as a lead it's better to cater to the majority opinion of the team. 99 percent of the time you never have full control.
Developers and leads typically evolve to fit into the overall team culture while inserting a bit of their own opinions.
I'm a rust and Haskell developer applying for a python job. Do I apply my entire opinion to control the team? No, I adapt to all the bad (in my opinion) and good practices of the team. I introduce my own ideas slowly and where appropriate.
> it is a mystery why they wouldn't give feedback.
This is easy. It's not a mystery. It's because they don't give a shit about me. They don't want to hire me so the business relationship is over. Legally they owe me zero time to explain anything and it's more efficient for the business to not waste time on feedback.
>Yes, but this is a small API -- you literally have to write a test that hits the server once. There are libs for doing this with flask, there is documentation showing you how. It's not rocket science, and it's crucial to catching bugs down the road.
No there isn't. You have to spin up redis. Flask documentation doesn't talk about that. Integration testing involves code that controls docker-compose. It's a bunch of hacky scripts with external process calls that are needed to get integration tests working.
It's not rocket science but not within the scope of a take-home. Additionally hacking with docker compose is not sustainable or future proof, eventually you will hit infra that can't be replicated with docker.
I will probably do it next time just to cater to people who share your viewpoints.
>If the prompt was "write this like you're at a startup that has no money and no time", then sure.
Lol. The prompt was write it in four hours ( no time) and they aren't paying me for it ( no money).
>Specs did specify test cases -- and none of them had a trailing slash.
What? In the specs all examples have trailing slashes. All of them.
What the specs didn't specify was what to do when there is no trailing slash. So why not a 302? The client can choose what to do here. Either follow the redirect or treat it as an error.
> In interpreted languages I believe the thing that gets saved to mem is a pointer to the code itself. So no inherit difference here in terms of allocation. For closures reference counting is increased for the referenced variables but this is not a closure. ...
This has all been tread over before -- I won't rehash it since it seems like we're going in circles.
The hot path is worth optimizing first, in my mind -- giving up perf there for stats which will be called much less doesn't make sense to me.
To each their own!
> 1st. Sorting on the web API is definitively wrong. This is especially true in nodejs where the mantra is always non blocking code.
This assertion is so wide that it cannot be true. No matter what language you're in, you can absolutely do sorting in the API layer.
> 2nd. Pipeline should 100 percent be known. Atomic operations are 100 percent required across shared storage and it's expected to occur all the time. Database transactions are fundamental and not unexpected. Pipeline should be extremely common. This is not an obscure operation. I commented about it in my code in case the reviewer was someone like you who isn't familiar with how popular atomic database transactions are, but this operation is common it is not a obscure trick.
You don't need to pipeline if the operation is O(1) -- it's not a question of whether transactions are good/bad, I don't know why you're assuming people don't know about them.
It's the question of taking in unnecessary complexity here when there's an O(1) operation available. You're optimizing for an operation that is not on the hot path. The reason you have to do the atomic operation is because you're using an access pattern that requires it.
I know why and how you used an atomic transaction -- I disagree on whether it was necessary.
> That'd be stupid in my opinion. 302 redirect is something I expected. Maybe I should have stated that explicitly in the docs.
>
> What? In the specs all examples have trailing slashes. All of them.
This is correct, I read wrong and for that I apologize, they do have all trailing slashes. I was thinking of "/api/".
> Unlikely. Even as a lead it's better to cater to the majority opinion of the team. 99 percent of the time you never have full control.
>
> Developers and leads typically evolve to fit into the overall team culture while inserting a bit of their own opinions.
>
I referred to the context of the project -- you did* have full control.
If you were thinking of it in a team context, you should have asked more clarifying questions and found out more of what they wanted or would look for, similar to a real team context.
> No there isn't. You have to spin up redis. Flask documentation doesn't talk about that. Integration testing involves code that controls docker-compose. It's a bunch of hacky scripts with external process calls that are needed to get integration tests working.
>
> It's not rocket science but not within the scope of a take-home. Additionally hacking with docker compose is not sustainable or future proof, eventually you will hit infra that can't be replicated with docker.
>
> I will probably do it next time just to cater to people who share your viewpoints.
Again, this is a matter of what people think is hard and what isn't. Integration testing does not require that you spin up docker compose -- I literally linked to a libraries that do this in a reusable (importable) way w/ docker.
You said you don't do this kind of testing regularly, so it is understandable that it's not easy for you.
If docker was the problem, you can open subprocesses and assume that redis is installed in the test environment (i.e. your laptop, or CI environment), or take a binary to the redis binary path from ENV.
You can even remove the burden of running redis from the test suite and just make sure it's running manually or in test suite invocation/setup. There are many options, and it's just not that hard in 2023.
Don't know what (if anything) turned them off, so can't say this matters -- but it would matter to me, at the very least it would be a large plus for people who found the time to implement it.
> Lol. The prompt was write it in four hours ( no time) and they aren't paying me for it ( no money).
This is true, you did not have much time and they weren't paying you so I guess the situations are similar.
Seriously look at O(logN) and look at NlogN. Databases everywhere already take logN as acceptable. You are optimizing past that and introducing an NlogN sort step that will block all hot path calls because redis is single threaded. I don't think it's a to each their own thing here, i believe the user experience metrics will definitively play out towards my methodology but we can't know until we actually have metrics so we're at an impasse here.
>This assertion is so wide that it cannot be true. No matter what language you're in, you can absolutely do sorting in the API layer.
No this is absolutely true. Especially if you're a nodejs programmer. Nodejs is single threaded, sorting imposes NlogN penalty on a single thread.
Backend web development for the api layer is all about transmitting as much compute as possible to the database layer. You can get away with small sorts and things can still work but in general the practice of backend optimization is indeed to let the database handle as much as possible.
We can debate it but from my experience I know I'm pretty much categorically right on this. It's a fundamental principle. And it has special application to things that use Async IO like NodeJS. That's why there's huge emphasis in nodejs in writing non-blocking code. A sort is a potential block.
There are very few cases where you would handle compute on the api server rather then the database. The first instance is obvious, if N is guaranteed to be small which is not the case in the takehome. In other instances of sort with large N, the server ideally needs to launch that sort in a separate non-blocking thread so that the server can continue handling oncoming requests. In nodejs, to my knowledge since I last used it, you can't easily spawn a thread at will, you just have to pray you have available workers that aren't blocked by a sorting operation. In python the GIL makes this more complicated, but slightly better then nodejs.
For node you probably have 8 workers on 8 CPU cores. If you have 100 requests/s, all you need is 8 of them calling a sort on large N and all threads become blocked.
NlogN stands at the cusp here. It could work for small N, but for large N it will fail. Anything above NlogN is a huge risk.
>You don't need to pipeline if the operation is O(1) -- it's not a question of whether transactions are good/bad, I don't know why you're assuming people don't know about them.
This is categorically false. The pipeline operation as I said HAS NOTHING to do with O(1) or O(N).
This is essentially the operation:
set["key"] += 1
# another thread can update the value here leading to print displaying something unexpected.
print(set["key"])
So to solve the above comment you have to place both operations in a transaction. No two ways about it. The above operation is in spirit what I am doing. I update a value, then I read the value. Two operations and a possible race condition in between.
>Again, this is a matter of what people think is hard and what isn't. Integration testing does not require that you spin up docker compose -- I literally linked to a libraries that do this in a reusable (importable) way w/ docker.
You linked? Where? I'd like to know about any library that will do this. Tell me of any library that does integration tests that spins up infrastructure for you. The only one closest I can think of that you can run locally is anything that would use docker-compose or some other IAC language that controls containers. I honestly don't think any popular ones exist.
>If docker was the problem, you can open subprocesses and assume that redis is installed in the test environment (i.e. your laptop, or CI environment), or take a binary to the redis binary path from ENV.
No way I'm going to assume the user has redis installed on his local machine. Many devs don't. It's all remote development for them or everything lives in containers.
Docker is not the problem. Docker or virtual machines makes this problem more amenable to a solution, but even using docker here with testing is overkill and hacky. A take home should not expect a user to build excessive infrastructure locally just to run tests.
>You said you don't do this kind of testing regularly, so it is understandable that it's not easy for you.
I'd like to know your perspective. How is spinning up infrastructure for integration tests easy? You have to write code that launches databases, launches servers and views the entire infra externally. Do-able? Yes? Easy? depends on what you call easy. As easy as unit tests? Definitely no. Easy enough for a take home? In my opinion, no.
Maybe for this take home project you could be right. I could do some integration tests by spinning up docker-compose from within python. Hacky but doable. But in general this solution is not production scalable as production involves more things then what can be placed inside a docker-compose.
>You can even remove the burden of running redis from the test suite and just make sure it's running manually or in test suite invocation/setup. There are many options, and it's just not that hard in 2023.
So I should expect the reviewer to have redis running on his local machine? I would expect in 2023, I should have the entire project be segregated from the context it's running in. That's the cutting edge I believe.
Yeah it's 2023, you tell me how integration testing should be done as easily as unit testing on a takehome. I had one takehome project involving S3 and aws lambdas. They expected me to get an AWS account and literally set up infrastructure because there's no way around even testing what I wrote without actual infrastructure. That entire project was just integration test nightmare. Much rather run asserts locally.
>I referred to the context of the project -- you did* have full control.
Well I mean 99% of hires get into a context where they aren't in full control. So why is it logical to test what a developer will do if he/she had full control? Isn't it better to test the developers ability to code and to adapt to different contexts? Your methodology sort of just tests the programmers personal philosophy and whether it matches your own. That was my point.
> You aren't getting what I'm saying. logN is already a huge optimization for a hot path. Look at that pic: https://dev-to-uploads.s3.amazonaws.com/i/ya7xk4n9ch50xgwibt...
>
> Seriously look at O(logN) and look at NlogN. Databases everywhere already take logN as acceptable. You are optimizing past that and introducing an NlogN sort step that will block all hot path calls because redis is single threaded. I don't think it's a to each their own thing here, i believe the user experience metrics will definitively play out towards my methodology but we can't know until we actually have metrics so we're at an impasse here.
Just to keep the comparison in line -- we're comparing O(log(N)) @ O(1) in redis-land.
On the stats side we're comparing O(N) on the redis side PLUS either O(Nlog(N)) or O(N) on the python side.
> No this is absolutely true. Especially if you're a nodejs programmer. Nodejs is single threaded, sorting imposes NlogN penalty on a single thread.
NodeJS has both child processes and workers for heavy computation:
((this is also part of the reason I think NodeJS is strictly superior to other "scripting" languages as mentioned briefly in the other comment))
Again, here you seem to be arguing against a strawman that doesn't know that blocking the IO loop is bad. Try arguing against one that knows ways to work around that. This is why I'm saying this rule isn't true. Extensive computation on single-threaded "scripting" languages is possible (and even if it wasn't, punt it off to a remote pool of workers, which could also be NodeJS!).
All of this is a premature optimization, but my point here is that the stats can be optimized later. You could literally compute it periodically or continuously and only return the latest cached version w/ a timestamp. This makes the stats endpoint O(1).
> This is categorically false. The pipeline operation as I said HAS NOTHING to do with O(1) or O(N).
> ...
I think this is where we're talking past each other, so let me explain more of how I see the problem -- the solution I have in mind is serializing the URL and using ONE call to INCR (https://redis.io/commands/incr/) on the ingest side.
There is a lot you can do with the data storage pattern to make other operations more efficient, but on the stats side, the most basic way you can do it is to scan
I will concede that given that we know the data should fit in memory (otherwise you just crash) your approach gives you O(N) retrieval time and it's definitely superior to not have to do that on the python side (and python just streaming the response through). I am comfortable optimizing in-API computation, so I don't think it's a problem.
Here's what I mean -- you can actually solve the ordering problem in O(N) + O(M) time by keeping track of the max you've seen and building a sparse array and running through every single index from max to zero. It's overkill, but it's generally referred to as a counting sort:
This is overkill, clearly, and I will concede that ZSET is lighter and easier to get right than this.
> You linked? Where? I'd like to know about any library that will do this. Tell me of any library that does integration tests that spins up infrastructure for you. The only one closest I can think of that you can run locally is anything that would use docker-compose or some other IAC language that controls containers. I honestly don't think any popular ones exist.
I am very sure that I linked that, but in the case I didn't, here it is again -- hope you find it useful.
> No way I'm going to assume the user has redis installed on his local machine. Many devs don't. It's all remote development for them or everything lives in containers.
>
> Docker is not the problem. Docker or virtual machines makes this problem more amenable to a solution, but even using docker here with testing is overkill and hacky. A take home should not expect a user to build excessive infrastructure locally just to run tests.
I don't think these statements make sense -- having docker installed and having redis installed are basically equivalent work. At the end of the day, the outcome is the same -- the developer is capable of running redis locally. Having redis installed on your local machine is absolutely within range for a backend developer.
Also, remote development is not practiced by many companies -- the only companies I've seen doing thin-clients that are large.
> Maybe for this take home project you could be right. I could do some integration tests by spinning up docker-compose from within python. Hacky but doable. But in general this solution is not production scalable as production involves more things then what can be placed inside a docker-compose.
I see it as just spinning up docker, not compose -- you already have access to the app (ex. if it was buildable via a function) so you could spawn redis in a subprocess (or container) on a random port, and then spawn the app.
I agree that it is not trivial, but the value is high (in mymind.
> Yeah it's 2023, you tell me how integration testing should be done as easily as unit testing on a takehome. I had one takehome project involving S3 and aws lambdas. They expected me to get an AWS account and literally set up infrastructure because there's no way around even testing what I wrote without actual infrastructure. That entire project was just integration test nightmare. Much rather run asserts locally.
I agree that integration testing is harder -- I think there's more value there.
That said I do think that's a weakness of the platform compute stuff -- it is inconvenient to test lambda outside of lambda.
> Well I mean 99% of hires get into a context where they aren't in full control. So why is it logical to test what a developer will do if he/she had full control? Isn't it better to test the developers ability to code and to adapt to different contexts? Your methodology sort of just tests the programmers personal philosophy and whether it matches your own. That was my point.
Ah, this is true -- but I think this is what people are testing in interviews. There is a predominant culture/shared values, and the test is literally whether someone can fit into those values.
Whether they should or should not be, that's at least partially what interviews are -- does the new team member feel the same way about technical culture currently shared by the team.
Now in the case of this interview your solution was just fine, even excellent (because you went out of your way to do async io, use newer/easier packaging methodologies, etc), but it's clearly not just that.
>Again, here you seem to be arguing against a strawman that doesn't know that blocking the IO loop is bad. Try arguing against one that knows ways to work around that. This is why I'm saying this rule isn't true. Extensive computation on single-threaded "scripting" languages is possible (and even if it wasn't, punt it off to a remote pool of workers, which could also be NodeJS!).
Very rare to find a rule that's absolutely true.. I clearly stated exceptions to the rule (which you repeated) but the generality is still true.
Threading in nodejs is new and didn't exist since the last time I touched it. It looks like it's not the standard use case as google searches still have websites with titles saying node is single threaded everywhere. The only way I can see this being done is multiple Processes (meaning each with a copy of v8) using OS shared memory as IPC and they're just calling it threads. It will take a shit load of work to make v8 actually multi-threaded.
Processes are expensive so you can't really follow this model per request. And we stopped following threading per request over a decade ago.
Again these are exceptions to the rule, from what I'm reading Nodejs is normally still single threaded with a fixed number of worker processes that are called "threads". Under this my general rule is still generally true: backend engineering does no typically involve writing non blocking code and offloading compute to other sources. Again, there are exceptions but as I stated before these exceptions are rare.
>Here's what I mean -- you can actually solve the ordering problem in O(N) + O(M) time by keeping track of the max you've seen and building a sparse array and running through every single index from max to zero. It's overkill, but it's generally referred to as a counting sort:
Oh come on. We both know these sorts won't work. These large numbers will throw off memory. Imagine 3 routes. One route gets 352 hits, another route gets 400 hits, and another route gets 600,000 hits. What's Big Oh for memory and sort?
It's O(600,000) for both memory and runtime. N=3 and it doesn't even matter here. Yeah these types of sorts are almost never used for this reason, they only work for things with smaller ranges. It's also especially not useful for this project. Like this project was designed so "counting sort" fails big time.
Also we don't need to talk about the O(N) read and write. That's a given it's always there.
>I don't think these statements make sense -- having docker installed and having redis installed are basically equivalent work. At the end of the day, the outcome is the same -- the developer is capable of running redis locally. Having redis installed on your local machine is absolutely within range for a backend developer.
Unfortunately these statements do make sense and your characterization seems completely dishonest to me. People like to keep their local environments pure and segregated away from daemons that run in a web server. I'm sure in your universe you are claiming web developers install redis, postgresql and kafka all locally but that just sounds absurd to me. We can agree to disagree but from my perspective I don't think you're being realistic here.
>Also, remote development is not practiced by many companies -- the only companies I've seen doing thin-clients that are large.
It's practiced by a large amount and basically every company I've worked at for the past 5 years. Every company has to at least partially do remote dev in order to fully test E2E stuff or integrations.
>I see it as just spinning up docker, not compose -- you already have access to the app (ex. if it was buildable via a function) so you could spawn redis in a subprocess (or container) on a random port, and then spawn the app.
Sure. The point is it's hacky to do this without an existing framework. I'll check out that library you linked.
>I agree that integration testing is harder -- I think there's more value there.
Of course there's more value. You get more value at higher cost. That's been my entire point.
Good finds. But what about SNS, IOT, Big Query and Redshift? Again my problem isn't about specific services, it's about infra in general.
>Ah, this is true -- but I think this is what people are testing in interviews. There is a predominant culture/shared values, and the test is literally whether someone can fit into those values.
No. I think what's going on is people aren't putting much thought into what they're actually interviewing for. They just have some made up bar in their mind whether it's a leetcode algorithm or whether the guy wrote a unit test for the one available pure function for testing.
>Whether they should or should not be, that's at least partially what interviews are -- does the new team member feel the same way about technical culture currently shared by the team.
The answer is no. There's always developers who disagree with things and just don't reveal it. Think about the places you worked at. Were you in total agreement? I doubt it. A huge amount of devs are opinionated and think company policies or practices are BS. People adapt.
>Now in the case of this interview your solution was just fine, even excellent (because you went out of your way to do async io, use newer/easier packaging methodologies, etc), but it's clearly not just that.
The testing is just a game. I can play the game and suddenly I pass all the interviews. I think this is the flaw with your methodology as I just need to write tests to get in. Google for example in spirit attempted another method which involves testing IQ via algorithms. It's a much higher bar
The problem with google is that their methodology can also be gamed but it's much harder to game it and often the bar is too high for the actual job the engineer is expected to do.
I think both methodologies are flawed, but hiring via ignoring raw ability and picking people based off of weirdly specific cultural preferences is the worse of the two hiring methodologies.
Put it this way. If a company has a strong testing culture, then engineers who don't typically test things will adapt. It's not hard to do, and testing isn't so annoying that they won't do it.
> Very rare to find a rule that's absolutely true.. I clearly stated exceptions to the rule (which you repeated) but the generality is still true.
>
> Threading in nodejs is new and didn't exist since the last time I touched it. It looks like it's not the standard use case as google searches still have websites with titles saying node is single threaded everywhere. The only way I can see this being done is multiple Processes (meaning each with a copy of v8) using OS shared memory as IPC and they're just calling it threads. It will take a shit load of work to make v8 actually multi-threaded.
>
> Processes are expensive so you can't really follow this model per request. And we stopped following threading per request over a decade ago.
>
> Again these are exceptions to the rule, from what I'm reading Nodejs is normally still single threaded with a fixed number of worker processes that are called "threads". Under this my general rule is still generally true: backend engineering does no typically involve writing non blocking code and offloading compute to other sources. Again, there are exceptions but as I stated before these exceptions are rare.
You seem to always fight the least knowledgeable strawmen. Process pooling and task distribution exist. Just because you didn't know about worker threads in NodeJS has nothing to do with me. Whether it's common or not has nothing to do with the validity of the solution.
Yet another reason why NodeJS is superior as a platform to CPython.
Anyway, I will take that you have relaxed your rule -- it needed to be relaxed. IIRC you said it was "definitively true" or something to that effect.
> Oh come on. We both know these sorts won't work. These large numbers will throw off memory. Imagine 3 routes. One route gets 352 hits, another route gets 400 hits, and another route gets 600,000 hits. What's Big Oh for memory and sort?
The array is sparse, and yes, this is exactly what I expect to happen. I wrote the scenario thinking of that.
> It's O(600,000) for both memory and runtime. N=3 and it doesn't even matter here. Yeah these types of sorts are almost never used for this reason, they only work for things with smaller ranges. It's also especially not useful for this project. Like this project was designed so "counting sort" fails big time.
>
> Also we don't need to talk about the O(N) read and write. That's a given it's always there.
Modern processors churn through numbers pretty quick, 600,000 operations is not a big deal -- still O(N)!
It's better to be clear than unclear -- it was a cause for confusion, evidently because you were comparing O(Log(N)) and O(NLog(N)). The comparison for ingest O(Log(N)) and O(1).
This is the last I'm going to say on this.
> Unfortunately these statements do make sense and your characterization seems completely dishonest to me. People like to keep their local environments pure and segregated away from daemons that run in a web server. I'm sure in your universe you are claiming web developers install redis, postgresql and kafka all locally but that just sounds absurd to me. We can agree to disagree but from my perspective I don't think you're being realistic here.
In the world before docker and docker compose, this is what people did when they had to run software to test locally. In the current world, people still do this, but it's made easier by docker and docker-compose.
That's the last I'm going to say on that.
> Sure. The point is it's hacky to do this without an existing framework. I'll check out that library you linked.
God forbid we write useful code without an existing framework.
> Of course there's more value. You get more value at higher cost. That's been my entire point.
Except "cost" here is relative -- not everyone considers that cost to be high. Relative to unit tests it is higher, but it is still trivial with knowledge.
It's simple, you just didn't have the knowledge so you thought it was prohibitively hard. That's fine -- that's just how technology goes.
> Good finds. But what about SNS, IOT, Big Query and Redshift? Again my problem isn't about specific services, it's about infra in general.
The goalposts seem to be moving, so time for me to get off the field. You noted those other tech specifically, so I answered.
Read up on ways you can replicate these environments locally, or don't. If you have some you can't replicate, don't use them, or find other ways to test them that aren't completely local (i.e. running in a test account).
That's the last I'm saying on that.
> No. I think what's going on is people aren't putting much thought into what they're actually interviewing for. They just have some made up bar in their mind whether it's a leetcode algorithm or whether the guy wrote a unit test for the one available pure function for testing.
Alright well I hope you're able to find a way through the minefield of reality, either way that's the last I'm saying on it.
> The testing is just a game. I can play the game and suddenly I pass all the interviews. I think this is the flaw with your methodology as I just need to write tests to get in. Google for example in spirit attempted another method which involves testing IQ via algorithms. It's a much higher bar
>
> The problem with google is that their methodology can also be gamed but it's much harder to game it and often the bar is too high for the actual job the engineer is expected to do.
>
> I think both methodologies are flawed, but hiring via ignoring raw ability and picking people based off of weirdly specific cultural preferences is the worse of the two hiring methodologies.
>
> Put it this way. If a company has a strong testing culture, then engineers who don't typically test things will adapt. It's not hard to do, and testing isn't so annoying that they won't do it.
Well, don't expect to win the game if you don't play.
Hope this was illustrative of at least how others think -- this is the last I'm going to post, thanks for the discussion!
I threw the docs in the README for the submission, but for HN readers they want to see the problem first. The submission didn't even include the specs.
>Yeah I can see this, but I think "automated tests" are quite big in peoples' minds. Feels like it would be a checkbox on the item.
Yeah you're completely right here. I'm gonna go with it next time.
>I disagree here -- table stakes for a good API is integration and E2E testing -- and they're the easiest to do (no need to manipulate a browser window, etc).
Yeah but many companies don't do integration testing as the infra on this is huge and writing tests is more complicated. It's certainly overkill for take home. I would say you're wrong here. Completely. This company was not expecting integration tests as part of the project at all.
I've worked for start ups most of my career, and I'm applying to start ups. It's mostly the huge companies that have the resources to spend the effort to have full coverage on that.
Integration tests are Not easy, btw. Infrastructure is not easy to emulate completely, how would I emulate google big query or say aws iot on my local machine? Can't, most likely integration tests involve actual infra, combined with meta code that manipulates docker containers.
>Sorry INPUT is what I meant to write there -- what happens if you get a -1 ?
You get a 500. Which is not bad. But you're right a 400 is better here.
>Right, but this is what I mean -- when I read that I wondered if it was complete, and it wasn't. That's an unnecessary ding to take.
I feel this is such a minor thing. I'm sure most people would agree if you dinged that you'd be the one going overboard, not the candidate.
>This is going to build objects dynamically every execution right? That's what I thought could be avoided. Less about lists vs generators, more about doing things that look like they allocate so much.
You're going to build objects dynamically every execution Anyway. This saves you the extra intermediary step of not having to save it to an extra list.
Look at the above. Same concept. Generators are drop in replacements for actual state. State and functions are the isomorphic (ie the same thing). Building a generator is cheaper then building a list. You should read SICP about this topic of how functions and data are the same.>Ah OK, but this isn't what they asked for, right? If we want to be really pedantic, your endpoint returns a 3xx for every single error.
It's a reasonable assumption that all users assume /xx/xx equals /xx/xx/. It's also reasonable for a client to handle a 302 redirect. The specs didn't specify the exact definition on either of these things.
>The measurement endpoint will be called far more (100x, 1000x, ...) than the stats endpoint -- it's the hot path.
So? logN is super fast. You have 100 objects binary or whatever indexed insert redis uses gets there in at most 4 or 5 jumps. So even if it's the hot path redis can take it.
NLogN is slow enough that if N is large there is noticeable slow downs EVEN when it's a single call. Recall that redis is single threaded sync. NlogN can block it completely.
Either way this is all opinion here right? Ideally the results are returned unsorted. the client is the best place to sort this, but that's not the requirement.
>Also, note that sorting is zero cost, but retrieving the data will still be in N time including network round trips, which will likely dwarf any time you spend sorting in local memory. Even the conversion to JSON will dwarf the time you spend sorting for large N.
Of course the act of reading and writing data will cost N regardless. But your reasoning is incorrect. Look at that picture again: https://i.stack.imgur.com/osGBT.jpg. O(NlogN) will dwarf O(N) when N is large enough. It will even dwarf IO round trip time and has the potential of blocking redis completely on processing the sort.
>Also, the point of memory usage being potentially large is still relevant. My instinct is that when I see something returning a possibly very large piece of data, setting up for streaming is the way to go. Doing a scan you have the possibility to use far less memory for an individual request, and you could actually stream responses back in extreme cases, so you can avoid holding the entire result set in memory at all.
Sure but look at the specs. It's asking for http requests. It's not a streaming protocol here. I mean at this point if you were evaluating my code for a job you'd be going to far already as your going off the rails completely.
>The point about error handling still stands with respect to reporting though -- you generally want to see error handling in code, rather than just leaving it up to the runtime to fail the task and carry on.
I mean I could build a full logger and error handler framework here. I'd put custom handling in a decorator and have the error handling invisible in the http route handlers. Flask already does this, but it just returns 500 on all exceptions. That is the most elegant way imo. But again this is a bit outside of the scope of a takehome, given the framework handles a lot it and the extra effort required.
>If you want to get really fancy, you can get creative with how data is stored on the redis side and get the size to N.
For takehomes and in general people prefer readable code. Only optimize if it's needed. I mean I can write the whole thing in assembly code for ultimate performance. I can def see some nitpicker dinging me for optimized but unreadable code.
>The things you include/don't include by instinct I think is valuable signal.
Not to be insulting here. But I think your instincts are pretty off here. There's a limited amount of time assigned to this project. All the basics were included with what I did in the time allotted.
When you have good instincts you know when to apply automated testing, when to use error handling. These aren't things you should do all the time. Your attitude is something I see in many younger engineers. This attitude of rule following and best practices is just adhered to without thinking it all through properly. For the scope of the project. most of what you wrote here isn't needed.
You're also not testing what's traditionally tested in interviews. They test intelligence, depth knowledge and ability to design and solve problems. You're testing the ability to follow written rules. I mean following these things is trivial. It doesn't take much brain power to write error handling or unit tests. Perhaps you're just testing for someone that agrees with your programming philosophy. I mean these things can be followed regardless
Your mention of integration tests is telling. I mean the code for integration tests can rival the size of the code in the project itself with the a minor benefit in safety . It's not just take home assignments but many many many companies loaded with senior engineers skip over automated integration tests and keep those tasks in todo lists for the longest time. There's a huge engineering cost to building out testing infra to the maximum ideal, and many companies just don't bother. It's mostly the bigger well established companies that have the resources to do this.
>I'd disagree that it's worse than algo interviews, because algos are rarely the most important thing in a professional code base, especially one with that's about web services and not something more demanding.
It is worse in my opinion because it's biased. What if your opinion was more correct then my opinion but I judged you based off of my world view? Would that be right? No it wouldn't. Programmers are diverse and the best organizations have diversity in opinions and expertise.
All in all I mostly disagree but I think your right on one thing: At least one of those job candidate evaluators that are looking at my code likely had a similar attitude as you do and I should bias my code towards that more. Likely won't write integration tests though, hard no on that one.
Also good catch on the /test/-1/. That's definitely a legit ding imo.