Seems highly unlikely to ever work, but why assume the technology is evil? If you could create a system that had a high accuracy of detecting terrorists, then that would be a good system, not an evil one.
I get your point that a system that claims to detect terrorists but only really detected Arabic people would be an evil one - but you're automatically calling the terrorist system evil without knowing if it really does detect terrorists or not.
As an extra hypothetical question, do you feel a system that could detect people who were really just about to commit terrorist attacks as good or evil? At a conceptual level, assume the system somehow scanned brain waves or some other truly difficult method.
Occam's razor. They are not 'scanning brain waves or some truly difficult method'. Not in this reality. You know it too. I didn't even have to suggest they built an Arab detector for you to bring it up ;)
I don't care about that hypothetical. Save it for your sci fi screenplay.
All reporting is always biased. Every word you select is a direct product of bias, and the choice between multiple sentence phrasing changes emotional direction of the reader.
I don't think many would argue that the content was in any way worse back when the internet was used by a much smaller population. At the end of the day, the content that is really useful and that you actually want is rarely created by an army of everyday internet users. It's created by the dedicated and the ingenious, and those are the types that will migrate quickly to the hidden places.
Look at how hard some of these creators are already struggling with getting enough patreons. How much do you think will survive in a hidden place? I'm not talking about the 1m channels that by now could just sign up with a publisher, but many of the sub 100k gems out there. Dedication and ingenuity will get you nowhere if you can't put bread on the table. Even with all its flaws YouTube has enabled a whole new world of creators to reach out to and be found by a crowd.
If an app (even badly coded) causes memory leaks in the OS, that's a fault of the OS still. Of course fast moving OSes like Android or iOS are always going to have these kinds of bugs and it's probably not related to Swift as Swift isn't used in the kernel internals.
I write kernel mode drivers as my day job, and this is certainly the first time I've ever heard of non-swappable being called "wired".
It might be called "non-pageable", "pinned", "non-swappable", etc. Out of those, I'd prefer "non-pageable", because it's the most descriptive term for it.
Non-standard terminology is not great for communication.
- Non-pageable memory: cannot be freed until explicitly released by the owner
- Userland memory can be wired by mlock(2) (subject to system and per-user limits)
- Kernel memory allocators return wired memory
- Contents of the ARC and the buffer cache are wired
- Some memory is permanently wired and is never freed (e.g., the kernel file itself)
> I write kernel mode drivers as my day job, and this is certainly the first time I've ever heard of non-swappable being called "wired".
Really ? Because it's literally called that in macOS Activity Monitor app, the term is also used in Apple's kernel documentation and API's (e.g. in the `mem_and_io_snapshot` struct defined in 'debug.h' in Kernel.framework). It's also mentioned in the Kernel Programming Guide.
Never did macOS drivers, just bare metal, Windows and Linux so far, macOS is not very big in our niche. Weird that Apple's terminology is so different.
Typing this on a Macbook Pro, and I can see that Activity Monitor does mention "wired" memory.
I did a little googling and I see the terminology is also used in the Mach microkernel, which macOS's XNU kernel was built upon. It looks like it's not something Apple came up with.
Companies are just people though - the guys in charge of decisions are the ones we're ultimately evaluating. That the left hand of the company doesn't know what the right hand of the company is doing isn't much of an excuse, as ultimately the buck stops at the executive and board positions.
There's no way a decision like this one didn't get approval from the top layers of the company as it no doubt had to go through different levels of legal, product and everything in between.
On the side, I've owned some chain restaurants for 32 years. Mine have always been noted as friendlier, cleaner, fresher and all that. I like to say that it's cause it runs downhill and I care about the customer experience.
Three years ago, I sold one of those restaurants inside a mall while still owning one just two blocks away. People from the mall come into this restaurant. Just yesterday someone told me that they hated the mall one. "They're just so mean to everyone!"
I don't think that's necessarily true. The binary line at which we judge objects begins with the concept of sentience/self-awareness. If our cells work to do something bad, we can't blame them - they aren't sentient. If people work to do something bad, we can and have always judged the group - they are sentient and are capable of making decisions within a moral framework.
There are group effects for groups of people, just as there are effects that organised groups of cells have that collections of cells do not.
If someone hands you a cupcake and punches you in the face with the other hand then you don't think that the two actions are from separate "organisations" of cells (maybe, in cases of schizophrenia, or similar).
People in an organisation often lack agency to fully control actions based on their personal ethics. This is often exploited by controlling influences.
if the cells are sick, the person is sick. you don't just say you have a little cancer but on the whole you're chilling and are ready to goto the beach
This design only requires the hosting server (e.g. Cloudflare, CDN, ...) to support encrypted SNI. It can be entirely transparent to user deployed servers. So it might well be here much sooner than you think!
People can be stabbed in the back if they go into dark alleys without watching behind them. Let's stab a few people who go into these alleys so that everyone will be afraid to do so and we have an opportunity to prevent people being stabbed in future by making them aware.
Why would you possibly think this is a good idea? The idea is to prevent pain, not cause more pain in some bizarre attempt at making people afraid. There's enough privacy violations - we don't need to be making more of them ourselves.
I actually agree with the parent's perspective. As I see it, there are three potential states for sensitive data:
1. Secured and private. This is data not exposed in any breach.
2. Unsecured and private. This is data which has been exposed in a breach, and which must be sought out by the reasonably tech savvy.
3. Unsecured and public. This is data which has been exposed and can be easily used by anyone.
We want all sensitive personal data to be in state 1. But because of the taboo of state 3, we end up in a situation where we're hostage to state 2, because everyone wants to treat published sensitive data as if it were still private. That takes power away from the non-tech savvy victims of breaches but doesn't diminish the power of tech-savvy criminals who want to use the data.
In my opinion, forcing all sensitive data to be considered either secure and insecure (instead of the weird, quasi-private state 2) would take power away from people who want to use it. Every time a new breach happens there is a race to use it before it's not useful anymore. I believe we could meaningfully defang these breaches by completely leaning in and demonstrating how public the data is. If there were a party truly committed to that and they couldn't be stopped, my hypothesis is that things would actually change.
Your analogy misrepresents the grandfather's point. A closer analogy for his argument might be:
- Some high number X of dark alley stabbings occur each year.
- But alleys still "feel" safe to people, because the stabbings aren't well-publicized. So people don't know to avoid them and the rate X remains the same.
- Let's publicize alley stabbings in an emotionally impactful way, so people know to avoid alleys and we can bring X down.
In the actual case at hand, the argument is that you break a few eggs so people understand the issue viscerally, and hope to achieve massive regulatory change because people now actually care. I don't know if it would work, but it's a more reasonable idea than you're making it out to be.
Solving the root problem here is orders of magnitude more important than any single data breach today is.
I don't think this is correct. For all the people who would have their data exposed in a public torrent, their data is likely safe at present and just needs to be removed from that website. If you put it in a torrent, you're hurting all of those people in a very direct way - you're the one stabbing them in the back.
What the authors here did is correct - they've publicized the issue. Releasing this data as a torrent is not
'publicizing' anything - it is stabbing millions of people in the back, and then waiting for the crowds to come and gape at the dead bodies.
> Let's publicize alley stabbings in an emotionally impactful way
The top post doesn't promote publicizing data breaches that already happened. It is promoting obtaining and publishing the data which weren't published before. It is completely different things. Like making a TV series about alley stabbings - and stabbing actual people in the alley to get better scenes for this video. The former is great, the latter is a heinous crime which can ruin the whole cause.
Doesn't matter. It's like justifying mugging by saying "well, criminals might have mugged you anyway, if not me then somebody else". If somebody might have committed the crime, does not justify committing it again.
There is logic at play here, even if you disagree with the approach behind executing it. It's pretty simple psychology that when your neighbor gets robbed it "hits home" with you more than hearing about nameless people on the news suffering the same fate.
With one of the major contributors to the paper being 'Committee on the Impacts of Sexual Harassment in Academic Science, Engineering, and Medicine'[1] who only get paid because sexual harassment exists surely can never have an agenda in proving such a thing exists.
In a follow up study, we could go and ask the labour unions how much benefit they give to the average worker. Surprise finding! Labour unions benefit all workers immensely, and they benefit workers who pay the most in membership fees. And then we can follow up with a study by dentists on how regularly you should visit your dentist (hint: it's more often than you think!)
We should probably keep doing studies to check how dentists think we should brush our teeth. We just shouldn't have dentists do a study on a variable that directly harms their income as we would already know the results they would give before the study is even done.
If you wanted a study on that variable, you'd likely need to at least get input from someone who says going to dentists is a bad idea - but at that point, you're just going to have a shouting match and a statistics measuring contest between the dentists and the guy who doesn't think anybody should see dentists. Your results would likely be contradictory and fairly useless.
We haven't yet found a way to do such studies in a useful manner, and it's an ongoing problem in politics that has yet to have a solution. And may never have a solution. And it's very important to be able to determine when such study variables align with the financial and ideological requirements of the researcher.
If you can tell without doubt what the result of a study by a certain group will be before the study takes place, it's usually a red flag.
I think your examples make your conclusions less likely, not more.
* Dentists, in particular, would secure more work if they organised and actively promoted dental practices that they knew were harmful. Nevertheless, that doesn't occur.
* Doctors as a group, similarly might stand to benefit from drawing out a disease for longer. Despite this, it is an absurd and baseless claim to make of doctors as a group, though presumably some unethical individuals might on occasion. An individual from a different year my medical school was recently charged with murder; this doesn't have me in a crisis over the legitimacy of the profession.
I note however it is a valid and compelling argument to make against pharmaceutical companies -which do have a theoretical financial incentive to pursue non-curative remedies over curative ones (guiding research priorities, and/or attempts to lessen the cost of production of known curative remedies), and this is followed by multiple documented examples showing this occurring in practice.
Among clinicians actually having to face sick people in person, this behaviour seems to be extremely rare.
--
Regarding your specific example about "asking dentists how often to brush your teeth", there is in fact quite a healthy debate between dentist organizations and medical organisations - the former arguing that 3x per day is best for dental health, while (some) medical organizations argue that 2x per day is a better balance between dental health specifically, versus the added risk to general health of harmful bacteria entering the bloodstream from minor cuts on gums caused by brushing.
In that debate, it is instructive to note that neither dentists nor doctors organisations accuse the other of acting in bad faith. In this instance, asking dentists as a group how often to brush your teeth is actually a quite sound idea, and the idea that they provide self-serving answers is comically absurd; if anything their current published guidelines on this question is too good for your dental health, at (very slight possibility) the expense of your general health.
For a more in-depth discussion of research and abuses of research on questions of health, I heartily recommend "Bad Science" (2008) by Ben Goldacre. One chapter is available free online here [1].
Isn't that already the case though, regardless of net neutrality? Seems the large scale surveillance and manipulation will happen, whether it is legal or not.
Kind of defeats the purpose of using golang for a task like this. The whole point of golang is using the little greenlet threads, but actually using them in this case is terrible on performance.
The remaining performance left behind is all in memory allocation and garbage collection - something you could optimize relatively easily if it were written in C. Such as by using a memory pool, so that you wouldn't need allocations or garbage collection at all.
Of course if performance isn't a big issue for your task, then none of this is really important.
Using Go doesn’t mean you have to use many goroutines or must not do some manual memory management where it is the right thing to do.
This article nicely shows how optimizing your program yields more speed than randomly throwing goroutines at it. Finally it does use goroutines for a good effect, but after proper consideration.
>Kind of defeats the purpose of using golang for a task like this. The whole point of golang is using the little greenlet threads, but actually using them in this case is terrible on performance.
The point of Golang is using them intelligently, not merely throwing at any problem like all you've got is a hammer...
It's surprising that the per-file Goroutines were so expensive, though. (The original per-line Goroutine, sure, that's excessive if you care about performance.) Just using long-lived workers seems non-idiomatic for Go, but it certainly pays big dividends in this example.
Per-file may have had other problems not related to the Go runtime, such as IO contention. I'm not going to check it, but it would be easy to verify that just by using a limited number of them at a time. Spawning a new goroutine in that case is not strictly necessary, but would still be good software engineering.
One of the problems I see repeatedly when people try to benchmark things with concurrency is when they don't specify a problem that is CPU-intensive enough, so it ends up blocked on other elements of the machine. For a task like this, I'd expect optimized Go to easily keep up with a conventional hard drive, and with just a bit of work, come within perhaps a factor of 2 or 3 of keeping up with the memory bandwidth on a consumer machine (including the fact that since you're going to read a bit, then write some stuff, you're not going to get the full sequential read performance out of your RAM), not because Go is teh awesomez but because the problem isn't that hard. To get big concurrency wins, you need a problem where the CPU is chewing away at something but isn't constantly hitting RAM or disk or network for it, such that those systems become the bottleneck.
"the benchmark was designed to repeatedly parse an in-memory byte slice (not the hard drive), thus IO contention is unlikely here"
You could still be getting IO contention from the RAM system. RAM is not uniformly fast; certain access patterns are much faster than others.
"concurrency is a big win when IO is a bottleneck : keep processing dozens of things while some of them are waiting for data from network or hdd."
Concurrency is a win when IO is a bottleneck on a single task. Once you've got enough tasks running that all your IO is used up, adding more may not only fail to speed things up, but may slow things down. I'm speaking of situations where you've used up your IO. The tasks you're benchmarking are so easy per-byte that I think there's a good chance you used up your IO, which at this level of optimization, must include a concept of memory as IO.
I think you'd be helped by stepping down another layer from the Go VM and thinking more about how the hardware itself works regardless of what code you are running on it. Go can't make the hardware do anything it couldn't physically do, and I'm getting a sense you deeply understand those limits.
Yes, it does feel that something is wrong somewhere, but I can't find out where. Nobody would be using the idiomatic goroutine-per-task with that kind of overhead, yet it's one of the most common building blocks of golang projects.
Hi iainmerrick, just for info the measured per-file cost didn't include reading from the filesystem. Only the in-memory parsing was taken into account.
I write single-threaded Go all the time, and you can use most of the same optimizations in Go that you could use in C (including pools). It's pretty easy to opt out of GC in Go. And you still keep all of the other benefits of using a modern, higher-level language (security, memory safety, straightforward tooling, etc).
It only does because the author is testing for their specific environment. At some number of cores, the concurrent calls will produce more performant code than running sequentially.
Ideally, in this case I would think one would want to check the number of cores and decide what route to take.
When the author removed parallelism the first time, I don't think this is the case. Running things in parallel has a cost. That cost often comes in the form of memory allocations and data copies so the unit of work can be stored and shared with another thread, and the synchronization costs of scheduling threads. If that aggregate cost is greater than the computational cost of what you're computing, you'll never win.
For the point at which the author removed parallelism, and the sequential code was faster, I think this was the case. The computation was too fine-grain. The author successfully took advantage of parallelism by applying it at a coarser granularity; each thread did more work. At this point, the author also does tune the solution for the execution environment, as he uses a fixed set of go-routines to process a bunch of messages rather than one go-routine per message.
FWIW I really mean the "take the numbers with a grain of salt" advice, i.e. "Your mileage may vary". What I'm sharing in this article is not a bunch of hard, strong, exact numbers ; It's a journey and an invitation to apply similar reasoning process to your own use case and hardware.
I get your point that a system that claims to detect terrorists but only really detected Arabic people would be an evil one - but you're automatically calling the terrorist system evil without knowing if it really does detect terrorists or not.
As an extra hypothetical question, do you feel a system that could detect people who were really just about to commit terrorist attacks as good or evil? At a conceptual level, assume the system somehow scanned brain waves or some other truly difficult method.