The word "safety" doesn't normally encompass lying or, more appropriately in this case, saying something untrue without realizing it. That's considered a very different kind of problem. Safety normally means there's a direct chance of physical harm to someone.
This kind of elasticity in language use is the sort of thing that gives AI safety a bad name. You can't take AI research at face value if it's using strange re-definitions of common words.
How exactly does honestly not clearly fall under safety? AI is an information system, and truthfulness from an information system that impacts human lives is clearly a safety concern.
This is not a redefinition, the harm results from the standard usage of the tool. If the AI is being used to predict the possible future behaviour of adversarial countries, then you need the AI to be honest or lots of people could die. If the AI concludes that your adversary would be more friendly towards its programmed objectives, then it could conclude lying to the president is the optimal outcome.
This can show up in numerous other contexts. For instance, should a medical diagnostic AI be able to lie to you if lying to you will statistically improve your outcomes, say via the placebo effect? If so, should it also lie to the doctor managing your care to preserve that outcome, in case the doctor might slip and reveal the truth?
How much software is safety critical in general, let alone software that uses deep learning? Very, very little. I'd actually be amazed if you can name a single case where someone has deployed a language model in a safety critical system. That's why your examples are all what-ifs.
There are no actual safety issues with LLMs, nor will there be any in the foreseeable future because nobody is using them in any context where such issues may arise. Hence why you're forced to rely on absurd hypotheticals like doctors blindly relying on LLMs for diagnostics without checking anything or thinking about the outputs.
There are honesty/accuracy issues. There are not safety issues. The conflation of "safety" with other unrelated language topics like whether people feel offended, whether something is misinformation or not is a very specific quirk of a very specific subculture in the USA, it's not a widely recognized or accepted redefinition.
> I'd actually be amazed if you can name a single case where someone has deployed a language model in a safety critical system. That's why your examples are all what-ifs.
AI safety is not a near-term project, it's a long-term project. The what-ifs are exactly the class of problems that need solving. Like it or not, current and next generation LLMs and similar systems will be used in safety critical contexts, like predictive policing which is already a thing.
Edit: and China is already using these things widely to monitor their citizens, identify them in surveillance footage and more. I find the claim that nobody is using LLMs or other AI systems in some limited safety critical contexts today pretty implausible actually.
"the worst-case scenario is unlikely or in the future ... loss of human life is virtually certain"
These two things are in conflict. We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war).
Yes, also for the climate. Look at the actual predictions and it's like a small reduction in GDP growth spread over a very long period of time, and that's assuming the predictions are actually correct when they have a long track record of being not so.
Really stuff like asteroids and climate is a good counter-argument to caring about AI risk. Intellectuals like to hypothesize world-ending cataclysms that only their far sighted expertise can prevent, but whenever these people's predictions get tested against something concrete they seem to invariably end up being wrong. Our society rewards catastrophising far too generously and penalizes being wrong far too little, especially for academics, NGOs etc. It makes people feel or seem smart in the moment, and they can punt the reputational damage from being wrong far into the future (and then pretend they never made those predictions at all or there were mitigating factors).
> We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war).
That's just incorrect. The Tunguska event was a nuclear-weapon scale asteroid. These are predicted to happen once every hundred years or so. If it happens over a populated city millions would die. If a person wrongly concludes this was a surprise nuclear attack, maybe everyone would die, to say nothing of the real risk that the asteroid itself could be big enough to wipe us all out.
There's a lot of uncertainty around climate change, but changing climate patterns will certainly change resource allocations (fresh water, arable land, etc.). This will lead to shortages in places that were once abundant, which could easily lead to wars in which millions die.
> Really stuff like asteroids and climate is a good counter-argument to caring about AI risk.
This. This boggles my mind. Long-tail risks exist, and burying your head in the sand and pretending they don't just places millions of lives at risk, and potentially the entire human race. You don't have to think these are top priorities, but to dismiss them as complete unimportant is frankly bonkers.
"The Tunguska event was a nuclear-weapon scale asteroid"
Which had very little impact on humanity because it exploded in the middle of the tundra.
"These are predicted to happen once every hundred years or so"
What is predicted exactly, by whom and how were these predictions validated against testable reality given the postulated rareness? If they're so common then why is it so hard to name the last 10? I think in reality these events are very rare and will almost always happen over the oceans, deserts, poles etc where not many people live.
"Long-tail risks exist, and burying your head in the sand and pretending they don't"
They exist and I am not pretending they don't. I am saying that this style of reasoning in which an extremely unlikely event is unfalsifiably and arbitrarily assigned near infinite downsides in order to justify spending time and resources on it, is problematic and as a society we are far too generous towards people who do this.
Luckily, it again happened near a depopulated zone, but the damage was still extensive.
> I think in reality these events are very rare and will almost always happen over the oceans, deserts, poles etc where not many people live.
Most car accidents will happen to bad drivers, so if you're a good driver you don't need to wear your seatbelt, amirite?
The fact that an easily preventable event typically happens without much damage is no consolation when that's not the case.
> They exist and I am not pretending they don't. I am saying that this style of reasoning in which an extremely unlikely event is unfalsifiably and arbitrarily assigned near infinite downsides
Your mistake is thinking the likelihoods assigned are arbitrary and unfalsifiable. By your logic, COVID-19 was unlikely as most outbreaks are small and isolated, and the likelihood of a global contagion so unfalsifiably remote it's not worth thinking about. Therefore pandemic preparation is a waste of time and money. Now that 6 million people have died, that view doesn't look so rosy in hindsight.
The calculations on asteroid threats have been done based on known data, and even with our current preparations we still miss some potentially devastating ones like Chelyabinsk:
It is a fruitful area for research! Truffle is an example of the sort of framework you mean. Implement a parser+interpreter using Truffle and you get JIT compilation, GC, debugging, profiling and more stuff for free on top of the JVM.
It does matter because the issue with wikis (not just confluence) is there's no approval or review workflow. Imagine trying to write a large program in which everyone could just commit at will, with no review process whatsoever, and where nobody had made any decisions about design up front. There'd be duplication, dead code, the organization would be crazy.
That's the average wiki. It's a commons and a tragic one. To make docs work you have to treat it more like a codebase: clear ownership, standards, review processes, approvals, up front design, refactoring efforts etc.
> To make docs work you have to treat it more like a codebase: clear ownership, standards, review processes, approvals, up front design, refactoring efforts etc.
Maybe true in large orgs.
But for smaller companies what I've seen is usually paralysis.
e.g. someone notes a problem (maybe just a typo) in the doc. Can they fix it within seconds? If instead they need to raise a ticket then most likely it ain't happening. They move on, and the next person experiences the same problem.
IMO the default should indeed be towards everyone committing at will. Yes that will result in the occasional snafu. Fix that when it happens. (obviously not good practice for the operating manual for a nuclear power plant - but for a <500 person Saas company it is).
Mandating a Jira ticket for simple typo fixes is overkill. But if you make it easy to create a PR directly on the documentation file, without leaving the tab, I don't see an issue. This is already a Github feature.
We did PR's on documentation files at my last place, it worked but it was more painful than getting reviews for code PR's. Tickets work because they can be reasoned about, shoved aside, brought back into the limelight, updated, other tickets can easily be related to them as more information is discovered, etc.
Overall the comments on this page fall into 2 camps, people who've tried it all and found what works is discipline and those who are still trying it all.
Disagree. A ticket should be created for any change, no matter how small. It takes seconds to write a title, body and hit submit. I've seen those small ad-hoc changes cause havoc because someone forgot to escape a single quote or didn't realize tabs were necessary and replaced them with spaces.
The default for Confluence is just that, everyone commits at will. There is no structure, tons of duplication, no standards when it comes to naming, formatting, audience, etc. I'm a huge fan of markdown/plain-text solutions, only because linters can be run that force you down one happy path. I don't believe Confluence has linters at all.
Yep, and that process also involves other people, to review/ approve the fix to the typo.
It then goes from being a few seconds of elapsed time and actual time (to just commit a fix to the typo) to taking hours, days or weeks of elapsed time and hours of actual time and forcing context switching on, and interrupting the workflow of, all of people involved.
The last thing I want is to raise the friction for writing down documentation.
It's hard enough to get technical minded people to contribute to a git (or style) based knowledge base.
Pick your poison I guess but I'm quite happy to have testers/BAs/directors/etc able to quickly jot down thoughts roughly than have it disappear into the ether.
> The last thing I want is to raise the friction for writing down documentation.
A better solution might be that anyone can write the documentation, and there is a maintainer who constantly refactors the wiki to keep it legible. Makes sure the information is not duplicated, adds hyperlinks to things, etc.
"Everyone's responsibility" sounds like an euphemism for "no one really cares". When people actually care about something, they hire an expert.
Why do you hire software developers, instead of making software development everyone's responsibility? Is that because most people suck at software development? Well, most people suck at writing documentation, too.
I mean. I guess? The difference between having it written down in Confluence and disappearing into the ether is academic though. Either way nobody will ever find the information again.
There are some really nice git-based wiki systems out there, and one is built into GitHub and GitLab. If you want that type of workflow for your wiki, it's easy to get.
No not to the same extent. Meetings proliferate because people love them. LOVE them. Programmers dislike them because of flow but everyone else can't get enough of them. I only realized after enough years outside the tech industry. Meetings are like eating junk food. They make people feel important and like they had a busy/productive day, even if the actual mental effort required was low and the measurable output minimal. Compared to sitting at a desk and focusing on a single task, it's far inferior for most people, who find that quite exhausting or demotivating.
About mental effort in meetings being lower than solving a technical issue, it is definitively true (most of the time), even though non-technical people cannot even grasp how much a difference there is.
In a previous software job, I was participating in a trade show. It was meeting after meeting, with both sales and technical aspects (I was paired with a sales guy). After several days like this, I remember vividly the sales guy saying how this trade show was good, but at the same time complaining how tired he was because all of these meetings. I kept it to myself, but for me, this was nearly like vacations compared to the day to day usual work
I think that's a bit ungenerous to think that programmers are somehow different in this regard. Everyone hates attending boring meetings. And non-programmers also have heads down individual work they need to / would rather be doing.
However I agree that the extent of it is the key. I think most people actually hate running meetings (it's basically public speaking), but they hate writing documents even more. It's much easier to "voice over" something than to sit and write it out. It's also easier to enforce attendance than to police opening and actually reading documents. So meetings are the path of least resistance/effort.
Well people say they hate attending boring meetings, but when you observe what people do it's normally the coders who actively find ways to skip / who aren't setting up new meetings / are requesting fewer meetings. Other job roles, at least in my experience, tend to jump to a meeting as the first reaction. Developers will say: let's discuss it over email. Others say: let's hop on a call / grab a room. The number of meetings I've been in where there are multiple participants who don't have any obvious reason to be there, and who don't say anything throughout the entire meeting, is uncountable.
Now you're right it's obviously not that black and white, I'm generalizing. But I think devs often under-estimate how many people in a typical company perceive meeting other internal employees as amongst their primary outputs, as an end in and of itself, not just a means.
A good way to observe this in action is to try and enforce a rule that meetings must have pre-published agendas. Good luck with that! People will just work around it or write useless non-agendas because often a meeting is not to get something specific done, but is used more like a sort of coffee break to split up the day and give people something to look forward to between desk time.
Something else worth remarking on - a lot of people in sales or marketing roles never seem to use word processors. They communicate ideas by sending PowerPoint decks around, often with a density of words in the slides too high to actually project (only readable on hi-dpi screens). Where I last worked there were people whose working hours boiled down to meetings and PowerPoints. They could spend a whole week making a deck, which would only be seen by their colleagues in a meeting. I found it odd but maybe the slide templates help them structure their thoughts.
> you observe what people do it's normally the coders who actively find ways to skip / who aren't setting up new meetings / are requesting fewer meetings
Interesting - I actually chalk up that to two things: first, people in SWE roles having historically been given a tremendous amount of latitude for behavior that does not conform to "professional" norms. The freedom to dress however they want, work from home, and skip out on meetings they don't want to attend are all of a piece. And second, I think software engineering work is often (generalizing as well) less cross functional than other roles. Gathering requirements and understanding how it fits into a larger company plan is usually tasked out to PMs or tech leads, which is not something I've seen for Finance, HR, Legal, or other functional roles. So the need to schedule their own meetings is also lessened.
I was in yet another meeting the other day, and we had more "observers" than actual contributors / workers. Totally absurd. At the end of the meeting, some of the "observers" simply don't understand the meeting, and want a follow up meeting to discuss the meeting, so they discuss it in yet another meeting (presumably to appear intelligent so even more meetings can be scheduled.)
I've also noticed the PowerPoint issue: densely packed slides look ridiculous and don't project well. I think a lot of people wind up simply reading off the slides.
I tried to create this type of culture at my last gig, where I had the unusual privilege of being able to hire almost the entire engineering team, alongside my manager who was also very document oriented. Unfortunately, it didn't work out. Maybe Tremendous has done tremendously better, it's certainly possible, but here is a list of things that went wrong, maybe it's useful.
1. Standard interviews don't assess reading/typing speeds. If you want a high documentation culture this is critical. It took way too long for us to figure this out but many people in the company were significantly slower at reading/typing than us; they found long documents overwhelming and would find excuses to not read them. Slack conversations became a massive sore spot because unknown to us some people felt like they couldn't keep up. They'd try to type a question or response and we'd already posted another two paragraphs before they got a chance to finish their thought. They'd complain to each other that if they asked us a question they got back an essay in response, etc.
2. Documentation requires ownership, otherwise it rapidly becomes useless. Standard corp tooling like wikis doesn't make such workflows easy. They are however optimal from a corp politics perspective (dispersal of responsibility). Maintaining markdown based websites works well as long as you have empowered maintainers who view document quality as a core job function, but you have to force people to submit changes by e.g. rejecting at code review time changes that don't update the docs. People will moan. They will ask you to do it for them. They will submit absolutely min-viable docs changes, they will demand you hire technical writers even if they're easily capable of doing it themselves. And of course the moment you're not using a git-style workflow, just forget it, you have no chance of preserving coherency in any sort of knowledge base.
3. Lots of people aren't just slow but actively HATE reading and writing. They will make things up on the spot, or lie, or just flat out refuse to do the work rather than sit down and read a long document. Jeff Bezos has said about why Amazon uses meeting time to force people to read the memo:
"If we don’t, the executives, like high school kids, will try to bluff their way through a meeting"
You will have to fire people for refusing to read things if you're serious about creating and maintaining such a docs-oriented culture, which in practice is so unpleasant nobody ever does it and so maintaining such a culture is nearly impossible. You will also have to flat-out refuse to meet people in order to force them to read, because otherwise they'll receive a document and just ignore it. I had several cases where one of my most senior engineers would assert that a product we used didn't have feature X, and I had to correct him by pointing out that the user manual discussed feature X in detail. I knew this because I'd actually read the user manual cover to cover. Basically nobody does this and guess what, if you're the one person on the team who reads stuff then you're going to come across as the awkward smart alec who makes people look stupid. Sometimes, ignorance is bliss.
This is brutal, but sheds light on why my own documentation-heavy style doesn't catch on.
I get questions from people, which can be answered by searching my wiki and just finding the right page. I can see the number of pages visits with the wiki tool I use, so I am led to believe that I'd get a ton more questions if not for my wiki.
So what's the problem? I am just one person in my group. There's a couple hundred of us, and I don't think the next most documentation-heavy engineer is producing half of what I am. (Probably more like a quarter)
Which is a real shame. Part of why I produce so much documentation is that I've created by own tooling and processes which let me generate vast amounts of useful content on the fly, and quickly. I've got 100+ hours of dev work into one tool, and I'm pretty sure I'm the only user of that tool (although I give presentations on it from time to time). Think: A tool which looks up details about an environment, and then aggregates those details in markdown format (including links to dig in further). Copy > Paste > Save page > Done.
Probably not IMO, it's very unlikely your source tree is structured the same way good docs should be. Docs are often task oriented, code is usually component oriented.
Ha, my tasks are usually updating those components. Mostly interested in developer efficacy. It is a real pain to find and reverse engineer components when starting a new gig. Would be nice to have an explanation and links to other components they interact with.
Sure. I tend to prefer a docsite generated using a rendering tool because when the docs look and feel high quality/high effort, people take it more seriously. But it can be that a README file in a set of directories is OK too. Chrome does this for example.
Could you talk a bit more about your success? What does it do, what pricing model do you use, how long did it take you to acquire customers, do you feel it's worth it? Asking for a friend, of course.
I don't want to talk about it in detail, because I am trying to stay anonymous on HN.
It's a Mac app, pricing model is perpetual license with paid upgrades for major new versions (every 3-5 years). Customers are roughly 50% individual devs and 50% companies who buy for their employees, and a bunch of educational licenses, but I don't earn much from them.
I got initial traction by being mentioned by some popular bloggers and twitterers, but I only started making a full time income after two or three years. I think marketing is mostly word of mouth, and lots of people using the trial version who eventually buy a license, but for the most part I have no idea how people find my app.
As for whether it's worth it -- it's a cushy job, I can take care of my kids and don't need to worry about putting in enough hours, because nobody cares how much I work.
On the other hand it does get somewhat boring after 10 years. I've been hearing the same feature requests and ideas for improvements for 10 years. My customers are all the same average tech dudes like myself. Sometimes an interesting bug shows up, and when I track it down that's the highlight of my week.
My understanding is there's Fleet, their VSCode competitor, which sounds like you're referring to, and the UI refresh, which parent is referring to.
The UI refresh is the same IDE under the hood, just way simpler. I control the IDE primarily through command palette (I think many do?) so decluttering would be great- the UI is unnecessarily complex when you can press a key and type a few words to do what you want.
"I thought trans people being oppressed was clear statistically and unambiguous."
Oppression isn't something you can even give a clear and unambiguous definition for, let alone measure statistically.
"my understanding is that they are a group with massively higher rates of suicide and depression"
This does not automatically imply they are oppressed. An obvious and common alternative interpretation is that choosing to surgically change your gender is at least sometimes caused by mental illness, which also causes depression and suicide, and/or that people who change gender discover that they don't actually like their new gender (for internal reasons, not due to the attitudes of others).
When definitions change organically over long periods of time, due to the masses of people concluding they like the new definition better, that's one thing. It may be disruptive but it's usually a slow process so there's plenty of time to adjust.
But here it's a small minority of exceptionally aggressive people with nothing better to do, trying to change language by forceful tactics. They need to receive strong pushback and possibly exclusion. Words exist purely to communicate, they have no other purpose. Once bored over-educated people are allowed to start redefining the language it's open season and they won't ever stop. See: newspeak, inspired by the Soviet Union. Given your username you should know all about that.
This verbal game-playing has already caused immense damage with the effort to redefine words like racism to mean "discrimination based on skin color that's bad unless it's against white people", an effort that now means accusations of racism no longer carry any weight. Ditto for the attempt to redefine fascism to mean "anything that isn't far left". Words have meanings and people may not simply try to blow their way through to a new meaning by viciously attacking everyone who uses the normal definition until they concede through exhaustion. It's anti-social behavior to do that regardless of whether you like the idea or not. Come up with new words if you care so much.
Do you believe that's what I'm doing? Are my comments "exceptionally aggressive"?
I'd like to think I'm rather the opposite - firm in my convictions but willing to walk anyone through my reasoning. You may not agree with my conclusions and you may walk away thinking I'm out of touch.
But I feel I'm watching a situation where both sides are painting with over broad brushes - you accuse everyone of being exceptionally aggressive, many folks in my camp accuse everyone of being a bigot. It's the same pattern, imo, and we'd get a lot farther if we all avoided such generalizations. (Imo)
No, see my other comment. You're doing fine - same as in the other thread, I'm making a general comment about the notion of language "evolving". Evolution is a natural and emergent phenomena. In this case the language is not evolving so much as being forced into strange new forms through a kind of epistemic eugenics programme.
But let me ask, do you really argue that the people arguing that women shouldn't be called women, are not exceptionally aggressive? These are people who routinely throw around accusations of "phobias" and "isms", publicly demand people be fired and often succeed for using language in ways that are entirely normal, routinely destroy people's entire lives overnight and are widely regarded as people you want to stay away from at all costs. There are views being expressed in this thread that are just flat out crazy, like the idea that if you go on a date with someone you should immediately ask them what genitals they have as a matter of course and if you don't and are then caught by surprise, the problem is you!
"you accuse everyone of being exceptionally aggressive, many folks in my camp accuse everyone of being a bigot"
That's exactly the point. Do you see the connection here? Accusing everyone of being a bigot is exceptionally aggressive behavior. There isn't some equivalence here. People who engage in those tactics will be described as aggressive as a consequence, they can't then say "well I think you're a bigot so we're even", that's not how it works. Aggression is about concrete, objective acts.
This kind of elasticity in language use is the sort of thing that gives AI safety a bad name. You can't take AI research at face value if it's using strange re-definitions of common words.