This integration is way more limited and frictioned. Whereas with search Apple's fully outsourced and queries go straight to your 3rd-party default, Siri escalates to GPT only for certain queries and with one-off permissions. They seem to be calculating that their cross-app context, custom silicon, and privacy branding give them a still-worthwhile shot at winning the Assistant War. I think this is reasonable, especially if open source AI continues to keep pace with the frontier.
The trendline is definitely toward increasing dynamic routing, but I suspect it's more so that MoE/MoD/MoDE enable models to embed additional facts with less superposition within their weights than enable deeper reasoning. Instead I expect deeper reasoning will come through token-wise dynamism rather than layer-wise -- e.g., this recent Quiet-STaR paper in which the model outputs throwaway rationale tokens: https://arxiv.org/abs/2403.09629
I think it's reasonable as an early-ish stage startup to say to a candidate that ~"eventually, with scale, there will be cool and impactful ML opportunities here" as long as you're realistic and upfront about the facts that ~"right now most of the impact is in simple but foundational analyses" and ~"there'll be some amount of fires to put out and rote work to automate".
Seems like permanent remote-optional is rapidly becoming the competitive equilibrium in tech.
As tooling improves, there is some point at which gains from expanded recruiting pool + real estate savings outweigh coordination/culture costs of remote. Beyond this it will be irrational to force physical presences (for many roles, at least) and there'll be no going back.
Approximately once per week I observe a comment on HN saying approximately "search doesn't work anymore." Why is this? Why do I only observe such complaints here?
Lots of people here who were and remember being "good at search" and able to make exactly the thing they're looking for shoot to the top 3 spots on the results screen and being able to craft a series of searches such that, if they all failed, one could be pretty damn sure the information wasn't online or at least was hidden from the Google bot, neither of which are really possible anymore. The 100th time one attempts to find something on Google that one knows is there but just cannot get to show up no matter how many rare keywords one uses, one concludes Google has lost some pretty significant utility it used to have and starts to wonder what one is not finding when one doesn't know exactly the page one is trying to find.
(of course it may be better for lots of other things, but for a fairly large set of "finding things on the Web" tasks it's way worse than it used to be, to the point of being nearly useless—in part I think this is because sometime around 2008-2010 they stopped trying to fight webspammers, choosing instead to embrace some set of them provided they play by Google's rules, and downrank anything that wasn't "well-behaved" webspam or well-known sites hard)
I definitely remember a time when Google was much, much better at returning relevant results for technical content. My feeling is that there are (at least) two factors at play:
1) Google and other search engines revised their algorithms more than a decade ago. Instead of showing you results for what you searched for, they now show you what they _think you meant_ based on your search history and the search histories of millions of other users. This means searching for uncommon topics and phrases get you useless results. And you can't disable this functionality because it's baked into the core of how Google indexes and categories the content that it slurps up from the web.
2) Blogspam authors and e-commerce sites have gotten SEO down to a science, to the point where searching for nearly _anything_ not super specific no longer gets you _information_ about that particular thing, it gets you blogspam articles filled with fluff and affiliate links, or half-broken e-commerce sites trying to sell you something vaguely related to what you searched for. This is not technically Google's fault but there is a lot they could do to curb this, but all that ad revenue on those sites is how they earn that sweet sweet lucre.
> This is not technically Google's fault but there is a lot they could do to curb this, but all that ad revenue on those sites is how they earn that sweet sweet lucre.
That had been going on for years, but there'd be clear times when Google got ahead of it and results would get much better for a while, and because search was so much more precise it was possible to work around the spam. That those good times stopped happening and results are now a consistent and fairly high level of "spam-filled" by content that's seemed pretty much the same sort of crap for years, leads me to conclude they stopped trying. IIRC right around then they stopped the "no no, our ads our different and good, they're just text and always formatted the same way so it's easy to tell what they are" and became just another banner ad slinger.
[EDIT] just mined Slashdot for that last bit, looks like that happened around the last half of '07, which roughly checks out with my recollection of Google search abruptly getting much worse around '08-'09 then never getting better again.
As with so many things, it seems like incentives are at the root of this problem: manufacturers want to sell more foods, and adding sugar (whose addictive properties are likely to increase consumption x retention) is an easy tactic to pursue.
What should be done to counter this? Tax sugar and it will be substituted for some other unhealthy, manipulative thing. Perhaps only shifting preferences via culture / education (s.t. sweetened products sell relatively worse) can work?
Sure, not releasing the full trained model probably delays it, but sooner or later a bad actor will do their own scraping and train their own model and share it around and the genie will be out of the bottle. Then what?
I think we need to be conducting AI research (and building software generally) under the assumption that all of it will eventually be repurposed by bad actors. How would our practices be different if we consistently and cautiously did this?
Here's a thought experiment: how would the Manhattan project have been different if it were carried out in the open and its products were instantaneously and infinitely reproducible? What is the MAD equilibrium of AI research? I think the impact potential is similar even before AGI.
A lot of the advancements that made the Manhattan project were published by the Germans. On hearing about Hiroshima, Otto Hahn was "'shattered,' and went on to say that he felt 'personally responsible for the deaths of hundreds of thousands of people,' believing that his discovery had made the bomb possible." (https://www.washingtonpost.com/archive/opinions/1992/03/01/t...)
wasn't that the point of this whole openai thing? they didn't like the idea of there being a club with just google in it that had access to resources and funding to collect and train on massive datasets so they were going to be the "bad actors" who would do their own scraping, train their own models and share them around?
isn't it supposed to be called OPENai?
they don't want to share the data because they don't want to throw away the edge they've gained by collecting it. :)
computer programs that generate human like text aren't dangerous, the internet is full of human like text that is mostly bullshit anyway.
I have had the same impression regarding their work on dota. They got a lot of publicity with it but their work is not open at all. They have released neither their code which runs the bots on dota2 nor their training code nor the final model. All we have is video recordings of a few games against humans.
CommonCrawl already has open dataset in petabyte size ready on AWS. Even if it didn’t exist, scrapping 80GB of data in AWS is trivial. I am surprised authors considered this as such a big deal. Also notice that performance is not anywhere close to humans. It sort of works and it’s astonishing that it does but long way to go before we have to fear weaponizing text generation.
I think the big deal is the size of model, BERT large is 300M params, and this one is 1.5B.
Bert has been trained on pod with 64 TPUs, and this model requires even larger GPU/TPU cluster. There is no way indie underfunded researcher can train such model.
1) low signal-to-noise ratio - i.e., lots of duplicative stories, publishers are commoditized, and there are no mechanisms to distinguish (and reward!) high quality content
2) paywalls and/or extreme ad loads lurking everywhere
3) local news is a second-class citizen
4) lack of user control in customizing topics, while at the same time all the (uncontrollable) targeted "for you" stories tend to overfit (e.g., Google seems to think tech news and the Patriots are all I care about)
In general I'm extremely dissatisfied with the functionality of available news products and with the consequences of attention-optimized digital news media on our world. There's a lot of data which indicates most people feel some degree of similarly [1][2][3][4] and I think there's opportunity and demand for creating better.
As such, last summer I set out to build Gatherscope - a news ecosystem designed from first principles to serve readers and sustain quality journalism. At the moment, the high-level plan is:
1) build an aggregator layer that's differentiated in the breadth, depth, quality, and customizability of content that's surfaced and in applying machine clustering and summarization to make it faster and easier for busy consumers to digest what current events are happening, see varying perspectives about them, go deeper where worthwhile, and then get on with their lives
2) stack other layers around this - either by building them or through 3rd party relationships - to further deliver a rich, insightful, frictionless UX (e.g., bundled multi-publisher subscriptions, community quality review, synthesized audio, contextual search and feedback loops, publisher tools for local journalists)…these should all work independently but best together
Today there's a (quite-rough) MVP of part 1 live at https://www.gatherscope.com and a very long way to go :)
All this to say: if any of this sounds interesting and/or you have strong, thoughtful opinions about news media, I'd love to hear from you. I've recently begun a hunt for collaborators, and even if that's not for you it would be good to hear how to build you a useful product faster.
I find the ethical boundary between colorizing old footage and 'deepfakes' to be fascinating to consider. The former is celebrated here while simultaneously the latter is an object of increasing apprehension, abhorrence, and calls for regulation.
An obvious difference is intent. The intent of They Shall Not Grow Old is (I assume) to memorialize historic events and enable modern audiences to understand them more viscerally. The intent of (most) deepfakes is deception.
Still, what should we do when the same technology that enables deepfakes also enables better They Shall Not Grow Old's, perhaps concerning subjects more poorly-documented than World War I? It's acceptable to make assumptions about the colors and sounds of a moment from 100 years ago and represent them indistinguishably from truths, but how about assumptions as to who was present and what actions they took and words they spoke?
Peter Jackson addressed the ethics of colorization in the behind the scenes extra after the credits. He believed that the original camera men wanted to get the most accurate picture of the front lines. However color cameras and steady frame rate capable cameras were not available. He believed the artists would have picked better cameras if they were available so he wasn't violating the vision of the original people making the films
Sure, but the point is that the colours aren't real, isn't it?
It's like, say, a movie or novel based on a historical event we don't know much about - it may aid understanding of the event, but the details are made up. It's fictionalized, "based on a true story", and to present it as fact or as history would be misleading.
I strongly recommend watching the behind the scenes extra with this film. Peter Jackson has this amazing personal collection of world war 1 stuff. Like the reason they knew what the uniforms color looked like was because peter Jackson has almost every type of uniform from world war 1. And the reason they knew how things like the wheels of the artillery sounded like was because peter Jackson has an extensive collection of world war 1 artillery. I’m sure it could have been better, but a lot of effort went into making this as close to life as is possible.
Sure, I haven't seen the movie. Sorry I didn't make it clearer - I was talking about what rfdearborn was saying more generally about colorizing, fakes, fabrication etc, not specifically Jackson or his movie at all.
I think using video as some source of truth is just as ethically shaky.
Synthesizing texture, adding faces, meh. It's all an array of pixels to aid hallucination. The sooner we break out of video = impartiality, video = truth, the better off society will be.
Choosing what to record, how to crop it, what to show destroys videos supposed fundamental impartiality.
I mean...corporations consist of people (i.e., employees) with feelings and values which - at least to some extent - govern their collective choices and actions. Granted it's tough / probably impossible to disentangle this from profit motive.
A company is not a collective. Most of the employees will receive an unequally small portion of the total profit compared to the upper echelon, and have very little say in the overall direction. In fact, a public company is run by its board, who are free to remove leaders who do not grow the company or lead it the way they like.
