> Unlike traditional Transformer designs, which suffer from quadratic memory and computation overload due to the nature of the self attention mechanism, our model avoids token to token attention entirely.
I skimmed the paper, and unlike transformers they basically can scale much more efficiently with longer context. While it's possible to fit 1M token, you need a significant amount of memory. Alrhough they benchmark against GPT2, so I would say quite preliminary work so far, although promising architecture.
I had a similar problem recently: I got this cheap 5.1 surround system soundbar that support up to Dolby TrueHD via HDMI. But here is the catch, it only works with eArc enabled devices (new gen TV). If you plug your PC you need to use SPIDF or aux which hamper the quality. One solution beside buying an audio ectractor/splitter is to fake the PC edid to be reconized as eARc by the soundbar. Still yet working on this, no strict guidelines existing unfortunately.
> A more concerning limitation is that when the city re-ran parts of its analysis, it did not fully replicate its own data and results. For example, the city was unable to replicate its train and test split. Furthermore, the data related to the model after reweighting is not identical to what the city published in its bias report and although the results are substantively the same, the differences cannot be explained by mere rounding errors.
Very well written, but that last part id concerning and point to one part: did they hire interns? How cone they do not have systems? It just cast a big doubt on the whole experiment.
To hammer one point though, you have to understand that researcher are desensitized to minor novel improvement that translate to great value products. While obviously studying and assessing the limitations of AI is crucial, to the general public its capabilities are just so amazing, they can't fathom why we should think about limitations. Optimizing what we have is bette than rethinking the whole process.
Interestingly the author mentions LoRa as a "special" way for fine-tuning thatis not destructive. Have you considered it or you opted for more direct fine-tuning?
It's not special and fine tuning a foundation model isn't destructive when you have checkpoints. LoRa allows you to approximate the end result of a fine tune while saving memory.
Haven’t tried it personally, as this was a use case where a classic SFT was effective for what we wanted and none of us had done LoRa before.
Really interested in the idea though! The dream is that you have your big, general base model, then a bunch of LoRa weights for each task you’ve tuned on, where you can load/unload just the changed weights and swap the models out super fast on the fly for different tasks.
During my research years, we had to grind on Combinatorial Optimization book by Korte and Bygen for the weekly book reviews. Safe to say, it was not an introductory work. Still it was fun seeing the different examples my colleague would bring up during those meetings.
I majored in CS and I had no idea that was possible: public websites you access have access to your local network. I have to take time to process this. Beside what is suggested in the post, are there any ways to limit this abusive access?
There are no mechanisms in browsers yet. Best you can do is using the OS to forbid your whole browser to access your local network. (And use another browser only for your local network.) Ask ChatGPT for methods to sandbox your browser.
reply