I remember already being able to use certain stackexchanges with kiwix before, as well as the arch wiki, wikipedia without images, and some other great resources. It is nice to see that they actually pay attention to this use-case and I look forward to updated workflows with kiwix or similar in the future. Latency is way better that way, even with good and stable internet. OpenZIM[1] is also useful in turning any page for use with kiwix.
I also have great memories from a University exam where we were allowed to have laptops that were not connected to the internet.
This has been available for a while but it's great to see some acknowledgement especially since the most recent data set was stuck in 2019 for a while.
> ... to ensure that an up-to-date version of our dataset is easily available for those who need it, and will work to improve its readability and reduce its size so there is less friction for end users...
This is great! Too many services today become completely unusable when they encounter technical problems, are hacked or are just lost over time. Having an easily accessible offline copy is always reassuring, showing that their survivability does not depend on just a few people and the projects are fundamentally about the information, not an organization.
This project has really been the make or break for us to be able to do the coding in prisons program, as well as the devshop that develops Ed Tech for prisons at Unlocked Labs. Love at the project has been formalized and will make a huge difference in the lives of many justice involved individuals who just want to return to their communities as productive citizens.
Never heard of Unlocked Labs before reading this comment, but that sounds like an outstanding project. Keep up the good work. You didn't promote yourself but I have no affiliation and this is what I found after searching: https://unlockedlabs.org/
I was once a guest in a maximum security prison for a few hours. It gave me the eerie realization that my own freedom is entirely up to the guards and staff. When that door closes you are no longer free - you're at the mercy of a huge system. It's a terrible feeling. Talking with some of the inmates made me realize how much I had subconsciously dehumanized them as a group. It was eye opening.
Listening to stories of people who have been in prison, even if you never meet them, can help build empathy. Here's a good interview with a guy who taught himself to program while in prison: https://corecursive.com/prison-programming-with-rick-wolter/
If you're ever interviewing a formerly incarcerated person for a job, I'd encourage you to try very hard to keep your biases in check. Building a life after prison seems like a supremely difficult task.
[edit] That's what I get for reading the comments before the article! Unlocked Labs is linked in the 2nd sentence. :)
Love this. Reminds me of the other Kiwix projects to make MediaWiki services like Wikipedia available offline[1]. The entirety of English Wikipedia is ~50GB of text and ~100GB of images.
I downloaded the databases earlier this year via torrent and attached them to a local SQL Server instance. Some notes:
- The files are big. The data files were 486 GB and the log file (I haven't tried shrinking it yet) is 1.4 TB
- There were no foreign key relationships defined. Nor indexes on commonly queried columns. Easy enough to add them as you work on tuning it.
- To be usable by humans you'll need to set up some full text indexes for searching - they use Elasticsearch. SQL Server's Full-Text catalog doesn't get you where you need.
So this is a desktop app, but it uses the server as part of it? The normal Kiwix desktop client doesn't do that right?
I'll throw in my own shameless plug: Self-host your Stack Overflow, Wikipedia etc on Sandstorm: https://apps.sandstorm.io/app/5uh349d0kky2zp5whrh2znahn27gwh... Obviously uses kiwix-serve as well. 3 years old, I need to make a better clip for updating it.
A rather small number of people are working on it, but it's chugging along. Kenton has the keys still so he does basic vital things. A few new apps are in the pipeline. I'm still rather interested, and am developing another app that I expect to get a lot of attention when it's done.
The bigger it gets, the more time can be diverted toward it. I was recently paid to upgrade the Etherpad app. But in theory it could go toward core development (which wouldn't be me).
My favorite consultant I ever worked with was a boat programmer. You hired him for super specialized MySQL magics so he, well his company, charged a pretty substantial hourly rate and he apparently had enough revenue/leverage to get his company to foot the bill for two separate satellite internet connections on his boat. I feel like I would get lonely but it's definitely a vibe.
To me this is programming during the first 10 years where the "Internet" were local BBS, magazines spoke about Compuserve and Prodigy, and the connection rates where impossible, so we had to get by with what came on magazines and local library.
I see the "/s" but I actually do wonder if integrating the "prompt" behavior into the question box would help cut down on the absolutely staggering number of duplicate questions. Regrettably, I'm not enough of a GPT expert to know what percentage of the time it would generate gibberish thus making the duplication question problem _worse_
Pretty sure they already show suggestions when the user is typing the question topic, though idk if they catch much. I guess the question here is whether some kind of AI would pop up more relevant suggestions, all the way to “your question is already answered, doofus”.
Copilot themselves probably swiped it all already, since SO answers are licensed under CC-BY-SA, which to Github is equal to ‘just take it all and ignore the author’. With the possible small difference that the uploader doesn't agree to the Github ToS.
https://devdocs.io/ exposes a huge catalog of indexed and searchable collections of documentation for a wide variety of languages, libraries, and subjects, including HTML, JS, and CSS – though, the only GL I see is WebGL – and _all_ of it can be downloaded to an IndexedDB for offline use.
This is awesome! At first I thought this only supported Stack Overflow and not the other 170+ StackExchange forums, but it looks like it does (or will?). From the blog:
> “We built the Sotoki (Stack Overflow to Kiwix) scraper in such a way that it can capture each and every one of the 180 Stack Exchange websites.”
Unclear to me if "can" means "does" or "will soon" or just "could"
I also have great memories from a University exam where we were allowed to have laptops that were not connected to the internet.
[1] https://wiki.openzim.org/wiki/OpenZIM