
I built an IPFS crawler with libp2p - adlrocha
https://adlrocha.substack.com/p/adlrocha-how-i-built-an-ipfs-crawler
======
kickscondor
Was hoping there was a link to the results - either a search box or some
statistics. But hey - free code. Not complaining.

Also, if you haven't checked out Beaker Browser now that it's nearing 1.0 -
great things are happening. This hasn't been broadcast much - but Beaker now
has a built-in editor pane and forking mechanism. It's the best shot I've seen
at a 'view source' 2.0.

There's been some healthy competition lately forking and editing the default
twitter/mastodon clone. (Which is here:
hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ But
forks of it are here:
hyper://0e0d1685c3575cf1df94191e14b4ecf068c43bcefee787ffbbed3d1f4666cab4/ and
here:
hyper://b798b86cc282ae065697f36a2a07072bfa28e5fdfa1d9a8c6fac0f07ff7d6676/.) If
you want to learn Javascript, this seems like an amazing way. (Kind of like
jsfiddle but for actual apps.)

~~~
koeng
Is beaker going to support IPFS?

~~~
anchpop
They used to, but they removed support. It makes sense to focus on just one
protocol I think since they're such a lean operation. Also the guy running
Beaker seems to be sort of anti-blockchain (describing himself as an
"Unblockchain consultant") which might make them biased against it.

Also, IPFS urls change when the content they're pointing to changes, which
makes them bad for decentralized web stuff. They are working on IPNS to handle
this but I don't know what the status is.

~~~
momack2
Actually, content-based addresses is really _critical_ for decentralized web
stuff - since you can't trust a single source to name data, and instead want
many different parties to participate in the same global address space in a
secure and trustless manner. Check out the dweb primer for why working with a
base-layer built on content addressing is so powerful:
[https://flyingzumwalt.gitbooks.io/decentralized-web-
primer/a...](https://flyingzumwalt.gitbooks.io/decentralized-web-
primer/avenues-for-access/lessons/power-of-content-addressing.html)

IPNS is used by many groups to create a mutable layer over IPFS (it recently
got much faster in our 0.5.0 release in April), but you can also use tools
like Textile, OrbitDB, or other mutable databases built on IPFS.

~~~
anchpop
I read that document and it seems to me like everything in there applies to
Dat also, which isn't really content-addressed. But if IPNS is fast now it
seems like it could be a good replacement.

------
momack2
For anyone looking at
[https://arxiv.org/pdf/2002.07747.pdf](https://arxiv.org/pdf/2002.07747.pdf)
to get an estimate on the actual # of IPFS nodes, note that these tools are
trying to crawl the number of nodes acting as "DHT Servers" in the Public IPFS
DHT. Nodes behind home firewalls or NATs don't become DHT routers because they
aren't dialable, and other large users intentionally set many of their nodes
into a lower-bandwidth client mode. Also, many applications run their own
private IPFS networks (like OpenBazaar for instance) and don't participate in
the Public IPFS DHT, so wouldn't be reached through a crawler here.

------
wrnr
How is the adoption of IPFS going, any example application that are using it
and are themselves being used? I'm thinking of integrading it into a product,
it has the best _blockchain_ like properties without the mining madness.

~~~
momack2
Going well! Check out all the awesome applications using IPFS in our editable
ecosystem diagram: [https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-
Appli...](https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-Applications-
Diagram)

A few highlights are Fleek, Textile, Audius, 3box, Anytype, Qri, Berty, and
MagicLeap! Specifically in the blockchain space, there's Peepeth, Augur,
Uniswap, Civic, Arbore, and Aragon.

------
carapace
So this is crawling the nodes not the content?

~~~
mirimir
Seems so.

There is [https://ipfs-search.com/](https://ipfs-search.com/)
[[https://github.com/ipfs-search/ipfs-search](https://github.com/ipfs-
search/ipfs-search)].

