Interesting, I've been looking for a system / tool that acknowledges that a dbt transformation pipeline tends to be joined-at-the-hip with the data ingestion mode....
As I read through the documentation, Do you have a mode in ingstr that lets you specify the maximum lateness of a file? (For late-arriving rows or files or backfills) I didn't see it in my brief read through.
great point about the transformation pipeline, that's a very strong part of our motivation: it's never "just transformation", "just ingestion" or "just python", the value lies in being able to mix and match technologies.
as per the lateness: ingestr itself does the fetching itself, which means the moment you run it it will ingest the data right away, which means there's no latency there. in terms of loading files from S3 as an example, you can already define your own blob pattern, which would allow you to ingest only certain files that fit into your lateness criteria, would this fit?
in addition, we will implement the concept of a "sensor", which will allow you to wait until a certain condition is met, e.g. a table/file exists, or a certain query returns true, and continue the pipeline from there, which could also help your usecase.
feel free to join our slack community, happy to dig deeper into this and see what we can implement there.
(A) not all environmentalists take a non-nuclear position. Be careful painting with a broad brush
(B) some people's opinion is based on emotional reaction to a technology rather than a cold calculus of how much a technology could do to reduce global warming effects and reduce humans environmental impact.
I was curious about the "not all" since it depends whether "not all" is 0.1% or 99%. So I did this:
1. Google for "largest environmental organizations"
2. Select the top 5
3. Search for their position on the subject
Here's what I found are the top five largest according to a rudimentary Google:
1. Laudato Si' Movement "In addition to being unsustainable and failing to contribute to climate objectives, nuclear power is also proven to be a dangerous, dirty and expensive source of energy"
2. WWF: "Nuclear energy cannot, and must not, be considered part of the urgently required energy transition."
3. 350.org "Nuclear: a deadly and costly distraction"
4. Climate Foundation (couldn't find anything)
5. Greenpeace "Nuclear power is dirty, dangerous and expensive. Say no to new nukes."
I think the environmentalists don't like it, man. I think I've seen enough. Nuclear power must be pretty bad if it is so broadly hated. Or the environmentalists are fools.
I was going to say, the fab is capital-intensive but at least has value as an asset so long as you can keep it printing out chips profitably, which is more likely in a trade war where certain foreign chips are banned...
As for what chips you print on them... x86, ARM, RISC-V, GPUs, NPUs, RAM, FPGAs, PLAs, print whatever the market asks for... It's not like there's fundamental superior value in x86 as an intellectual property, just inertia.
But local manufacturing gets protected in a trade war, because you'll need it if push comes to shove. Intellectual property just gets stolen and copied.
I have found postcards to be a great way to send encouragement, sympathy notes, or thank-you notes.
The small space on the post card for writing means you can usually only fit a few sentences in (keeping the time investment short) and you don't even need to fold up the letter into an envelope. You can start with a few post cards bought from a gift shop, and if you like the habit, you can get themed boxes of 100 postcards from an online retailer for $20. (e.g. birds, or cats, or Disney Princesses).
A buyer is not always required to select the bid of the highest monetary value; sounds like the Onion had a proposal that was "a reasonable sum of money, and also we help lead a healthy way to find a path to redemption for this website and make it a kinder place than before"
My most recent foray into MQTT was using the MQTT 3.1.1 compatibility mode in NATS, which was pretty easy to set up once I got the permissions sorted out.
Works great for my smart home needs while also letting me explore latency and throughput load testing if I want to have a little fun.
"The God Virus: How Religion Infects Our Lives and Culture" by Darrel Ray follows in the footsteps of The Selfish Gene to certain logical conclusions about clusters of related memes/concepts (specifically: how religion(s) formed and evolved as clusters of overlapping concepts)
As I read through the documentation, Do you have a mode in ingstr that lets you specify the maximum lateness of a file? (For late-arriving rows or files or backfills) I didn't see it in my brief read through.
https://bruin-data.github.io/bruin/assets/ingestr.html
Reminds me a bit of Benthos / Bento / RedPanda Connect (in a good way)
Interested to kick the tires on this (compared to, say, Python dlt)
reply