Just curious, what would you consider, "absurdly small amounts of data around using big data tools like spark" and what do you recommend instead?
I recently worked on some data pipelines with Databricks notebooks ala Azure Fabric. I'm currently using ~30% of our capacity and starting to get pushback to run things less frequently to reduce the load.
I'm not convinced I actually need Fabric here, but the value for me has been its the first time the company has been able to provision a platform that can handle the data at all. I have a small portion of it running into a datbase as well which has been constant complaints about volume.
At this point I can't tell if we just have unrealistic expectations about the costs of having this data that everyone wants, or if our data engineers are just completely out of touch with the current state of the industry, so Fabric is just the cost we have to pay to keep up.
One financial services company has hundreds of Glue jobs that are using pyspark to read and write less than 4GB of data per run. These jobs run every day.
Divorce is a legal procedure to dissolve a legal union; local rules vary, but while it can be accomplished without standing in court, in the US at least the terms of the dissolution must still be documented and--even if signed by both parties--approved by a judge. One of the most notable pieces of the divorce is the division of the union's assets, which especially in a "no fault" divorce must be "fair" to both parties since neither one is accusing the other of wrongdoing.
_In general_ (I am not a lawyer, I don't know your situation or laws of your area), anything acquired during the marriage is joint property. That can include every paycheck you receive in the span of the marriage, even if the salary is established before marriage, deposited in a personal account, and you were the only one working the entire time.
And that means everything you pay for with that money is also jointly owned, including any mortgage payments using those joint funds. Therefore your (to be ex-)spouse may have stake in any property you own that must be fairly divided. If you bought the house after getting married this may at least be a simple 50/50 split, but if one of you put your whole pre-marriage savings as the down payment you can bet this gets a lot messier.
Speaking of money, if you're making 100K and your spouse makes 50K the union has 150K to sustain a standard of living. After the divorce neither party will be able to keep that same standard of living of course, but one party is more greatly affected than the other. In a "no fault" divorce the outcome must be "fair" to both parties.
But WTF does that even mean? Is it "fair" to have to pay your ex-spouse for being less successful in their career? According to the courts the answer may be, "Yes," especially for any income gained (or lost) during the marriage because, "Pursuing a new job is something you decided _together_, right?"
All that is to say that even if both parties agree that holding onto a failing relationship isn't to anyone's benefit, divorce is something else that either (or both) might not find so agreeable.
When my ex-wife asked for a divorce, I wasn't going to fight over what had already felt by that point a completely one-sided relationship. That helped a lot getting over the emotional shock of the situation, and we did manage a pretty amicable no fault divorce. But it was still was months of debating how much she should get from the two years paid into our 30 year mortgage, and ended with both of us being worse off financially for several more years.
Divorce is not just a break up that you state your waning interest and walk away from; it is a complicated legal process that will force some uncomfortable conversations about things you probably would have never imagined being an issue while you believed the relationship was going well.
Which is not to say you shouldn't do it if your relationship is bad, but if you think your relationship is bound to end on some frivolous grounds you either shouldn't get married in the first place or find relationship help immediately.
I'll be showing my utter lack of romanticism, but I think any long lasting relationship (marriage or not) is on the same boat, the only difference will be how the couple discusses these issues, and whether they have a leg to stand on when shit hits the fan or they're just SOL.
Imagine being in a non-married relationship and taking a mortgage for a house you'll both live in and potentially both pay. You'll still need to have these uncomfortable discussions, probably upfront and not when it goes south. Same if you have a kid, if one quits their job to take care of that kid (or take care of the other if needed).
A marriage will package a defined set of rules to apply to these situations, where not being married will force a lot of case-by-case examination, with probably one end of the relation getting shafted. Some countries (Japan is one, there must be others) have a "not married but could as well be" status for these kind of situations.
What I'm saying is, being in a marriage or not is akin to having a contract or not. It doesn't change what you're supposed to be doing, it will only help to frame the discussion in the dire times. By the same token, breaking up a long lasting relationship shouldn't be about whether the paperwork is a PITA or not, not being in a marriage doesn't make it OK to just screw the other side for instance, hopefully you'll still have the uncomfortable discussions either way.
Sorry, that's the point I was making to which I may have got off track. Marriage can't be ended by one party "calling it quits on frivolous grounds" because its a legal contract between two people.
Both parties must come to an agreement about how to break the contract, including conditions they may or may not have realized they agreed to, otherwise the must demonstrate how one has already broken the contract.
If you want a relationship you can just walk away from, or believe your parter does, you should not get married.
When I get frustrated enough to consider putting a kill switch in my work, I cool off by reminding myself that would prompt them to make a huge effort recovering from backups and have someone go through my code to get it working again. If I just do what they ask me instead, it will slowly decay without anyone having any idea what it even does, until its too late to fix or recover anything.
There are almost certainly different approaches depending on the environment and situation. But, especially if this is a "culture" problem one of the best fixes I've found is to make shorter meetings with a defined agenda, such that you can always pull out a, "I'd love to take this offline, but we need to get back to..."
I've personally found you should almost never need more than 30 minutes unless you specifically want to get into rabbit holes. And if you do need more than 30 minutes, it's probably better to split it into multiple sessions of no more than 30 minutes anyways to prevent this from happening. If you still have this problem at 30 minutes, shave 5 from either side (or both), which you can even use the excuse of giving time to transition between meetings.
That's not to say you shouldn't genuinely allow room for brainstorming, but if you're going to take an entire room of peoples time, make sure it's something the room agrees is worth discussing and find another time to do it instead of getting sidetracked now. If not, offer some 1-on-1 time, and move on.
In a college chemistry lab we would have to write lab reports of our work. The instructor made it very clear that he has never given 100% on a lab reports because there's always something to improve.
A one of my CS cohorts happened to be in the same class so we teamed up for the first lab project. It was pretty straightforward, we collected whatever information, and started working on our report. We didn't bother spending much time on it because we already knew we'd lose points for something or another.
When we got it back, there was a big, red "100" on top. We checked around and it did look like we were the only ones that got a perfect score, so we went to the instructor and, mostly jokingly, said, "What's up with this?" to which he stayed on beat and replied, "Do you want me to take another look?"
It's not hard to do good work, but you do have to make a habit of it. Re-read what you write, preferably out loud, to make sure it actually makes sense.
You'll still make errors and mistakes and you won't catch them all, but no one's going to care about a typo or two unless you draw attention to it with more glaring problems. And I think this is where metrics, especially things like code coverage, can actually be detrimental, because they bring attention to the wrong things.
Specifically, in places I've seen code coverage enforced, tests (written by consultants making x5-10 more than I do) tend to look like, `assert read_csv("foo,bar") == [["foo", "bar"]]` that execute enough of the function to satisfy the coverage requirements, but everyone is surprised when things break using a real CSV document.
The corollary of the author's trick is that if you keep making excuses to produce poor work, you may subconsciously decline instead.
A "lifetime warranty" often refers to the "lifetime" of the product, not your lifetime or the expected lifetime of any person.
In that sense, T-Mobile has played this pretty straight. Customers got a price lock for the lifetime of the offer.
Just on a quick read, it looks like a "lifetime warranty" must actually define a length of time though. Which for a product you can, at least claim to, stress test or base on the weakest component.
Physical attributes are easy to recognize, so if these weigh heavily in your expectations, you can more rapidly estimate the distribution. Being able to do this quickly also means you can break out populations to make more accurate estimates of your current environment; i.e. while the media bombards us with beautiful celebrities, you can maintain an independent estimation of your actual dating pool.
Social attributes take more time. You need to spend time with someone to understand where they are, and you would need to spend time with a lot of people to understand the distribution. You can try to use heuristics, like being in a relationship with someone you believe has similar expectations, but as noted those aren't actually very accurate.
Reading about aphantasia was a trip. I couldn't even understand it at first, so curious if this condition fit me I looked up the test for diagnosing aphantasia which started describe scenes of a beach, now with a sunset, and so on, to which I was hopelessly lost because the responses seemed to suggest I should be "seeing" something. I could "imagine" a beach, and think about how wet sand feels and what waves sound like; but of course I couldn't "see" it, it was just imaginary. I had to read an essay from someone else with this condition discovering that other people do "see" things before I had my little spit-take moment.
I think I _used_ to be able to visualize things. I remember feeling frustrated when I was younger and did a more art because I could never imagine the same thing twice when I wanted to draw it. Every time I tried to think about about it, it would have a different pose or texture or orientation. But even then I think I just had a different "level" of aphantasia. I could never figure out how to use color. I kept to pens and lead pencils for art and couldn't get into painting. Even now, staring at a wall and thinking about repainting it, I cannot visualize it with any other color, much less two or three colors for trim and accents.
One of the advantages, perhaps, is that I don't need to close my eyes to imagine things. There's no point, there's nothing there. I may let my focus drift so I don't get distracted by shiny or moving things. But I can stare at a wall, or go for a hike or a run and just let my mind go wild.
That said, I do enjoy fiction, though I never really get "into" anything in particular. In light of this discovery, perhaps it is because I'm not really able to "visualize" scenes to recreate memorable moments, I can only really enjoy it in the moment and maybe recall a few quotes and descriptions, which are harder to get excited about.
I've been involved in a product review at my work. The tool hits a sweet spot of identifying a real problem and demoing impressively. I have little doubt we will purchase this tool unless the beancounters simply reject the expense.
But I find myself against it. This is somewhat ideological; the tool is, at its core, a telemetry tool, and I don't believe we have the maturity to manage and leverage that data effectively. And the data and features to product enables? We already know where the problems are and have other tools to address them. It's just that everyone is always "too busy" to actually listen to the customers and do anything about it.
Pondering how to express this then, I ended up labeling the product (at first a "luxury", but realizing people want those and doesn't help my argument) a "toy," like a jewel-encrusted hammer: It's pretty, but if a plain hammer isn't solving your problem this isn't going to either. Worse, the extra time and care needed to maintain this tool, in an organization that's already "too busy", is likely going to be even less effective if not a net loss.
However, it occurred to me, knowing one of the people trying to push this tool, calling it a "toy" would only be an opportunity:
Toys can be incredibly powerful in the hands of a good imagination.
And, I agree.
And this is where I struggle. Collectively, we don't have a "good imagination." We're all too busy being busy to do anything creative and solve the problems we have. But individually there is a lot a creativity that just lacks the means to express itself. And enabling these people is why _I_ do software.
I'm still not sure this tool is the right way about it, but that fact we're even here is testament that the current technologies aren't inspiring anyone.
I recently worked on some data pipelines with Databricks notebooks ala Azure Fabric. I'm currently using ~30% of our capacity and starting to get pushback to run things less frequently to reduce the load.
I'm not convinced I actually need Fabric here, but the value for me has been its the first time the company has been able to provision a platform that can handle the data at all. I have a small portion of it running into a datbase as well which has been constant complaints about volume.
At this point I can't tell if we just have unrealistic expectations about the costs of having this data that everyone wants, or if our data engineers are just completely out of touch with the current state of the industry, so Fabric is just the cost we have to pay to keep up.