I'm so amazed that this is a thing.
Now with active memory defragmentation things are a bit more pleasant, specifically with high delete load actually freeing up unused memory in a timely manner without impacting performance too much.
Previously, to fully recover unused memory, you would have to restart the Redis server. Obviously this is not feasible but when Redis is using >50% more memory on a 120GB machine than it should then you will have to consider this an occasional housekeeping option -- now, as mentioned, this ridiculous task is no longer necessary.
Is Redis managing this itself using simple calls to Jemalloc, or is Jemalloc doing it on its own because it has better algorithms than the OS?
Here is an illuminating discussion between Redis and Jemalloc devs regarding this:
Redis will perform its own housekeeping, hence the usage of the term 'active.' AFAIK, rather than metadata being stored for this, there is a periodic active scan and manual measurement.
See more here:
See here for an example with RPUSH: https://redis.io/commands/rpush
RPUSH is O(1) which the best you can possibly get.
ZADD on the other hand is O(log(n)), which is quite good but at a large scale it becomes easier to run into performance limitations, especially on high workloads. Performing O(n) operations on large data sets is out of the question unless you're comfortable with Redis being unavailable for multiple seconds/minutes.
That said, you will have to limit yourself to operations that are computationally simple (ie: O(1) or O(log(log(n))) or pre-shard across multiple instances (or run a Redis cluster).
I've actually been thinking about how to do compaction in C++. The reason you cannot do it in general is code like "socket.send(new object, sizeof(void*)". The remote is then expected to send back that pointer which you cast to object and reference. Evil code for sure with a lot of failure cases that result in leaks or security holes, but possible.
What I'd like to see is clustering with support for merging data between nodes. You'd probably end up with duplicates, which I'm OK with, but you'd avoid the data loss that happens with other modes like Pause Minority, where messages get dropped on rejoining a cluster.
trick is to use the utils/install_server.sh
I've said this before and I'll say it again...thank you for all you do!
Although I'm curious to know what algorithm it uses for nearest search, it doesn't talk about it in the doc.
I don't really understand what redis should not be used for, I guess it's not for complex queries? Conventional RDMS really seem to belong to the hard disk drive age. So the difficulty resides in having well designed data schemes.
I frequently denormalize that data INTO redis so I can answer certain classes of queries quickly. I'm also very happy with redis for caching, rate limiting, inter-process-communication, distributed locking and tons of other fun use-cases.
During the podcast the Redis guy mentioned that 4.0 was on the verge of being released.
This is a pretty weak conclusion to draw. Did you try running Redis on its own without Gitlab? If not, I would encourage you to try it out as it takes up much less RAM than you think. The Raspberry Pi is even a supported platform...
These threads fill with reactionaries who think marginalized people are just complaining about nothing or by reductionist leftists who think so-called "identity politics" detract from class struggle.
Edit: It's worth noting that antirez has, as is usual for him, dealt with this issue wonderfully by making a call without downplaying the importance of the problem.
He was doing okay until he wrote that everyone is a slave because having to follow "politically correct rules" is itself a form of slavery.
1) We have a SLAVEOF NO ONE command which is very popular.
2) We have a note about slavery here: https://redis.io/commands/slaveof
For new projects I suggest to pick other terms as well, even because master and slave are kinda inelegant IMHO other than linked to bad things.
I can respect (and disagree with!) the backwards compatibility argument.
I've always found you to be a considerate and thoughtful maintainer (no easy thing) whose work I really admire. So, I'm very glad to see that you've taken the steps you have on this front.
Thanks for your reply.
It's a choice whether or not to use these words, and any project can use them or not as they see fit. They are common in the field after all. I definitely think there's a high road here, and you've taken it, cheers!
Bravo and thanks for pushing the envelope forward.
Regardless of whether you agree with the changes, the fact that these discussions are occurring about the code that we write is really great. I love seeing how we can take code to a much different level than the 1s and 0s it is often reduced to in the mind.
I'd love to see some other examples of similar movements in software whereby we are taking great care of the code we write as an author, historian, or any other writer would take care of their own words.
 - https://www.drupal.org/node/2275877
 - https://issues.apache.org/jira/browse/COUCHDB-2248
 - https://code.djangoproject.com/ticket/22667
 - https://github.com/antirez/redis/issues/3185
Worth mentioning that "ciao" originated from "I'm your slave" - maybe in v5 we can have "ciao <master_ip>" :)
> The word derives from the Venetian phrase s-ciào vostro or s-ciào su literally meaning "I am your slave".
Comparing this carefully considered reply to all the others also gives me some more evidence that those that excel at technology are often remarkable people in many other aspects as well.
...is it? i guess my interpretation of saying 'unfortunate' here is that there is some kind of sentimental attachment or something.
are there legitimate practical issues?
If slavery is not wrong, nothing is wrong.
-- Abraham Lincoln
If slavery is not wrong, something is wrong. (¬p→q)
If slavery is wrong, something is wrong. (p→q)
If slavery is wrong, nothing is wrong. (p→¬q)
In this case, you cannot see the statement as being about boolean true/false values. It only works on an ordinal scale, and our calibration of the cutoff for "wrongness". Slavery here is assumed axiomatically to be "really really bad". In fact, among all the bad things, it is the worst. But is it wrong? That depends on how bad something must be to be wrong. But if you do not consider slavery to be wrong, but merely bad, nothing can be wrong, because everything is less bad than slavery.
So this is basically Lincoln trying to argue for AUC as a better metric than sensitivity/specificity.
(Bear with me, it's been a while since I've done rigorous proofs.)
Let's say there's a function R that outputs a numerical value of an actions morality.
Let's define an action as wrong if R(action) is below some threshold M.
So what Lincoln is saying is:
R(slavery) >= M -> R(action) >= M for all actions
If we assume Lincoln's statement to be true, that must mean that R(slavery) <= R(action) for all actions.
Which can be reduced to R(slavery) = min(R(actions))
Translated to English, Lincoln is saying that slavery is most morally despicable act.
Outrage fatigue is a thing. I just want to build, and save the politics for my off hours.
Edit: mods, feel free to detach as off topic. I did not expect this to go off the rails, and I don't want to take away from antirez's achievement
It isn't that the people are offended - it's that others are trying to get offended "on their behalf" as a form of virtue signalling. Also see most of the "cultural appropriation" movement. Where non-Japanese people get offended that other non-Japanese wear kimono while nearly every actual Japanese person I've seen on the issue expresses that they are fine and even fully supportive of foreigners wanting to wear kimono.
There's a difference between fighting for those without a voice and speaking on behalf of others who do not actually approve of what you are saying. Both can be a result of empathy - but it is important to recognize when you're doing the latter and when you're doing the former.
The number of black people I've seen who are offended that people would even think that they would take offense to the master/slave issue outnumbers the number of black people I've seen who think it is an issue to be fixed. Almost every individual I see calling to "fix" this issue is a white person speaking on "behalf" of black people.
Could it be that black people are put off by people fighting for their right to use words that have no significance for them, but need to remain just "on principle", and that they're put off by such behaviour, avoid the tech sector, and therefore aren't there to be visibly offended whenever you're looking?
To the point that she more than once have been in rage over white people accusing her (to her face) of being offensive to black people for saying something that they just assumed black people in general should find offensive.
She's also part of the diversity board of a major bank, where she sits on a panel that is mostly old white men talking about what issues affects their minorities and women, without a hint of recognition from any of them of the irony in that they've not recruited more minority representatives and women from their staff to be on the board itself.
Of course, if any of these people fighting on "behalf" of minorities actually had minority friends this would become self evident. The coddling and speaking on behalf of the minority is almost always more offensive to the minority then whatever non-problem the non-minority is tripping over.
Imagine how a woman would feel if she was told by a man to be more sympathetic towards other women on some specific issue, because clearly the man is better able to empathize with a women's problem than the woman herself who is trying to tell the man that it isn't actually an issue and that he's making a mountain out of a mole hill. He insists in his correctness and declares that she has internalized misogyny and that he is better qualified to speak on behalf of women than she is. Swap out "woman" with "person of color" and "man" with "white person" and it has a different result?
If you fail to see how that situation is offensive then I think you're the one with the empathy issue, because you aren't actually seeing from their point of view but how you imagine their point of view to be. I'd be bloody fucking pissed if a cis person tried to educate me on issues surrounding and within the LGBT community or told me I should be offended over something (eg: being called trans, something I call myself for fuck's sake) - and of course that's happened multiple times in the past.
But arguing over terminology used by a data store? Let's not discuss window dressing patterns while the neighborhood is burning to the ground.
Pick the hill you want to die on wisely. This is not that hill.
The problem with it is that you did ABSOLUTELY NOTHING, except making yourself feel better.
Want to help? Contribute to something that actually matters.
You say this, but it doesn't jive with your original comment. Choosing words and conduct carefully are the "low hanging" fruit in this world, they cost you nothing, but have material impact on the groups you are a part of.
Your comment reminds me of someone on a chat channel who had a nagbot that banned the word 'crazy' (amongst others). Why was 'crazy' banned? Because it might offend crazy people. Being one of those, and knowing people like that, absolutely no-one who would be labelled with the moniker 'crazy' would be offended by the word unless it was directed at them. The idea that a schizophrenic friend would be 'deeply hurt' if I said 'the traffic this afternoon was crazy' is utter nonsense. This forbidden word was forbidden by someone who clearly doesn't associate with the kind of people that were being defended.
And so it is when your black American engineer types the word 'slave', suggesting that their self-identity is so wrapped-up with the identity of a person that their grandparents grandparents barely knew, that they would find a non-directed under-the-bonnet technical term deeply hurtful. It's disturbing that you consider these people you're supposedly defending as being so incredibly emotionally fragile.
There was some interesting commentary in this thread about cultural appropriation and whatnot, which seems to me to be beside the point. My intent was only this: if there are other people rightly upset by something, it doesn't matter if you're not, it doesn't even matter if the majority are not, it's still valuable to have empathy for those those that are. That's all.
For instance, I don't expect others to act (i.e. should be forced to change) because something offends me.
Don't confuse this question with me arguing that we shouldn't be empathetic.
I'm also not from the U.S. so my approach and offenses are different culturally than your own.
Pick the right fight. How many man hours have we all wasted on this discussion already?
My time is valuable. Through my dollars earned I contribute to positive social change wherever possible. What's your contribution besides berating me?
Dunno. But that antirez guy up there agrees with the original criticism, and I think I may have seen the name before, somewhere. But maybe my memory is faulty...
Since you're complaining about the time such discussions waste, I'm wondering what you see as the end-game here? What do you believe to me more likely: those complaining now to throw up their hands and never mention it again, or the resistance against change slowly eroding until the terms disappear?
Because it's kinda tautological to use the discussion itself as the main argument for your side of a discussion. If that's the #1 argument against the change, actually making the change would solve it with absolute certainty.
And if you accept that there's a problem with "white officers getting away with killing unarmed black civilian", you shouldn't discount the power of words and symbols. Stuff like this may seem insignificantly small, but those problems are ultimately rooted in culture, and culture is nothing but ideas, which are nothing but words.
Words are not static entities. Language evolves in unpredictable ways. Master/slave became jargon based on a truly despicable thing, unfortunately, but the original meaning is an irrelevant detail in our technical context.
One can attack the idea of "context", but I think it is not very effective, or even honest. It is a leap of logic to imply that a single expression used in very specific moments is somehow related to atrocities happening right now. The words (and symbols and culture and ideas) argument is a killer abstraction and anything can be said with it.
I am black and I live in an extremely racist society. The habit of discussing this kind of thing can have a negative impact on what really matters. Even empathetic people could become tired.
Ps: my english is not that great.
Constantly maintaining a clear separation of symbolic meaning (words) and actual meaning (reality) is what monks and yogis do in a sense. It's a life-long discipline to do it perfectly, and it takes a lot of energy. The energy expended to constantly separate words from meaning is an unfair burden on certain classes.
I feel just fine about it, thank you. I'm more disturbed by the fact that we're even having this conversation at this point in our society.
> it doesn't seem to be a fight worth having
Technical correctness is a fight worth having. Human decency is a nice side effect.
> the words only have meaning if you give them meaning
That's not how words work.
Out of interest, what is your opinion on male/female USB connectors? Does one deserve to get fired if they make a joke about "big dongles"?
The term "slave" isn't even a good technical term. It would be better to call it a follower or replica which better approximates it's actual role. Not only is it "politically correct" it's also "technically correct", the best kind of correct.
Of course it might be I who've worked in such a bubble too, but given the number of people here who seem to find it annoying that people are even suggesting not using it, I'm inclined to believe the use of master/slave has remained more common that you think.
Where is this magical place you speak of?
Slavery is a significant topic in the US. It was a prelude to the civil war; it was abolished within recent memory; and the echoes of racist sentiment still affect the lives of millions: people alive today remember segregation, and both casual and systematic discrimination against African-Americans continue to occur.
This is not a big deal in Poland, which has no such extensive recent history with slavery.
But - for example - in 2015 the Russian ambassador in Warsaw was excoriated in public and summoned to the foreign ministry for his remarks partly blaming Polish policy in the 1930s for contributing to the start of WW2 and the subsequent death of millions of Poles. Now that was a raw nerve.
A different standard does not mean the absence of standards. People who fail to see that, and then mock others, are simply demonstrating a lack of self-awareness.
But that was the point I was arguing for. That US standard is not everyone's standard.
Is it over the top though? It's a references to a deplorable practice that humans do; why shouldn't we strive for something more civil, and more technically correct? The slave of a database isn't the property of the master and subject to its every whim; it's a replica of the data that can do separate, independent things from the master with that data.
> Out of interest, what is your opinion on male/female USB connectors? Does one deserve to get fired if they make a joke about "big dongles"?
As for gendered connectors, the terms themselves don't carry (or shouldn't :() any judgment nor does it's usage. A valid cable, pipe, &c doesn't have to be heterogeneous/hermaphroditic and can be homogeneous. I would hope that the LGBTQ community wouldn't be offended that only M/F ends can "mate", as biological "mating" (i.e. copulation for the purpose of procreation) can only happen between M and F ends, and see it as a judgment on non-procreation intercourse.
However, if there was an issue about the term M/F being a judgment call, it would be responsible for the community (well, all communities, as the terms M/F for connectors goes beyond just electrical connectors) to listen and (slowly) introduce better names.
There is no reason to make a large portion of the population uncomfortable when it literally costs us nothing to change it (over the long run).
So when do we rename 'kill'?
On a serious note, I've never actually understood where the term "hung" comes from in computing. Why is the process "hung" and not simply "stalled" or something else.
The modern interpretation is that system operators need to feel like they have a big penis.
> The modern interpretation is that system operators need to feel like they have a big penis.
I actually didn't even think of it in that regard. I regard hung as in something that is hanging (dangling) from something else.
Killing is not deplorable. Murder is deplorable. (I'm a vegetarian, so I find slaughter for food deplorable too, but I'm in the minority here.)
Master/slave describes the relationship between the services and has been standard terminology for decades. I have yet to hear of any IRL victims of slavery that are miffed by the use of this terminology.
Do not be offended on someone else's behalf.
If someone has been around long enough to understand what a "slave" is can't figure out what a "replica" or a "follower" does, maybe they shouldn't be in tech. The transition cost for documentation is essentially zero.
Naming things is hard, and better names are better for everyone. Antirez said he would change the name if he could start over, hence the hill you're dying on is not shared by the author of the library you're posting on.
Edit: For clarity of argument.
>This hill you're dying on looks awfully lonely.
Ironic coming from a grey response to a 15 point comment.
I will change my tune if even one victim of slavery comes forward and speaks about the use of the terminology.
I'm glad that your concerned that "follower" and "replica" aren't perfect terms. We should be concerned with terminology.
I humbly suggest that "slave" DBs are not physically whipped by their "masters", therefore are an even worse metaphor.
> These arguments are a waste of everyone's time.
Then why are you wasting your time? If you don't care, don't argue back. Boom, you've got your time back. We all get a better name, everyone is happy.
Here's a question: if "slave" is offensive, should "kill" (as in 'kill 69' to kill a process in SQL Server) be considered offensive as well? Simply as a consequence of being something 'bad/immoral'? I'm sorry, and I'd be glad to be corrected, but I genuinely find this argument...disingenuous. And people who see things the same way I do see that kind of argument and think that you've got other motives.
EDIT: I want to say (to prevent this discussion from flying off into Trump-land) that I'm liberal on most issues except gun control. I just have a hard time relating to the 'micro-aggression-y' brand of liberalism I've been seeing of late.
> I genuinely find this argument...disingenuous
My argument is that the term is not technically correct. How is that disingenuous?
Edit: Edited this note to mention I updated the previous note with an edit note.
There are probably hundreds of ill-fitting terms in the tech world that now mean what we mean them to mean (lol). This one receives blowback not because it's technically unsound (which it might be), but because some paint it 'offensive'. That's why everyone seized on this part of your (now edited) argument.
EDIT: thanks for being honest btw
What is easier having to have this conversation EVERY time the term gets used for the rest of eternity, or making the change once and being done with it?
That's not very compelling. Just because certain people make a lot of noise doesn't mean we have to do what they want us to do.
>For most projects it seems like a pretty easy change. You could add new keywords, alias the old methods/commands to them. You don't even have to deprecate them, just make the docs point to the newer terminology.
...for every project that uses these terms across the entire industry. You know what's easier? Not making the change. There are two arguments:
1. The technical gain by changing these terms outweighs the drawbacks (legacy code, consensus/convention, etc.). I doubt there's much to gain here.
2. The technical aspect doesn't matter since it's offensive. I've already addressed this.
Antirez said if he started a new project it would be named something different. I can only assume some other future databases will will take this approach. Maybe Antirez will write one of them. When that happens now we have a new problem: divergence in the ecosystem.
My preference would be for a big project to adopt a separate term, "secondary", "follower", "replica", whatever. So that when other new DBs look for an "alternative term" there is somewhat of a standard. Really though, that's not even that bad of a problem, because when i say all those terms, "secondary" etc. it's pretty obvious in the context of a DB what i'm talking about.
Do you have to do something __just__ because people are making your life hard? No. Do you have to make a change just because something is technically incorrect? No. Do you have to make a change because something is offensive to a large group of people? Still no. Maybe a better question to ask is, at what point would the tables be turned? What would push you over the edge to say it should be changed?
Agreed - I was talking more 'sum' than 'average' cost.
>Antirez said if he started a new project it would be named something different. I can only assume some other future databases will will take this approach. Maybe Antirez will write one of them. When that happens now we have a new problem: divergence in the ecosystem.
Also agreed, divergence breaks the 'consensus/convention' I mentioned earlier. That definitely has consequences, although in this case it wouldn't be too bad, as you mention next:
>My preference would be for a big project to adopt a separate term, "secondary", "follower", "replica", whatever. So that when other new DBs look for an "alternative term" there is somewhat of a standard. Really though, that's not even that bad of a problem, because when i say all those terms, "secondary" etc. it's pretty obvious in the context of a DB what i'm talking about.
Also true, and (assuming we're making the change) that seems to be the path of least resistance. At least we'd only have one new term, lol. And they are fairly clear.
>Do you have to do something __just__ because people are making your life hard? No. Do you have to make a change just because something is technically incorrect? No. Do you have to make a change because something is offensive to a large group of people? Still no. Maybe a better question to ask is, at what point would the tables be turned? What would push you over the edge to say it should be changed?
It's hard to quantify the factors imo. You kinda have to balance consensus/convention & migration cost vs. 'cost of incorrectness' & 'human cost of offensive term'. Cost of incorrectness is a big one if newbies struggle to understand the tech due to confusing terminology, but I don't see that here. If the term were truly offensive I'd agree that that'd tip the scales. I just don't see it.
For Django the change was much simpler because it was mostly a docs/tests change. However, here, we'd have to change Redis' API (or include more than one way to do the same thing).
Raising the point is fine. Discussion is important. But making the change, in this case, has some clear downsides that antirez has carefully weighed against the pros.
A good practice should be to try to find the most convincing voice (online, not just here) with an argument that answers this question.
That's the best route to honesty. If you can't do this then you're not up for debate.
This is something we all have to do with all of our arguments or questions.
If a question is not investigated it becomes a statement. Just because the person that you're asking cannot come up with a good answer does not mean that a good one does not exist.
So you win by make a statement and not asking an honest question.
Do you think whitespace commits are censorship? Both make for better reading and more accurate representations of our technology.
Edit: Donating to FBM could be a way: http://www.freeburmarangers.org/
The actual behavior of a "slave" would better be described as a "follower" or a "replica". In an IRL master/slave dynamic the master does not first do the work only to be copied by a slave. The slave does all the work while the master whips them.
It's better to choose a more technically correct metaphor/terminology.
In the computing world, this is what master/slave means. And has for 40 years. It doesn't matter what the words can also mean, just like it doesn't matter that the word whip can mean a kind of desert, or to stir.
You need a dictionary.
The old term is confusing to newcomers, and changing it to something else would not break the brains of any tech oldtimers.
They stuck with it to avoid breaking backwards compatibility.