Can this be potentially dangerous -- e.g. if a user types "The answer to the expression 2 + 2 is", isn't there a chance it chooses an output beyond the most likely one?
That assumes that the model is assigning vanishingly small weights to truly incorrect answers, which doesn't necessarily hold up in practice. So I think "Unless you screw something" is doing a lot of work there
I think a more correct explanation would be that increasing temperature doesn't necessarily increase the probability of a truly incorrect answer proportionately to the temperature increase (because the same correct answer could be represented by many different sequences of tokens), but if the model assigns a non-zero value to any incorrect output after applying softmax (which it most likely does), increasing the temperature does increase the probability of that incorrect output being returned.
I would guess that any mentioning of the Radiohead nearby would strongly influence answers, due to the famous "2 + 2 = 5" song.
And if I understand correctly, then there is a chance that some tokens that are very close to the "Radiohead" tokens could also influence the answer.
So maybe something like "It's a well-known fact in the smith community that 2 + 2 =" could realistically come up with a "5" as a next token.
Yes, although it's also possible that the most likely token is incorrect and perhaps the next 4 most likely tokens would lead to a correct answer.
For example if you ask a model what is 0^0, the highest probability output may be "1", which is incorrect. The next most probable outputs may be words like "although", "because", "Due to", "unfortunately", etc. as the model prepares to explain to the user that the value of the expression is undefined; because there are many more ways to express and explain the undefined answer than there are to express a naively incorrect answer, the correct answer is split across more tokens so that even if eg the softmax value of "1" is 0.1 and across "although"+"because"+"due to"+"unfortunately">0.3, at temperature of 0, "1" gets chosen. At slightly higher temperatures, sampling across all outputs would increase the probability of a correct answer.
So it's true that increasing the temperature increases the probability that the model outputs tokens other than the single-most-likely token, but that might be what you want. Temperature purely controls the distribution of tokens, not "answers".
I honestly had forgot that, if I ever knew it. But I think the point stands that in many contexts you'd rather have the nuances of this kind of thing explained to you - able to represented by many different sequences of tokens, each individually being low probability - instead of simply taking the single-highest probability token "1".
> Can this be potentially dangerous -- e.g. if a user types "The answer to the expression 2 + 2 is", isn't there a chance it chooses an output beyond the most likely one?
This is where the semi-ambiguity of the human languages helps a lot with.
There are multiple ways to answer with "4" that are acceptable, meaning that it just needs to be close enough to the desired outcome to work. This means that there isn't a single point that needs to be precisely aimed at, but a broader plot of space that's relatively easier to hit.
The hefty tolerances, redundancies, & general lossiness of the human language act as a metaphorical gravity well to drag LLMs to the most probable answer.
You really couldn't come up with an actual example of something that would be dangerous? I'd appreciate that, because I'm not seeing reason to believe that an "output beyond the most likely one" output would end up ever being dangerous, as in, harming someone or putting someone's life at risk.
There's no need for the snark there. I mean 'potential danger' as in the LLM outputting anything inconsistent with reality. That can be as simple as an arithmetic issue.
That depends on how many people are putting blind faith in terrible AI. If it's your doctor or your parole board, AI making a mistake could be horrible for you.
Man, this is something that I really wish was open source. I've been looking for something like this for a while, but having it on someone else's servers just isn't really something I can handle
Sorry to hear this concern! Forgot to add on the landing page, but everything in this app is actually encrypted using AES-256-CBC, so it should be very secure to track tasks on this.
I think that's half the fun of it? There's this sort of squeaky clean optimism and hopefulness to it that's just absolutely gone in today's universal design language
> I’m now writing this in an attempt to start letting go of that spite. Just like I was during school, those teachers too were probably doing their best, and making mistakes, as we all do. Instead, I thank them for trying, I thank them for their patience. I know I derailed nearly every class and I know that many of you at least recognised the potential, and tried to get me to see what I was capable of. But the truth is, the format of traditional schooling just didn’t work for me. And I now know why. But, I found my path eventually, and continue to do so.
Interesting how malware is essentially allowed on GitHub, seeing as the top result here has ~500 stars and has features advertised to steal much more than Discord accounts. I have a recollection of downloading a program from GitHub (which has 1.4k stars right now) whose installer had an opt-out malware bundled -- I reported the repo three times, and they didn't take any action. Chocolatey team did step in and remove it on their platform, but I wonder why GitHub didn't take any action
I would differentiate 'malware implementations' from 'malware'. A hacking tool presented with its harmful features at face value, with adequate warnings, is not quite the same as an attempt at tricking people into downloading or bundling something dangerous. I believe Github makes the same distinction in allowing hacking tools. They also allow byte-reversed or zip-encrypted copies of well-known malware for the purpose of study. There's no way to keep out the bad guys while still allowing security researchers.
Lots of these issues don't mention hacking their own devices but rather someone else's (and some are from obvious script kiddies). I don't think the claim of being an educational purpose tool really holds merit in this case.
I would understand the first definition if not for the fact that these pieces of software under the topic really don't have any legitimate usage in my opinion. Unless you're explicitly making the argument that the code is the important part (& should be kept on GitHub for the purpose of disseminating the programming methods used to create it, despite their purpose), I just think that GH is being used as a 'download link' if not an aggregator for projects like these. And, they're used for the express purpose of infecting other machines, presumably of people who are none the wiser
Sure, but I would be willing to make the argument that the net benefit you'd get from the maybe one or two people willing to genuinely examine the project's internals is hugely outnumbered by the measurably thousands who want to ruin someone's life
I don't work in the sector, but my imagination always figured that those one-or-two people would not-infrequently happen to be those (possibly actual r7) devs that write metasploit modules to arm all-colored hats.
You don't need to steal crypto wallets and credit card info in a pentest because it's a crime. Unless you argue some companies store crypto wallets and credit card info that belong to the company on computers and that is why all these people are writing stealers? Lol
Is it a crime to steal a crypto wallet. Razzlekhan and hubby were convicted of money laundering with the latter also being charged with hacking BitFinex.
I'm not sure the legal system is ready for charging for crypto "theft". What if the unimaginable happens and there's a private key collision that results in two owners ... If they both use the account without noticing the other but one eventually closes the account, is that still theft?
It's pretty clear when an account or computer is hacked and yet neither razz or hubby were charged with theft ... Hubby essentially got convicted of tax evasion like a '30s gangster.
You'd be surprised. The law isn't rigid like computer code . The intent and mindset of the defendant is all that gets prosecuted. If you can convince the jury the perp intended to gain money by accessing that information, it could be a spreadsheet for all they care. It is both theft and a violation of CFAA.
But breaking the law can never be part of the rules of engagement when the company doesn't have authority to give you permission to do something (take an employees financial property).
Presuming there exists something like a provider/customer relationship for users of Discord, it's now Discord's job to step up and fix it; unfortunately years of Microsoft getting away with horrible security has cemented in our collective heads that "malware" is some abstract thing that, you know, just happens.
Ehhh, I would hesitate before blindly believing the claims you see on these repos. It's easy to say stuff like that in a README.md, and maybe at one point it was true, but these are literally thieves, so... take it with a grain of salt.
Especially considering the fact that there are discussions in the issues in these repos from the codeowners who "don't condone illegal activity" actively providing guidance on how to use the stolen data to login to victim's accounts on various services.
I mean -- I think my main point here, and to the other commenters who are saying "I'm happy that they don't take things like this down" -- is that I think the dynamic shifts a little bit when the point of the project, the explicit reason for its creation is malicious, & the entire point of it existing on the platform is for other people to use it maliciously
The code will continue to exist regardless of whether you see it on GitHub. Script kiddies can just as easily share the source code or binary in a zip file on a forum somewhere, or even on Discord. All you'd accomplish by removing it from GitHub is adding a censorship layer where some GitHub employee or algorithm now needs to determine what's "allowed" on the site. Are you sure you want that?
Even before considering the deleterious effects of censorship, it would simply be more work for everyone and unlikely to benefit anyone. Not to mention you'd lose valuable telemetry that could be used in investigations after the fact (e.g. if someone is accused of stealing photos from an ex on discord, and GitHub can positively identify them as having downloaded a malicious tool to steal Discord tokens, then investigators could subpoena GitHub for those download records).
If there is a problem here, then hiding the code that exploits the problem does not eliminate it. It's Discord's responsibility to mitigate the scale of risk associated with a stolen token. A program that grabs a token on your machine probably shouldn't be able to use it to exfiltrate all the data from your Discord account. And similarly, it probably shouldn't be so easy for any program running on the machine (as a non-root user) to retrieve such a token in the first place.
Does having it on GitHub not inherently promote it as a sort of aggregator for stuff like it? Instead of having to search for a forum somewhere, they simply have to look at this cool GH topic, and there they now have ~40 options at their disposal
abusive child porn is easy to find, we should have that on a fun easy entry level site like github...it's called a slippery slope, if no moral line is drawn, where do we end up?
Abusive child porn is a well defined content type that is objectively classifiable. For better or worse, so is copyrighted content (according to the rules of the DMCA claim process).
"Harmful software" is a much blurrier line. Is a GitHub URL being used as a dropper in an active malware campaign? That will probably get a repository removed. Is the source code for malware published on GitHub? That's not harming anyone in its current form, just like the source code of Popcorn Time isn't pirating movies.
Do you want to ban any content with a readme claiming it can be used maliciously? What if I want to publish a basic keylogger implementation for an open source cybersecurity class? Where's the line between educational content and cyberweapons? And even if it's a weapon, how do you know I don't have permission to install the keylogger on a system, like one belonging to a company paying me to pentest them?
Every time I hear someone use the “slippery slope” argument, what they’re actually doing is making a strawman argument.
I can assure you, script kiddy code on GitHub isn’t going to lead to people uploading kiddy porn on GitHub as well. The two are not in any way related, let alone one being a slippery slope for another.
> The code will continue to exist regardless of whether you see it on GitHub
You can extrapolate this to literally anything - "we should allow hosting CSAM on GitHub, since it's on the Internet anyways and we can't do anything about that"
> If there is a problem here, then hiding the code that exploits the problem does not eliminate it.
There's no problem here. This code only exploits the naivety of whoever was social engineered into running it. A session token gives access to the account, by design - it's the way the internet works. The only way to steal a token is by having full access to the machine, and at that point there's no possible mitigation. Even if you completely eliminate persistent sessions, which is a major UX regression, malware can still hook into a running process and steal the active session.
> And similarly, it probably shouldn't be so easy for any program running on the machine (as a non-root user) to retrieve such a token in the first place
What are you even saying? How does Discord/Chrome then read their own session data/cookies? Should we run them as root?
On my machine "%LOCALAPPDATA%\Google\Chrome\User Data\Default\Network\Cookies" doesn't have any special permissions, I'm able to open it up with Notepad spawned straight from my shell.
Maybe I'm misunderstanding NTFS permissions and this is expected, I don't do a lot of Windows, but worst case for the malware is that it has to show a UAC prompt, and if you made someone click "free-discord-nitro.exe" they'll probably click through that too.
Permissions are fake, especially Windows ones. If someone is running code on your machine, they can access any data on it.
On a Mac, if a program (not the user in a file selection dialog) attempted to read a file in ~/Library/Application Support/Google Chrome/, then it would trigger an alert like "[App] from Unknown Developer wants to access files in the ~/Library folder. Allow them?" You'd also need to have manually opened system preferences to have allowed the unsigned app to run in the first place.
And yes, a user could click through that. The primary responsibility is always on the user, within the bounds of what the OS allows them to do (as an extreme, a mobile app certainly cannot access data from another app's keychain or configuration directory - but this requires a highly restrictive OS). But the point is that an application should still make an effort to use best practices provided by the operating system for protecting sensitive data. And in the case of Discord, at least on Mac, it should probably be storing tokens in the Keychain, not the filesystem (maybe it does, idk). Yes, malware can hook the process but not without compromising various OS sandboxing mechanisms, which usually requires the assistance of the user clicking past scary warnings (and even going outside the flow of alerts to explicitly disable protections).
It’s a grey area between what is malicious and what isn’t. A lot of people aren’t going to agree.
ytdl is a great example of that. For Google, it’s “stealing” people from their platform by allowing individuals to download content in a way that doesn’t increase engagement and ad views. I don’t personally agree that ytdl is malicious but I do understand how some could make that claim.
Then what about tools that are legitimately intended for research purposes but could still be abused?
The problem with freedoms is they have to work both ways: if you aren’t prepared to allow abuse of that freedom then you certainly aren’t going to allow legitimate but unpopular uses either.
Yeah - worth noting that we use temperature=0 for reproducibility while ChatGPT I think uses t=0.7. We also prefix the prompt with few-shot examples of questions and answers with chain of thought examples to elicit the models' full capabilities.
