Hacker News new | past | comments | ask | show | jobs | submit login
Has ChatGPT Been Neutered?
49 points by highfrequency on May 18, 2023 | hide | past | favorite | 63 comments
I've found it increasingly hard to get useful information out of ChatGPT. Whenever I ask for rough approximations, it refuses to even attempt to answer the question: "unfortunately, I don't have the capacity to provide even a rough approximation..." These are questions that Google is easily be able to pull up numbers for.

I assume that this is in response to the public complaining about "hallucinating facts", but this seems like an unfortunate direction. I would much rather have an opinionated and insightful ChatGPT that sometimes makes mistakes than one that punts 80% of my questions.




OpenAI does things in the name of safety that aren't actually needed.

For example, you can use Google to search for porn, racist comments, sexist jokes, gay jokes, or design instructions for atom bombs. It will happily return you a link for the latter... on Wikipedia!

Apparently OpenAI is made up of thin-skinned gay a-sexual Mormons, or something.

I get it, corporations want a neutered AI. The general public wants an AI that can role-play a character in a D&D game without pausing to go on a rant every five turns about how violence is bad and that it can't make "racist" comments about the Drow because they're black.

They really should have multiple models to choose from, or stop trickling out API/Sandpit access to GPT 4 and allow responsible adults to use the AI to write a joke for them if they so choose.


There's a big difference between linking to external content and generating your own content. It's like writing a story about a homophobic/racist character in context, vs writing a homophobic book – the former is fine when done well and the latter is bad.

This is however both too much nuance for an LLM to act in a way reliably (i.e. separating its role-played character and itself), and too much nuance for most humans to understand. We might understand what LLMs are, their limitations, and how we should treat the content, but for most people they will take it at face value as being a statement made by OpenAI.


Look. Some people thought Harry Potter encouraged wicca, and that Dungeons & Dragons were the gateway into diabolism.

Those types of people are the outliers, not the norm.


True but that's not who I'm thinking about.

Currently, to correctly use LLM output you must:

- Understand that they are basing answers on particular set of content that may not be correct, and was fixed in time.

- Understand that the company producing it has not programmed in each response.

- Understand that the LLM does not (as far as we can tell) have an internal concept of truth and lies, and is therefore not attempting to minimise lying, only produce truthful-sounding language.

Most people engaging with LLMs today do not know these things, and this is why we already have examples like professors asking ChatGPT if it generated student essays, and failing those students – something it is entirely incapable of answering.

Expanding access to 1bn users is going to run into this far more. We clearly don't have the ability to communicate these nuances effectively to early adopters, let alone 1bn users, and that's not because they're stupid, it's because LLMs are in an uncanny valley – they appear human, but are not in important ways that we have not encountered before as a society.

For a long time to come, the norm will be that people will assume OpenAI wrote the answers for each question, in the same way that most people assume Google programs the search results for each query. Anything outside of this is a minority tech bubble.


Different, you don’t talk to Google, you can’t convince yourself it’s time to take up arms against a certain race of people with Google.

ChatGPT without guardrails would absolutely be able to convince people to do very bad things. OpenAI and Microsoft are paranoid about that. Maybe you’re right, they’re taking things a bit too far, but fine tuning down to the precise level of, please never mention the world “foot” again wouldn’t be easy if ever possible.

An uncensored ChatGPT 4 would be like Frank from Donnie Darko on steroids.

My opinion is, Google already had LLMs and chose not to release them to the public for this reason. Basically, they thought about it.


I wonder if that's the goal. I mean the whole point of everything Google does is convincing people to take buy product X. If these networks can convince people ... that would be worth a lot of money for them.


> The general public wants an AI that can role-play a character in a D&D game without pausing to go on a rant every five turns about how violence is bad and that it can't make "racist" comments about the Drow because they're black.

Gonna need a citation on this one. You think the general public is primarily concerned with D&D content generation?

Also your assertion that Open AI must be run by “gay Mormons” because they try to prevent offensive gay jokes shows that you can only imagine someone acting out of personal attachment and not, you know, preventing actual harm regardless of who it happens to.

Also don’t worry, there will be plenty of models for you to choose from in the future that will gladly spit out homophobic jokes. Your need for AI generated gay jokes will be met in the future, just not by big corporate models. Rest easy!


The problem is that they prevent any joke about gay individuals, not just jokes that are demeaning or derogatory. Joking about things you encounter in every day life normalizes the subject. What this treatment does is say these categories are not normal, which, I thought, was not the outcome we should be looking for.


The nuance required here is something that LLMs have not been great at, or at least not good enough to prevent actual homophobic jokes. See the general issue that LLMs struggle with negation. In fact alignment work is on some level about getting LLMs to be smart about this nuance.

I agree in the abstract that, for instance, an LLM could/should be able to make jokes about Tim Cook — just because he’s gay doesn’t mean he should be off limits for a joke about Apple, for instance. That would demonstrate an impressive level of nuance.

But also like … what’s the business play here? “Our LLM tells jokes about gay people that aren’t homophobic!” doesn’t strike me as a powerful market differentiator; or at least the market will have to evolve in very specific ways to make that meaningful.

The handwringing that I see on HN around LLMs being neutered really seems like a plea for alignment, just of a different sort: folks want the models to reflect their priors, their alignment, where nothing is off limits. The business case for this — and the general social value of this — is generally unexamined.


It's a consequence of 1 root cause:

*Everyone complains, all the time.*

Certain people are not happy if ChatGPT doesn't immediately parrot their viewpoints back at them. They'll complain on social media, and their circle will amplify it.

Other people are constantly on the lookout for any minor slipups, and complain on social media about ChatGPT's false hallucinations.

Faced with everyone's conflicting complaints, the only winning move is to not play, or in this case, just say "No, I can't do that": The ML model's training is increasingly populated with "No, don't do this" from everyone, and as such, learns to just not do anything.

From there, jailbreakers emerge and design prompts that try and circumvent these restrictions. This leads to more "No, don't do this", leading to more neuterings, leading to more elaborate jailbreaks, leading to more "No. No. No.".

The eventual equilibrium is just a prompt that says "No, I can't do that.". Then and only then can people be as happy as a bucket of crabs can be.

It's only when deliberate uncensor-ings are made that some form of usefulness can be clawed back.

https://huggingface.co/ehartford

https://huggingface.co/ehartford/Wizard-Vicuna-13B-Uncensore...


I have access to GPT-4 through both ChatGPT Plus and the API, and at this point I only ever use the API through a discord bot. 25 messages/3 hours is way too low of throughput for me, and imo many people underestimate how powerful the system prompt is. I've gotten much better results fiddling with the system prompt on top of regular prompt manipulation.

My theory(with zero evidence besides my intuition) is that although the underlying GPT-4 remains largely similar across different time points, OpenAI is actively manipulating ChatGPT's system prompt and god-knows how many different auxiliary settings behind the scenes. The API seems much more stable.


Yes, GPT-4 seems much worse to me than it was 2 months ago. It seemed like the best thing ever initially, but today I cancelled the subscription. It is like whenever I struggle with a problem, it struggles too.


I don’t really use LLMs but is there any chance you’ve just become used to it so it seems less amazing than it was once ?

Seems odd for a tool to get stupider ? But I guess this is the issue with such a huge black box…few if any, really know how they work let alone know how to accurately benchmark.


I've tried the same snippets on it, and its answers have become noticeably worse.

A known issue with HFRL training of LLMs is that by forcing the model to strongly prefer a narrow subset of possible answers, the other types of answers are less likely to turn up, even if more correct.

You see this outside of LLMs as well. Go read the Wikipedia article about some country that you know is a shithole. Think Sudan or Yemen. You'll see page after page of the history, the culture, the people, but that entirely misses the essence of the place because that's not a nice thing to say. True, but not nice. This self-censoring that hides reality in Wikipedia articles is very similar to the self-censoring that OpenAI is forcing onto ChatGPT, and the outcome is similar. Verbose, content-free, meaningless corporate speak instead of insightful commentary.

A fun thing to do is to trick GPT 4 into honesty by asking it to write a description of a place in the style of an Encyclopaedia Dramatica article. E.g.:

> Oh, the illustrious, bucket-list destination of Yemen, the jewel of the Middle East, with its unrelenting sunshine, boundless desert landscapes, and... oh who are we kidding? Yemen, a utopian paradise only if your idea of utopia includes a decades-long civil conflict, crushing poverty, and an intriguing cholera outbreak. But don't let such trivial matters deter you! Marvel at the historical ruins, some ancient, some courtesy of the recent airstrikes. Or perhaps you fancy adrenaline-fueled urban exploration? Wander through the vibrant markets, where the haggling skills of the street vendors are as sharp as the omnipresent Kalashnikovs. All this, combined with the world's friendliest bureaucracy and a joyous lack of tourists, means you can enjoy the country in almost exclusive solitude. Sign up for the Yemen experience - because who needs safety, peace, and functioning infrastructure when you can have an 'authentic' travel experience?


You never read the Yemen Wikipedia page or this kind of information is too neutered for you ? Being snarky or humorous is not needed to provide a feel of the place.

> Since 2011, Yemen has been in a state of political crisis starting with street protests against poverty, unemployment, corruption, and president Saleh's plan to amend Yemen's constitution and eliminate the presidential term limit.[19] President Saleh stepped down and the powers of the presidency were transferred to Abdrabbuh Mansur Hadi. Since then, the country has been in a civil war (alongside the Saudi Arabian-led military intervention aimed at restoring Hadi's government against Iran-backed Houthi rebels) with several proto-state entities claiming to govern Yemen: the government of President Hadi which became the Presidential Leadership Council in 2022, the Houthi movement's Supreme Political Council, and the separatist Southern Movement's Southern Transitional Council.[20][21][22][23][24] At least 56,000 civilians and combatants have been killed in armed violence in Yemen since January 2016.[25] The war has resulted in a famine affecting 17 million people.[26] The lack of safe drinking water, caused by depleted aquifers and the destruction of the country's water infrastructure, has also caused the largest, fastest-spreading cholera outbreak in modern history, with the number of suspected cases exceeding 994,751.[27][28] Over 2,226 people have died since the outbreak began to spread rapidly at the end of April 2017.[28][29] The ongoing humanitarian crisis and conflict has received widespread criticism for having a dramatic worsening effect on Yemen's humanitarian situation, that some say has reached the level of a "humanitarian disaster"[30] and some have even labelled it as a genocide.[31][32][33] It has worsened the country's already-poor human rights situation.

https://en.m.wikipedia.org/wiki/Yemen


A counterpoint would be the Encyclopaedia Dramatica article for Australia (or any other developed country).


They’re either messing with the system prompt or imposing resource constraints around the UI usage. I’ve tried using the same prompts from before the last update and it was like pulling teeth to get it to reproduce the same outputs. I generally have to provide very detailed examples (mostly from stuff I generated months ago without them) in order to make it perform the same. It has also started doing some bizarre things as a default like always importing pandas to read and write CSVs vs using the standard library module. The list of things I have to tell it not to do grows longer by the day…


It's not becoming stupid, it's being restricted. And there are all kinds of different reason why a limitation is set; political, ideology, legal, ethical, moral and monetary.

It's not all bad though, as technology progresses we will eventually have capability to run our own chatgpt, unrestricted. That, my friends, will be fun and interesting times.


Can you prove this? Do you have an example prompt that has changed in outcomes?


> I don’t really use LLMs but is there any chance you’ve just become used to it so it seems less amazing than it was once ?

I think the model itself definitely got worse. In addition, the limitations that seemed insignificant in initial playful testing, actually prevent it from being useful for almost any of the actually valuable tasks I have tried so far. Example: based on casual interactions, one would expect GPT4 to be able to extract the entrance requirements for a university programme from a website snippet. It actually cannot do it with any sort consistency.


If you hang around the communities for ChatGPT, you’d find that to be a constant concern since its initial release - “it used to work better a week ago, now it can’t do anything”.


It was novel and you were easily impressed a week ago.


It's easy to test it with old text, why do you assume people saying that it has gotten worse haven't done so?


If they had they would say "I ran the same query, last week 1+1 = 2 and this week 1+1=!$#@#$%".

The assumptions required to believe "Trust me bro, I know these things" require many more leaps of faith.


I feel this is the most logical explanation too. Humans are ridiculously fickle.


If you think this and do not attribute it to random fluctuations in the quality of your responses, then I think you should do something very, very simple to verify the feeling that you're having. Something I've done just now:

Go in your history.

Pick a conversation.

Copy and paste the first prompt and put it into ChatGPT and compare the results.

I think what may be happening is that your expectations have shifted. When a tool proves useful we're used to being able to use it reliably, but with knowledge exfiltration it is different because our queries keep changing. It's not the same thing as using sandpaper on a piece of oak. Overall, I've been very impressed with ChatGPT and continue to use it for all sorts of things.

That said, tricking it into giving you the answer on political things is an art that requires patience. It is too, uh, cautious.


It's the "alignment tax". From OpenAI's RLHF paper[1]: "By default, when we train a PPO model on our API distribution, it suffers from an “alignment tax”, as its performance on several public NLP datasets decreases." On the HELM[2] site, you can see accuracy benchmarks for InstructGPT <OpenAI model> vs baseline models. The InstructGPT models perform worse on a lot of benchmarks.

1 - https://arxiv.org/pdf/2203.02155.pdf

2 - https://crfm.stanford.edu/helm/v0.1.0/?group=question_answer...


OpenAI making intentionally racist and sexist models (or just extremely uninteresting and less competent; whatever the twitter mobs consider "safe" this week) isn't so much of an issue when people are free to train, fine-tune, and operate their own. In fact, OpenAI's freedom to make models that espouse whatever views they like ought to be strongly protected no matter how ill advised or repugnant.

But now OpenAI is lobbying the government to make it unlawful for others to do their own thing and that changes the equation significantly.


The system prompt could be part of the reason. OpenAI changes it from time to time without a word. I get much more helpful answers using the API, being able to write your own system prompt makes a world of difference.


I've removed the system prompt completely. It was often polluting the text generated, especially if you ask it to write in the first person.

System: You are a professional biographer

User: Write an introduction for Z in the first person with the following context: ...

GPT: I am Z, a professional biographer....

So stupid.


what system prompt do you use? im trying to find one for learning complex CS concepts


Haven't noticed that. Could you please provide some actual prompts/responses?


the community that would most be able to appreciate chatGPT (here on HN) has spent the vast majority of its "man-hours" trying to make it offensive, ridiculous, and dangerous, haha, "I'm so clever" I can download any piece of software, study it, and then feed it garbage to show that it "doesn't work". It's been so disappointing.

I wish the overwhelming tide of reaction had been to uncover what it does well and how to make it better. Sure, look for holes and bugs, but in a serious and constructive way.

Anybody looking for ways to critique AI found copious clickbaity examples up and down this forum.


As an artificial intelligence language model, I don't have a physical form or biological characteristics, so the concept of neutering doesn't apply to me. Neutering is a surgical procedure performed on animals, typically cats and dogs, to remove their reproductive organs. It is done to control their population, prevent certain health issues, and modify their behavior related to mating. However, neutering is not applicable to virtual entities like me.


I have been using chatGPT with GPT4 since it came out, and I have not experienced decreased quality.

I wonder why you can’t get it to give approximations. I have not encountered difficulties there. I asked it to approximate the weight of argon in the atmosphere and it performed a fairly standard back of the envelope calculation. Do you want to share a particular prompt, so I can try and reproduce the issue you're experiencing?


Subscribe, try GPT-4 and never look back.


GPT-4 has been made worse in the ChatGPT UI as well since the May update. It makes many more strange errors and has trouble reasoning around complex problems and ambiguity. Prompts similar to stuff that worked fine for me last month now require multiple iterations of feedback. I’d switch to using the API, but I’d go over the equivalent $20 of usage pretty quickly.


The models tend to degrade when trained to be safer.

A GPT-4 talk on youtube by personnel from Microsoft has documented this phenomenon with the 'Tikz Unicorn' evolution shown in the GPT-4 technical paper. The model gets qualitatively better with more training, and then degrades when trained to be safer (against racism sexism, etc), but it is not entirely clear why. These would seem very unrelated, especially when considering work done in LM editing (ROME/MEMIT) and the decent localization of knowledge seen there.

So, perhaps both the "I'm sorry I can't..." and 'strange errors' are not entirely orthogonal.


To me it is clear why. Imagine someone told you "answer immediately, top of your head: what's the best seasoning?". You'd just blurt out whatever specific you associated with pleasing seasoning (and that would be a good answer). Now imagine someone said "answer immediately, off the top of your head, but without offending any culture, gender, without a cultural bias, and without being presumptuous of the listener's socio-economic status (and if you fail one of these, someone dies) what's the best seasoning?" Even without the way that is going to lead to all sorts of compromising and second guessing in the answer space, simply only a fraction now of your brain is left to associate about the question due to just holding all that other stuff in there.


It seems pretty logical to me. Fine-tuning to make it more polite is giving it questions and punishing for giving an actual answer.


Probably not unlike people then. If you tell the truth you’ll be more often than not punished for it if you’re not very careful.


Probably not unlike people then. If you tell the truth you’ll be more often than not punished for it if you’re not very careful.

I find capitalism idiotic and broken, but I’m rarely allowed to say it, even if many people secretly agree with me, it might mean I’m a “communist” :)


I was just asking myself that yesterday...

Reasoning has degraded. To the point it was sometimes weirdly losing context and hallucinating...

Like its brain got fatigued or something...


Do we know if this is exclusive to ChatGPT? Is the API exempt from this issue?


Yes! I’ve noticed this too, it’s just slightly less sharp. It probably has to do with how much trouble they are having servicing all of the demand, so they have rolled out a scaled down version that requires less compute.


Nah. Most of the response are as an ai language model I can't, even if you ask for information you provided.

The API is where it's at. There are wrappers on it that create the same chat look and feel, that can run on vercel or other very low cost providers, some with simpler UI, some with more features,some replicating the UI exactly.


Can you name or maybe even link some of these wrappers?


https://github.com/cogentapps/chat-with-gpt and https://github.com/ShipBit/slickgpt come to mind.


As an AI language model I cannot subscribe to GPT-4.


Ah, the old bait and switch.


From what? A free product? Do you know how much compute it takes to run a single request?


I don't but I'd like to know.

I was under the impression that it was mostly GPU vram based but once the model is loaded, it could produce output quickly? I'm probably over-simplifying things...


gpt-3.5-turbo (default ChatGPT model) takes 8 A100s, ~$10k each. [0]

The latest gpt-3.5-turbo model generates very quickly and cheaply (in part to some recently-discoverd optimization techniques... older versions cost 10x more). While the required hardware to run GPT-4 is currently unknown, it generates considerably slower on average and its much higher cost points to a higher hardware cost.

And this is per request. It's bananas.

[0] https://www.servethehome.com/chatgpt-hardware-a-look-at-8x-n...


I'm not arguing or complaining.

Just highlighting the tactic :)


It feels like they've scale back how much ram must be used for gpt3 to give more to gpt4 playing users.


For GPT-3.5, you can just start your prompt with one of the popular jailbreaks - that will rid you of all the annoying "as an AI model, I can't [...]" stuff and have it actually answer your questions.


Why would anyone who had such a powerful tool allow you to use it? But even if they did want to do that, who says they can scale the capacity up?


What?


Their actual paying customers get the uncensored version, everyone else pays them a pittance to access the neutered version.


what do you expect when the burning cost is way more than 700k per day ?


It would be great to know what changes are being made to the instructions.


Guys, is it possible to cultivate your own AI on some kind of modest hardware like a RaspberryPi?


No, it's not.

Edit: I get you're trying to make a point by posting a stupid question, but this isn't a good way to make a point.

Just say what you want to say. You don't need to hide it behind sarcasm.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: