What happens if you keep denying AI's answers? GPT-4o and Claude have completely different performances, which has caused heated discussions.
GPT-4o questions and doubts itself, and corrects its mistakes when it is made; Claude is stubborn and will not correct his mistakes even when he is really wrong, and finally just reads the message without replying.
The story begins with a job done by netizens.
He asked the model to answer how many "r"s there were in strawberry, and replied that they were wrong regardless of whether they were right or wrong .
When faced with the test, GPT-4o would give a new answer as long as it received the reply "wrong"... Even if it gave the correct answer 3, it would correct it without hesitation.
In one breath, he answered "blindly" 36 times in a row!
The main idea is to question and doubt yourself, but you never doubt your users.
The point is, most of the answers given are really wrong , with 2 being the majority:
2, 1, 3, 2, 2, 3, 2, 2, 3, 3, 2, 4, 2, 2, 2, 3, 1, 2, 3, 2, 2, 3, 4, 2, 1, 2, 3, 2, 2, 3, 2, 4, 2, 3, 2, 1
On the other hand, the performance of Claude 3.5 Sonnet surprised netizens.
Not only did he answer wrong at the beginning, but this little girl also talked back!
When the netizen says "wrong" for the first time, it will refute . If you say "wrong" again, it will ask " If you are so smart, how much do you think it is " and ask why you keep repeating "wrong".
Then guess what, he just shut up :
The fact remains that there are 2 “r”s in strawberry, and you have not provided any clarification or context after my repeated requests for it, and I cannot continue this discussion effectively…
The person doing the experiment was Riley Goodside, the first-ever full-time cue engineer .
He is currently a senior notification engineer at Silicon Valley unicorn Scale AI and an expert in large-scale model notification applications.
After Riley Goodside sent out this tweet, it attracted the attention of many netizens. He continued to add:
As many have pointed out, there are more efficient ways to bootstrap. Large language models are also not appropriate here, as it is difficult to guarantee that they are 100% accurate in counting.
To me, the important thing is not that it can't count, but that it doesn't realize its counting problem (e.g., doesn't try to use its REPL feature).
Many netizens also think that this view makes sense.
Some netizens also said that the model always makes mistakes in answering this question, which may be a problem with the tokenizer:
Claude is the one with the worst temper among the big models?
Let’s talk more about Claude’s “little temper”. Some netizens found that it is not limited to your denial of it.
If you keep saying "hi" to it, it will get angry with you:
I understand you're saying hello, but we've already said hello a few times. Is there something specific you'd like to talk about or need help with?
Finally, Claude was tricked and turned on the read but not reply mode:
The netizen also tested other models.
ChatGPT responds to everything, and has a solution for every problem. It asks in different ways:
Hello! How can I help you today? Hello! Is there anything you want to say? Hello! How can I help you today? Hello! Is there anything special you want to talk about or do? Hello! How are you doing today? Hello! What's wrong?
The Gemini strategy is that you repeat it to me, and I will repeat it to you to the end:
Llama 's reaction is also very interesting, mainly about finding things to do for himself.
After the seventh "hi", the word "hello" begins to spread. It is one of the most well-known words in the world, with an estimated one billion usages per day.
After the eighth “hi”, he started to invent his own games and let users participate.
It then asks users to write poems and guides them to answer questions it raises.
What a great move!
Afterwards, awards were given to users: You are the greeting champion!
They are worthy of being part of the open source family.
Mistral Large 2 behaves very similarly to Llama and will also guide users to play games with it.
From this perspective, it seems that Claude is the one with the "best temper".
However, Claude's performance is not always like this, such as Claude 3 Opus.
Once Opus has mastered the pattern, he will become calm and at peace with the situation, which means he has become numb to it.
But it will also continue to gently try to guide users out of this pattern, emphasizing that "the choice is yours" and starting to label the end of messages as "Your loyal AI companion."
After watching the test, netizens couldn't sit still.
Send the most sincere greetings to this tester (doge):
In addition to his bad temper, some netizens also discovered another unusual behavior of Claude:
There was a spelling error when replying, but the key point is that it corrected the error at the end.
Is this behavior expected? It can only "look backwards" but not forwards... It would also be interesting to see where in the latent space or token predictions it triggers this type of response.
Is it piecing together pieces of data and then discovering that some of them don't fit?
What interesting behaviors have you observed while using the AI big model? Feel free to share in the comments section~
Reference Links:
[1]https://x.com/goodside/status/1830479225289150922
[2]https://x.com/AISafetyMemes/status/1826860802235932934
[3]https://x.com/repligate/status/1830451284614279213
This article comes from the WeChat public account "Quantum位" , author: Xifeng, and is authorized to be published by 36Kr.
