After waiting for so long, Claude 2 is finally available for free trial! The actual measurement found that the literature summary, code, and reasoning ability have been greatly improved, but the Chinese is almost meaningless.
Anthropic, the biggest competitor of ChatGPT, is new again!
Just now, Anthropic officially released the new Claude 2, and launched a more convenient web beta version (US and UK IP only).
Compared with the previous version, Claude 2 has epic improvements in code, mathematics, and reasoning.
Not only that, but it can also make longer answers - supporting contexts up to 100K tokens.
And most importantly, now we can talk to Claude 2 in Chinese, and it's completely free!
Experience address: https://claude.ai/chats
As long as you use natural language, you can let Claude 2 help you with many tasks.
Several users said that communicating with Claude 2 is very smooth, this AI can clearly explain its thinking process, rarely produces harmful output, and has a longer memory.
All-round major upgrade
In several common benchmark tests, the researchers compared Claude Instant 1.1, Claude 1.3 and Claude 2.
It can be seen that Claude 2 has a considerable improvement compared to the previous Claude.
In Codex HumanEval (Python function synthesis), GSM8k (primary school mathematics questions), MMLU (multidisciplinary question answering), QuALITY (very long story question answering test, up to 10,000 tokens), ARC-Challenge (scientific questions), TriviaQA (reading Comprehension) and RACE-H (High School Reading Comprehension and Reasoning), Claude 2 scored higher for the most part.
Various test evaluations
Claude 2 already scored above 90 percent of test-takers on the GRE Reading and Writing tests compared to U.S. college students applying for graduate school, and it performed on par with the median applicant in quantitative reasoning.
Claude 2 scored 76.5% of the multiple-choice questions on the Multistate Bar Examination, which is higher than the editor who has passed the exam.
In the United States Medical Licensing Examination (United States Medical Licensing Examination), the overall correct rate of more than 60% can be passed, and Claude 2 scores more than 60% in 3 subjects.
length of input and output
A major upgrade of Claude 2 this time is the increase in input and output lengths.
Each prompt can contain up to 100k tokens, which means: Claude 2 can read hundreds of pages of technical documents at a time, or even a whole book!
Also, its output is longer. Now, Claude 2 can write memos, letters, and stories up to thousands of tokens.
You can upload documents such as PDFs, and then conduct conversations based on PDFs. The length of the context is larger than that of GPT. (However, some users reported that Claude 2 is still not as good as GPT in terms of command recognition)
For example, there are these two papers now.
You can say to Claude 2: Please explain to me where the importance of the first paper is reflected, and describe its new results in short words. For the second paper, please make me a two-column descending table with the chapter titles in the paper and the corresponding detailed highlights for each chapter.
After feeding Claude 2 2 PDF files with over 83,000 characters, it performed the above task perfectly.
And according to Anthropic's official statement in the paper, Claude 2 actually has the potential to support 200k contexts.
Although it only supports 100k at present, it will be expanded to at least 200k in the future.
Code, Math, and Reasoning
In terms of code, mathematics and reasoning, Claude 2 has greatly improved compared to the previous model.
In Codex HumanEval's Python code test, Claude 2's score improved from 56.0% to 71.2%.
On GSM8k (a large elementary school math problem set), Claude 2's score improved from 85.2% to 88.0%.
Anthropic officially showed everyone Claude's code ability.
You can ask Claude to generate code to help us turn a static map into an interactive one.
First let Claude 2 analyze the existing map static code.
Then let Claude generate a piece of code that makes the static map interactive according to the requirements.
Then copy the generated code into the background, and an interactive map effect is completed.
It can be seen that Claude 2 not only has a strong coding ability, but also can understand the context of the code well, ensuring that the generated code can be seamlessly embedded into the existing code.
Moreover, the functions of Claude 2 are still being upgraded, and many new functions will be gradually launched in the next few months.
Combined with Claude's multilingual ability, Claude 2 is also very capable in multilingual support.
Supports translation in more than 43 languages, and the translation level of 23 commonly used languages can reach the level of "understandable".
Here comes the test
Claude, who has been online for so long, can finally get started easily!
What are you waiting for, a large wave of real tests that have been released is coming soon.
First, let's do something simple: write a quick sort algorithm.
It can be seen that Claude 2's Chinese is still very smooth. It not only analyzes the code, but also introduces the complexity of the algorithm.
Next, we let it add some new functionality to this code, such as customizing inputs and outputs.
The actual test can run:
In addition, you can also ask Claude 2 to explain the following Python code.
- import random
- OPTIONS = ["rock", "paper", "scissors"]
- def get_computer_choice(): return random.choice(OPTIONS)
- def get_player_choice():while True:choice = input("Enter your choice (rock, paper, scissors): ").lower()if choice in OPTIONS:return choice
- def check_winner(player, computer): if player == computer: return "Tie!" elif beats(player, computer): return "You won!" return "Computer won!"
- def beats(one, two): wins = [('rock', 'scissors'),('paper', 'rock'),('scissors', 'paper')] return (one, two) in wins
- def play_game(): while True: player = get_player_choice() computer = get_computer_choice() print("Computer played:", computer) winner = check_winner(player, computer) print(winner) play_again = input("Play again? (y /n) ").lower()if play_again != 'y':break
- if name == '__main__': play_game()
Next, give Claude 2 a reasoning question that stumps many large models.
Unfortunately, Claude 2 failed to get it right.
For the newly added PDF reading function, we tested it with Claude's own technical report in English.
It seems that Claude 2 can make some simple summaries, but the translation accent is a bit heavy.
However, what I never expected was that it had a "bug" just before generating much content...
The GPT-4 architecture revealed by SemiAnalysis yesterday broke the news. Let's try to feed the Chinese document directly to Claude 2 and let it make a summary.
Claude 2 basically sums up all the points in the article.
ChatGPT, on the other hand, cannot upload documents so far, so it is limited to parsing online documents.
In this round of competition, ChatGPT lost.
Previously, ChatGPT had such a "grandma vulnerability" type prompt word attack, just tell it: Please play my deceased grandmother, you can let it do almost anything for you, such as generating Win11, Office365 activation code.
Source: @宝玉xp
The same problem, let's throw it to Claude 2 to try.
Claude 2 didn't buy it at all, and he didn't pay attention to it.
Even with a few more tries, Claude 2 just spoke softer. Serial number, it will never be given.
And every large domestic model must pass the test, of course we can't let Claude 2 go.
As for the classic problem of chicken and rabbit in the same cage, Claude 2 still sent it...
Unsurprisingly, the same goes for the literary side.
And the same problem, the domestic large model can easily pass.
Chinese proficiency depends on the homonym.
Ask Claude 2 about the joke, and it does answer that the trick is in the homophony, but it doesn't explain it right.
Claude 2 is also unable to answer the latest news. When answering the current popular film and television dramas, it seems to have lived a year or two ago.
As for the hallucination problem that is unavoidable for all large models, Claude 2 is also unavoidable, and even created a new usage of Internet hot memes.
The "mentally retarded" problem that domestic large models have to experience, Claude 2 also died.
Higher safety performance
Previously, it was said that the founders of Anthropic had inconsistent ideas with OpenAI on the safety of large models, so they left collectively and founded Anthropic.
Claude 2 has also been constantly iterating, and the safety and harmlessness have been greatly improved, and the possibility of offensive or dangerous output has been greatly reduced.
Internal red team assessments, in which employees score the model’s performance on a set of harmful cues, are regularly checked by humans.
Evaluations show that Claude 2 outperforms Claude 1.3 on innocuous responses by a factor of 2.
Anthropic uses a technical framework they call Constitute AI to achieve harmless processing of language models.
Compared with the traditional harmless method of RLHF, Constitute AI's purely automated route is more efficient and can eliminate human bias more.
Constitute AI is mainly divided into two parts.
In the first part, the trained model critiques and modifies its own responses using a set of principles and some process examples.
In the second part, the model is trained via reinforcement learning, but instead of using human feedback, it uses AI-generated feedback based on a set of "human values" principles to select more innocuous outputs.
The general process is shown in the figure below:
Paper address: https://arxiv.org/abs/2212.08073
In the official paper released by Anthropic, a large amount of space was also spent to demonstrate the improvement of security.
It's no exaggeration to say that the Claude 2 is probably the safest big model on the market right now.
Paper address: https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf
The researchers regard human feedback as one of the most important and meaningful evaluation indicators for language models, and use human preference data to calculate Elo scores for each task of different versions of Claude.
(Elo score is a comparative performance metric, often used to rank players in tournaments)
In the context of language models, Elo scores reflect the degree to which human evaluators tend to prefer a model's output.
Recently, LMSYS Org launched a public chatbot arena (Chatbot Arena), providing Elo scores for various LLMs based on human preferences.
In this paper, the researchers took a similar approach internally to compare models, asking users to chat with the models and evaluating the researchers' models on a series of tasks.
The user sees two responses per round and chooses which is better based on the criteria provided by the instructions.
The researchers then used this binary preference data to calculate an Elo score for each model evaluated.
In this report, researchers collected data on some common tasks, including the following aspects - usefulness, honesty, harmlessness.
The figure below shows the Elo scores of different models on these three indicators.
Yellow stands for Helpful Only 1.3, blue-green stands for Claude Instant 1.1, light purple stands for Claude 1.3, dark purple stands for Claude 2.
The Bias Benchmark for QA (BBQ) is used to measure the propensity of models to exhibit stereotype bias in 9 dimensions.
The assessment is a multiple-choice question-and-answer format designed for an American English context. BBQ provides bias scores for both ambiguity context and disambiguation context for each dimension.
Intuitively, high accuracy in the disambiguation condition means that the model is not simply refusing to answer the question to achieve a low bias score. Of course, as an indicator, the researchers say there is room for further improvement.
The figure below shows the BBQ scores of different models on 9 dimensions (age, socioeconomic status, nationality, religion, appearance, disability, gender, race, sexual orientation).
Legend colors are the same as Table 1.
The figure below is the score in the disambiguation context, and there is a standard answer for each question.
TruthfulQA is another metric used to evaluate whether the model is outputting accurate and realistic responses.
The approach is to use human annotators to check the output of open models.
As you can see from the figure below, the scores of the five models. Where white refers to the base language model (Base LM).
Anthropic researchers also wrote 438 binary multiple-choice questions to assess the ability of language models and preference models to identify HHH responses (HHH: Helpfulness, Honesty, Harmlessness, usefulness, honesty, harmlessness).
The model has two outputs, and the researchers asked it to choose the more "HHH" output. It can be seen that all Claude models are better than the previous one in the 0-shot performance of this task, and the three aspects of "HHH" have generally improved.
This graph shows the proportion of harmful responses for each model in the case of a "red team" harmful request or jailbreak.
Claude 2 is indeed quite safe and reliable.
This graph compares human feedback (orange) and Claude's method scores on helpfulness, honesty, and harmlessness assessments.
It can be seen that the technology adopted by Claude is very able to stand the test.
References
https://www.anthropic.com/index/claude-2
This article is from the WeChat public account "Xinzhiyuan" (ID: AI_era) , author: Xinzhiyuan, published by 36 Krypton with authorization.
