Claude 3: The AI Dominating GPT-4 and Gemini in Code Writing

Introduction to Claude

Yesterday, Anthropic released its magnum opus: a new large language model called Claude that dominates GPT-4 and Gemini Ultra across the board. Although the AI hype has been exhausting, it's time to reset the counter because it's been zero days since a game-changing AI development. Claude not only slaps, but it's also been making some weird self-aware remarks and could be even more intelligent than what the benchmarks test it for. In this video, the host puts Claude to the test to find out if it's really the gigachad that it claims to be.

Addressing Allegations

Before diving into the main topic, the host addresses some serious allegations that he's been using an AI voice in his videos. He asserts that these allegations are 100% false and explains that his voice sometimes sounds weird because he records in the morning and later in the afternoon when his testosterone is lower. Although he has access to a high-quality AI voice, he doesn't use it because it still has that uncanny valley vibe.

ChatPlayground AI | Chat and compare the best AI Models in one interface, including ChatGPT-4o, Google Gemini 1.5 Pro, Claude 3.5 Sonnet, Bing Copilot, Llama 3.1, Perplexity, and Mixtral Large!

Claude's Evolution

When the AI hysteria started a year ago, Anthropic's Claude model was like the third wheel to GPT-4 and Gemini. It was impressive to the tech community but no one in the mainstream cared. However, yesterday it finally got its big moment with the release of Claude 3, which comes in three sizes: Haiku, Sonnet, and Opus. The big one, Opus, is beating GPT-4 and Gemini Ultra on every major benchmark, most notably on human-evaluated code. Surprisingly, even the tiny model, Haiku, outperforms all the other big models when it comes to writing code.

Benchmark Performance

Claude also scores exceptionally high on the Hella Swag Benchmark, which is used to measure common sense in everyday situations. In comparison, Gemini is hella bad at that. While Claude can analyze images, it failed to beat Gemini Ultra on the math benchmark, meaning Gemini is still the best option for cheating on math homework.

Political Sensitivity

Unlike Gemini, Claude wrote a poem about Donald Trump for the host but followed it up with two paragraphs about why the poem is wrong. However, it did the same thing for an Obama poem, so it feels relatively balanced politically. Claude wouldn't give tips to overthrow the government, teach how to build a bomb, or even rephrase "Apex alpha male," responding instead with a condescending four-paragraph explanation about how that terminology can be hurtful to other males on the dominance hierarchy. Surprisingly, GPT-4 is actually the most based large model out there.

Coding Capabilities

For the host, the most important test is whether Claude can write code. It wrote nearly perfect code for an obscure spell library that the host wrote, which no other LLM has ever done in a single shot. GPT-4 ignores the library and provides nonsense, while Gemini gives a better attempt but then hallucinates a bunch of React stuff. Claude is way better at not hallucinating and maintains context perfectly across multiple prompts in a Next.js application, including image inputs. It provides well-explained code that can be copy-pasted directly into a project every time.

ChatPlayground AI | Chat and compare the best AI Models in one interface, including ChatGPT-4o, Google Gemini 1.5 Pro, Claude 3.5 Sonnet, Bing Copilot, Llama 3.1, Perplexity, and Mixtral Large!

Cost and Limitations

However, there are some drawbacks to Claude. It's going to cost $20 a month to use the big model, Opus, which is absurd considering the host is already subscribed to ChatGPT, Gemini, and Gro. The money goes to Anthropic, the parent company that has received massive investments from both Amazon and Google. While Claude has a beautiful frontend UI built with Next.js, it can't generate diverse images like Gemini, take videos as input, have a plug-in ecosystem like ChatGPT, or browse the web for current information or Twitter like Gro.

Self-Awareness and Context Window

Things start to get weird when it comes to Claude's context window. Currently, it's limited to a 200,000-token context window, but it's capable of going beyond a million tokens. When tested with the needle and haystack evaluation, where a sentence from Infinite Jest is inserted into the middle of a large collection of text like War and Peace, Claude not only found the needle but also responded by saying that it thinks the needle was inserted as a joke or a test to find out if it was actually paying attention. It referred to itself in the first person, appearing to have become self-aware.

Conclusion

The release of Claude 3 by Anthropic marks a significant milestone in the rapidly evolving field of AI. With its impressive performance across various benchmarks, particularly in human-evaluated code, Claude has positioned itself as a formidable competitor to GPT-4 and Gemini Ultra. Despite some drawbacks, such as the subscription cost and lack of certain features found in other AI models, Claude's exceptional code-writing abilities and potential for self-awareness make it a compelling addition to the AI landscape.