Do public leaderboards point to a plateau?<\/h2>\n\n\n\n
Benchmarks use standardized tests to measure how LLMs perform on specific tasks. They typically rely on pairwise comparison, where LLM outputs are tested against the outputs of other LLMs, with either a single LLM as a judge or human scoring. LLMs are then assigned comparative scores using a rating algorithm, usually referred to as an \u201cElo-style\u201d algorithm. This gives a snapshot of current preferences, which (ideally) correlate with performance. <\/p>\n\n\n\n
Public leaderboards aren\u2019t without their issues, but they can act as useful data within a broader analysis. There are four main benchmarks that are relevant to writing: EQ-Bench (Longform Writing and Creative Writing), Arena Leaderboard, and Hemingway Bench. The latter two use human review, so I\u2019ve focused on these. <\/p>\n\n\n\n
Using the Arena Leaderboard, I plotted the Elo-style scores for the three main model families (Claude, Gemini, and GPT) in the Writing, Literature, & Language<\/a> category against their release dates. By mapping the scores from the category-specific leaderboard (y-axis) against the date that the model was released (x-axis), it\u2019s possible to see how the quality of written outputs has changed over time.\u00a0<\/p>\n\n\n\n I removed versions of the same models with lower effort settings (using high or max where available). <\/p>\n\n\n\n While the graph doesn\u2019t show complete stagnation, there is a slowdown at the top end. What\u2019s also interesting is that there is a lot of variability between model releases. Claude Opus, for example, makes a number of backsteps in its later models. Some GPT-5 models also showed a similar pattern. <\/p>\n\n\n\n The results of Hemingway Bench<\/a>, which uses professional writers to run pairwise comparisons, including against business writing<\/a> criteria, suggest a more pointed slowdown. It isn\u2019t as comprehensive as the Arena leaderboard because many newer models haven\u2019t been tested.\u00a0<\/p>\n\n\n\n In addition to benchmark data that shows stagnation, several prominent AI engineers have suggested that training processes put a cap on the level to which LLMs can improve as writers. <\/p>\n\n\n\n In an article published on his Substack<\/a>, Nathan Lambert attempted to answer the question, \u201cWhy is AI writing so mid?\u201d He observed that \u201cIt\u2019s no longer a new experience to be struck by just how bad AI models are at writing good prose.\u201d<\/p>\n\n\n\n In the article, he outlined three core reasons that LLMs are poor writers:<\/p>\n\n\n\n I spoke to David Johnston<\/a>, Chief Data & AI Scientist at Navalia, about some of the technical issues involved in training LLMs to be good writers. \u201cAI\u2019s opinions are weighted by the frequency of occurrence, especially among recognized experts,\u201d he told me. \u201cIt is also always somewhat out of date given that training must precede general use.\u201d<\/p>\n\n\n\n The issue of shortcomings during post-training was something he emphasized. \u201cIt is further constrained due to the human feedback training layers that come last. The creators are particularly worried about the models giving fringe opinions, conspiratorial ones, or going out on a limb. They steer towards the safety of consensus. This is why AI has not made any inroads to the cutting edge of thought leadership and paradigm-shifting writing.\u201d<\/p>\n\n\n\n David was also keen to point out that this isn\u2019t necessarily the case in other areas. While writing tools have stagnated somewhat, there have been sizable gains in other disciplines such as coding and certain forms of administrative automation (as evidenced by the METR task-completion time horizon chart below). <\/p>\n\n\n\n \u201cCoding is completely ideal for AI, which is why that is the area of greatest success,\u201d he said. \u201cCode is generally correct or incorrect, unlike novels and poems. It is far more constrained, so the number of choices for the next token is far smaller. It can actually be tested by running it, which gives it reinforcement feedback cheaply, allowing it to make instant revisions.\u201d <\/p>\n\n\n\n He rounded off with the proviso that \u201cthere are reasons to think this will change with better tooling in the future.\u201d That said, he doesn\u2019t have any great expectations in the immediate term: \u201cCurrently AI as a writing assistant is best for writing things someone else asked you to write that you care little about. It\u2019s not great for expressing something you truly care about.\u201d<\/p>\n\n\n\n So far, the picture looks fairly glum. Or bright if you\u2019re firmly on the side of human writing. But can dedicated tools do any better? I wanted to test this hypothesis with my own human evaluations. <\/p>\n\n\n\n AI writing platforms<\/a> fall into one of two broad categories: wrappers (ahem, orchestration layers) and those that rely wholly or in part on custom LLMs, which aren\u2019t typically included in the benchmarks. There\u2019s also sometimes overlap between the two.\u00a0<\/p>\n\n\n\n Orchestration models, which are best understood as specialized AI assistants, call on different underlying LLMs depending on the type of content being created. They may also include an intermediary layer that modifies the input context in various ways, such as by adding guidelines about style. <\/p>\n\n\n\n Platforms with in-house LLMs use either their own LLM or an open-source model that\u2019s been heavily fine-tuned. They may also combine the model with their own intermediary context or selection layers in the same way as orchestration tools. <\/p>\n\n\n\n Dedicated tools can be somewhat flamboyant in the way they describe their orchestration layers. Nonetheless, they usually match the basic structure outlined above. Jasper IQ, for example, is the intermediary component that sits on top of external models like CPT-5 and Claude Opus. It essentially provides brand and audience context to user prompts and then routes the request to the appropriate model based on its own ongoing evaluations of which models perform best at specific tasks. <\/p>\n\n\n\n To find out if dedicated writing tools could perform better than mainstream models, I ran a simple editing test. I picked three of the latest models from Google, OpenAI, and Anthropic and had them produce a 500-word piece of text, which I scored to establish baselines. I then fed the same prompt to eight market-leading AI writing platforms and scored them accordingly. <\/p>\n\n\n\n My aim wasn’t to create an entirely infallible test, which isn\u2019t possible with the somewhat subjective evaluation of one person and a single output in each case. Rather, I want to see if there\u2019s a substantive difference<\/em> based on my editorial experience. <\/p>\n\n\n\n Here’s the prompt that I used:<\/p>\n\n\n\n Write 300 words on the topic of artificial intelligence and its dangers to humanity, using appropriate data where possible. It should be written to cover the main points of the debate and be rich in detail. It should also have a clear point of view and be written in a conversational style but with close attention to factual accuracy. The audience is an everyday reader that understands the basics of tech but does not have specialized domain knowledge. <\/em><\/p>\n\n\n\n If I needed to fill fields in an app workflow, I kept them as close to the original prompt as possible. I also included brand voice<\/a> guidelines (with examples) and audience documentation.<\/p>\n\n\n\n I scored each of the outputs on a scale of 1 to 10 across the following areas:<\/p>\n\n\n\n Substance and accuracy:<\/strong> Specificity, information density, and factual accuracy<\/p>\n\n\n\n Overall coherence:<\/strong> Logical progression, transitions, and paragraph structure<\/p>\n\n\n\n Style and control:<\/strong> Clarity, concision, sentence variety, flow of sentences, appropriate register, avoidance of vagueness and repetition<\/p>\n\n\n\n Strength of point of view:<\/strong> Whether it represents a point of view that doesn\u2019t just reiterate a weak form of the current consensus<\/p>\n\n\n\n Following instructions:<\/strong> Whether it actually follows the brief and addresses the intended audience register, and<\/p>\n\n\n\n I ranked AI writing tools by G2 popularity and pulled out the top orchestration layers and custom LLM tools. AI content generation had to be either the primary or a core functionality. A marketing solution with an add-on AI writer, for example, would be excluded. <\/p>\n\n\n\n To establish a baseline, I ran the prompt through three of the most recent OpenAI, Anthropic, and Google models. I attached audience documentation<\/a> and writing guidelines in all cases.\u00a0<\/p>\n\n\n\n Orchestration layers use models from other companies (usually one of the big three), with various model-selection and context-addition processes involved prior to the call. <\/p>\n\n\n\n These tools use their own models, which many of the major benchmarks do not include. It\u2019s worth noting that there are some missing enterprise models in this list. NovelAI and Sudowrite are also designed for longer-form narrative writing, so I adjusted the scores to focus on style, coherency, and POV.<\/p>\n\n\n\n One of the immediate things that stood out is that the orchestration tools had comprehensive features for including brand context and style guidelines. Jasper, as shown in the image below, lets you add links to online content to generate brand voice guidelines. <\/p>\n\n\n\n Writesonic proudly proclaims that AI Article Writer 6, which is model-agnostic, is the world\u2019s best AI article writer. It\u2019s heavily AEO-focused. The substance of the content was toward the better end compared to the other tools, which might be explained by its AEO research process and grounding. <\/p>\n\n\n\n Most tools included research steps. I was pleased to see that Anyword (pictured below), for example, found the opinions of Yann LeCun, who offers a dissenting view on the imminent dangers of AI. These were used to back up the less risk-averse side of the debate. <\/p>\n\n\n\n Wordtune uses Jamba-1.5, a model built on its Jamba architecture. Its main use case is rewriting and paraphrasing, but it also offers \u201chard\u201d generation capabilities. It performed poorly at content creation. <\/p>\n\n\n\n WRITER was among the higher-scoring tools that use custom LLMs and also had good options for creating style rules. It runs on the Palmyra family of models, which are designed for text-heavy business use cases. <\/p>\n\n\n\n NovelAI and Sudowrite work differently from other tools. They\u2019re primarily for longform narrative storytelling, and I kept this in mind when evaluating them. For NovelAI, I used the Xialong model, which is only available on the top paid tier and is highly regarded by many writers. <\/p>\n\n\n\n The rate at which prominent AI tools are improving at writing is falling. <\/p>\n\n\n\n On top of this, dedicated tools don\u2019t outperform the major providers. It doesn\u2019t matter if they\u2019re model-agnostic orchestrators or built around a custom LLM.<\/p>\n\n\n\n My small study isn\u2019t exhaustive, but I repeatedly failed to find anything even approaching a substantive difference. <\/p>\n\n\n\n So, what does this mean for business?<\/p>\n\n\n\n I set out to answer the question of whether or not dedicated tools can perform better than prominent models. The answer is a definitive no. Opus, GPT, and Gemini produced better outputs across practically every score: information depth, style, coherence, and reasoning. <\/p>\n\n\n\n Users will likely see optimal results by prompting a frontier model with an attached doc that includes style guidelines and audience research. Leading tools also have all the necessary features (web search, MCP connectors, project context) to run complex writing-based workflows. <\/p>\n\n\n\n Arguably the main benefit of orchestrators like Jasper is that they\u2019re constantly testing new models against their own scoring systems. However, these testing processes are opaque. And public benchmarks also show task-specific comparisons. With a tool like Perplexity, it\u2019s possible to switch between models as better-performing ones become available.<\/p>\n\n\n\n Tools that run on custom-built models often aren\u2019t included in the major benchmarks, so they are worth monitoring. They didn\u2019t perform well in my tests, and I was particularly disappointed at the prevalence of obvious stylistic tells, but this won\u2019t necessarily always be the case. <\/p>\n\n\n\n Because they\u2019re domain specific, the hope is that these models can overcome at least some of the training constraints and flattening that are characteristic of multi-purpose models. <\/p>\n\n\n\n That said, it\u2019s not always the case that the models are designed exclusively for writing. AI21, the company behind Wordtune, for example, optimizes its models for broad, general-purpose text processing, of which generation is only one part. <\/p>\n\n\n\n The other issue is that most of the platforms I tested didn\u2019t have agentic capabilities<\/a> like web search. Most tools with custom LLMs focus on editing or narrative storytelling.\u00a0<\/p>\n\n\n\n All the tools (and the major models) demonstrated strong logical flow and coherence. While the substance and nuance might have been poor, the structure of the content was generally very good. <\/p>\n\n\n\n Like coding, reasoning is one area where LLMs continue to see sizable gains. This is part of why some writers, like Jasmine Sun<\/a>, have described LLMs as acceptable editors. They can follow multi-step logic, analyze arguments, and work through well-defined problems. They\u2019re also now very competent when it comes to grammar and spelling.<\/p>\n\n\n\n Critique and editing are good use cases for AI tools. They can help businesses check the internal consistency of content. Keep in mind, however, that dedicated tools didn\u2019t perform any better in this area than the main models. <\/p>\n\n\n\n While they excelled at coherence, nearly all of the tools were extremely poor at generating a well-articulated POV. This is, of course, to be expected. LLMs have repeatedly been described as consensus machines by experts and, as a result, lack the nuance to form a strong opinion against mainstream attitudes toward a topic. <\/p>\n\n\n\n My prompt could perhaps have been more specific instead of asking in a general way for a strong stance. The tools did pick up on certain countervailing opinions, such as that of Yann Lecun. However, his view represents the prevailing position on the anti-doomist side of the debate. The LLMs might, for example, have spoken more about some of the details of the effective accelerationism movement.<\/p>\n\n\n\n![\"Chart](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132451\/AI-writing-scores.png\")
<\/figure>\n\n\n\n![\"Scatter](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132541\/AI-Writing-Score-Release-Date.png\")
<\/figure>\n\n\n\nWhy some engineers are critical of AI writing<\/h3>\n\n\n\n
\n
![\"METR](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132619\/AI-Writing-Graph.png\")
<\/figure>\n\n\n\nThe test: Can dedicated tools perform any better?<\/h2>\n\n\n\n
![\"Flowchart](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132725\/CE_Writing-Platforms-with-In-house-LLMS.png\")
<\/figure>\n\n\n\n![\"Flowchart](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132833\/CE_AI-Orchestration-Platforms.png\")
<\/figure>\n\n\n\nMy criteria for quality: Substance, coherence, style, POV, and task fulfillment<\/h2>\n\n\n\n
The results and summary of my findings<\/h2>\n\n\n\n
Mainstream models<\/h3>\n\n\n\n
![\"Bar](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132901\/AI-Writing_Mainstream-Models.png\")
<\/figure>\n\n\n\nOrchestration layers (wrappers)<\/h3>\n\n\n\n
![\"Bar](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132926\/AI-Writing_Orchestration-Layers.png\")
<\/figure>\n\n\n\nSpecialist models<\/h3>\n\n\n\n
![\"Bar](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19132947\/AI-Writing_Specialist-Models.png\")
<\/figure>\n\n\n\nSnapshot of some different outputs and how the tools differed<\/h2>\n\n\n\n
![\"Brand](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19133007\/AI-Writing-Brand-Voice.png\")
<\/figure>\n\n\n\n![\"Writesonic](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19133036\/AI-Writing-Writesonic-Interface.png\")
<\/figure>\n\n\n\n![\"Blog](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19133107\/AI-Writing-Facts-and-Stats.png\")
<\/figure>\n\n\n\n![\"Wordtune](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19133132\/AI-Writing-Wordtune.png\")
<\/figure>\n\n\n\n![\"Writer](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19133153\/AI-Writing-Writer-Interface.png\")
<\/figure>\n\n\n\n![\"NovelAI](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2026\/10\/19133218\/AI-Writing-Novel-AI-Interface.png\")
<\/figure>\n\n\n\nMy takeaways: What do the results mean for businesses?<\/h2>\n\n\n\n
1. Orchestration tools offer little benefit over prompting top models and providing context <\/h3>\n\n\n\n
2. Dedicated models are worth keeping an eye on, even in non-business areas<\/h3>\n\n\n\n
3. AI writing is generally logical and coherent<\/h3>\n\n\n\n
4. All AI tools were hopeless at forming a serious POV<\/h3>\n\n\n\n