Generating a paragraph with a language model is a solved problem. Generating ninety thousand words that a human will read start to finish, and that still make sense on page four hundred, is not. The difference between the two is not model quality. It is state management, and almost every hard problem in long-form generation turns out to be a state problem wearing a costume.
We run a book writing platform. Just over 11,000 writers have used it, and it has produced more than 190 million words. About 4,000 of those runs reached book length. That is a decent sample to watch failures in, and the failures are consistent enough to be worth writing down.
The context window is a red herring
The first instinct when a long generation goes wrong… the first instinct is to reach for a bigger context window.
One problem is attention dilution.
It mostly does not work, for two reasons.
One problem is attention dilution. A model with a million token window does not attend evenly across a million tokens. Detail introduced in chapter two competes with four hundred thousand tokens of everything else by the time you are drafting chapter thirty. Although the information is technically present, it becomes practically invisible.You will see the model contradict something it was given, in full, three thousand tokens earlier in the same prompt.
Another is the cost curve. Naive full-context replay is quadratic in the number of chapters. Chapter one costs one unit. Chapter fifty costs fifty. Sum that over a book and you are paying for roughly n squared over two units of input to produce n units of output. On a ninety thousand word manuscript that is the difference between a feature you can ship and one you cannot.
So the interesting question is not how much context you can fit. It is what you choose to carry forward.
Three failure modes, in order of how often we see them
Entity drift. A character’s eye colour changes. A city that was two days away becomes an afternoon’s drive. A minor character who died reappears. This is the most common and the most damaging, because readers notice it and it destroys trust in everything around it.
Structural repetition. Two chapters do the same beat. A revelation lands twice. The model, having no memory of narrative debt already paid, pays it again. This is subtler than entity drift and harder to catch with any automated check, because both chapters are individually fine.
Register collapse. Around chapter fifteen or twenty the prose flattens toward a generic middle. Voice is the first thing to go when the model is working from a summary rather than from the actual text, because summaries preserve plot and discard style.
Each of these needs a different mechanism. Treating them as one problem called “consistency” and throwing more context at it is how you end up with an expensive system that still drifts.
What we carry forward instead
The working shape is a small, explicit, structured state object that is updated after every chapter and passed into the next one, alongside a much smaller slice of raw text.
The state object holds facts, not prose. Characters with their fixed attributes and their current situation. Locations. Objects that matter. Open threads, with the chapter they opened in. Things the reader knows that a given character does not, which is the one piece of bookkeeping that human writers do intuitively and models do not do at all.
It is worth being strict about the schema here. If you let the model write freeform notes to itself, the notes grow, drift, and start contradicting each other, and you have recreated the original problem in miniature. A fixed schema with typed fields, updated by a separate call whose only job is to extract state changes from the chapter that was just written, stays bounded. The extraction call is cheap because it is short and mechanical, and it is the single highest leverage component in the whole pipeline.
Alongside the state object, carry the last chapter in full. That is what protects voice. Summaries tell the model what happened; the actual previous chapter tells it how the book sounds. Losing that is why register collapses.
Where retrieval helps and where it does not
Vector retrieval over previously generated chapters sounds like the obvious fix for entity drift, and it helps less than you would expect.
The reason is that the drift you care about is usually a fact the model never thought to look for. It writes “she pushed her red hair back” without any awareness that hair colour is a retrievable fact with a prior value. A retrieval step only fires when something in the generation triggers it, and contradictions are exactly the case where nothing triggers.
Retrieval does earn its place for callbacks. When you want chapter thirty to reference a specific scene in chapter four, semantic search over the manuscript is the right tool, and it works well. So keep it, but keep it for the job it is actually good at, and let the structured state object handle consistency. Deterministic beats semantic when the fact has a known slot.
The pragmatic addition is a validation pass: after a chapter is generated, check the new text against the state object for direct contradictions, and flag rather than auto correct. Auto correction produces confident nonsense. Flagging produces a list a human can clear in ten minutes.
The economics of a long run
There is a design constraint people underestimate. A book generation is not a request, it is a job that runs for a long time, and every architectural decision downstream of that is shaped by it.
You cannot hold an HTTP connection open for it. You cannot put it behind a function with a short execution ceiling. You need durable job state, because the run will fail partway through and it must resume from the last completed chapter rather than from zero. You need per chapter checkpointing to storage, not in memory, because the process will be recycled.
We learned the resumability part the expensive way. Early on, a failure at chapter twenty eight meant regenerating twenty eight chapters. That is not just cost, it is a user watching a progress bar reset, which is worse.
The other economic reality is that streaming matters more than throughput. Nobody waits twenty minutes staring at a spinner. Streaming each chapter as it completes turns an unacceptable wait into an acceptable one, and it changes nothing about the underlying compute.
The number that should worry you
Here is the figure from our own data that we find most instructive. Over 11,000 writers, more than 190 million words generated, about 4,000 runs that reached book length, and fewer than 2,000 users who ended up with a substantial finished book.
Generation is not the bottleneck. Completion is. The gap between words produced and books finished is where the actual product problem lives, and no amount of model improvement closes it, because it is not a model problem. It is a problem of people losing the thread, generating faster than they can read, and ending up with a large object they do not feel ownership of.
That reframes what a good system does. The job is not to produce text as fast as possible. It is to keep a human in the loop at a rate they can sustain, which means chapter level review gates, editable outlines, and making it cheap to change your mind at chapter ten without discarding chapters one through nine.
Most of the engineering effort in a serious AI book writing tool goes into that loop rather than into the generation itself, which is the opposite of where you would expect it to go when you start building one.
If you are building something similar
A short list of what we would tell ourselves at the start.
Design the state object before you design the prompts. It is the schema everything else negotiates through, and changing it later is expensive. Separate extraction from generation. One call writes prose, a different call updates state. Mixing them makes both worse and makes failures impossible to attribute.
Checkpoint every unit to durable storage and make resume the default path, not the recovery path. Carry the previous unit in full, always, for voice. Validate and flag, never silently correct.
And measure completion, not volume. Words generated is a vanity metric that will look excellent while your users quietly abandon their manuscripts.
None of this is specific to books. Any system that generates a long, internally consistent artifact from a language model hits the same wall in roughly the same place. Books just make the failures easy to see, because a human reads every word in order and notices when the eye colour changes.
Leave a Reply