Sponsored by Inngest. This section is not related to the post below it.
New in agents-towards-production: durable RAG ingestion. Ingestion is the stage most RAG write-ups skip. It works on your laptop, then a real corpus kills it halfway through, and the rerun re-embeds everything you already paid for and leaves duplicate chunks behind. The new tutorial builds the version that survives that: every stage recorded so a retry resumes at the document that broke, upserts keyed on a content hash so nothing duplicates, and a replay path so you fix one parser and re-run one step.
Read it here:
https://github.com/NirDiamant/agents-towards-production/tree/main/tutorials/durable-rag-ingestion-inngest
The durable execution platform it runs on is Inngest:
https://inngest.link/diamantai-nl
The architecture, the tradeoffs and the section on when not to use it are my own.
Your image prompt has a word in it that does nothing, and another word that quietly repaints the whole frame. Reading the prompt won’t tell you which is which, so hold everything else still: same model, same random seed, and one phrase added to the prompt, on a white tablecloth.
Both pictures come from the same model and the same seed, and only the prompt changed, from a glass of wine to a photograph of a glass of red wine on a white tablecloth.
The glass stays where it was. The table, the wall, the window and the light all change.
But here is the catch. A tablecloth covers a table. It doesn’t get to pick the window. So the phrase did something other than add a tablecloth, and until you know what, every prompt you write is a guess.
Once you know what a word does inside the model, you can read any prompt and tell which words pull and which are dead weight. The video runs the experiment on one real photograph, and the post gives you the mechanism.
If the player does not show in your mail app, watch it on YouTube here.
How Diffusion Is Trained
Training starts by destroying a photograph on a schedule. At each step you mix a fixed fraction of fresh Gaussian noise into the image and keep a slightly smaller fraction of what was there. After enough steps the image is statistically indistinguishable from pure noise. Modern models do this on a compressed latent and decode to pixels at the end, but the idea is the same.
One real photograph with noise added ten times, frames 0, 3, 6 and 9.
The network never learns to draw. It learns one regression task: given a noisy frame and the step number, predict the noise that was mixed in. The loss is the mean squared error between the predicted noise and the noise that was really added. Subtract the prediction from the frame and you have an estimate of the clean image.
Do that over billions of image-caption pairs and the estimate gets good.
Generation runs the schedule backwards. Start from pure noise, predict the noise in it, remove a fraction, and repeat for a few dozen steps, each one the same network call with a lower step number. The picture that comes out was never in the training set. It is what the noise predictor believes a clean image looks like, refined step by step.
Training asks one question, which noise was added, and scores the answer by mean squared error. Generation asks the same question from pure noise and subtracts a fraction each step. The bottom row is a real reverse pass from the experiment.
Some newer models predict the direction toward the clean image instead, and some build the picture as tokens, but the prompt enters the same way in all of them.
How the Prompt Gets Inside Every Step
Your prompt does not go to a separate understanding stage. A text encoder turns it into one vector per token, and those vectors are fed into the noise predictor through attention layers on every step. The call at step forty sees the prompt, and so does the call at step one.
The same static run twice, once prompted with red fox and once with wine glass.
Take the same starting noise twice, condition one run on red fox and the other on wine glass, and the identical noise resolves into two different images. The noise never changed. The prediction changed at every step, because the prompt vectors sat inside every prediction.
Most tools push this harder with classifier-free guidance. At each step the model predicts the noise twice, with your prompt and with an empty prompt, then moves the guess further in the direction the prompt added, scaled by a number you can usually set. That is why one phrase can change the wall behind the glass. It was never painted in at the end. It tilted every one of the few dozen predictions.
The prompt becomes one vector per token, and those vectors enter every denoising step through attention. Guidance pushes each step further in the direction the prompt added.
Why the Early Guess Is an Average
Now the part that decides everything. Minimizing mean squared error has a known consequence: the optimal prediction is the expected value. So on a frame that is still mostly noise, the clean-image estimate is the average of every training image that could plausibly sit under that noise and match your prompt.
So a prompt doesn’t describe one picture. It selects a region of the training distribution, the images whose captions contain your words, and the early estimate is the mean of that region. Later steps sharpen the estimate, but toward whatever the mean already committed to.
A vague prompt selects a huge region, so the mean is mush. A specific prompt selects a small region, so the mean already has a room, a light and a lens in it.
Your words select a region of the training distribution, and the early estimate is that region’s mean. The two means shown are the real averages of eight generations each.
The Experiment: One Phrase Per Step
One open model, eight fixed seeds, and a prompt that grows one phrase at a time: a glass of wine, then a photograph of one, then red wine on a white tablecloth, then warm window light from the left, then shot on 35mm film. The seed fixes the starting noise tensor, so the composition is anchored and any other change belongs to the words.
Two seeds across four prompts. Down a column the glass moves and the world stays. Across a row the glass stays and the world moves.
Add the tablecloth and the model doesn’t lay a cloth over the old wooden table. It moves to a brighter room with a window and daylight, because the mean of the tablecloth region already contains the window.
People add praise words for luck, masterpiece, ultra detailed, award winning, and expect them to do nothing. In the experiment they changed the whole scene: a dim bar, a warm glow behind the stem, and on one seed raspberries in the wine. Praise words select a region too, the images captioned with that vocabulary, and that region has its own look.
Now average the eight vague pictures pixel by pixel, and do the same for the eight specific ones.
Eight vague-prompt pictures averaged into one, next to eight specific-prompt pictures averaged into one.
In the vague mean the only thing all eight runs share is the glass, and everything behind it averages into brown mud. In the specific mean the cloth, the left-hand window and the streak of light survive, because all eight runs put them in the same place. What your words pin down survives the average. What you leave open gets decided per seed and blurs away.
Write the Caption, Not the Wish
The rule follows from the mechanism. The model was trained on image-caption pairs, so the words that select a tight region are the words that appear in real captions: the subject, its setting, the light, the camera. Write the caption of a photograph that could exist. A quality adjective selects a style region, not a content region, and you rarely want it.
The specific prompt from the experiment, split into its four parts, each pointing at the part of the real picture it fixed.
The workflow for your next prompt. Lock the seed first: a seed field in the Stable Diffusion and ComfyUI interfaces, a seed parameter in Midjourney, a seed argument in every hosted image API. Leave it alone for the whole session, because that setting separates what the words did from what the starting noise did.
Then keep a plain text file, prompt notes, with four lines per attempt: subject, place, light, camera. Build the prompt from those lines only, generate once, change one line, generate again. After three rounds you know which line moves the picture in that tool.
Delete every word that grades the picture instead of describing it. The bad prompt from this experiment reads a glass of wine, masterpiece, ultra detailed, award winning. The good one reads a photograph of a glass of red wine on a white tablecloth, warm window light from the left, shot on 35mm film. Same length, and only the second is a caption. If a picture still comes back over-styled, lower the guidance scale before touching the words, because a high scale exaggerates whatever region the prompt selected.
Choose Seed or Words When
Keep a word when it could appear in a caption under a real photo of what you want: subject, place, light, lens, time of day.
Change the seed, not the prompt, when you like the room, the light and the camera and only want the object somewhere else.
Add a word when two runs on the same seed disagree about something you care about. Your prompt left that detail open and each run picked from the whole region.
Cut a word when it is a quality word, a mood word, or a word you would never see under a real photograph.
The same four rules as a flow: generate on two seeds, then add, cut, or change the seed depending on where the runs disagree.
What This Cannot Fix
Narrowing only works while there is a region to narrow to. Ask for something people don’t photograph, and the caption region is empty, so the model reaches for the nearest populated region instead. No rewrite fixes that. Stop typing and hand the model a picture, as a reference image or an edit, because the words were never the problem.
This test used one open model and eight seeds. Another model picks a different bar or window, but the way words move the picture stays the same.
The Same Words in a Different Order
An image generator walks noise toward the mean of a region of its training data, and every word you write moves that region at every step. Read your prompt as a caption, and delete every word that selects a region you didn’t want.
One question is still open. Put the same specific words in a different order and you land in the same room, same cloth, same window, but the camera has stepped back and the glass is small. The text encoder is position-aware, so order does change the vectors, yet nothing in the sentence said where to stand. What the model was listening to is the next video.
The specific sentence, and the same words in a different order, on one seed.