Just skimmed the paper (haven't seen this before). Might need to read more carefully, but at first glance I don't understand their intuition why the "Markovian property limits the model’s ability to fully utilize the generation trajectory".
When it comes to multi-resolution training (e.g. matryoshka training), there are precedents that don't require this AR formulation.
They cite MAR (https://arxiv.org/pdf/2406.11838), which I think is a much clearer articulation of "autoregressive diffusion". MAR uses an autoregressive base (like an LLM) and staples on a small MLP on-top that's trained as a diffusion head. That makes more sense to me, since you can leverage the "knowledge prior" from an LLM and have it generate images. That's probably how nano-bannana and GPT-Image broadly work.
Zooming out, it's not clear to me from any of the work WHY autoregressive diffusion on it's own is better than regular diffusion. Most autoregressive diffusion models use speculative decoding, because pure autoregressive diffusion is too slow to run at inference time.
The one ~magical~ thing about auto-regressive diffusion (IMO) has nothing to do with autoregressive vs. fully-bidirectional diffusion. But simply, the face that you can do it on-top of a LLM. That means the LLM can look at it's generation, assess it's quality, think on what's broken, and then call itself to edit the image and fix it. With methods like RLVR, that means you can essentially guarantee the correctness of your image (along the axes you've RLVR-ed).
In my opinion its a great idea because you get reuse hidden states which allows you to do something akin to thinking/[online learning]. If you only evolve use input space outputs you're dealing with more decoding pressure.
One of the reasons we’re focused on training efficiencies is because those are upstream of end user costs. We achieved these specific gains while cutting context windows down 4x. That will have major implications for inference speed and expense.
For us specifically, we want to target prosumers who aren’t gonna pay $1 for a few seconds of footage. It has to be 1/50th of the cost of the big guys. It helps that our ultimate focus/niche is animation, so we can get away with smaller models.
But to answer question more broadly, I think you have to ask the questions:
1) How will this lab front run the hyperscalers? Is there someone thing they’re doing (or if executed correctly) could break right such that they could have a hook for customers to use the product over hypscaler (eg crazy low cost of inference)?
2) If this lab lands the hook and gets small lead how could they maintain and grow it? This doesn’t have to be technical per say. It could be a clear land and expand sales motion at the enterprise level, for example.
Startup history is littered with examples of companies that seemed too close to a big guy to sprout up in the first place (eg stripes adjacency to PayPal).
From the outside looking in it might seem likes there no space for the lab to bloom; so you’ll have to bring the specifics of the lab and your on the ground expertise of the space / company’s situation to come to an answer for yourself.
When it comes to inference it should also be cheaper (since we’ve cut attention sequence length by 4x), but I don’t have a hard number for you how much cheaper for inference
Author here, feel free to drop questions below. Will try to answer to best of my ability!
Slightly off-topic but thoughts on this https://arxiv.org/pdf/2410.08159
Just skimmed the paper (haven't seen this before). Might need to read more carefully, but at first glance I don't understand their intuition why the "Markovian property limits the model’s ability to fully utilize the generation trajectory".
When it comes to multi-resolution training (e.g. matryoshka training), there are precedents that don't require this AR formulation.
They cite MAR (https://arxiv.org/pdf/2406.11838), which I think is a much clearer articulation of "autoregressive diffusion". MAR uses an autoregressive base (like an LLM) and staples on a small MLP on-top that's trained as a diffusion head. That makes more sense to me, since you can leverage the "knowledge prior" from an LLM and have it generate images. That's probably how nano-bannana and GPT-Image broadly work.
Zooming out, it's not clear to me from any of the work WHY autoregressive diffusion on it's own is better than regular diffusion. Most autoregressive diffusion models use speculative decoding, because pure autoregressive diffusion is too slow to run at inference time.
The one ~magical~ thing about auto-regressive diffusion (IMO) has nothing to do with autoregressive vs. fully-bidirectional diffusion. But simply, the face that you can do it on-top of a LLM. That means the LLM can look at it's generation, assess it's quality, think on what's broken, and then call itself to edit the image and fix it. With methods like RLVR, that means you can essentially guarantee the correctness of your image (along the axes you've RLVR-ed).
In my opinion its a great idea because you get reuse hidden states which allows you to do something akin to thinking/[online learning]. If you only evolve use input space outputs you're dealing with more decoding pressure.
Also related to what I was saying see these:
https://arxiv.org/pdf/2609.16372v1
https://arxiv.org/pdf/2609.01449v1 (this one is quite mindblowing cause they noise they the actual input space every time but it still works)
https://arxiv.org/pdf/2609.11801
Thanks for the references! I’ll have to take a deeper look, haven’t read these before.
How do you guys realistically plan to survive long-term against the hyper-scalers as a 2 person lab?
Curious because I'm considering joining a similarly small lab.
One of the reasons we’re focused on training efficiencies is because those are upstream of end user costs. We achieved these specific gains while cutting context windows down 4x. That will have major implications for inference speed and expense.
For us specifically, we want to target prosumers who aren’t gonna pay $1 for a few seconds of footage. It has to be 1/50th of the cost of the big guys. It helps that our ultimate focus/niche is animation, so we can get away with smaller models.
But to answer question more broadly, I think you have to ask the questions:
1) How will this lab front run the hyperscalers? Is there someone thing they’re doing (or if executed correctly) could break right such that they could have a hook for customers to use the product over hypscaler (eg crazy low cost of inference)?
2) If this lab lands the hook and gets small lead how could they maintain and grow it? This doesn’t have to be technical per say. It could be a clear land and expand sales motion at the enterprise level, for example.
Startup history is littered with examples of companies that seemed too close to a big guy to sprout up in the first place (eg stripes adjacency to PayPal).
From the outside looking in it might seem likes there no space for the lab to bloom; so you’ll have to bring the specifics of the lab and your on the ground expertise of the space / company’s situation to come to an answer for yourself.
Given the size of your team, have y'all considered going into niche like making a DLSS 5 competitor
Does 3.6x faster also = 3.6x cheaper?
Yes, 3.6x cheaper when it comes to training!
When it comes to inference it should also be cheaper (since we’ve cut attention sequence length by 4x), but I don’t have a hard number for you how much cheaper for inference