I really like this approach! I sort of think of the vision encoder here as an expensive high fidelity RAG encoder.
The thing I’d love to do with a system like this is train it to be KV cache ordering independent (ie permutation invariant at the page level). Basically each page’s KV cache should be understandable by the model in any ordering - which would allow you to go one step further and treat the KV cache of the vision encoded page as the chunk for the model to reason over.
Then all these zoom in for more detail tricks will extend naturally.
I remember reading a paper entitled "A Picture is Worth a Thousand Tokens" or something similar like 2-3 years ago. The reality is that no one really wants/needs contexts that big, anyway. It's hard enough making LLMs truly useful even with a small/medium context.
Taking a bit of a longer-stick approach to AI/ML, what I am struck by is that this is clearly a utility model intended for use alongside other models/tooling, and - has the 'feel' of being a bit like as if someone composed a batch script to do something important - and I mean that in a good, positive, useful way, just to be clear.
We are living in the age where people can compose a model to do almost anything you can describe, literally. I personally have accrued a rough 'gang' of models I use to check each other before they wreck each other, and nothing gets past a repo review cycle. All I am now is the final builder, distributable packager, for a fleet of little purpose-built byte-chuckers.
Am I correct in assuming that a principle use case for LensVLM is in the training side of things .. ?
>Say, what? When would a compressed image of text be smaller than just text?
Since every character in text shares compressible ligaments with every other character, maybe? Or, more finite, every pixel representing a ligature component in a character can be compressed against other pixels in the path set.
I think, at scale, this is very important - folks training models on near-petabyte sized corpus would have cause to want to use this in pipelines, I imagine ..
I always found it weird that we don't have glacial type input for llms or any kind of active-working memory.
There's no reason why we shouldn't be able to expose active relevant information that is only relevant for the next request: current agents running, time, etc.
There's also no reason why we shouldn't have a cheaper lossy input which uses way less bytes per token - see deepseek flash 4.1.
I really like this approach! I sort of think of the vision encoder here as an expensive high fidelity RAG encoder.
The thing I’d love to do with a system like this is train it to be KV cache ordering independent (ie permutation invariant at the page level). Basically each page’s KV cache should be understandable by the model in any ordering - which would allow you to go one step further and treat the KV cache of the vision encoded page as the chunk for the model to reason over.
Then all these zoom in for more detail tricks will extend naturally.
Oh My Pi has done this for a while now, they call it Snap compact.
I remember reading a paper entitled "A Picture is Worth a Thousand Tokens" or something similar like 2-3 years ago. The reality is that no one really wants/needs contexts that big, anyway. It's hard enough making LLMs truly useful even with a small/medium context.
yep, deepseek
Ha! Didn’t realize that’s what it was doing, I’d compact and it would say snap compact with a little icon of a camera, so this all makes sense now.
Taking a bit of a longer-stick approach to AI/ML, what I am struck by is that this is clearly a utility model intended for use alongside other models/tooling, and - has the 'feel' of being a bit like as if someone composed a batch script to do something important - and I mean that in a good, positive, useful way, just to be clear.
We are living in the age where people can compose a model to do almost anything you can describe, literally. I personally have accrued a rough 'gang' of models I use to check each other before they wreck each other, and nothing gets past a repo review cycle. All I am now is the final builder, distributable packager, for a fleet of little purpose-built byte-chuckers.
Am I correct in assuming that a principle use case for LensVLM is in the training side of things .. ?
Say, what? When would a compressed image of text be smaller than just text?
Maybe to save on doing the client-side layout and rendering?
I'm reminded of the MSPaint IDE:
https://www.youtube.com/watch?v=eyH4aXlB1Js
https://ms-paint-i.de/
>Say, what? When would a compressed image of text be smaller than just text?
Since every character in text shares compressible ligaments with every other character, maybe? Or, more finite, every pixel representing a ligature component in a character can be compressed against other pixels in the path set.
I think, at scale, this is very important - folks training models on near-petabyte sized corpus would have cause to want to use this in pipelines, I imagine ..
I always found it weird that we don't have glacial type input for llms or any kind of active-working memory.
There's no reason why we shouldn't be able to expose active relevant information that is only relevant for the next request: current agents running, time, etc.
There's also no reason why we shouldn't have a cheaper lossy input which uses way less bytes per token - see deepseek flash 4.1.
Interesting approach. thanks
[flagged]
[dead]