I'm glad Figma is pushing in this area. I do think that Figma has one of the best UI datasets out there in the world, and for the most part they've been sitting on it letting it get unused. Excited to see where they take this, but I do think they should be looking at image models in addition to LLM-based edit models (I'm biased as I left figma to start a image-mode-based UI design tool) as I find they're more capable in this area.
Totally agree with bitter lesson problems here. It’s clear one of the next things taking off right now is the ability to understand and use graphical UIs. I bet they could find some way to get OpenAI to specifically train on Figma, it’s a good use case on both sides. They might do it anyway I guess.
So... I'm not quite sure why I would use this over using GPT/Claude/Gemini directly for design?
Bitter lesson, etc, and training an entire VLM to prevent text overlap seems a bit much, especially since GPT 6.* /Claude 5.*'s vision are good enough that it rarely ever happens for me, and if they do make a mistake, it's generally a 5 minute fix?
I work as a product designer, and your right, mostly people aren't using Figma's AI products (at least in my circles). 80% of my time is spend in Claude Code/Design. 20% in Figma doing manual work when I really need to explore things and spec out in detail.
Figma's current AI Agent is essentially useless, especially when compared to these other tools. Not sure what effect the work in this post will have, but even if it's great I feel that designers have already left Figma and are working elsewhere.
You've misunderstood the bitter lesson if you think it applies here. It's not "frontier LLMs are always better", it's "algorithms that scale with compute/data will usually end up surpassing algorithms that are based on human understanding".
Fine-tuning an open-source LLM with reinforcement learning in order for it to be better at a specific task does not go against the bitter lesson at all.
Ex-Figma.
For context, this blog was released the same day Canva made their announcement for Brush, which looks to be further along in this area: https://www.canva.com/newsroom/news/canva-brush/
I'm glad Figma is pushing in this area. I do think that Figma has one of the best UI datasets out there in the world, and for the most part they've been sitting on it letting it get unused. Excited to see where they take this, but I do think they should be looking at image models in addition to LLM-based edit models (I'm biased as I left figma to start a image-mode-based UI design tool) as I find they're more capable in this area.
Totally agree with bitter lesson problems here. It’s clear one of the next things taking off right now is the ability to understand and use graphical UIs. I bet they could find some way to get OpenAI to specifically train on Figma, it’s a good use case on both sides. They might do it anyway I guess.
So... I'm not quite sure why I would use this over using GPT/Claude/Gemini directly for design?
Bitter lesson, etc, and training an entire VLM to prevent text overlap seems a bit much, especially since GPT 6.* /Claude 5.*'s vision are good enough that it rarely ever happens for me, and if they do make a mistake, it's generally a 5 minute fix?
I work as a product designer, and your right, mostly people aren't using Figma's AI products (at least in my circles). 80% of my time is spend in Claude Code/Design. 20% in Figma doing manual work when I really need to explore things and spec out in detail.
Figma's current AI Agent is essentially useless, especially when compared to these other tools. Not sure what effect the work in this post will have, but even if it's great I feel that designers have already left Figma and are working elsewhere.
The Figma MCP is so bad, it is almost unusable. Letting the calling AI write some javascript to implement a design?
Bitter lesson learned in 3... 2..
I know Dylan Field recently posted they made the Llama's 70B model do amazing things but the AI training field is an absolute snake pit.
But then again they have money to burn ig.
You've misunderstood the bitter lesson if you think it applies here. It's not "frontier LLMs are always better", it's "algorithms that scale with compute/data will usually end up surpassing algorithms that are based on human understanding".
Fine-tuning an open-source LLM with reinforcement learning in order for it to be better at a specific task does not go against the bitter lesson at all.
they need to use nemotron for true sovereignty and stop paying the tax (just like salesforce and servicenow)