A very long time ago I wrote an Eclipse plug in that would read from source comments an ascii diagram describing the state/transition flow of a java class object, and generate/update the necessary state fields and transition methods.
It was pretty cool in principle but nearly unworkable in practice, purely because maintaining an ascii diagram in a text editor for anything more than "Hello world" is a massive PITA.
The simple text editor has a lot to answer for when it comes to how we think about programming, in a Sapir-Whorf kind of way. It's a shame there has been no cast iron standard for mixing live text and visuals. Our industry might look very different.
> The simple text editor has a lot to answer for when it comes to how we think about programming
I wonder whether those that used punch cards said the same about those punch cards?
It’s weird how our mobile devices aren’t programmed using keyboards/text as input devices. Or our microwaves. Or refrigerators.
Programmers are stuck with text because their programming paradigms are stuck in a text-based paradigm, hence AIs spew out reams and reams of simple to understand text.
There is definite room for improvements and room for keyboards. However the focus should move on from keyboard to mouse to XR environments using 3D glasses. Our programming paradigms have to move aswell.
Using a visual first paradigm means using higher level constructs, things such as Blockly from Google isn’t a good example of visual programming. For music, things such as noisecraft are. For more general programming approach, Node-RED is a good example of visual programming based on flow based programming paradigm which is well suited to visual programming.
> I wonder whether those that used punch cards said the same about those punch cards?
Oh, absolutely. One statement/line per card, the first 5 columns reserved for labels (how's that for significant whitespace?), the 6th to signal line continuation, the last eight for arbitrary comments/line numbers? People loathed it.
Punch cards are just text editors in primitive form though, ie columns across and lines down. All very 2D.
A visual programming canvas is free of that, and thus can display connections and relationships far more deeply and conveniently than text-based languages.
It's the 2D nature of our languages that has a lot to answer for.
The language was just python and the IDE swapped out certain functions' arguments with the image and targets within the image that the code referred to. It also had a bunch of built-in functionality that let you manipulate the images and targets (the little red crosses in that screenshot) directly within the IDE instead of having to edit the underlying python code.
Looks like it has a successor called Oculix but I know nothing about that.
That's interesting. The most I had to do with SikuliX recently is its Robot Framework library, which is very neat way of both keeping the tech somewhat alive and making it easier to write the scripts.
Every to often a project will come along which reminds me of the visions which Hermann Hesse's _The Glass Bead Game_ (originally published as _Magister Ludi_) inspired in me --- this looks to be the latest.
Looking forward to reading the paper and hope that it is widely read/discussed, and that it influences projects/developers such as: https://www.inkandswitch.com/
I'd dearly love to see a 3D CAD program which used this methodology.
Visual languages often fail when it comes to concurrent development where the state of the art is textual diff and merge. The IBM Rational suite attempted visual model diff/merge but I recall it being unworkable in practice.
Modern collaborative visual tools like Miro sidestep the concurrent development problem by being live multiplayer and by essentially having no version management at all.
Most visual BPM tools that I'm aware of try to mix flowchart style programming with e.g. Javascript on activity nodes, but they fail provide any developer-time syntax checking or completion. They also tend to serialise to XML which is unworkable in practice for diff/merge of the visual logic.
In a previous job we developed an in-house flowchart based language that embedded Javascript. We put the effort into writing a first-class Eclipse plugin that had syntax checking and completion across all graphical and Javascript constructs. It even had interactive debugging that interleaved graphical and textual single-stepping. We never solved the diff/merge problem in a satisfactory way.
for a long time I've believed that the way to bridge the visual and the textual worlds is to generate the visuals from a plain text source of truth. I've never gone so far as to make a visually editable programming language, but I have done multiple projects where, for example, process diagrams in an IDEF0 style were generated from human readable text propositions of the form:
Prepare Meal is a Process
Oven enables Prepare Meal
Utensils enable Prepare Meal
Prepare Meal transforms Raw Ingredients
Prepare Meal produces Finished Meal
Menu governs Prepare Meal
Customer Order governs Prepare Meal
You can map out a business process using a very simple (and XML-free, and diffable) plain text DSL and generate interactive diagrams from it. My most recent example is for concept maps along similar lines.
I work daily in TouchDesigner and am constantly wrestling with the integration of its node and Python scripting contexts. In particular order of operations, delegation of responsibility, and store of state are very unsolved areas when expressing intent across both domains.
So cool! Love that Clojure(Script) is the language of choice here. Seeing a graphical tile nestled in a function call blew my mind for a hot minute, there.
I feel it would be remiss of me not to mention the humble TempleOS support for mixing code and sprites. I can't say I've ever used the system myself, but its DolDoc format [0] offered some pretty nifty features for directly interspersing sprites into source code viewable together as one. You can see some screenshots on the hackmag page [1] under the section that reads: "If that’s not enough to impress you, open any game file and behold the unthinkable: the sprites are embedded right in the code!"
because it's unusable in practice. It's fun as an idea and as a toy, but when you try to use such a visual system to do work, you quickly realise that text is way more flexible and simpler to operate on.
Oh dear. I think they may fall over in surprise when they realize that this technology become commercially viable while they were busy writing this academic paper...
I don't know, my vibe coder friends always talk about wanting a visual programming language so they can understand the systems they make.
So I feel like your last sentence doesn't really track - yes they are like you
, just telling specifications to the machine, but they can struggle for hours trying to get those specific instructions right because they can't understand what their systems are doing.
You won't find it in any of the academic literature because it's not an academic project: https://bablr.org/
BABLR is just the extensible streaming parser framework, but it has a few key things going for it:
1. Gap support like Hazel has, but for any language it can parse
2. Streaming parse results, which make multi-pass stream transformation easy
3. The ability to consolidate and take over the work currently done by Treesitter, LSP, Git, and the IDE itself.
"BABLR is a parser framework roughly comparable to Tree-sitter, but built from the ground up for the web"
I have to admit, I don't know why I would stop using my wasm build of treesitter that works amazing on the web for something that is "conditionally production ready". Also I don't see where your project mixed visual and textual code like this paper here explores?
Yeah it's by me. It isn't the purpose of the project to be able to make visual widgets, just an ability that arises naturally when your state layer holds both structure and text.
You are bringing the idea that a specific programming language must be made to do visual programming, not me. That is the opposite of what I think!!
If your solution to global problems is for everyone in the world to adopt Hybrid Clojure Script as their programming language, you don't really have an economically viable solution.
> I think they may fall over in surprise when they realize that this technology become commercially viable while they were busy writing this academic paper...
"this technology" is an actual concrete language that mixes text and graphics. Not a version control system or an extensible parser framework or whatever it is that you have.
Maybe one day you will build an actual language that mixes text and graphics but unless you can show me a screenshot of that it seems like a dick move to say that they've wasted their time because you've already got a commercially viable version, when actually you haven't at all.
I certainly invite people to disrespect me for superficial reasons, and that is on purpose. The people who actually judge me on my merit I find to be much more worth my time.
I assume he's referring to the massive commercial success of Holy-C and TempleOS.
(It's the only programming language with inline graphics I can think of, at least, your average esoteric visual language tend to not mix with normal code.)
I was going to mention holyC here, but I don't think it is the same thing. holyC lets you embed images etc in the code, but those images don't become part of the logic. This paper is talking about mixing graphical programming languages with textual ones (like a file with both scratch blocks and c code in)
Yeah I'm not talking about that. I'm thinking more like "a C program is just a parse tree, so why couldn't a visual tool help you visualize or rearrange the parse tree?"
This academic paper assumes two largely unmixed disciplines just barely kissing each other. I see a complete blending coming that will melt the boundaries between textual and visual programming.
Once you start thinking like this, you stop seeing semantic editing as a feature of a particular language: it should just be a native ability to have on any parseable code: the ability to reassemble the parse tree nodes like lego bricks.
I entirely agree with you, and I'll raise you one -- coding via AST's lets you formally note and track the boundary/linking facts and concepts via comments. That might finally get us out of the stone-age!
A very long time ago I wrote an Eclipse plug in that would read from source comments an ascii diagram describing the state/transition flow of a java class object, and generate/update the necessary state fields and transition methods.
It was pretty cool in principle but nearly unworkable in practice, purely because maintaining an ascii diagram in a text editor for anything more than "Hello world" is a massive PITA.
The simple text editor has a lot to answer for when it comes to how we think about programming, in a Sapir-Whorf kind of way. It's a shame there has been no cast iron standard for mixing live text and visuals. Our industry might look very different.
> The simple text editor has a lot to answer for when it comes to how we think about programming
I wonder whether those that used punch cards said the same about those punch cards?
It’s weird how our mobile devices aren’t programmed using keyboards/text as input devices. Or our microwaves. Or refrigerators.
Programmers are stuck with text because their programming paradigms are stuck in a text-based paradigm, hence AIs spew out reams and reams of simple to understand text.
There is definite room for improvements and room for keyboards. However the focus should move on from keyboard to mouse to XR environments using 3D glasses. Our programming paradigms have to move aswell.
Using a visual first paradigm means using higher level constructs, things such as Blockly from Google isn’t a good example of visual programming. For music, things such as noisecraft are. For more general programming approach, Node-RED is a good example of visual programming based on flow based programming paradigm which is well suited to visual programming.
I can assure that mobile devices and embedded systems are programmed with text and keyboards as well.
As a small challenge you could try expressing the contents of your comment using visuals only.
> As a small challenge you could try expressing the contents of your comment using visuals only.
Hah! That's a trick question, isn't it, because HN filters out emojis?
I mean just as an exercise. Also, emojis (as well as hieroglyphics ) are still text.
> I wonder whether those that used punch cards said the same about those punch cards?
Oh, absolutely. One statement/line per card, the first 5 columns reserved for labels (how's that for significant whitespace?), the 6th to signal line continuation, the last eight for arbitrary comments/line numbers? People loathed it.
Punch cards are just text editors in primitive form though, ie columns across and lines down. All very 2D.
A visual programming canvas is free of that, and thus can display connections and relationships far more deeply and conveniently than text-based languages.
It's the 2D nature of our languages that has a lot to answer for.
SikuliX was a UI automation tool from over a decade ago with an IDE that looked similar to their screenshots: https://avleonov.com/wp-content/uploads/2017/02/script_windo...
The language was just python and the IDE swapped out certain functions' arguments with the image and targets within the image that the code referred to. It also had a bunch of built-in functionality that let you manipulate the images and targets (the little red crosses in that screenshot) directly within the IDE instead of having to edit the underlying python code.
Looks like it has a successor called Oculix but I know nothing about that.
That's interesting. The most I had to do with SikuliX recently is its Robot Framework library, which is very neat way of both keeping the tech somewhat alive and making it easier to write the scripts.
Every to often a project will come along which reminds me of the visions which Hermann Hesse's _The Glass Bead Game_ (originally published as _Magister Ludi_) inspired in me --- this looks to be the latest.
Looking forward to reading the paper and hope that it is widely read/discussed, and that it influences projects/developers such as: https://www.inkandswitch.com/
I'd dearly love to see a 3D CAD program which used this methodology.
Hm...the screenshots don't really sell it;
Maybe the authors should just vibe code a cljs port and put it in a browser?
And showcase some program written in this language that sells it better?
Yes, I think the best avenue for exploration in this space would be to start with literate programming tools like Jupyter notebooks or Observable.
https://new.observablehq.com/@observablehq/notebooks-2-0-sys...
Visual languages often fail when it comes to concurrent development where the state of the art is textual diff and merge. The IBM Rational suite attempted visual model diff/merge but I recall it being unworkable in practice.
Modern collaborative visual tools like Miro sidestep the concurrent development problem by being live multiplayer and by essentially having no version management at all.
Most visual BPM tools that I'm aware of try to mix flowchart style programming with e.g. Javascript on activity nodes, but they fail provide any developer-time syntax checking or completion. They also tend to serialise to XML which is unworkable in practice for diff/merge of the visual logic.
In a previous job we developed an in-house flowchart based language that embedded Javascript. We put the effort into writing a first-class Eclipse plugin that had syntax checking and completion across all graphical and Javascript constructs. It even had interactive debugging that interleaved graphical and textual single-stepping. We never solved the diff/merge problem in a satisfactory way.
for a long time I've believed that the way to bridge the visual and the textual worlds is to generate the visuals from a plain text source of truth. I've never gone so far as to make a visually editable programming language, but I have done multiple projects where, for example, process diagrams in an IDEF0 style were generated from human readable text propositions of the form:
Prepare Meal is a Process Oven enables Prepare Meal Utensils enable Prepare Meal Prepare Meal transforms Raw Ingredients Prepare Meal produces Finished Meal Menu governs Prepare Meal Customer Order governs Prepare Meal
You can map out a business process using a very simple (and XML-free, and diffable) plain text DSL and generate interactive diagrams from it. My most recent example is for concept maps along similar lines.
I work daily in TouchDesigner and am constantly wrestling with the integration of its node and Python scripting contexts. In particular order of operations, delegation of responsibility, and store of state are very unsolved areas when expressing intent across both domains.
Related research: Elliot Evans' Polytope (https://elliot.website/editor/).
https://bsky.app/profile/watwa.re/post/3lizpdz34522g
I'm working on a similar idea, it's been a while since I published about it but open to input and collabs!
So cool! Love that Clojure(Script) is the language of choice here. Seeing a graphical tile nestled in a function call blew my mind for a hot minute, there.
I feel it would be remiss of me not to mention the humble TempleOS support for mixing code and sprites. I can't say I've ever used the system myself, but its DolDoc format [0] offered some pretty nifty features for directly interspersing sprites into source code viewable together as one. You can see some screenshots on the hackmag page [1] under the section that reads: "If that’s not enough to impress you, open any game file and behold the unthinkable: the sprites are embedded right in the code!"
0) https://tinkeros.github.io/WbTempleOS/Doc/DolDocOverview.htm... 1) https://hackmag.com/security/templeos
We have the complete AST in almost every IDE. Why are we not learning to render abstracted code and systems from that??
because it's unusable in practice. It's fun as an idea and as a toy, but when you try to use such a visual system to do work, you quickly realise that text is way more flexible and simpler to operate on.
Oh dear. I think they may fall over in surprise when they realize that this technology become commercially viable while they were busy writing this academic paper...
I mean visual programming is standard in game development. Did not get traction in the rest of the dev community.
Now it does not matter anyways. We are not reading/ writing code anymore. Just specifying it and testing it.
Grasshopper is massively successful in the CAD space. If you’ve looked at a building and thought “that doesn’t seem possible”, it’s probably grasshopper. See: https://www.grasshopper3d.com/page/architecture-projects
But I drive grasshopper with Claude code, so still 100% agree with your last graf.
Yeah but that's pure visual programming, not a hybridization of the best parts of visual and textual programming.
Anyone who thinks code doesn't matter anymore is selling something. Trying saying "math won't matter anymore" while clicking your heels three times
I don't know, my vibe coder friends always talk about wanting a visual programming language so they can understand the systems they make.
So I feel like your last sentence doesn't really track - yes they are like you , just telling specifications to the machine, but they can struggle for hours trying to get those specific instructions right because they can't understand what their systems are doing.
Can you elaborate?
You won't find it in any of the academic literature because it's not an academic project: https://bablr.org/
BABLR is just the extensible streaming parser framework, but it has a few key things going for it: 1. Gap support like Hazel has, but for any language it can parse 2. Streaming parse results, which make multi-pass stream transformation easy 3. The ability to consolidate and take over the work currently done by Treesitter, LSP, Git, and the IDE itself.
So bablr is from you?
"BABLR is a parser framework roughly comparable to Tree-sitter, but built from the ground up for the web"
I have to admit, I don't know why I would stop using my wasm build of treesitter that works amazing on the web for something that is "conditionally production ready". Also I don't see where your project mixed visual and textual code like this paper here explores?
Yeah it's by me. It isn't the purpose of the project to be able to make visual widgets, just an ability that arises naturally when your state layer holds both structure and text.
That does not look like a visual programming language?
It isn't! It's a version control system and IDE state layer that can make EVERY programming language visual at the same time
So, where is this commercially viable visual programming language you claimed exists?
You are bringing the idea that a specific programming language must be made to do visual programming, not me. That is the opposite of what I think!!
If your solution to global problems is for everyone in the world to adopt Hybrid Clojure Script as their programming language, you don't really have an economically viable solution.
Your original comment was:
> I think they may fall over in surprise when they realize that this technology become commercially viable while they were busy writing this academic paper...
"this technology" is an actual concrete language that mixes text and graphics. Not a version control system or an extensible parser framework or whatever it is that you have.
Maybe one day you will build an actual language that mixes text and graphics but unless you can show me a screenshot of that it seems like a dick move to say that they've wasted their time because you've already got a commercially viable version, when actually you haven't at all.
...Their name might be descriptive...
I certainly invite people to disrespect me for superficial reasons, and that is on purpose. The people who actually judge me on my merit I find to be much more worth my time.
My judgment is that you were lying in your original comment, to draw attention to one of your unrelated projects. Then I noticed your name.
I assume he's referring to the massive commercial success of Holy-C and TempleOS.
(It's the only programming language with inline graphics I can think of, at least, your average esoteric visual language tend to not mix with normal code.)
You can use images inline in Racket. Decidedly less esoteric :)
I was going to mention holyC here, but I don't think it is the same thing. holyC lets you embed images etc in the code, but those images don't become part of the logic. This paper is talking about mixing graphical programming languages with textual ones (like a file with both scratch blocks and c code in)
Yeah I'm not talking about that. I'm thinking more like "a C program is just a parse tree, so why couldn't a visual tool help you visualize or rearrange the parse tree?"
This academic paper assumes two largely unmixed disciplines just barely kissing each other. I see a complete blending coming that will melt the boundaries between textual and visual programming.
Once you start thinking like this, you stop seeing semantic editing as a feature of a particular language: it should just be a native ability to have on any parseable code: the ability to reassemble the parse tree nodes like lego bricks.
I entirely agree with you, and I'll raise you one -- coding via AST's lets you formally note and track the boundary/linking facts and concepts via comments. That might finally get us out of the stone-age!