I highly recommend watching "Its Quieter in the Twilight" (https://www.dailymotion.com/video/x9rrxr)
a cool documentary about the engineers still running voyager (a lot of them have been on it for decades)
As the should. Voyagers are still active and this maintenance is needed in case issues occur. In a way due to the +24 hours oneway communication to correct software issues should they occur, this will help speed corrections up.
Now I wonder how the test it ? Is it on a software emulator on modern equipment or do they have a Voyager replica ?
What they have is surprisingly fragmentary. For the recent software issue, they had to be very conservative in what they changed because they aren't 100% sure about the details of the ISA on the actual spacecraft (due to amiguity between different revisions of the wiring of the CPU), and so they don't really have a simulator they can trust.
NASA hasn't had infinite budget for a long time. They didn't even have it when they were building the voyager probes (they were the budget version of what they wanted to build).
They have a simulator for one of the three computers on Voyager, the Computer Command Subsystem. They don't have a simulator for the other two computers, the Attitude and Articulation Control Subsystem and the Flight Data Subsystem. They used to have a Voyager replica, the Capability Demonstration Lab, but they got rid of it after they moved the Voyager team into a satellite office following the end of the main mission. There's more information here if you're interested: https://arc.aiaa.org/doi/pdf/10.2514/6.2016-2415
That would be an interesting project, I assume a lot of not so young engineers will want to play with it, as a hobby. Also if that project exists, I'm sure that someone will try to port DOOM.
Per the article, “much of the original documentation has been lost or fragmented. Voyager paperwork from the 1970s and 1980s was largely paper, and each time the project moved offices, more of it disappeared”.
Maybe not years ago, but scanning documents with the phone in your pocket has become incredibly efficient. That combined with AI transcription and indexing for search makes such a project faster and cheaper in 2026 than at almost any other time in the past.
I scanned an entire book in 2016. It was way faster than scanning with the phone in my pocket. It’s just not ergonomic to use a general purpose device (a phone) instead of a specialized device (dedicated scanner).
Agreed. I scanned a short book with my phone, and a dedicated scanner would have been nice to have.
But with page flattening and separation and automated capture, it went much faster than I would have thought. If I were going to do a lot more, I'd want something like a scan tent [1]. It's not as ergonomic as a dedicated solution, but in 2026 a phone and some light can get you a lot of the way there, pretty fast.
I mean, they were never meant to last this long. NASA has a shoestring budget. I understand not taking the time and resources to do that when it could stop working a week later.
Regarding "that almost nobody on Earth fully understands anymore", I claim this is nonsense, and definitely not an obstacle.
I've audited codebases in languages that I haven't programmed in. It is a matter of grasping a few basic concepts, like branch execution, branch destination, where data is stored, how it is communicated. Don Lancaster told us how to do this: https://www.tinaja.com/ebooks/enhance_vI.pdf.
I was puzzled by this claim, too. I think that the article is wrong, and that the code is written in HAL/S, a NASA-only language that sort of started off as a preprocessor to Fortran, though it has some PL/I-like features. If it really was written in Fortran, it probably was in a vendor-extended Fortran IV, which a lot of old guys like us know. But NASA used HAL/S for a number of projects, including the Shuttle. (And, by the way, thanks to perplexity.ai, here's a link to the HAL/S language manual: https://archive.org/details/nasa_techdoc_19750002029.)
The parent comment is apt. Of course, languages have their own quirks. But, as Christopher Strachey is once claimed to have said, “I use the same language no matter what compiler I run.”
Now what is more likely to be true is that the code is strangely structured (both because structured programming was new then, and because of memory and processor limitations), and also that much of the internal documentation has been lost. I wish the article had been clearer on that.
I think that the Voyager software is in assembly languages (there are several distinct computers on board), and that it is the program preparation software is written in a “Fortran V” extension.
> Regarding "that almost nobody on Earth fully understands anymore", I claim this is nonsense, and definitely not an obstacle.
It seems like you aren't very familiar with the history of the Voyager or the computer systems it uses. The Voyager has three onboard computers, each of which has a custom ISA. Two of the three computers were only used on Voyagers. After the main Voyager mission ended in the late 1980s and they were repurposed to collect data on interstellar space, the probes were reprogrammed to only require limited commands on one of the computers, and no intervention whatsoever on the other two. They also got rid of the testbench for testing code on the ground, so the only working Voyager systems are billions of miles away from Earth. Since then, Voyager has been operated by a skeleton crew that has been shrinking over time as people retire. The result of all of this is that they legitimately do not have a full understanding of how the hardware and software operates. Here's one example from a talk about how the crew fixed an issue with the Flight Data Subsystem, one of the computers that hadn't needed any significant software changes since the interstellar mission began (https://www.youtube.com/watch?v=dF_9YcehCZo):
> So our top priority was to figure out what the FDS was and how it worked. Because unfortunately the person who was the real expert had retired decades ago and the person who was their fill-in, their backup, had retired two years ago. So it was the worst place it could have hit us. These are some examples of some of the documentation we dug up on the FDS. So they're all hand written. These are some very dim circuit diagrams and these are hand written timelines for how to change FDS processors. We were lucky to find these. Some things we couldn't find. A lot of it was like this, that had been scanned. And frequently, these sources were contradictory, ambiguous. Why? Because we changed the way the spacecraft worked with every planetary encounter and entering the VIM and when things broke. So there was a lot of opportunity for ambiguity to arise in the documents over the course of 50 years. This is my favorite example. So this is a page out of an important FDS document. The change bar on the far side indicates it's a change, but somebody went in and made a very cryptic circle of the sentence and crossed it out. I have no idea to this day what it means. I have no idea. Maybe it was important. Maybe they crossed it out, because they thought crossing out was important. I'm not sure. So there were other cases like this. So we were even so confused in some cases, we weren't sure if we were sending data to the FDS - should it be least significant bit first or most significant bit first? We had that level of uncertainty.
> We didn't know we had the right instruction set. I showed you the instruction set there. We didn't know that was the one used to build the software, back when there was an assembler. We didn't know the source code version was the same as what was running onboard the spacecraft. We had a listing of the code, but it was a Microsoft Word document, with optical character recognition scans. We didn't know if there were errors in it. We had no assembler. So they're just playing with bits. There's no simulator and there's no test bed. So there were a few challenges. So basically this was bare knuckle binary manipulation. And they had to think of all the problems. I'm sure you guys are far better than I am of thinking of the problems associated with playing with memory like this. You could overwrite something. You can fail to recognize jumps. How do you debug it in flight on a flying spacecraft? And how do you know you're not missing something in those instruction sets and codes? And the list goes on and on.
>NASA still maintains some of the Voyager spacecraft code in a 1970s-era programming language that almost nobody on Earth fully understands anymore
I did not address the issue of the hardware documentation, just the issue raised by the headline. I fully understand the issues that the hardware and the problems of not having a full map of the hardware or a test bed to check out the code.
Well the 1970s era programming language would be the assembly language for the bespoke Voyager processors, and the spotty institutional knowledge and lack of a Voyager on the ground means that there's no full understanding. I'm not sure what's inaccurate here.
> Microsoft Word document, with optical character recognition scans
It's incredible that there is someone out there in this world who thought that OCRing documentation into a Word document is a good idea. And implemented it.
> younger engineers often have the capability but not the inclination
Kids these days... Why would someone in their right mind think working on the Voyager project could damage their careers? You can work on new and fancy tools all you want to improve supporting tools, and it's still one of the coolest space missions active. Plus, it has a real end - at some point, support will be further reduced and the person will move on to another space exploration job, with the extra golden star of having been on the Voyager.
I would go further... This project gives a rare opportunity for a young engineer to learn to build truly mission critical, resilient software while requiring complete, top to bottom understanding of the software and hardware stack.
I am no longer a junior, but would have been upset to be tasked with refreshing the old historical obsolete laundry (no matter how sacred or distinguished), expecially when I already had experience delivering safety critical products packing much more modern technologies.
The opportunity they would be offering is not rare at all! The opportunity to research and design something truly new on the other way is very scarce.
Also, many decisions taken Probably can be traced to limitations / idiosincracies of the era
And you're left with a codebase that has been in hands of 6 Decades of probably great engineers that have already done a lot, plus any of the arcane cruft of such a long lived and esoteric project
It's a great CV highlight, but I don't know if it's the best learning opportunity
What have you worked on that is as cool as a space probe that's cruising in interstellar space and still collecting valuable data?
There are a lot of things as cool as, done by people I know, such as the gyros on the Webb telescope, the APU in the F-35, or a small rack-mountable Cesium reference clock, but there aren't many opportunities like that.
That's the thing. You only have the cool factor, but that wears off very quickly when you are maintaining legacy code and tools and then your collegues are playing with the new hot and shiny toys.
I won't write about the projects I've been involved with for privacy, but to give you an idea some of my old team members were involved in ams-02 for example.
Is that because juniors want to leave their name on something? I ask honestly since I shared a lot of the same sentiment as you, and never quite got an understanding as to why working on the cool new thing was "more fun" even if a lot of the projects under-the-hood were recycled.
> Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand.
> We’re not excited by incremental renovation: tinkering, improving, planting flower beds.
>> The opportunity they would be offering is not rare at all!
The opportunity to maintain software running on a spacecraft is not rare? I don't think so. And those two particular spacecrafts? I'd take that job in a flash.
I agree, and I would think the same, but I also feel like many things I've been sold as "door openers" for interviews unfortunately tend to ultimately be things that no one cares about.
I think people tend to squander door openers with bad layouts or information density. Most CVs are essentially the same as each other, just a bullet point list of jira ticket titles.
Do I care if someone has won the world championship for ping-pong 3 years in a row? Not particularly. Does it make them stand out against a sea of slop? Only if I actually see that info when skimming! But if I do see it, I'm probably going to stop and re-read the whole thing, which is a tactical advantage.
So you're giving me an idea for my midlife career change CV. Lead with the cool and interesting fact above all else, then have the normal CV menu fare at the end.
Hmm this is also drumming up the hard question of: What have I actually done which is ACTUALLY an attention-grabber...
I would assume they have built some key problem-solving skills that can be valuable. Training in tooling is much easier than building the right mindset.
It's a tragedy some of our best minds are dedicated to that, and digital surveillance so their corporate masters can sell better targeted ads with a higher click-thru rate.
I really think it's overly generous to throw people who do not understand the importance of extending Voyager's scientific mission into the "our best minds" category. I guess I agree with the larger point.
> Kids these days... Why would someone in their right mind think working on the Voyager project could damage their careers?
It's an isolated legacy-project with no future. Mostly everything you learn for it will be only useful for this specific project, so all time to invest there is time to can't invest into something useful. Sure, there are probably some parts to learn from this too, but It's less than what your competitors will learn on their fancy modern projects.
> You can work on new and fancy tools all you want to improve supporting tools
Voyager is in maintenance, there is no big innovation or big progress to be made there. It's just work to hold the line as long as possible. And I guess nobody want's to be the one killing it because of a poor attempt to innovate something.
Understanding assembly language (any architecture) give insights into how computers work that are still valuable and relevant today. Almost nobody still writes code in assembly, but understanding it at least conceptually is still worth something.
If I were hiring, I would almost always prefer a candidate who had some experience at machine/assembly programming to one that didn't, all else being roughly equal.
I've been a developer for 20 years, I do some reverse engineering stuff on the side using assembly.
There hasn't been a single time in 20 years that it was actually relevant for real work in any way.
Unless you are doing very specific work knowing assembly is about as useful as knowing COBOL (which is also useful for a very specific kind of work that most devs will never do)
Learning new and weird things builds brain elasticity and vocabulary for you to express new ideas. I always recommend people to learn FORTH, or Lisp, or APL. Learn to think with different paradigms.
As an extracurricular activity sure, but if your resume just has FORTH on it recruiters are going to throw it in the garbage because there's no evidence you can work in a modern development environment
You should hire for personality characteristics, not knowledge. I'll take anyone who's worked on a weird obscure system and figured it out from first principles over Front End React Dev #8482828 with Opinions on algebraic effects.
> You should hire for personality characteristics, not knowledge.
I'm not sure what personality characteristic are supposed to be in this context. But if you mean skills, then you have a false assumption. NASA and the surrounding industry is not really the environment where they grow boring react devs. For every legacy project like voyager, they have a dozen frontier-projects and other weird obscure jobs where one can gain the same skills, but better, with more potential for further growing. And someone early in their career will naturally choose something with more potential for their future.
You don't need to push the needle as a junior though. It's a "completed project" where one can glean many insights. And the matter of transferrable skills is simply a matter of being able to say how what you learned applies in a different context. A useful skill in-and-of-itself
> You don't need to push the needle as a junior though.
Doesn't matter. Many young people have strong motivations to shine and push, especially when they are in high profile-jobs. Legacy-projects are simply not alluring enough for this.
Because software development back in the day wasn’t like how it’s now now, the charade so called software development now is a clown show: scrum, daily stand ups, open office style, tickets, tons of ci/cd BS, and of course, the wrangler aka PM and all politics involved, none of this existed like the cult it is now, I only had one experience in such environment and despite the effort I had to ask for some common sense, it was like insulting someone’s religion, “how dare you challenge the sacred methods that the silicone valley companies are using?!!”
Additionally, back in the day there was true ownership for the code you write, the code is owned by you not the company, and I know few old engineers that until now (they are retired) the companies still pay them for using their code they wrote while working there. That sense of ownership encourages you to tackle hard issues rather feeling like a machine spewing code for someone else’s business, I have seen some contracts too where the company will have ownership for anything you do while you are in the contract, including your personal projects on your own free time.
Or like many a company that prefers to paper old hands with cash to keep an old system running until it is really, really no longer feasible.
Banks and airlines are the most common example, many of them run on mainframe systems with code that's old enough for humans to go into retirement - and replacement projects usually tend to go into the billions of euros range with many of them failing catastrophically.
Even paying some greybeard 500k a year to deal with that stuff despite him being retired is far more profitable in the short term. That's the problem with letting beancounters run the show because eventually, there will literally be barely anyone left alive who is capable - and even less who know all the "implicit knowledge" behind edge cases.
Unfortunately in many places it is indeed a cult and serves to ossify management decisions. We ARE doing Agile, what do you mean? No this person is the scrum master and they tell us what Agile is and then we do it, see?
I have worked at 8 different software places and none of them implemented things in a way I would call "genuinely agile" and most of them were just bad waterfall with more meetings and telling the engineers they are accountable for the bad ideas that are now their job.
CI/CD? How about make that only "Devops" job and then make the exact same undemocratized system before with gatekeepers who spend a significant amount of time blocking your work because they are afraid of things changing.
My point is you would never even have a clue that there is dysfunction at all without these methodologies. These days you at least have some idea in your head how it should be vs how it is.
I think there's something critical being lost in younger people who have never been exposed to the bad old ways and only understand their current situation through memes. We need to get a grip here.
That's somewhat fair, though I dont think you need to know about the beforetimes to understand that the current times get nowhere fast with all their talk of velocity.
I worked in a mainframe team at a major financial network (very explicitly not a bank) and we were agile. The entire company was agile. The company was so agile that when we moved to our new building we did hot-desking where no team had a fixed location and every worker had a little safe where they could put their valuables and leave it in modular lockers throughout the building, so that anyone could sit anywhere there was space to sit on that day.
I mean of course it didn't exactly work like that in practice because directors had to have the corners and they wanted their teams close to them, so that, e.g. our team of 15 mainframe engineers could file in to a 5 x 5 meter cubicle and one-by-one give a report on what they had done the previous day a.k.a. The Scrum. We had a certified Scrum Master.
The one time I tried to sit in a different team's space I got yelled at by their director and had to go back. I had moved in his team's space out of protest because the only space left for me to be near my team was right in front of the gents and it smelled a bit funny. I was supposed to be sitting next to my (official) Mentor but I was studying part-time for a MSc and so was not there in the mornings and another engineer would always take my place next to the Mentor, who was the most experienced Cobol programmer in the team. He always looked at me with a wry smile whenever I came in to the office and found him sitting at ... well, it wasn't really my place so I couldn't really say anything. But then the only place for me to sit was in front of the loos. So I tried to go and sit somewhere else, farther away, in protest. And then I got yelled at.
But in principle we had hot-desking and we could sit anywhere throughout the building while communicating remotely with our colleagues.
>"So you'd prefer for all this project management drama and power struggle to be invisible?"
Well project managers can have their dramas. Just do not involve developers. Or what is even better - get the fuck out and leave it to people who can do things without drama.
>"All this scaffolding is not a cult. It exists to democratize the process. Your personal comfort is irrelevant to the results."
Pile of BS. It exists to feed whole layer of self serving people who contribute very little and grow like a cancer.
In my career I was lucky. I am an independent software developer. Designed and developed many products for various clients (including some of my own ventures) and have managed without Agile, Scrum and the likes. My largest products - I had teams of up to 35 people under me and somehow we've survived.
On few occasions I had pleasure to be on some of those meetings as a visitor - felt nothing but disgust. Again luckily I was spared from direct participation
>"Why would someone in their right mind think working on the Voyager project could damage their careers"
Assembly? Understanding how things actually work? No Agile? No K8s? No Rust, No React? - death knell for someone's resume
>"and the person will move on to another space exploration job, with the extra golden star of having been on the Voyager"
this is the best case with the result of being tied to another single project for years and unemployable anywhere else. in more realistic case - warm goodbye in few years and start your life from scratch with no credits for the thins done.
> The succession problem matters most in the next decade. After that, the question becomes academic. There will be no Voyagers left to maintain.
Well, the Voyagers will still be there, there will be just no way to contact them anymore once the power runs out or communication is lost (whichever happens first)...
I don't think you want LLMs touching projects that cost over $800.000.000, even to assemble "documentation" (since the LLM can't really document in as much it's translating what it's reading, because documentation includes much more information than what's stored in the code itself).
It's a cool idea, though, I'd like to see this done as an experiment :)
My thought on this is that LLMs should be allowed to touch high stakes projects, but they shouldn't be left completely unsupervised.
Here me out.
Would you let AI manage your investments/retirement savings/etc. completely autonomously and unsupervised? Or would that make you a little nervous?
What if you had to undergo a X-Ray or a MRI. Would you ONLY want AI reading the images? Would you ONLY want a human radiologist reading the images? This is an interesting one because I would want both. In fact, I would find it questionable if AI wasn't given the opportunity to look at the images.
> i can certainly imagine that LLMs help with the process of understanding and editing old code.
Can you trust it with assembly language for a custom-built CPU with its own instruction set? Even if you digitized all of the documentation you have no idea what was lost since 1977 (or earlier), or what was never written down to begin with.
Arguably, Anthropic and Boeing are NOT what you should use to determine whether your coding practices are reasonable. Software problems with Boeing have literally killed hundreds of people due to faulty code and Claude Code is know to be a pretty buggy CLI (although extremely useful, sure).
Though I agree with the expensive factor. Perhaps, what I actually believe is that LLMs shouldn't touch code that's as mission critical as what NASA works on, even though it might be great to develop user-facing frontend software and CRUD backend code for huge corporations and projects.
> Many of the issues could potentially be solved by modern LLMs?
Yesterday I asked an LLM what customizations niri made to the KDL language.
It said niri modified the language to add single line comments with //. However if you visit the official home page of KDL https://kdl.dev/, the very first example shows single line comments as being part of the official spec. There's also a whole page dedicated to comments in the spec that mention this.
The moral of the story is LLMs are honestly really really bad and I'm sincerely concerned at how they manipulate people into thinking what they produce is accurate or trustable. I didn't believe the AI because my spidey sense said that doesn't feel right, so I double checked the real source.
It's gotten to the point where I'm finding a huge majority of the time, it provides incorrect information on really basic things. Pure hallucinations. I'm at the stage now where even if you paid me, I wouldn't use it. There's a 0% chance I'd ever consider paying for it in its current state and it's upsetting because it is killing off the web in real-time. It's becoming harder and harder to find useful and accurate information.
Of all the possible projects, this is the one where it's both highly feasible for humans to learn and historically critical that we have a full understanding and control of its operation.
I highly recommend watching "Its Quieter in the Twilight" (https://www.dailymotion.com/video/x9rrxr) a cool documentary about the engineers still running voyager (a lot of them have been on it for decades)
Video isn't there... but I found it on YT.
https://youtu.be/RIP1p5gAoak?si=ftTEGYFJscXJPaJm
Unfortunately geoblocked it seems.
https://www.dailymotion.com/video/x9rrxrg
ty!
Content not available. Any chance it wasn't like that 10 minutes ago?
he cropped one character from address, the correct address is this
https://www.dailymotion.com/video/x9rrxrg
you copypasted link missing "g" in the end
this is correct link: https://www.dailymotion.com/video/x9rrxrg
As the should. Voyagers are still active and this maintenance is needed in case issues occur. In a way due to the +24 hours oneway communication to correct software issues should they occur, this will help speed corrections up.
Now I wonder how the test it ? Is it on a software emulator on modern equipment or do they have a Voyager replica ?
What they have is surprisingly fragmentary. For the recent software issue, they had to be very conservative in what they changed because they aren't 100% sure about the details of the ISA on the actual spacecraft (due to amiguity between different revisions of the wiring of the CPU), and so they don't really have a simulator they can trust.
Maybe I'm used to think of NASA as this infinite budget agency, but considering Voyager's significance...
To me that souns like we should test all revisions of the CPU, not none of them
NASA hasn't had infinite budget for a long time. They didn't even have it when they were building the voyager probes (they were the budget version of what they wanted to build).
> Now I wonder how the test it ?
They have a simulator for one of the three computers on Voyager, the Computer Command Subsystem. They don't have a simulator for the other two computers, the Attitude and Articulation Control Subsystem and the Flight Data Subsystem. They used to have a Voyager replica, the Capability Demonstration Lab, but they got rid of it after they moved the Voyager team into a satellite office following the end of the main mission. There's more information here if you're interested: https://arc.aiaa.org/doi/pdf/10.2514/6.2016-2415
That would be an interesting project, I assume a lot of not so young engineers will want to play with it, as a hobby. Also if that project exists, I'm sure that someone will try to port DOOM.
https://github.com/Zaneham/voyager-fds-emulator
I believe they have both.
Simulators for the spacecraft and replicas for some bits of hardware.
It's mind boggling that they didn't digitize every last scrap of paper around the project years ago, for starters.
Keeping a filing cabinet in the basement is awfully cheap...
Per the article, “much of the original documentation has been lost or fragmented. Voyager paperwork from the 1970s and 1980s was largely paper, and each time the project moved offices, more of it disappeared”.
Tearing down the project is even cheaper. But presumably we're alive for a reason
Maybe not years ago, but scanning documents with the phone in your pocket has become incredibly efficient. That combined with AI transcription and indexing for search makes such a project faster and cheaper in 2026 than at almost any other time in the past.
I scanned an entire book in 2016. It was way faster than scanning with the phone in my pocket. It’s just not ergonomic to use a general purpose device (a phone) instead of a specialized device (dedicated scanner).
Agreed. I scanned a short book with my phone, and a dedicated scanner would have been nice to have.
But with page flattening and separation and automated capture, it went much faster than I would have thought. If I were going to do a lot more, I'd want something like a scan tent [1]. It's not as ergonomic as a dedicated solution, but in 2026 a phone and some light can get you a lot of the way there, pretty fast.
[1] https://www.transkribus.org/scantent
I mean, they were never meant to last this long. NASA has a shoestring budget. I understand not taking the time and resources to do that when it could stop working a week later.
NASA has a significantly larger overall budget than SpaceX for info
And far more responsibilities too. You can't just directly compare budgets
How is that at all relevant to my comment? Spacex isn't part of this conversation.
A budget is a budget and all I said was I understood digitizing being down prioritized as a project when you have to get other things done (like SLS).
I'm pretty sure volunteers will happily do this if NASA gave permission.
So it's 60 years old codebase still running. And in the two human-made objects the further away from earth.
Maybe only a few COBOL codebases still active can beat that? Or not?
Why does the picture at the top of the article look more like Cassini–Huygens than Voyager?
How else will we be able to assist the crew of the Enterprise?
Regarding "that almost nobody on Earth fully understands anymore", I claim this is nonsense, and definitely not an obstacle.
I've audited codebases in languages that I haven't programmed in. It is a matter of grasping a few basic concepts, like branch execution, branch destination, where data is stored, how it is communicated. Don Lancaster told us how to do this: https://www.tinaja.com/ebooks/enhance_vI.pdf.
I was puzzled by this claim, too. I think that the article is wrong, and that the code is written in HAL/S, a NASA-only language that sort of started off as a preprocessor to Fortran, though it has some PL/I-like features. If it really was written in Fortran, it probably was in a vendor-extended Fortran IV, which a lot of old guys like us know. But NASA used HAL/S for a number of projects, including the Shuttle. (And, by the way, thanks to perplexity.ai, here's a link to the HAL/S language manual: https://archive.org/details/nasa_techdoc_19750002029.)
The parent comment is apt. Of course, languages have their own quirks. But, as Christopher Strachey is once claimed to have said, “I use the same language no matter what compiler I run.”
Now what is more likely to be true is that the code is strangely structured (both because structured programming was new then, and because of memory and processor limitations), and also that much of the internal documentation has been lost. I wish the article had been clearer on that.
I think that the Voyager software is in assembly languages (there are several distinct computers on board), and that it is the program preparation software is written in a “Fortran V” extension.
That could well be, the article is still wrong.
> Regarding "that almost nobody on Earth fully understands anymore", I claim this is nonsense, and definitely not an obstacle.
It seems like you aren't very familiar with the history of the Voyager or the computer systems it uses. The Voyager has three onboard computers, each of which has a custom ISA. Two of the three computers were only used on Voyagers. After the main Voyager mission ended in the late 1980s and they were repurposed to collect data on interstellar space, the probes were reprogrammed to only require limited commands on one of the computers, and no intervention whatsoever on the other two. They also got rid of the testbench for testing code on the ground, so the only working Voyager systems are billions of miles away from Earth. Since then, Voyager has been operated by a skeleton crew that has been shrinking over time as people retire. The result of all of this is that they legitimately do not have a full understanding of how the hardware and software operates. Here's one example from a talk about how the crew fixed an issue with the Flight Data Subsystem, one of the computers that hadn't needed any significant software changes since the interstellar mission began (https://www.youtube.com/watch?v=dF_9YcehCZo):
> So our top priority was to figure out what the FDS was and how it worked. Because unfortunately the person who was the real expert had retired decades ago and the person who was their fill-in, their backup, had retired two years ago. So it was the worst place it could have hit us. These are some examples of some of the documentation we dug up on the FDS. So they're all hand written. These are some very dim circuit diagrams and these are hand written timelines for how to change FDS processors. We were lucky to find these. Some things we couldn't find. A lot of it was like this, that had been scanned. And frequently, these sources were contradictory, ambiguous. Why? Because we changed the way the spacecraft worked with every planetary encounter and entering the VIM and when things broke. So there was a lot of opportunity for ambiguity to arise in the documents over the course of 50 years. This is my favorite example. So this is a page out of an important FDS document. The change bar on the far side indicates it's a change, but somebody went in and made a very cryptic circle of the sentence and crossed it out. I have no idea to this day what it means. I have no idea. Maybe it was important. Maybe they crossed it out, because they thought crossing out was important. I'm not sure. So there were other cases like this. So we were even so confused in some cases, we weren't sure if we were sending data to the FDS - should it be least significant bit first or most significant bit first? We had that level of uncertainty.
> We didn't know we had the right instruction set. I showed you the instruction set there. We didn't know that was the one used to build the software, back when there was an assembler. We didn't know the source code version was the same as what was running onboard the spacecraft. We had a listing of the code, but it was a Microsoft Word document, with optical character recognition scans. We didn't know if there were errors in it. We had no assembler. So they're just playing with bits. There's no simulator and there's no test bed. So there were a few challenges. So basically this was bare knuckle binary manipulation. And they had to think of all the problems. I'm sure you guys are far better than I am of thinking of the problems associated with playing with memory like this. You could overwrite something. You can fail to recognize jumps. How do you debug it in flight on a flying spacecraft? And how do you know you're not missing something in those instruction sets and codes? And the list goes on and on.
I was responding to the headline that says:
>NASA still maintains some of the Voyager spacecraft code in a 1970s-era programming language that almost nobody on Earth fully understands anymore
I did not address the issue of the hardware documentation, just the issue raised by the headline. I fully understand the issues that the hardware and the problems of not having a full map of the hardware or a test bed to check out the code.
Well the 1970s era programming language would be the assembly language for the bespoke Voyager processors, and the spotty institutional knowledge and lack of a Voyager on the ground means that there's no full understanding. I'm not sure what's inaccurate here.
> Microsoft Word document, with optical character recognition scans
It's incredible that there is someone out there in this world who thought that OCRing documentation into a Word document is a good idea. And implemented it.
[dead]
> younger engineers often have the capability but not the inclination
Kids these days... Why would someone in their right mind think working on the Voyager project could damage their careers? You can work on new and fancy tools all you want to improve supporting tools, and it's still one of the coolest space missions active. Plus, it has a real end - at some point, support will be further reduced and the person will move on to another space exploration job, with the extra golden star of having been on the Voyager.
I would go further... This project gives a rare opportunity for a young engineer to learn to build truly mission critical, resilient software while requiring complete, top to bottom understanding of the software and hardware stack.
I am no longer a junior, but would have been upset to be tasked with refreshing the old historical obsolete laundry (no matter how sacred or distinguished), expecially when I already had experience delivering safety critical products packing much more modern technologies.
The opportunity they would be offering is not rare at all! The opportunity to research and design something truly new on the other way is very scarce.
Also, many decisions taken Probably can be traced to limitations / idiosincracies of the era
And you're left with a codebase that has been in hands of 6 Decades of probably great engineers that have already done a lot, plus any of the arcane cruft of such a long lived and esoteric project
It's a great CV highlight, but I don't know if it's the best learning opportunity
What have you worked on that is as cool as a space probe that's cruising in interstellar space and still collecting valuable data?
There are a lot of things as cool as, done by people I know, such as the gyros on the Webb telescope, the APU in the F-35, or a small rack-mountable Cesium reference clock, but there aren't many opportunities like that.
That's the thing. You only have the cool factor, but that wears off very quickly when you are maintaining legacy code and tools and then your collegues are playing with the new hot and shiny toys.
I won't write about the projects I've been involved with for privacy, but to give you an idea some of my old team members were involved in ams-02 for example.
You would be able to go to your grave saying "I worked on the Voyager project." That will never not be interesting, or memorable.
Is that because juniors want to leave their name on something? I ask honestly since I shared a lot of the same sentiment as you, and never quite got an understanding as to why working on the cool new thing was "more fun" even if a lot of the projects under-the-hood were recycled.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
> We’re programmers.
> Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand.
> We’re not excited by incremental renovation: tinkering, improving, planting flower beds.
He's very wrong.
>> The opportunity they would be offering is not rare at all!
The opportunity to maintain software running on a spacecraft is not rare? I don't think so. And those two particular spacecrafts? I'd take that job in a flash.
It also gives a rare opportunity to fuck things up and fail a mission that nobody else fucked up in ~50 years.
Even all that to the side, it lets you say you worked on the Voyager project!
If I'm reviewing CVs and I see that you worked at NASA on the Voyager code, you're getting an interview just so I can ask about it.
I wouldn't normally approve of CV driven development, but for this?!
I agree, and I would think the same, but I also feel like many things I've been sold as "door openers" for interviews unfortunately tend to ultimately be things that no one cares about.
I think people tend to squander door openers with bad layouts or information density. Most CVs are essentially the same as each other, just a bullet point list of jira ticket titles.
Do I care if someone has won the world championship for ping-pong 3 years in a row? Not particularly. Does it make them stand out against a sea of slop? Only if I actually see that info when skimming! But if I do see it, I'm probably going to stop and re-read the whole thing, which is a tactical advantage.
So you're giving me an idea for my midlife career change CV. Lead with the cool and interesting fact above all else, then have the normal CV menu fare at the end.
Hmm this is also drumming up the hard question of: What have I actually done which is ACTUALLY an attention-grabber...
You could work on the Voyager project.
But would you give them a job? Would they even match the requirements?
> But would you give them a job?
That's what the interview is to find out, isn't it?
> Would they even match the requirements?
I would assume if they got a job at NASA working on mission critical systems, they probably exceed the requirements of my startup
I would assume they have built some key problem-solving skills that can be valuable. Training in tooling is much easier than building the right mindset.
But the required midset today is "move fast and break things"...
There are a whole lot of people in this thread who would rather tell their grandkids that they worked on web apps and sold ads.
It's a tragedy some of our best minds are dedicated to that, and digital surveillance so their corporate masters can sell better targeted ads with a higher click-thru rate.
I really think it's overly generous to throw people who do not understand the importance of extending Voyager's scientific mission into the "our best minds" category. I guess I agree with the larger point.
Well… some of the best minds are doing actual useful stuff. Sadly, most aren’t.
Q: What was the hardest debugging problem you had?
A: I had a very slow network connection to the computer, and it was 23 light-hours away from me.
> Kids these days... Why would someone in their right mind think working on the Voyager project could damage their careers?
It's an isolated legacy-project with no future. Mostly everything you learn for it will be only useful for this specific project, so all time to invest there is time to can't invest into something useful. Sure, there are probably some parts to learn from this too, but It's less than what your competitors will learn on their fancy modern projects.
> You can work on new and fancy tools all you want to improve supporting tools
Voyager is in maintenance, there is no big innovation or big progress to be made there. It's just work to hold the line as long as possible. And I guess nobody want's to be the one killing it because of a poor attempt to innovate something.
Understanding assembly language (any architecture) give insights into how computers work that are still valuable and relevant today. Almost nobody still writes code in assembly, but understanding it at least conceptually is still worth something.
If I were hiring, I would almost always prefer a candidate who had some experience at machine/assembly programming to one that didn't, all else being roughly equal.
I've been a developer for 20 years, I do some reverse engineering stuff on the side using assembly.
There hasn't been a single time in 20 years that it was actually relevant for real work in any way.
Unless you are doing very specific work knowing assembly is about as useful as knowing COBOL (which is also useful for a very specific kind of work that most devs will never do)
Learning new and weird things builds brain elasticity and vocabulary for you to express new ideas. I always recommend people to learn FORTH, or Lisp, or APL. Learn to think with different paradigms.
As an extracurricular activity sure, but if your resume just has FORTH on it recruiters are going to throw it in the garbage because there's no evidence you can work in a modern development environment
You should hire for personality characteristics, not knowledge. I'll take anyone who's worked on a weird obscure system and figured it out from first principles over Front End React Dev #8482828 with Opinions on algebraic effects.
> You should hire for personality characteristics, not knowledge.
I'm not sure what personality characteristic are supposed to be in this context. But if you mean skills, then you have a false assumption. NASA and the surrounding industry is not really the environment where they grow boring react devs. For every legacy project like voyager, they have a dozen frontier-projects and other weird obscure jobs where one can gain the same skills, but better, with more potential for further growing. And someone early in their career will naturally choose something with more potential for their future.
It would also be an interesting job, even if off the beaten track.
As a hiring manager, if I saw this on a resume interspersed with various web development work, I would be intrigued.
You don't need to push the needle as a junior though. It's a "completed project" where one can glean many insights. And the matter of transferrable skills is simply a matter of being able to say how what you learned applies in a different context. A useful skill in-and-of-itself
> You don't need to push the needle as a junior though.
Doesn't matter. Many young people have strong motivations to shine and push, especially when they are in high profile-jobs. Legacy-projects are simply not alluring enough for this.
> Voyager is in maintenance, there is no big innovation or big progress to be made there.
This is just so blatantly ignorant. As if you believe there's some other space agency taking scientific measurements out past the heliopause.
> As if you believe there's some other space agency taking scientific measurements out past the heliopause.
We are talking about the maintenance, not the research. Nobody has doubts about the scientific value of the collected data.
Because software development back in the day wasn’t like how it’s now now, the charade so called software development now is a clown show: scrum, daily stand ups, open office style, tickets, tons of ci/cd BS, and of course, the wrangler aka PM and all politics involved, none of this existed like the cult it is now, I only had one experience in such environment and despite the effort I had to ask for some common sense, it was like insulting someone’s religion, “how dare you challenge the sacred methods that the silicone valley companies are using?!!”
Additionally, back in the day there was true ownership for the code you write, the code is owned by you not the company, and I know few old engineers that until now (they are retired) the companies still pay them for using their code they wrote while working there. That sense of ownership encourages you to tackle hard issues rather feeling like a machine spewing code for someone else’s business, I have seen some contracts too where the company will have ownership for anything you do while you are in the contract, including your personal projects on your own free time.
Still paid for the code you wrote while working there sounds like a consultant with a hell of a contract, not an employee.
Or like many a company that prefers to paper old hands with cash to keep an old system running until it is really, really no longer feasible.
Banks and airlines are the most common example, many of them run on mainframe systems with code that's old enough for humans to go into retirement - and replacement projects usually tend to go into the billions of euros range with many of them failing catastrophically.
Even paying some greybeard 500k a year to deal with that stuff despite him being retired is far more profitable in the short term. That's the problem with letting beancounters run the show because eventually, there will literally be barely anyone left alive who is capable - and even less who know all the "implicit knowledge" behind edge cases.
Employees, auto industry, although it was in Europe not US.
Rose-tinted glasses, eh? They ain't what they used to be.
> none of this existed like the cult it is now
So you'd prefer for all this project management drama and power struggle to be invisible?
All this scaffolding is not a cult. It exists to democratize the process. Your personal comfort is irrelevant to the results.
Unfortunately in many places it is indeed a cult and serves to ossify management decisions. We ARE doing Agile, what do you mean? No this person is the scrum master and they tell us what Agile is and then we do it, see?
I have worked at 8 different software places and none of them implemented things in a way I would call "genuinely agile" and most of them were just bad waterfall with more meetings and telling the engineers they are accountable for the bad ideas that are now their job.
CI/CD? How about make that only "Devops" job and then make the exact same undemocratized system before with gatekeepers who spend a significant amount of time blocking your work because they are afraid of things changing.
https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
My point is you would never even have a clue that there is dysfunction at all without these methodologies. These days you at least have some idea in your head how it should be vs how it is.
I think there's something critical being lost in younger people who have never been exposed to the bad old ways and only understand their current situation through memes. We need to get a grip here.
That's somewhat fair, though I dont think you need to know about the beforetimes to understand that the current times get nowhere fast with all their talk of velocity.
I worked in a mainframe team at a major financial network (very explicitly not a bank) and we were agile. The entire company was agile. The company was so agile that when we moved to our new building we did hot-desking where no team had a fixed location and every worker had a little safe where they could put their valuables and leave it in modular lockers throughout the building, so that anyone could sit anywhere there was space to sit on that day.
I mean of course it didn't exactly work like that in practice because directors had to have the corners and they wanted their teams close to them, so that, e.g. our team of 15 mainframe engineers could file in to a 5 x 5 meter cubicle and one-by-one give a report on what they had done the previous day a.k.a. The Scrum. We had a certified Scrum Master.
The one time I tried to sit in a different team's space I got yelled at by their director and had to go back. I had moved in his team's space out of protest because the only space left for me to be near my team was right in front of the gents and it smelled a bit funny. I was supposed to be sitting next to my (official) Mentor but I was studying part-time for a MSc and so was not there in the mornings and another engineer would always take my place next to the Mentor, who was the most experienced Cobol programmer in the team. He always looked at me with a wry smile whenever I came in to the office and found him sitting at ... well, it wasn't really my place so I couldn't really say anything. But then the only place for me to sit was in front of the loos. So I tried to go and sit somewhere else, farther away, in protest. And then I got yelled at.
But in principle we had hot-desking and we could sit anywhere throughout the building while communicating remotely with our colleagues.
We were very agile.
Our certified Scrum Master moved on before I did.
>"So you'd prefer for all this project management drama and power struggle to be invisible?"
Well project managers can have their dramas. Just do not involve developers. Or what is even better - get the fuck out and leave it to people who can do things without drama.
>"All this scaffolding is not a cult. It exists to democratize the process. Your personal comfort is irrelevant to the results."
Pile of BS. It exists to feed whole layer of self serving people who contribute very little and grow like a cancer.
In my career I was lucky. I am an independent software developer. Designed and developed many products for various clients (including some of my own ventures) and have managed without Agile, Scrum and the likes. My largest products - I had teams of up to 35 people under me and somehow we've survived.
On few occasions I had pleasure to be on some of those meetings as a visitor - felt nothing but disgust. Again luckily I was spared from direct participation
>"Why would someone in their right mind think working on the Voyager project could damage their careers"
Assembly? Understanding how things actually work? No Agile? No K8s? No Rust, No React? - death knell for someone's resume
>"and the person will move on to another space exploration job, with the extra golden star of having been on the Voyager"
this is the best case with the result of being tied to another single project for years and unemployable anywhere else. in more realistic case - warm goodbye in few years and start your life from scratch with no credits for the thins done.
> The succession problem matters most in the next decade. After that, the question becomes academic. There will be no Voyagers left to maintain.
Well, the Voyagers will still be there, there will be just no way to contact them anymore once the power runs out or communication is lost (whichever happens first)...
Many of the issues could potentially be solved by modern LLMs?
Reading, analyzing and assembling documentation could be probably done by LLMs.
And by including old code and snippets into the training set, the LLM could be fairly proficient in writing this code probably too?
Maybe someone knows more about the use/not-use of LLMs in this context?
I don't think you want LLMs touching projects that cost over $800.000.000, even to assemble "documentation" (since the LLM can't really document in as much it's translating what it's reading, because documentation includes much more information than what's stored in the code itself).
It's a cool idea, though, I'd like to see this done as an experiment :)
My thought on this is that LLMs should be allowed to touch high stakes projects, but they shouldn't be left completely unsupervised.
Here me out.
Would you let AI manage your investments/retirement savings/etc. completely autonomously and unsupervised? Or would that make you a little nervous?
What if you had to undergo a X-Ray or a MRI. Would you ONLY want AI reading the images? Would you ONLY want a human radiologist reading the images? This is an interesting one because I would want both. In fact, I would find it questionable if AI wasn't given the opportunity to look at the images.
> It's a cool idea, though, I'd like to see this done as an experiment :)
Since the Bun AI LLM 'exeriment' conversion got merged I'm a lot less trusting of 'experiments' with LLMs. They seem to get shipped
I am fully aware of the costs and so on. But i can certainly imagine that LLMs help with the process of understanding and editing old code.
And of course you need to test and debug before you ship to production.
> i can certainly imagine that LLMs help with the process of understanding and editing old code.
Can you trust it with assembly language for a custom-built CPU with its own instruction set? Even if you digitized all of the documentation you have no idea what was lost since 1977 (or earlier), or what was never written down to begin with.
Anthropic / Google / Meta / Airbus / Boeing / ASML EUV machines, etc, etc... are all developed using LLMs and they are much more expensive than this
Arguably, Anthropic and Boeing are NOT what you should use to determine whether your coding practices are reasonable. Software problems with Boeing have literally killed hundreds of people due to faulty code and Claude Code is know to be a pretty buggy CLI (although extremely useful, sure).
Though I agree with the expensive factor. Perhaps, what I actually believe is that LLMs shouldn't touch code that's as mission critical as what NASA works on, even though it might be great to develop user-facing frontend software and CRUD backend code for huge corporations and projects.
> Many of the issues could potentially be solved by modern LLMs?
Yesterday I asked an LLM what customizations niri made to the KDL language.
It said niri modified the language to add single line comments with //. However if you visit the official home page of KDL https://kdl.dev/, the very first example shows single line comments as being part of the official spec. There's also a whole page dedicated to comments in the spec that mention this.
The moral of the story is LLMs are honestly really really bad and I'm sincerely concerned at how they manipulate people into thinking what they produce is accurate or trustable. I didn't believe the AI because my spidey sense said that doesn't feel right, so I double checked the real source.
It's gotten to the point where I'm finding a huge majority of the time, it provides incorrect information on really basic things. Pure hallucinations. I'm at the stage now where even if you paid me, I wouldn't use it. There's a 0% chance I'd ever consider paying for it in its current state and it's upsetting because it is killing off the web in real-time. It's becoming harder and harder to find useful and accurate information.
That's insane.
Of all the possible projects, this is the one where it's both highly feasible for humans to learn and historically critical that we have a full understanding and control of its operation.
You don't want your spacecraft that's 15.8 billion miles away pointing backwards because the AI's fairly proficient code misunderstand something.
“You’re absolutely right, I shouldn’t have jettisoned the RTG”