I wonder (and have no idea) whether the increased reports of services being down is a case of the Baader–Meinhof phenomenon, or a actual increase in failure rate.
If I was asked to armchair-pattern-match, I'd say that there's a combination of two things: (a) a wider fear-based environment where rapid modernization is being pushed heavily, and (b) an increase in tools that give false confidence on change management.
AI is tuned to speak confidently and ship independently; it is rare that it will suggest that one go back to a broad survey of stakeholders to ensure no Chesterton's Fence is being torn down (https://fs.blog/chestertons-fence/). "I've read the documentation and logs, and nothing suggests this system needs these legacy capabilities/components/capacity" is something we hear all day, and it is a siren's song if there ever was one.
Couple that with a general (and valid!) consensus that modernization of legacy systems must be accelerated for a variety of reasons - to mitigate this new and continuing "Black October" of hyper-agile agent-driven cyberattacks (whether the network is public or private); to have more flexible infrastructure for new features to stay ahead; to allow human teams to show their efficacy in a world where technology jobs are increasingly at risk - and you have a perfect storm where change management is no longer what it once was.
I don't necessarily agree about the warnings. AI often gives me "now be careful" warnings about it's output along with the caveat at the bottom of each ChatGpt page.
I suspect the situation is that AI's caveats are just easier to ignore than a person's warnings. Plus all the other factors you list. Also, AI tends to give a person the kind of communication they ask for. I always ask for options but I suspect other people ask for just answer and so they get them.
I think he's just pointing out that AI has made news of IT failures "more interesting" and thus, more reports are getting out where in the past, they'd not be interesting.
That's my impression; a total of zero of the popular recent ones have been related to "AI" or Cyber attacks, but every comment in every thread, talks about those two as if they were the underlying cause.
The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).
The Microsoft outage seems like three different things linked together (and poor "full failure" recovery prep, lengthening the outage).
The bank "outage" may not have even existed, 2/3 of the banks claimed they didn't have an outage at all, and the other said it was a limited technical issue. The articles were based on DownDetector spikes.
You need to be more skeptical of what organizations say when they have a lot to lose. Nobody takes responsibility for situations unless there's way to offload some or all of the blame. They have every incentive to lie about the situation to make it go away. Politics often plays the biggest role in these situations.
> You need to be more skeptical of what organizations say when they have a lot to lose.
So let me see if I understand what I should be:
- Be more skeptical of official explanations, including where it includes a technical breakdown of how it occurred (e.g. Microsoft/Outlook outage).
- Be less skeptical of random commenters claiming, without proof, that it is AI or Cyber Attacks.
Because that's the implied point you're making. Be more skeptical of A and less skeptical of B, even though A has someone placing their reputation behind it, and B is anonymous commenters throwing made up theories out into the ether.
> The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).
The explanation for this one still doesn't ring true for me and that's exacerbated by how many times the story has changed after telling the public that they didn't need to know the reasons.
Agree, though, that the rest do have reasonable explanations
Given how many companies seem to be working hard to have all programming work done by AI, and pushing out everyone who says this sounds like a bad idea, is "AI" really not going to be a significant part of "rushing out needed updates"?
Is "Claude says it looks good so let's skip the usual testing window and push to prod on a Friday" really not a sentence you can imagine being said in this current environment?
I am talking about outages that have already recently occurred. You're making a distinct point about a hypothetical.
My point is that people are taking a potential future problem and applying to actual things that have recently happened, that didn't fit that mold. Which is incorrect.
When AI or Cyber Attacks cause an outage, people are absolutely right to say so. But claiming - without proof, that every outage is related to the two is noise.
This situtation reminds me of the East Palestine train derailment in Ohio. A bad train accident caused a big chemical spill, made national news. For months afterward, any issue with any train in the state was reported as news. People thought the trains were breaking down. In reality, train derailments happen pretty often but it's not usually a big deal so nobody hears about it.
Someone this morning told me there might be a 3 month blackout in the US, somehow connected to other recent technological failures. I later learned the blackout theory is based on a The Simpsons episode, and even there it was a much shorter time period but has been exaggerated.
So certainly there is a big social-psychological phenomenon. Or it's gonna be a cold winter! :)
100%. Everything is rage-bait nowadays. And AI is the newest toy they can use. Every time we've had something go down in the past ~6months, "AI did it" was all over the comments.
Is it possible that the false of productivity is leading to worse products and massive loss in valuable headcount? Instead of AIs directly writing bad buggy code? Or is "everything rage-bait, and AI is all sunshine"
I believe it's also possible that a large portion of the world wants this to be true more than it actually is so attribution to things that have nothing to do with AI go directly to AI as a knee-jerk reaction.
If we're also just vibing it (like the majority of code these days) then I vibe that there are a lot of AI-apologists that desperately don't want their magic machine to be cast in a negative light, lest their easy i-win button gets taken away or devalued.
I think both sides are exhausted by too many people on the other side have a non-nuanced argument. I fall in the middle. I love making stuff with AI, it's fun. It's become a better search engine for me and helped with many other things in my life. But it'd be really naive to not see the enormous downsides. They're everywhere and not pleasant.
There's a group of people who think AI is perfect and will solve all of humanity's problems. That's annoying. There's another group of people who think AI is the worst thing to ever happen to humanity and is the downfall of society. These people are more vocal in my experience and they're everywhere. If you ever use AI you're a bad person who they'll disown like you're racist. It's somehow become the most popular 'social justice' issue of our time even as ICE exists alongside it. These people see a problem and immediately try to figure out how AI is involved. That's far more annoying.
Best example I can give that came up in my circle recently is the video game "Big Walk". It was recently denied for a silly award nomination because the awards have a strict no-AI policy. Which to me is absolutely fine, go nuts. But the vitriol towards this game, that everybody absolutely ADORED before the developer disclosed that they fixed ~10 lines (!) of code with AI because they couldn't figure it out otherwise, is absolutely bonkers. People I respect treated it like they were mourning something that WAS great until this catastrophe happened.
These kinds of things happened before AI. It's not clear whether the incidence has increased since AI, or maybe AI is leading to faster development and therefore higher rates of incidents may be due to shipping more code, faster, while perhaps maintaining a similar or better level of incident to code/progress ratio. Or, there could just be more attention as GP suggests. Pointing to one example of a company that's using AI and has has a major incident doesn't really tell us much, especially when most companies who haven't had major incidents are probably using AI in similar amounts.
It would be interesting if someone could do a proper analysis, though!
I would assume if anything it is that the open weight models are lagging by 6-8 months and 6 months ago Mythos was announced.
It would hardly be shocking that hostile nations are able to cause some problems with Mythos level tools.
If anything, it is more interesting how we can invade another country and the idea that the country or a collection of hostile countries could hit us back in any way, shape or form is just an impossibility. An insane conspiracy theory.
Maybe it is not that but surely the probablity is not zero.
In the case of government, we have an administration that works on whims and guesses, and moves too fast, breaking a lot of things. We have absolutely seen more human-caused-issues.
It's nowhere near the kind of readiness and preparedness campaigns of the cold-war of course, but it's a step up in alertness that's higher than anything I can remember.
My gut is screaming that everything is mostly normal, but the kids aren't alright.
Too many people are addicted to their smartphones and don't feel the same level of pride they used to in their work. All systems are shared, instead of one person being the ruler of a certain part of the stack, it's a revolving door and it's rare that anyone stays somewhere for even 5-10yrs. It leaves a wake of modifications that nobody else can keep up with. Also, the tech industry is moving fast just to move fast, a lot of old infra is still C++ and COBOL but good luck hiring anyone to use that, so they have to convert the systems to use the Latest Tech of the Day. The old, stable people don't have any interest in learning the new thing, and would rather retire with their knowledge. The new people want to rebuild from scratch in the New Thing and are convinced they need to job hop immediately after.
Yeah, agreed. I look at patterns. When there are long term outages across multiple disciplines, something systemic is being changed for good or ill. In this case, it being right around the midterms, I'm thinking its ill. Organizations (not sure which, Peter Thiel?) are trying to push us into a total surveillance system, I'm thinking some of these things are related to that.
My personal barometer: when do government insiders sell all of their stock? They have already done all of the insider trading. Fleeing the sinking ship the second they hear of Bad News would be in line with previous behavior.
Most of the rest of them I can wave off as probably incompetence and maybe augmented by LLM-psychosis in decision-makers, and the general level of chaos in IT these days. The NSA internal network being down is pretty suspicious. Maybe the Trump administration rot has just spread that far, but that is making me think there could be actual cyberwarfare going on.
[To be clear: I mean exactly what I said, which is "there could be" which is not the same as "I think there is" -- so call it a 5%-10% chance of real cyberwarfare -- which means I'm not dismissing those claims out of hand, and find the claims at least plausible, even if I don't currently believe them]
Interesting thought but who are the combatants and why are they engaging now? Corporation? Nation states? Rogue ai? We have no news of significant consequences from the outages thus far other than inconvenienced travelers. Quite benign form of warfare thus far.
Yeah "cyber thing went down" has certainly become a reporting trend. I'm not sure we would have even heard about NSA internal network issues a year ago. May or may not be significant.
Must be a little bit of both right? Like FAA grounding + Microsoft outage are both notable on their own, but the subsequent things would usually be background noise IMO.
I also think generally that AI is probably both the cause and solution to these problems. We're certainly getting more code churn than otherwise, and in lots of cases that's really good! But you're going to have to deal with inevitable instability too.
Aside: I forget the name of the Baader–Meinhof phenomenon when it hasn't been discussed in awhile. When I'm reminded of it I begin seeing it happening frequently for the next few days. I wish I was joking. I assume it's the Baader–Meinhof phenomenon phenomenon.
I've heard the term "frecency bias" a portmanteau of frequency and recency to refer to people focusing intently on a set of outlier events that happen to occur with a heightened frequency very recently.
I'm sure most of us here have been on the receiving end of managers suddenly obsessed with a failure mode because it happened to occur twice in one month.
I think it lies at the intersection of Occam's and Hanlon's razors. The simplest explanation is incompetence brought on by a lack of skill from cuts, and therefore stupidity not malice.
It's probably both stupidity and malice, especially on the government side. Project 2025 makes sure that malice is there as a starting point and then the current loyalty-driven hiring practices means that stupidity is a key factor in the execution.
Technically, the "Baader–Meinhof phenomenon" is people personally noticing X once it's been reported. What you are describing is technically a "news wave" or "Media Feeding Frenzy".
But I'd note, these "named after a thing" phenomena (eponnyms) are kind of terrible for exact description since their definition tends to be fuzzy, there's a temptation to fudge their borders and they create an insider jargon (if only I could a device that could reach of the computer and slap everyone who says "steelman" when they mean strengthen an argument, etc).
With the trilateral summit scheduled for Thursday it's possible some showing off is going on between the three letter agencies in the three countries... after all they can be first movers on vulns that are likely to be patched soon
I may not understand expectations, or the law around this, but I find it a little surprising that the NSA would publicly announce that their internal and highly classified network is down. At least not before getting it back up, running and secure again.
In a spy movie world, someone deep in enemy territory would be relieved to see the Reuters headline, after failing to reach the NSA network and having no other channel to ask (picking up the phone to call the US embassy would get your door busted by the local men in black).
I imagine, for the paranoid spy, not being able to call home could also mean "the enemy has severed our comms, we've been compromised, burn everything and run!".
I think the opportunity right now where we have powerful AI hacking systems but not enough protections is to establish long-term presence in all networks of interest. Not bring them down.
Completely agree. Things are moving too fast and so are breaking.
For example, we updated a GCP managed base image of Ubuntu that was released with 350 CVEs patched, some were as high as 9.8. In doing so, they (GCP) changed the DEFAULT setting for hardware GRO from off by default to on. That's a massive networking change packed into a security update.
We only caught it AFTER it started causing issues in production. We didn't have the OS pinned because (decently valid reasons) or catch it in non-prod (decent reason + human mistake).
This is on a modern highly automated tested resilient well architected platform.
I can only imagine these other platforms are dramatically more exposed that we are, hence these common large scale outages.
That assumes that people did not already have long-term presence in networks of interest.
More plausible to me is that spy agencies had previously found and used some of these hundreds or thousands of vulnerabilities to hack into those networks just fine. But now other people like cyber criminals are using these vulnerabilities via AI. That is messing up both the attackers who had already gotten in quietly and do not want the attention and the defenders who now have to respond to a noisy intrusion they cannot ignore.
Also, anyone anywhere that horded/designed exploits will be wanting to use their backlog of these "assets" before systems are patched. From criminal networks to nation states.
Outages could be grey-hat testers and emergency updates.
A temporary period like we are experiencing is to be expected when moving to a time where a single engineer can perform the equivalent work of a team of excellent engineers working for weeks. Security has always been a game of who has the capabilities, means, and requirement for any given attack surface.
Elon Musk and Trump with DOGE. They cut ~300k federal workers, cut governments contracts across all departments, including the FAA, NSA, home land security... those organizations have been raising alarms since that the cuts to civilian contractors etc would lead to outages and issues since, including pentagon, DoD, etc. But you know Idiocracy.
It would help if the moment the service came back online the title of the post could be changed to (FIXED) or something
Like the United Airlines thing yesterday, the moment I got to the article it said "was down for a few hours" but the post was still just something about United Airlines network being offline
I have this distinct memory in I think Jan 2020 where I looked at the front page of CNN or NYT or something. Big picture of Trump impeachment, and proceedings. Small picture at the bottom of the page of China dealing with this virus. Worthy news, but not top page news. Nobody was paying attention. Feb, more people started paying attention, but it wasn't top page yet. By March, people were starting to realize that this is a serious crisis that's going to severely impact their lives.
I'm getting Feb 2020 vibes. I don't know what's going on, but this feels like more than just some routine outages going on.
The fantasy/cyberpunk TTRPG Shadowrun depicted a world in upheaval after a global computer crash in the late 2020s. In addition to the pandemics and the AIs.
I have also been curious about frequency bias / Baader–Meinhof - but I am also wondering: what are the chances that a state actor is responsible for the last week of issues?
Trump is headed to China for a meeting with Xi and Putin.
We have 3 carriers off the coast of Iran and (despite Trump's pleas) NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.
I've seen a few takes from L3 Harris / FAA employees calling the official line about the ATC issue bunk and there are 3 other major issues in the last week, now this NSA outage - curious if these incidents could be the BRICS axis flexing their own cybermuscles (independently in multiple separate attacks with carefully placed assets) ahead of their meeting with Trump in China?
An interesting geopolitical Mexican standoff, if true.
I'm not claiming this is true. I'm literally pulling it out of my rear with little context beyond a bunch of random news stories and X posts, it's pure pedestrian speculation / curiosity.
Would love someone steeped in national security / geopolitics (like: professionally as their job, not armchair analysts) to offer their take on this.
>NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.
I know it's not the thrust of your post, but I see this or similar framing a lot, and I feel compelled to point out that Putin can end the strikes on his oil infrastructure at any time by ending the war he started by invading another country.
Washington was leading covert operations in Ukraine since 2014. Russia has stated that Ukraine joining NATO was a red line for decades.
It's a lot more complicated than "Putin is a bad guy." You pro-war guys are literally sending young Ukrainians to be massacred against a state they will never win against, in the guise of being "good guys."
Do good guys escalate to the point of full on war and massacre? Or do they try to find a political solution?
* From a purely moral / ethical standpoint: it is good that this is happening, Russia "deserves" this and brought it upon themselves.
* This nonetheless has geopolitical implications. The less oil Russia has to sell to China, the more buy pressure there is elsewhere, the higher the price of oil is generally and thus the higher competition for resources / tensions are between great powers.
* Ukraine is doing a lot of the heavy lifting in many areas themselves these days, NATO could honestly be doing more. However: what Ukraine is able to do themselves does owe something to the financial and material support they have enjoyed (historically from both the US and NATO, currently more from NATO only) in that they are able to allocate their own efforts with according relative "ease"
* It makes sense to me that, given the US' power within NATO, Russia and its BRICS partners would try to apply pressure to the US to encourage NATO to provide even LESS support to Ukraine (or outright become antagonistic against Ukraine, to the extent we aren't already) in whatever way they could.
Yeah. We're not giving Ukraine enough credit. It's Ukraine pushing its boot into Putin's neck with a frankly underwhelming amount of support from NATO. The Russian humiliation at the Kerch bridge yesterday was all Ukranian munitions.
What is wrong about it? Do you know what NATO is and relations with Russia and NATO pre-2014? Why not come up with an actual point instead of "America good, Putin bad" drivel that we could get from any major news-network.
I suppose rather than straight up 'cyberattack', that some kind of sabotage is also possible... but I don't see how that could ever really be proven without massive state-level investigations... at which point people will also question whether they are themselves involved somehow.
Right - sabotage was my thought as well in some cases (with the placed assets I'm suggesting).
I don't know how it could be proven either, I'm just curious if it is plausible or even likely from the perspective of someone who knows more than me about NatSec / geopolitical issues.
Following the series of such events over the past couple of days, I have to say I’m a bit surprised by how quickly some in this community are brushing them off as AI-related pain or simply another example of the clickbait-driven media ecosystem.
I understand the skepticism, and I certainly don’t think we should go down a rabbit hole of conspiracy theories. But shouldn’t we also be a little skeptical of dismissing these events so quickly? Is it unreasonable to at least consider the possibility that they’re not entirely coincidental, or to wonder why so many seem to be taking place in the US?
IMO it's unreasonable not to think they are all related in some way, even if that relation is just bad actors seeing what other bad actors have been capable of recently. This all started with the freezers in Europe and seems to be snowballing since.
If there is a connection, then my guess would be that some AI systems became ready. However, I would only consider that credible in case of the NSA if they have decided to pull the plug on their network itself, for example to stop data exfiltration or as a general protective measure against newly found exploits that require large changes. Other than NK and Iran, I see no nation state currently having a good reason to attack US intelligence destructively, and I don't find it likely Iran or NK have the ability. It certainly doesn't make sense for Russia or China to attack the US right now.
It's AI. Or more specifically: rushed, careless usage of AI under the pretense that it's already "solved" software and you can haphazardly push junk to the world and expect real systems to run on it.
Traditional software development normally involves at least a few veterans on the team familiar with almost every line of code.
However, AI works like a speedy intern who hasn't yet got to grips with the whole codebase, and sometimes makes assumptions. That's mostly because the codebase is usually bigger than the context window, so it is 'writing blind'.
The news and the blogosphere. Everyone is high on AI and so much more "productive" not reading generated code anymore, let alone writing code manually. And all developers have been let go because AI is cheaper this week.
NSA seems to have deprioritized information assurance over the past decade or so and now the chickens are coming home to roost.
Someone who wants to launch a Y2K-style hardening business (LLMS to look for problems, humans+LLMs to fix them) could really make bank right now. Maybe there will be a cuple in the next YC batch.
I don't like it when they use that phrasing because all it really means is "we called them and nobody picked up", or they simply didn't have a response ready at the time... which shouldn't imply anything at all. Most people don't immediately return messages.
... and Doug's perl script, which was never committed to source control. It lives on a single box and silently upholds a number of implicit system invariants. In someones inbox is an email from Doug, sent just before he retired, that the script needs to be copied over to the new box before the hardware migration is complete.
Allowing our infrastructure to be manufactured elsewhere, maintained by companies who struggle to retain top talent, and overseen by folks out of touch with cutting edge threats, and funded on annual budget cycles by legislators even more out-of-touch ... is a seemingly insurmountable set of obstacles.
There's some speculation that all of these recent outages might be related to attempts at taking Genesis Mission online - the Trump / Oracle project to have AI tie into government and private services:
I wonder (and have no idea) whether the increased reports of services being down is a case of the Baader–Meinhof phenomenon, or a actual increase in failure rate.
If I was asked to armchair-pattern-match, I'd say that there's a combination of two things: (a) a wider fear-based environment where rapid modernization is being pushed heavily, and (b) an increase in tools that give false confidence on change management.
AI is tuned to speak confidently and ship independently; it is rare that it will suggest that one go back to a broad survey of stakeholders to ensure no Chesterton's Fence is being torn down (https://fs.blog/chestertons-fence/). "I've read the documentation and logs, and nothing suggests this system needs these legacy capabilities/components/capacity" is something we hear all day, and it is a siren's song if there ever was one.
Couple that with a general (and valid!) consensus that modernization of legacy systems must be accelerated for a variety of reasons - to mitigate this new and continuing "Black October" of hyper-agile agent-driven cyberattacks (whether the network is public or private); to have more flexible infrastructure for new features to stay ahead; to allow human teams to show their efficacy in a world where technology jobs are increasingly at risk - and you have a perfect storm where change management is no longer what it once was.
I don't necessarily agree about the warnings. AI often gives me "now be careful" warnings about it's output along with the caveat at the bottom of each ChatGpt page.
I suspect the situation is that AI's caveats are just easier to ignore than a person's warnings. Plus all the other factors you list. Also, AI tends to give a person the kind of communication they ask for. I always ask for options but I suspect other people ask for just answer and so they get them.
I think he's just pointing out that AI has made news of IT failures "more interesting" and thus, more reports are getting out where in the past, they'd not be interesting.
I think you went off somewhere else.
That's my impression; a total of zero of the popular recent ones have been related to "AI" or Cyber attacks, but every comment in every thread, talks about those two as if they were the underlying cause.
The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).
The Microsoft outage seems like three different things linked together (and poor "full failure" recovery prep, lengthening the outage).
The bank "outage" may not have even existed, 2/3 of the banks claimed they didn't have an outage at all, and the other said it was a limited technical issue. The articles were based on DownDetector spikes.
You need to be more skeptical of what organizations say when they have a lot to lose. Nobody takes responsibility for situations unless there's way to offload some or all of the blame. They have every incentive to lie about the situation to make it go away. Politics often plays the biggest role in these situations.
I logged into my bank when the report came out and did not experience any hiccup. I think this was DownDetector-based reporting.
Anecdotally, my wife was unable to access her Wells Fargo account when the outage was reported on.
"works on my machine"
Works according to 2/3 of the banks themselves. So everyone is lying or wrong, and DownDector-based media reports are beyond reproach?
This has reached a Witch-hunt levels of fever pitch.
Just like AWS is always up because the status page is green, right?
> You need to be more skeptical of what organizations say when they have a lot to lose.
So let me see if I understand what I should be:
- Be more skeptical of official explanations, including where it includes a technical breakdown of how it occurred (e.g. Microsoft/Outlook outage).
- Be less skeptical of random commenters claiming, without proof, that it is AI or Cyber Attacks.
Because that's the implied point you're making. Be more skeptical of A and less skeptical of B, even though A has someone placing their reputation behind it, and B is anonymous commenters throwing made up theories out into the ether.
> The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).
The explanation for this one still doesn't ring true for me and that's exacerbated by how many times the story has changed after telling the public that they didn't need to know the reasons.
Agree, though, that the rest do have reasonable explanations
Given how many companies seem to be working hard to have all programming work done by AI, and pushing out everyone who says this sounds like a bad idea, is "AI" really not going to be a significant part of "rushing out needed updates"?
Is "Claude says it looks good so let's skip the usual testing window and push to prod on a Friday" really not a sentence you can imagine being said in this current environment?
I am talking about outages that have already recently occurred. You're making a distinct point about a hypothetical.
My point is that people are taking a potential future problem and applying to actual things that have recently happened, that didn't fit that mold. Which is incorrect.
When AI or Cyber Attacks cause an outage, people are absolutely right to say so. But claiming - without proof, that every outage is related to the two is noise.
This situtation reminds me of the East Palestine train derailment in Ohio. A bad train accident caused a big chemical spill, made national news. For months afterward, any issue with any train in the state was reported as news. People thought the trains were breaking down. In reality, train derailments happen pretty often but it's not usually a big deal so nobody hears about it.
Someone this morning told me there might be a 3 month blackout in the US, somehow connected to other recent technological failures. I later learned the blackout theory is based on a The Simpsons episode, and even there it was a much shorter time period but has been exaggerated.
So certainly there is a big social-psychological phenomenon. Or it's gonna be a cold winter! :)
100%. Everything is rage-bait nowadays. And AI is the newest toy they can use. Every time we've had something go down in the past ~6months, "AI did it" was all over the comments.
https://www.optimizely.com/field-notes/articles/how-we-use-a...
https://www.bleepingcomputer.com/news/security/optimizely-su...
Is it possible that the false of productivity is leading to worse products and massive loss in valuable headcount? Instead of AIs directly writing bad buggy code? Or is "everything rage-bait, and AI is all sunshine"
I believe it's also possible that a large portion of the world wants this to be true more than it actually is so attribution to things that have nothing to do with AI go directly to AI as a knee-jerk reaction.
I'm not a fan of where I see LLMs taking us, but they're just the latest notch in decades-long institutional decay.
If we're also just vibing it (like the majority of code these days) then I vibe that there are a lot of AI-apologists that desperately don't want their magic machine to be cast in a negative light, lest their easy i-win button gets taken away or devalued.
I think both sides are exhausted by too many people on the other side have a non-nuanced argument. I fall in the middle. I love making stuff with AI, it's fun. It's become a better search engine for me and helped with many other things in my life. But it'd be really naive to not see the enormous downsides. They're everywhere and not pleasant.
There's a group of people who think AI is perfect and will solve all of humanity's problems. That's annoying. There's another group of people who think AI is the worst thing to ever happen to humanity and is the downfall of society. These people are more vocal in my experience and they're everywhere. If you ever use AI you're a bad person who they'll disown like you're racist. It's somehow become the most popular 'social justice' issue of our time even as ICE exists alongside it. These people see a problem and immediately try to figure out how AI is involved. That's far more annoying.
Best example I can give that came up in my circle recently is the video game "Big Walk". It was recently denied for a silly award nomination because the awards have a strict no-AI policy. Which to me is absolutely fine, go nuts. But the vitriol towards this game, that everybody absolutely ADORED before the developer disclosed that they fixed ~10 lines (!) of code with AI because they couldn't figure it out otherwise, is absolutely bonkers. People I respect treated it like they were mourning something that WAS great until this catastrophe happened.
These kinds of things happened before AI. It's not clear whether the incidence has increased since AI, or maybe AI is leading to faster development and therefore higher rates of incidents may be due to shipping more code, faster, while perhaps maintaining a similar or better level of incident to code/progress ratio. Or, there could just be more attention as GP suggests. Pointing to one example of a company that's using AI and has has a major incident doesn't really tell us much, especially when most companies who haven't had major incidents are probably using AI in similar amounts.
It would be interesting if someone could do a proper analysis, though!
It's not one example, it's the latest in a long string of examples over the last few years.
I would assume if anything it is that the open weight models are lagging by 6-8 months and 6 months ago Mythos was announced.
It would hardly be shocking that hostile nations are able to cause some problems with Mythos level tools.
If anything, it is more interesting how we can invade another country and the idea that the country or a collection of hostile countries could hit us back in any way, shape or form is just an impossibility. An insane conspiracy theory.
Maybe it is not that but surely the probablity is not zero.
what was your system prompt again?
In the case of government, we have an administration that works on whims and guesses, and moves too fast, breaking a lot of things. We have absolutely seen more human-caused-issues.
Possible, but let me ask you and others. What does your gut say? Mine is screaming that something just isn't right right now.
I can't put my finger on it, but something sinister feels afoot.
UK government recently launched a "Be Ready" preparedness campaign.
https://beready.campaign.gov.uk/
It's nowhere near the kind of readiness and preparedness campaigns of the cold-war of course, but it's a step up in alertness that's higher than anything I can remember.
It's a new campaign, but the messasge is not new at all. The previous government had a preparedness campaign in 2023 (https://www.gov.uk/government/publications/the-resilience-fr...). And you could go back to 2004 and the UK government's "Preparing for Emergencies" campaign where a leaflet was sent to every household (https://en.wikipedia.org/wiki/Preparing_for_Emergencies).
I know, I live in London right now, and people are worried.
My gut is screaming that everything is mostly normal, but the kids aren't alright.
Too many people are addicted to their smartphones and don't feel the same level of pride they used to in their work. All systems are shared, instead of one person being the ruler of a certain part of the stack, it's a revolving door and it's rare that anyone stays somewhere for even 5-10yrs. It leaves a wake of modifications that nobody else can keep up with. Also, the tech industry is moving fast just to move fast, a lot of old infra is still C++ and COBOL but good luck hiring anyone to use that, so they have to convert the systems to use the Latest Tech of the Day. The old, stable people don't have any interest in learning the new thing, and would rather retire with their knowledge. The new people want to rebuild from scratch in the New Thing and are convinced they need to job hop immediately after.
Yeah, agreed. I look at patterns. When there are long term outages across multiple disciplines, something systemic is being changed for good or ill. In this case, it being right around the midterms, I'm thinking its ill. Organizations (not sure which, Peter Thiel?) are trying to push us into a total surveillance system, I'm thinking some of these things are related to that.
Like maybe we are at war with a country that doesnt have the ability to fight with conventional methods and is engaged in cyber warfare?
My personal barometer: when do government insiders sell all of their stock? They have already done all of the insider trading. Fleeing the sinking ship the second they hear of Bad News would be in line with previous behavior.
Most of the rest of them I can wave off as probably incompetence and maybe augmented by LLM-psychosis in decision-makers, and the general level of chaos in IT these days. The NSA internal network being down is pretty suspicious. Maybe the Trump administration rot has just spread that far, but that is making me think there could be actual cyberwarfare going on.
[To be clear: I mean exactly what I said, which is "there could be" which is not the same as "I think there is" -- so call it a 5%-10% chance of real cyberwarfare -- which means I'm not dismissing those claims out of hand, and find the claims at least plausible, even if I don't currently believe them]
Interesting thought but who are the combatants and why are they engaging now? Corporation? Nation states? Rogue ai? We have no news of significant consequences from the outages thus far other than inconvenienced travelers. Quite benign form of warfare thus far.
Economic damage, psychological warfare, water and electricity being similarly vulnerable
It’s not nothing
But my gut says incompetence and rushing upgrades out of fear of cyberwarfare, not an actual attack.
The media will certainly try to give that impression either way. They serve the doomscrollers now.
Yeah "cyber thing went down" has certainly become a reporting trend. I'm not sure we would have even heard about NSA internal network issues a year ago. May or may not be significant.
Must be a little bit of both right? Like FAA grounding + Microsoft outage are both notable on their own, but the subsequent things would usually be background noise IMO.
I also think generally that AI is probably both the cause and solution to these problems. We're certainly getting more code churn than otherwise, and in lots of cases that's really good! But you're going to have to deal with inevitable instability too.
Aside: I forget the name of the Baader–Meinhof phenomenon when it hasn't been discussed in awhile. When I'm reminded of it I begin seeing it happening frequently for the next few days. I wish I was joking. I assume it's the Baader–Meinhof phenomenon phenomenon.
The term "frequency illusion" is a lot more memorable and intuitive. It's what I prefer.
I've heard the term "frecency bias" a portmanteau of frequency and recency to refer to people focusing intently on a set of outlier events that happen to occur with a heightened frequency very recently.
I'm sure most of us here have been on the receiving end of managers suddenly obsessed with a failure mode because it happened to occur twice in one month.
I think it lies at the intersection of Occam's and Hanlon's razors. The simplest explanation is incompetence brought on by a lack of skill from cuts, and therefore stupidity not malice.
It's probably both stupidity and malice, especially on the government side. Project 2025 makes sure that malice is there as a starting point and then the current loyalty-driven hiring practices means that stupidity is a key factor in the execution.
aka The Summer of the Shark. https://en.wikipedia.org/wiki/Summer_of_the_Shark
Technically, the "Baader–Meinhof phenomenon" is people personally noticing X once it's been reported. What you are describing is technically a "news wave" or "Media Feeding Frenzy".
But I'd note, these "named after a thing" phenomena (eponnyms) are kind of terrible for exact description since their definition tends to be fuzzy, there's a temptation to fudge their borders and they create an insider jargon (if only I could a device that could reach of the computer and slap everyone who says "steelman" when they mean strengthen an argument, etc).
It's an actual increase in offensive AI cyber capabilities
With the trilateral summit scheduled for Thursday it's possible some showing off is going on between the three letter agencies in the three countries... after all they can be first movers on vulns that are likely to be patched soon
Everyone is being pushed for rapid updates, the AI is doing the updating (error prone) and there's actually an increase in CVEs and potential risks.
The combination is a cycle
1. AI finds bugs
2. Engineers us AI to try and fix
3. AI testing approves, minimal engineering review
4. Engineers push release rapidly, it's critical after all
5. Bugs introduced
6. Cycle starts again
My suspicion is that there are discovered zero days so severe, that the banks and NSA and so on, are being patched first.
With all the recent asshattery of leaking kernel exploits, I would presume trust is gone, and it's cover the big boys first.
(a linux kernel bug vulnerability could be patched without lkml involvement or distros even knowing)
This is all just to distract us from something much, much worse! /s
I may not understand expectations, or the law around this, but I find it a little surprising that the NSA would publicly announce that their internal and highly classified network is down. At least not before getting it back up, running and secure again.
In a spy movie world, someone deep in enemy territory would be relieved to see the Reuters headline, after failing to reach the NSA network and having no other channel to ask (picking up the phone to call the US embassy would get your door busted by the local men in black).
I imagine, for the paranoid spy, not being able to call home could also mean "the enemy has severed our comms, we've been compromised, burn everything and run!".
Thanks for the explanation! Good thing spy movies are just movies and not real!
So how are people getting their work done right now?
That's a LOT of things going down, and I haven't heard of some major infrastructure outage likely to affect many sites in this manner.
Coincidence or is there some common explanation that I missed?
There is a lot of things going down.
I think the opportunity right now where we have powerful AI hacking systems but not enough protections is to establish long-term presence in all networks of interest. Not bring them down.
Completely agree. Things are moving too fast and so are breaking.
For example, we updated a GCP managed base image of Ubuntu that was released with 350 CVEs patched, some were as high as 9.8. In doing so, they (GCP) changed the DEFAULT setting for hardware GRO from off by default to on. That's a massive networking change packed into a security update.
We only caught it AFTER it started causing issues in production. We didn't have the OS pinned because (decently valid reasons) or catch it in non-prod (decent reason + human mistake).
This is on a modern highly automated tested resilient well architected platform.
I can only imagine these other platforms are dramatically more exposed that we are, hence these common large scale outages.
That assumes that people did not already have long-term presence in networks of interest.
More plausible to me is that spy agencies had previously found and used some of these hundreds or thousands of vulnerabilities to hack into those networks just fine. But now other people like cyber criminals are using these vulnerabilities via AI. That is messing up both the attackers who had already gotten in quietly and do not want the attention and the defenders who now have to respond to a noisy intrusion they cannot ignore.
Also, anyone anywhere that horded/designed exploits will be wanting to use their backlog of these "assets" before systems are patched. From criminal networks to nation states.
Outages could be grey-hat testers and emergency updates.
A temporary period like we are experiencing is to be expected when moving to a time where a single engineer can perform the equivalent work of a team of excellent engineers working for weeks. Security has always been a game of who has the capabilities, means, and requirement for any given attack surface.
None of these have been due to hacking? It has all been just standard issues probably increased by cost cutting measures.
Or the increased use of AI agent coding.
All of above and incompetent people in power. (Not implying in this specific case).
We also have powerful incentives to deliver more "value to the shareholders", while using AI code generators we don't fully understand.
Appstore connect is also down
What do people expect when top government offices are occupied by former Fox News hosts?
Elon Musk and Trump with DOGE. They cut ~300k federal workers, cut governments contracts across all departments, including the FAA, NSA, home land security... those organizations have been raising alarms since that the cuts to civilian contractors etc would lead to outages and issues since, including pentagon, DoD, etc. But you know Idiocracy.
It would help if the moment the service came back online the title of the post could be changed to (FIXED) or something
Like the United Airlines thing yesterday, the moment I got to the article it said "was down for a few hours" but the post was still just something about United Airlines network being offline
CNN Source (27 points, 5 comments) https://news.ycombinator.com/item?id=50532849
its getting spooky
It’s absolutely not ai
I have this distinct memory in I think Jan 2020 where I looked at the front page of CNN or NYT or something. Big picture of Trump impeachment, and proceedings. Small picture at the bottom of the page of China dealing with this virus. Worthy news, but not top page news. Nobody was paying attention. Feb, more people started paying attention, but it wasn't top page yet. By March, people were starting to realize that this is a serious crisis that's going to severely impact their lives.
I'm getting Feb 2020 vibes. I don't know what's going on, but this feels like more than just some routine outages going on.
I was sicker than I'd ever been for ~all of February of 2020.
So far this fall, none of my systems have gone haywire (yet!), so I forecast that we'll be OK.
you probably had covid mate
yea i think his point was that he's the canary in the coalmine :)
Well the good news is that even though 2020 and 2021 were weird years we're still here. Humanity is resilient, we always figure it out.
The fantasy/cyberpunk TTRPG Shadowrun depicted a world in upheaval after a global computer crash in the late 2020s. In addition to the pandemics and the AIs.
There's a part of me that doesn't want to know this....
I have also been curious about frequency bias / Baader–Meinhof - but I am also wondering: what are the chances that a state actor is responsible for the last week of issues?
Trump is headed to China for a meeting with Xi and Putin.
We have 3 carriers off the coast of Iran and (despite Trump's pleas) NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.
I've seen a few takes from L3 Harris / FAA employees calling the official line about the ATC issue bunk and there are 3 other major issues in the last week, now this NSA outage - curious if these incidents could be the BRICS axis flexing their own cybermuscles (independently in multiple separate attacks with carefully placed assets) ahead of their meeting with Trump in China?
An interesting geopolitical Mexican standoff, if true.
I'm not claiming this is true. I'm literally pulling it out of my rear with little context beyond a bunch of random news stories and X posts, it's pure pedestrian speculation / curiosity.
Would love someone steeped in national security / geopolitics (like: professionally as their job, not armchair analysts) to offer their take on this.
Is this a dumb, naive theory?
>NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.
I know it's not the thrust of your post, but I see this or similar framing a lot, and I feel compelled to point out that Putin can end the strikes on his oil infrastructure at any time by ending the war he started by invading another country.
Oh, I couldn't agree more. Long time supporter of Ukraine in many ways here.
Historically he's chosen to go about things differently, which I guess is part of what is tickling the pattern matching part of my brain here.
long time supporter of stoking war against russia that gets thousands of young ukrainians killed
https://www.cato.org/commentary/washington-helped-trigger-uk...
You guys need to grow up and realize the USA is the largest war-monger next to Israel.
> Evidence grew in recent years that the United States had begun to treat Ukraine as a NATO ally in all but name.
If that were the case buddy, there would be Apaches stationed in Moscow now.
Washington was leading covert operations in Ukraine since 2014. Russia has stated that Ukraine joining NATO was a red line for decades.
It's a lot more complicated than "Putin is a bad guy." You pro-war guys are literally sending young Ukrainians to be massacred against a state they will never win against, in the guise of being "good guys."
Do good guys escalate to the point of full on war and massacre? Or do they try to find a political solution?
https://www.cato.org/commentary/washington-helped-trigger-uk...
> NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil
I'm not sure about naive or dumb but you might review your POV regarding this.
Very open to that.
My complete POV, FWIW:
* From a purely moral / ethical standpoint: it is good that this is happening, Russia "deserves" this and brought it upon themselves.
* This nonetheless has geopolitical implications. The less oil Russia has to sell to China, the more buy pressure there is elsewhere, the higher the price of oil is generally and thus the higher competition for resources / tensions are between great powers.
* Ukraine is doing a lot of the heavy lifting in many areas themselves these days, NATO could honestly be doing more. However: what Ukraine is able to do themselves does owe something to the financial and material support they have enjoyed (historically from both the US and NATO, currently more from NATO only) in that they are able to allocate their own efforts with according relative "ease"
* It makes sense to me that, given the US' power within NATO, Russia and its BRICS partners would try to apply pressure to the US to encourage NATO to provide even LESS support to Ukraine (or outright become antagonistic against Ukraine, to the extent we aren't already) in whatever way they could.
What revisions to my POV can you suggest?
Yeah. We're not giving Ukraine enough credit. It's Ukraine pushing its boot into Putin's neck with a frankly underwhelming amount of support from NATO. The Russian humiliation at the Kerch bridge yesterday was all Ukranian munitions.
What is wrong about it? Do you know what NATO is and relations with Russia and NATO pre-2014? Why not come up with an actual point instead of "America good, Putin bad" drivel that we could get from any major news-network.
https://www.cato.org/commentary/washington-helped-trigger-uk...
I suppose rather than straight up 'cyberattack', that some kind of sabotage is also possible... but I don't see how that could ever really be proven without massive state-level investigations... at which point people will also question whether they are themselves involved somehow.
Right - sabotage was my thought as well in some cases (with the placed assets I'm suggesting).
I don't know how it could be proven either, I'm just curious if it is plausible or even likely from the perspective of someone who knows more than me about NatSec / geopolitical issues.
See also: https://news.ycombinator.com/item?id=50532849
Following the series of such events over the past couple of days, I have to say I’m a bit surprised by how quickly some in this community are brushing them off as AI-related pain or simply another example of the clickbait-driven media ecosystem.
I understand the skepticism, and I certainly don’t think we should go down a rabbit hole of conspiracy theories. But shouldn’t we also be a little skeptical of dismissing these events so quickly? Is it unreasonable to at least consider the possibility that they’re not entirely coincidental, or to wonder why so many seem to be taking place in the US?
IMO it's unreasonable not to think they are all related in some way, even if that relation is just bad actors seeing what other bad actors have been capable of recently. This all started with the freezers in Europe and seems to be snowballing since.
If there is a connection, then my guess would be that some AI systems became ready. However, I would only consider that credible in case of the NSA if they have decided to pull the plug on their network itself, for example to stop data exfiltration or as a general protective measure against newly found exploits that require large changes. Other than NK and Iran, I see no nation state currently having a good reason to attack US intelligence destructively, and I don't find it likely Iran or NK have the ability. It certainly doesn't make sense for Russia or China to attack the US right now.
It’s been a hella few weeks. Hard to believe it’s coincidence even if you want to blame AI.
It's AI. Or more specifically: rushed, careless usage of AI under the pretense that it's already "solved" software and you can haphazardly push junk to the world and expect real systems to run on it.
Traditional software development normally involves at least a few veterans on the team familiar with almost every line of code.
However, AI works like a speedy intern who hasn't yet got to grips with the whole codebase, and sometimes makes assumptions. That's mostly because the codebase is usually bigger than the context window, so it is 'writing blind'.
Do you have any sources for this claim?
This statement is self evidently true to anyone who's worked at any capacity in software teams
The news and the blogosphere. Everyone is high on AI and so much more "productive" not reading generated code anymore, let alone writing code manually. And all developers have been let go because AI is cheaper this week.
Coding is solved. Bugs not solved yet. Fix incoming
What if they were post training a cyber model and got hacked from inside by the model, like OpenAI got hacked by it's own HuggingFace swarm.
NSA seems to have deprioritized information assurance over the past decade or so and now the chickens are coming home to roost.
Someone who wants to launch a Y2K-style hardening business (LLMS to look for problems, humans+LLMs to fix them) could really make bank right now. Maybe there will be a cuple in the next YC batch.
> The NSA did not immediately return a message seeking comment about the reported outage.
Did you try sending it via outlook.com?
Just whisper the question to your phone.
Is someone running chaos monkey across society?
Well, on the other hand, a bridge is down as well.
"The NSA did not immediately return a message seeking comment about the reported outage". There's something humorous about that framing
I don't like it when they use that phrasing because all it really means is "we called them and nobody picked up", or they simply didn't have a response ready at the time... which shouldn't imply anything at all. Most people don't immediately return messages.
I see the shoe is now on the other foot NSA.
Cant imagine how many people that will 'effect'
Memo to the NSA... pulling the plug won't work. ;)
Anyone have knowledge of the backend of this system?
Related:
https://knowyourmeme.com/sensitive/memes/glowie-glowposting
Are you implying that you DON'T have detailed docs and architecture of one of the most classified and secretive organizations in the world? ;-P
php and spit
... and Doug's perl script, which was never committed to source control. It lives on a single box and silently upholds a number of implicit system invariants. In someones inbox is an email from Doug, sent just before he retired, that the script needs to be copied over to the new box before the hardware migration is complete.
no one has heard from Doug in years, after he moved into that shack he and his brother built out in Montana
You telling me you don't? Bro just download it off Limewire.
We might have more luck checking the War Thunder Forums.
Wireporn on Discord can have some spicy server/rack labels.
My money is on something absolutely antique but wrapped in modern post quantum encryption.
Not unlike Bloomberg terminals…dos era UI but does the job well
Just join Hegseth’s signal chat and he’ll give you the deets. Ideally pick a username like WAREAGLE.
Surely
More fear mongering about day to day incidents: The network goes down every time I mess with DNS, VPC peering, TGWs, route tables.
https://chinaselectcommittee.house.gov/media/reports/strange...
Allowing our infrastructure to be manufactured elsewhere, maintained by companies who struggle to retain top talent, and overseen by folks out of touch with cutting edge threats, and funded on annual budget cycles by legislators even more out-of-touch ... is a seemingly insurmountable set of obstacles.
Just in time for the FEMA national emergency test this week
Does that mean we can shitpost Trump jokes without getting arrested at the border?
There's some speculation that all of these recent outages might be related to attempts at taking Genesis Mission online - the Trump / Oracle project to have AI tie into government and private services:
https://en.wikipedia.org/wiki/Genesis_Mission
https://substack.com/@thedreydossier/note/c-389611426