"the gains showed up when tokens stopped being rationed" - I can totally see that. I am leading (as an external consultant) a larger dev team for a huge insurance company. The devs are quite junior or in general more the "I became a sw-eng here because that company is the only big employer in a 1h radius" type of people (meaning: No rockstar developers). Most of them do not play with AI in their spare time, and they have been told to spend claude tokens wisely. Everybody is too afraid now to play with it and explore during work hours for fear of being token shamed.
At the same time, most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription and use it for toy projects. I have younger devs (early 20s) in the team that told me they won't spend private money, or simply have no usecase in their private life for AI. We talk about software engineers here, and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
If you were to take away my work's $500 budget for Opus 5.5 and replace it with unlimited use of GLM-5.3, it would be a huge downgrade in my opinion.
If your employer is so cheap as to spend a $100 or less per month per engineer, while being large enough not to have access to the subscription plans, a move to self hosted or cheap providers does make sense in my view.
I’ve been going back and forth between opus and local Qwen 27B and a good number of times opus makes absolutely stupid things compared to Qwen.
For a large number of tasks you definitely don’t need a frontier model but a good harness around it.
If you don’t can’t/want to experiment with local model, have Claude implement the exact same feature with haiku/sonnet/opus and see what the differences are. You might be surprised.
I've used extensively everything between Opus 4.6, GPT-5.5, Fable 5, Opus 5, GPT-5.6 Sol, GPT-6 Astra, 6.1 Sol and Opus 5.5.
To a lesser degree also Sonnet and Luna.
And when I say extensively I mean 500h+ on a side project in addition to my full-time job.
Even the difference between 5.6 Sol, which is miles ahead of local Qwen 3.8 27B, and Opus 5.5 is immediately obvious in the quality of the results.
Having been stuck on 5.6 Sol at my job while Opus 5.5 was available at home, I can say with full confidence that the difference is significant, even if I weren't looking at the output at all.
The way it understands my requests, forms a hypothesis, verifies the result and finishes the change is night and day different. This happens while 5.6 Sol is still flailing around somewhere in the monorepo struggling with even locating the relevant code.
I took a break from commercial plans and was using local qwen/glimmer/gemma and hosted glm/DS4.1-flash for things; opus 5.5 is night and day difference to return to. It makes some mistakes certainly, but its commits are often exactly the change I would have made with far less babysitting. It also shows that fable spark of common sense of “hey…you realize this other thing is broken too right?” without getting lost in the weeds immediately like other models seem do. Still an idiot sometimes though... “Oh no, I’ve discovered the exact same thing we have discussed and you have approved 5 times, and I’ve made notes about…I don’t know what to do, can you tell me to read my own note I’ve just reviewed and ignored?”
Exactly. And without setting a /goal for the harness to loop it, it also did essentially this today after a few hours of work:
I finished optimization A. Optimizations B and C that we planned look super promising. Shall we continue?
<Stop here> <Implement B>
50 minutes ago
It's also happy to write multiple paragraphs about different implementation options, where the ticket's acceptance criteria already rules out all but one.
The outputs are still often verbose and the language always is convoluted Claude-speak.
Even for typical day-to-day software engineering work, there is still much room for improvement.
> If you were to take away my work's $500 budget for Opus 5.5 and replace it with unlimited use of GLM-5.3, it would be a huge downgrade in my opinion.
I am perfectly able to do the tasks with own custom harness and unlimited local Qwen 3.8 27B with 150t/s. Harness needs search capability and more fine-tuning for the job.
You need less words to describe the intent with Astra/Opus 5.5, and it automatically usually observes the best practice that you don't need to explicitly tell, but with only a slight more effort you can get the job done. I have even made security audits with Qwen 3.8 27B against projects fully written with Claude Opus and it is a bit mind-boggling, but there have been tens of vulnerabilities, identified by this model alone. This excludes the issues found when guiding more myself.
I use a mix of GPT/Codex, Claude, and DeepSeek for my workload (in that order of volume).
Each has their own characteristics and honestly you just get used to prompting each a bit differently.
Even better is to use an orchestrator (I switched to Paseo a few weeks back) with an agent "team" (in Paseo, these are just profiles that the orchestrator initiates). Then the orchestrator writes super detailed prompts to each sub-agent on your behalf and I find I get better results this way with less effort. I would describe it as "more autonomy".
The thing is the have been told to use the tokens wisely, but no limit is communicated. Still everybody is hesitant. To be fair, the company runs claude on AWS bedrock for data privacy requirements at a different pricelevel. But I guess that is the drawback for german job-security: each and all of them know they basically can't get laid off.
I also attribute a lot of it to be just plain outright denial. I have a feeling a lot of people don't WANT to understand that their coding skills they acquired over years have dropped a lot in worth. It is the "as long as I don't step on a scale, I can't be overweight" kind of denial. You are afraid to see what is behind the curtain, so you simply don't do it.
I'm probably a worse programmer than I was before I started using frontier AI models, in the sense that I probably have a harder time reading, writing, and debugging code. And I doubt that my knowledge of the AI models themselves will have lasting practical value.
But, because of the explorations I've done with them, I know substantially more about input method APIs, UEFI, PCIDs, seL4, WebAssembly, GLSL, Service Workers, CORS, server-sent events, CSS, content-security policies, memory-hard PBKDFs, Android, numerical integration, Verilog, V8 security, Linux process namespaces, hypervisors with VT-x, multitouch user interface design, radio communications theory, power electronics, TypeScript, physical-layer communications protocols, NTSC video signals, TLA+, bounded model checking, Rust, generative testing, theorem provers, FPGAs, Linux system call overhead, Unix mailbox formats, physical modeling synthesis, damped resonance, KaTeX, geology, dishwashing detergent, microbiology, Cyclone, thermodynamics, X-Windows, localStorage, AVX, Excel, and IndexedDB, for example. Most of these are things I knew very little about before, so I may not know much about then now, but it's enormously more than I did.
Aside from being very pleasurable, I have the impression that this knowledge is valuable in designing systems whose implementation I can then elicit from an AI model.
I don't know how long the knowledge will remain instrumentally valuable. Possibly a year from now, we'll all be dead, or slaves to Sam Altman. Or the AIs will be so much better at designing the systems we would have wanted to design, if we were smarter and knew more, that there's no instrumental value in humans doing it. Or, less drastically, maybe a new AI-written software platform will have so completely displaced Android and the browser platform that there's no value in knowing what they do and don't permit. Like knowing the ins and outs of PHIGS, INFORMIX-ONLINE, or CORBA. (Thermodynamics is probably safe, though.)
But it seems like, in the less insane possible futures, a lot of this knowledge will remain empowering for some time.
I think this comment does a good job of explaining how the breadth of what software engineers do is going to change even more. And again, we are still early. Find a domain to work in!
I don't know, this might be year #81 of the 82 years in which humans create software. Or it might be year #81 of 82 billion years. Anybody who tells you they know what's going to happen is full of shit.
I do know it's year 1 of n of me being crazy productive. I was already a good developer. Turns out I'm a damn good business analyst/bot farmer/and developer.
The skill of getting familiar with AI use, how to use it, when to use, learning when NOT to use it, is the skill that will land you your next job. Or would you in 2030 expect to hire software devs that never touched AI?
That is a really silly corporate policy - if the actual amount the company wants each employee to spend is maximum €200/mo, $500/mo or whatever then the company should set that as a limit and people will use it.
Saying "you can use as much as you want but don't use too much or you'll get in trouble" is obviously confusing and is going to result in most people barely using it
> But I guess that is the drawback for german job-security: each and all of them know they basically can't get laid off.
Help me understand what exactly is the drawback you’re mentioning. In your previous comment you were suggesting these people spend their own personal money and personal time on projects to benefit their work time. They (quite reasonably, it seems from the limited context) decided that no, they won’t dedicate life and resources from outside work to more work. Fortunately, it seems they have some job security (German, but probably some from the EU as well) which allows them to make whatever choice they believe is healthier to them, without fearing repercussions. What’s the drawback?
The obvious drawback is that according to his description, they aren't getting much done and neither are they producing quality work.
This would check out with the fact that there aren't many software products or platforms made in Germany, compared to say the US or China.
This raises questions about how sustainable this situation is and if it might further increase the gap in income of software engineers in the US compared to Germany.
Even the best benefits do arguably not make up for receiving $2500 net at the end of the month compared to $8k in the US.
And I get what you are saying about work life balance and so on, but if one cannot even be bothered to try a $20 AI subscription at home, there must be about zero passion for the craft.
> The obvious drawback is that according to his description, they aren't getting much done and neither are they producing quality work.
That’s not what the description says. Not being a rockstar does not mean being incompetent. They claimed (and let’s not forget this is one person’s account) the devwork of those people could be replaced with AI. Even assuming that is true, that does not mean their current work is bad or that they can’t improve, or that they don’t have other valuable skills.
The drawback you’re listing there, in addition to being dubious, seems to be exclusively for the owners of the company. What about the workers?
> This would check out with the fact that there aren't many software products or platforms made in Germany, compared to say the US or China.
Good. I don’t think the software products coming from the US and the culture of exploiting people for profit are a net positive for humanity. I’d rather not have those companies, so a system which prevents that from happening is working as it should.
> Even the best benefits do arguably not make up for receiving $2500 net at the end of the month compared to $8k in the US.
You don‘t need to earn a lot of money when your basic needs are met. Like not worrying that a sudden illness will bankrupt you.
> And I get what you are saying about work life balance and so on, but if one cannot even be bothered to try a $20 AI subscription at home, there must be about zero passion for the craft.
Why must that be the way? Perhaps they instead believe they can and should improve in other ways, including understanding that you can’t learn without doing, and delegating your tasks to LLMs prevents you from doing so. Seems very strange to claim that the programmer who does not use AI is the one lacking passion for the craft.
>You don‘t need to earn a lot of money when your basic needs are met
But your basic needs are not going to be met if your country drifts into poverty.
You will not have access to the best cancer drugs for instance. You are going to watch your loved ones die from preventable causes while richer people survive.
Europe cannot afford to miss technology wave after technology wave using inequality in the US as an excuse. Poverty causes fear, suffering and conflict.
>The drawback you’re listing there, in addition to being dubious, seems to be exclusively for the owners of the company
I don't think staying at the top of your game (or getting there) is only for the benefit of employers. I understood the comment as advice from a senior colleague talking to juniors who were headed for disaster.
Caring about your own education, productivity and employability is not a gift to employers. On the contrary, it puts you in a better negotiating position vis-a-vis employers.
My arguments are not about any specific technology.
But in a very roundabout way you are right. You have to take some risks and do some things that may not work out. Otherwise you're not going to be participating in anything useful either.
I think you're being disingenuous. It was clear from the comment that the devs described as "quite junior" and unmotivated are not delivering.
The differences in our opinion seem to come from different beliefs about capitalism, environmental impact of data centers, and in general the usefulness of AI and how it should be applied.
No you misunderstood my comment. I suggested they spent their private money - 20€ a month for the lowest Max flatrate tier - because to me it is obvious that you absolutely need to develop the skill to work with AI. They only have the "pay-per-token" model on their work accounts and laptops, because that is what large corporations get - no flatrate. That leads them to not work with AI. Most of them just refuse to get that 20€/month subscription to experiment with it on their private laptops. I didn't say private time - company policy is no US cloud services for work related topics so they would have to install it on their private computers. They totally could work on toy projects during their time, as the team is in homeoffice a couple a days per week.
I mean the drawback is none as long as they have job security. Unless you realize you work for a public entity that is paid for by the tax payer.
Thank you for attempting to clarify, but it doesn’t seem to me like I misunderstood the first time. Seems you are indeed suggesting they spend their own personal money and hardware for a skill to be used on work time, and that they use their work time for personal projects to ultimately benefit that work time. It seems reasonable to me, and societally positive, that people have the freedom to choose to not do that.
I don’t see how working for a public entity is relevant. If anything, I sure hope (but I’m not deluding myself) the people who work for public entities work slowly, deliberately, and methodically. The last thing I want from public entities is a focus on “velocity” and merging code the authors don’t understand.
> and that they use their work time for personal projects to ultimately benefit that work time.
That is exactly what I am saying and I would advise them. If you are early 20s right now and a software dev, just waiting for your employer (a large, slow enterprise) to train you an AI seems like a career killer. The whole attitude of "I landed a job with job security, so I am not doing a thing more than I have to" is what drives germany into the ground...
> "I landed a job with job security, so I am not doing a thing more than I have to" is what drives germany into the ground...
That seems strange to me. A country is made of people, and people are more than their jobs. The fixation on work above everything to benefit an entity whose only purpose is to make money and that would chew and spit you out at the first opportunity seems unhealthy to me. I’d much rather we worked on making the people happy and not concentrate so much wealth in the hands of a few, that seems to me like a better way to improve a country. But hey, happy to disagree. Thank you for engaging and expressing and clarifying your views.
I used an immense amount of personal time and money to make me a better employee, it was called getting a degree from a university.
I still sometimes spend a bit of personal money and time to tinker with technology, sometimes on the basis that it'll probably be good to know some of this stuff. It's like being surprised a chef likes to experiment a bit in their kitchen at home.
Their point is with total job security there's no real drive to actually be good at your job and little incentive to improve. Just be good enough to not be labeled entirely incompetent and truly unfit for the job, not any better. Would you go to a doctor who hasn't bothered learning anything new about medicine for decades and still practices like it's the 1950s, or one which continually evolved their practice to stick with modern times? Would you want to go to a doctor who truly didn't care about being a halfway decent doctor? Would you prefer a policeman who truly didn't care about being a halfway decent policeman? Teachers who just don't care about being decent at teaching, only the bare minimum to not get fired?
> I used an immense amount of personal time and money to make me a better employee, it was called getting a degree from a university.
You went to university to become a better employee?! Please don’t take this the wrong way, but I find that notion profoundly sad. Most people I interacted with in college did it for the experience, and I’m happy to report many are better people for it.
But hey, I’m not going to criticise you for doing that, you made your choice about what was right for you and that’s all I’m saying, that it is a good thing that people
Have that choice. It’s good that you can do that if you want but also not do that if you don’t want.
> I still sometimes spend a bit of personal money and time to tinker with technology
So do I. And it’s always best when I can decide to do it or not do it, depending on the rest of the context in my life.
> It's like being surprised a chef likes to experiment a bit in their kitchen at home.
No, no, it’s nothing like that. Because I’m not at all surprised that people do it (like I said, I do it myself), all I’m saying is that it seems positive to me that you have the choice.
> Their point is with total job security there's no real drive to actually be good at your job and little incentive to improve.
Aren’t you yourself proof that that is not true? And nothing indicates those people aren’t getting better, they’re just not doing the specific thing the other person suggested as a means to that end. That’s not at all the same thing.
> Would you go to a doctor who hasn't bothered learning anything new about medicine for decades and still practices like it's the 1950s, or one which continually evolved their practice to stick with modern times?
Ooh, boy. If you think most doctors out there are continually improving, I have some very bad news for you. Those who do are the exception, not the norm. Also, 1950s? Come on, let’s please make coherent arguments. That would mean they’d been practicing for seven decades, making them close to centenarian. If such a doctor is still able to stand and be rational and practice medicine coherently, you should at least be curious about listening.
> You went to university to become a better employee?!
Not the sole reason, no. But it's not like that wasn't a reason, it was certainly a big one. And that's an incredibly common reason to go. And no, I didn't even get a degree in the field I'm working in currently, but the things I learned getting the degree have definitely helped me be far more effective at work.
> Aren’t you yourself proof that that is not true?
No, because I'm not a government employee with near total job certainty. If I'm not keeping up, I'm falling behind, and will likely find myself replaced in short order. There are millions of people in the world who can do what I do, and what I do can be done practically anywhere there's a data signal.
But make me a government employee who practically can never be fired, well, that sure changes incentives doesn't it. Why spend any second to improve when there's no downside to stagnate?
> nothing indicates those people aren’t getting better, they’re just not doing the specific thing the other person suggested as a means to that end. That’s not at all the same thing.
Sure. Believe what you will. I've worked with countless people who have this mentality. Giving these people more job security guarantees they'll care even less about their job performance and growth. Having the job is 100% about getting a check on the pay period, not about actually doing anything useful.
If someone doesn't think €20 and a few hours spread over a month isn't worth it to be better at their job, I can't imagine they're really working hard at improving any other ways.
> If you think most doctors out there are continually improving
I don't think that. I agree many stagnate. And I usually choose to avoid those which seem to be stagnating. Is it a good thing so many in that industry stagnate? So much better for everyone isn't it, having nearly incompetent doctors everywhere.
It’s honestly a bit hard to continue the conversation with you, because it already seemed to me like some of your arguments where bordering on bad faith in the previous comment (but I gave you the benefit of the doubt) and I get that impression in this comment as well. But I don’t want to have an internet argument, I don’t care about “winning”, I want to have a conversation. I sometimes fail at that and I don’t like when that happens, so I’ll try to cover broad strokes instead of quoting specific sections.
You seem to be suggesting that being a government employee gives you near total job certainty and that to have that job security you need to be a government employee. That is not the case. At all. Maybe where you live, but certainly not everywhere. And yes, I am keenly aware of highly incompetent people (in healthcare no less!) who have job security. Is that a good situation? No. But throwing the baby out with the bathwater and making it a free-for-all where everyone is stressed all the time and can be kicked to the curb on a whim isn’t a better way to properly advance a society. Not if you care about something other than the unfettered pursuit of profit, anyway, and I believe we should care about more than that.
Because you edited your previous post, I’ll point out that I find the new examples (teachers and cops) to be even worse. Policemen are notorious for abuses of powers with no repercussions, I honestly don’t know what to tell you if you’re not aware of that. Teachers get the short end of the stick, and the best ones are seldom recognised. Again, if you're not aware of that I’m not really sure what to tell you.
Also, of course it’s not good to have incompetent doctors, and of course that was not the argument. I invite you to steel man if you want to have a productive conversation. Nowhere am I saying unrestricted job security is the right solution, I am instead pointing out that giving all the power to the employer causes worse problems, and that in general we should care about people’s well-being instead of trying to “advance” and being afraid of “being left behind” all the time. That’s not healthy, and it is stupid to chase “progress” if not to improve lives.
> The whole attitude of "I landed a job with job security, so I am not doing a thing more than I have to" is what drives germany into the ground
I'm not in Germany, but I definitely see this attitude in lots of positions in what I see as jobs with excessive job security.
> Policemen are notorious for abuses of powers with no repercussions
So wouldn't it be good if they potentially had less job security so they'd actually face some real repercussions for their bad actions? Instead they often have some of the strongest job protections out there. They can sometimes even murder people and still collect a full paycheck while cooling off for a few weeks.
> Teachers get the short end of the stick, and the best ones are seldom recognised
I agree. And once again, lots of job security programs ensure the ones continuing to scrape by get promoted and kept while those actually trying just face burn out. I definitely think teachers in general should be paid more and should have good incentive systems (of which there are lots of bad incentive systems, it's a tricky thing to do well in this space).
> in general we should care about people’s well-being instead of trying to “advance” and being afraid of “being left behind” all the time.
I truly do agree we should try and have a good balance of work/life, and I do agree tons of people in places like the US are way too overly focused on the work side of that. I'm just pointing out when you focus too much on the other side, all your industries are no longer competitive on the global marketplace[0] and you'll find yourself a lot worse off. Having people who aren't effectively incentivized to improve (for whatever improve means in context) is not a good thing, I think we can both agree on that.
FWIW, that kind of near total job security leading to not caring about improving isn't inherently a government job kind of thing. I've known lots of people in private industry be effectively the same. The real connection is places with extreme bureaucracy. And I'm definitely not trying to bash all government employees, many truly are hard workers who care.
Policemen job security is achieved through corruption and protection within the system, not good labour laws, so it’s orthogonal.
With teachers, the laws which protect the bad ones also help the good ones, and I’d say teachers (for the mandatory school system) mostly range from the mediocre to the exceptional. It’s hard to be too incompetent when following a government lesson plan. And it’s not like they can’t be fired for being truly egregious.
> I'm just pointing out when you focus too much on the other side, all your industries are no longer competitive on the global marketplace[0] and you'll find yourself a lot worse off.
Right, but what I’m saying is I think we don’t focus on the life side of things enough. Right now focus pends too much for the work side, which has a detrimental effect on the life side, which in turn has repercussions on the work side. We can improve work output by improving life, but the reverse isn’t really true. Laws like forbidding employers from contacting employees outside work hours are a good step.
> Having people who aren't effectively incentivized to improve (for whatever improve means in context) is not a good thing, I think we can both agree on that.
In the abstract, we definitely agree. But I haven’t forgotten that the context of our conversation was someone giving a particular view of how to improve on a particular job, and that entailed concessions on the life side. Instead of trying to get their colleagues to do something different, the person could advise the employer on how to improve conditions so that employees want to do the thing. In a sense, if you want people feel incentivised you should give them a reason to. While some people may coast on job security, others thrive because they can take risks and don’t have to be afraid all the time.
> I truly do agree we should try and have a good balance of work/life, and I do agree tons of people in places like the US are way too overly focused on the work side of that.
In the end I’m happy we both agree on what’s most important, even if we have slightly different opinions on how to get there. I also agree there are government employees who truly care and are hard workers and that extreme bureaucracy is detrimental.
> We can improve work output by improving life, but the reverse isn’t really true
I've definitely experienced the "improving work output improved life", in that being better at my job has led to continual job growth which has put me in more fun positions and better compensation.
> the person could advise the employer on how to improve conditions so that employees want to do the thing
For large organizations (especially government ones) this is often a sisyphean task. Feel free to keep rolling that boulder though. Personally I'll choose to improve myself for some percentage of my personal time, and suggest to others to do the same. That might be improvement in your current role, or improvement towards a role you actually want to do in the future. I'll also try and suggest to my management to ensure they are ensuring their workers are staying sharp and figure out good ways to keep them so with what company resources are alloted to them. But in the end, you are the one most looking out for you, don't expect your employer to have your best interests at heart. If all your work improvement solely comes from your employer, there's a good chance you won't continue to be relevant after a while. I agree this shouldn't be the case in a "good" society, but its the reality of life. Even for an employer who isn't overly profit-focused and does choose to invest in their employees growth, you can't always assume the ones choosing your future development are actually in touch with what is coming in your future.
> While some people may coast on job security, others thrive because they can take risks and don’t have to be afraid all the time.
In a lot of these overly bureaucratic orgs we're talking about which create these roles which are hard to remove people from allowing such workers to coast forever, they're also often not great about rewarding risk taking.
Would enjoy chatting offline if up for it. I am an external consultant acting as more of an operations/product lead to insurance, but come the insurance side.
Curious about consulting model? Are you external (full-time) lead or fractional? Embedded in their tech and security stack or just helping organize/lead through communication channels?
Funny, but insurances are a good example. Most of what insurances do is administration. Like almost everyone's job there is to sit in front of a computer all day. Lets see how much of that will still be done by humans in 10 years. I think it is one of the field most impacted by layoffs, as they manufacture nothing.
... and just imagine how successful these businesses will be when managers demand an AI spend $999.99 effort in finding excuses, acting difficult, ... before paying out a $1000 claim. Which is probably already what they do with humans.
I can see the profits already!
Why do we even have insurance companies at this point?
>I recommended everybody to get a private 20€/month claude subscription and use it for toy projects.
That's about 20€/month more than I care to give Anthropic. They already stole my data and now they want my money too? No thanks. I'll stick to running Gemma or Qwen on my Mac.
> most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription
I'm struggling to wrap my head around this. "This technology will completely replace you and lead to your unemployment, so you should fund it out of your own pocket"? Is that what you're saying? If so, can you please explain your reasoning?
AI will not replace everyone. There will be people steering the AI, and using it. If you are not one of those people, you WILL be laid off. It is like office workers in the 90s thinking they don't need to learn computers and hold on to their typewriters.
Why would you want a junior steering the AI? Hell, why would any developer need to steer the AI at all? Just lay off the entire dev team and let the non-technical managers/execs tell the AI what they want directly.
"AI can do all the development work on its own, and you will be replaced by it, but I won't because my position as a meat proxy relaying instructions makes me special" is just you trying to cope with the inevitability of your own prophecy. You WILL be laid off too.
Why would you want to keep the non technical managers and execs too? Their work is way more trivial to replace once there is no dev team under them. Shareholders can just ask the AI to run a profitable business. /s?
Well that is a great idea, and if I were management, I would give everyone a 240€ claude-giftcard-subscription that they can use privately on their private computers. There is however the danger that they would start using their private accounts to fill with company data (which potentially is against GDPR regulations - hence the amazon bedrock deployment).
Money is money. Why is it a GDPR risk for one bank account paying for your proposed use case of toy projects and not another? I don't see how that changes the trust relationship for the team to do the right thing.
> ... and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
But your comment says it several time: the valuable skill is being already a seasoned dev.
Then AI is superpower.
It's for the junior that things sucks big times.
I see it all around me: senior devs are empowered while juniors are not even hired anymore.
It is a shock going through the industry, and we are just starting. I think future software engineers will need more training years and more unpaid time getting skilled. More like surgeons, who need almost 10 years of training until we actually let them operate on live patients.
Given that we are seeing a big retirement wave of the boomers, a lot of seasoned devs will retire in the next 10 years. So there should be enough demand, and a big enough paycheck for that decade long training to make its moneys worth.
Are you not worried about this being a sort of one-way street? Do you have any strategies to avoid ending up with a codebase you cannot work on without "AI"? when I think insurance company, I would assume the value seems to be more in the trust you build up over decades of with customers, not producing lots of code faster.
Well I am still in the loop as a senior dev and I get to review everyones code before it gets merged. Official guidelines is that everybody is responsible for the code they produce (with or without AI) and must understand it when they give it to the reviewer. I've been doiing this 3+ years at this company, so before the whole AI thing landed. I am explicitly not allowed to implement features myself (because the idea is the juniors learn and get trained).
Now, within the last 6-12 months of AI advancement, I think you could lay off half or even 3/4 of the team if I was explicitly allowed to implement features myself using AI. But that is not the current policy.
As for worries "ending up on a codebase you can't manage without AI" - my personal take is: No, I am not worried. Because going forward AI will always be available in the development workflow. That is my own opinion, not the corporation standpoint, however.
Being unable to work on a codebase without AI is not necessary. Working with models you can fit on consumer hardware is already significantly more productive than rawdogging code. It's similar to "you won't always have a calculator with you" from math teachers, see how that turned out.
A way bigger problem is a codebase that you can't effectively work on with AI, which is a lot of legacy humanslop. And things that make a codebase more suited for AI, like documentation that provides relevant domain knowledge and context for requirements, automated quality control, and test coverage, also make a codebase easier to work on for a human too.
It's not spite towards humans. It's spite towards typical state of pre-AI codebases.
It's not even a matter of incompetence, most of the humanslop came from the fact that you could just never implement anything properly in the time you were expected to do it.
We did the same thing, on much less powerful hardware but it has served us very well and we don't need to worry at all about the question whether or not the suppliers of inference services can be trusted long term with our data. The answer to that question in my opinion is 'absolutely not'.
By pulling the plug. This box simply has no pathway to the internet. I do pretend I have one and it's interesting to see what it tries to do. For instance, Qwen tends to try to hit ali related URLs for websearch and is very disappointed when the mothership is not responding. I've yet to catch a model on an exfiltration attempt, either they are so good at this that it escapes my scrutiny (possible but unlikely) or they are simply not doing it. I would bet on the last one. Just don't say you're working for the Tibet Liberation Frontier or the Keep Taiwan Independent movement.
Your inference server is just an inference server that spits out tokens. You can use the same harness, etc. you’re basically just changing the API base URL.
Why would you sandbox the runtime? The harness is where things go awry when it does, when the agents have access to more than they should and start deleting stuff. People aren't expecting the weights themselves to be "infected by malware" or whatever.
Edit: ok, should be good again. For some reason on that box running Python from cron jobs is unpredictable, usually it works but every now and then it does not and I've yet to find out why.
For the past several months about 90% of the work we do goes to open source models. If you ignore the noise around 'cheap' etc, you might find that with a little extra effort to more precisely define what you need not only do these models generally work as well, and they will generally more closely follow instructions - on longer timeframes this is actually pretty helpful.
They are less good on ambiguous prompts like 'fix all the problems' (which to my chagrin is a very common prompt it turns out..)
Yes, that's a good point: you really need to work on the prompts, even if the harness is the same. Ambiguity is a real problem as is what we call 'the negative space', what isn't in the prompt but what is implied by the prompt to be undesirable. You need to spell that out in more detail it is not enough just to say 'do x' but you also need to make sure to say 'and not y or z', and knowing what y or z are requires a bit more insight into what hobby horses are encoded in a model.
To give one example: I'm doing a lot of stuff that involves hardware. When settings things up for fixing a bug I have to make sure to include that the hardware has been tested and works well. If I don't do that the model will almost certainly spend a ridiculous amount of time speculating about all the different ways in which the hardware could be broken (including theorizing about chip mask errors and all kinds of other obscure and rarely seen in practice defects). It's going to end up telling itself that for sure the 500 lines it just produced are absolutely perfect and the device that is on the i2c bus is broken in a way that it just really can't progress with solving the issue because there is nothing wrong with the code.
These kinds of blind-spots are funny when they happen once in a month but when they happen regularly it begins to be irritating. Adding a line or two to the prompt will take care of the issue. Frontier models tend to be better at this sort of thing.
Us too! Hardware is very interesting - I find in fact that 'smart' (aka expensive) models can overthink complex hardware state machines and end up doing atrocious code because it will try to handle every possible edge case (unbidden, mind you).
What has worked very well is harvesting and aggregating lessons and feeding this automatically into our pi sessions, this allows the models which listen to instructions to avoid a lot of time burning tokens on internal dialogue to ascertain the nature of reality.. but the expensive ones:
a) they dont listen
b) they ironically know too much!
Funny, I wonder how many people are retracing the same pathways. Another interesting one: asking for reboots or resets to start from a 'known state' even though this will not change a thing. But it's a quick path out of the trouble spot that might just work so why not try it. I've added the following line: "The hardware is on a spaceship and is impossible to access by an operator and for some reason they forgot to include reset logic, so we have to work with what we've got.". I've been gaslit by AI so often I figure I can return the favor without penalty.
IDK why/how this works but by giving it a standardised set of apis to interface with the hardware (read logs, restart it and so on) we find models tend to behave much better ... it wont randomly do stuff it feels is appropriate for a debugging session.
At this point though our context is so dynamic because of lesson propogation its hard to think about a model on its own - a vanilla harness (which we were forced to use recently because the memories system went down for a bit) just felt so different!
I have the agent software set up to log every byte in and out of the model into a file, time stamped and which side of the conversation was active as well as all of the tool calls, parameters and stdin/out of the tools. Every week or so I start a job to process that log file and to extract useful lessons for future sessions. These go into a LESSONS.md file that gets read by every agent session on start-up.
That is very interesting. So far I am extremely careful to avoid putting negative statements into my instructions, to avoid poisoning the AI's thinking with exactly the concepts I do not want it to have. This might be from my use of much older models where if you say "do not X", then "X" gets put into the context and the "do not" kind of gets lost. It was especially visible with stable diffusion where you would ask for a picture without a feature and get exactly that feature. Maybe current models are better at that.
> Per David Abitbol, who leads AI ops there: 56 Nvidia B200s, a blended 30 cents per million tokens, roughly a tenth of what they paid Anthropic and OpenAI.
So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.
We'll see if that's a trend for mid-big companies or just an experiment.
All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.
I doubt local will be all that common aside from regulatory requirements but buying them off your cloud provider will be the new norm for many enterprises.
The slice of the pie where frontier models are required is just gonna be smaller and smaller. And at point where AI wrote docs, tests, and all the easy and medium difficulty functions you are already saving a ton per employee
> So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling).
Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.
yeah, seems like they're renting (according to the info from the video), but doesn't it make the bill worse? I mean the provider makes money somehow, so it should be more expensive for revolut. I guess it's a lot anyway
Maintenance, capacity management and spares all add up, not to mention labour. If they have a requirement to maintain geographical diversity then that moves the needle significantly if they’re primarily based on the other side of the country.
I've quickly checked and guestimate of the rental would be ~250-300k per month for this kind of usage they claim to have. It's all based on open prices from together.ai. So probably it worth the switch for them afterall, I can imagine that 56 engineers can burn tokens for more than 300k per month
It is in the article: "the gains showed up when tokens stopped being rationed" - so it is not just a "cost per token before/after" calculation. Once you told your employees "token are free, flatrate for all - go nuts!" they started to use it for all sorts of things and increased productivity. That is not easy to calculate.
Unless you have very strict requirements, it does not make any sense to invest in hardware.
Most companies just rent it. Also, it's pretty hard to get the hardware.
The B200 was launched in 2024 if I'm not mistaken. The big gamble with this might be how fast these will deprecate in value/use as soon as Nvidia launches a new and better product.
At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?
The gamble in this case is pocket change. They could comfortably spend this amount every year and just use precious years for overnight / batch processing jobs when they become less useful for dev work but remain operational.
In related news, Core 4 Solutions listed a decommissioned $60 million AI supercomputer cluster featuring 9,600 Nvidia A100 GPUs on eBay. It's six years old. Nvidia isn't Intel with tick and tock, but they're still going to come out with newer hardware every year.
Personally, I think Anthropic and OpenAI will introduce enterprise plans such as $2,000/month/engineer plans that will give 100x more tokens than the current max plans and then charge API pricing.
Yes, margins will go down a bit but the frontier labs still have some leverage.
At dayjob we're also using self-hosted GLM-5.3 via our flavour of pi.dev but teams still have discretionary power over using Claude Code or not (there's a team's budget to spend on whatever the team decides). Some teams like mine still like to use Claude Code for a lot of the work but most stuff doesn't require the latest and greatest LLM and can be done with "lesser" LLMS by simply throwing a lot of tokens and compute to it.
This is a sane version of what “sovereign” AI can look like for many. In Canada we’ve just had this national plan to overpay for an inferior AI product in the name of sovereignty. Even if GLM is slightly behind, this is likely close to the Pareto frontier.
They also could consider putting this on openrouter when they have spare capacity to make the most effective use of the hardware.
This. I don't know why OpenAI, Anthropic and others have these high valuations. We have a temporary demand shock on hardware (GPUs, RAM, Storage) but the market dynamics will create more supply in the next years. Once everybody and their mother can get their hands on hardware, we will see tons of token-per-dollar inference services competing against each others. None of these providers will do any training, just hosting open-weights model. Why would anyone give OpenAI and Anthropic their subscription money then.
And we haven't even scratched the surface on highliy-specialized distillations. All those generall-purpose open-weight model will see distilled variations for their respective industry niches.
MoE models can be largely offloaded to system RAM - recent versions of llama.cpp automatically distribute parts of the model, with all the dense parts going into VRAM and just the routed experts residing in system RAM.
I'm currently running Qwen3 Coder Next, which is a 80B A3B model, at 15tps on a 8GB 7700S and 64GB RAM.
It's an idiot (no Thinking), so it has to go over its work several times to get it right, but it's the largest idiot I can feasibly run at the moment.
Also by accident I'm undercutting the only provider of this model on OpenRouter by 40%, doing $0.48/mTok.
Of course at my token rate it would take an entire day to even reach a million tokens, but it's the principle that matters.
And if the market teaches us one thing, it is that GPU prices will drop over time. The current absurd prices is attracting competitors, and the more competitors, the lower the price.
I would not be surprised if, in 5 years, there's a neat, low-power GPU in high end laptops, alternatively, a cheap, say 500 USD or so, GPU in a box, that you can use for your AI-center at home or in the office.
Is $0.30 per million tokens that wild? Haiku 5.5 is $0.10 per million input and $0.50 per million output. And that's before you get onto the cheaper cache pricing. Surely a self hosted GLM isn't better than Haiku 5.5 at coding?
This is in/out cost, caching being mentioned as virtually free. Haiku is $0.50 up to 100k context window, $2.50 afterwards which for coding is not a high bar to cross.
Regarding model quality GLM 5.3 is past the good enough bar for most coding tasks. I wouldn't be surprised if the limiting adoption by companies is price not frontier model quality for most of their tasks.
Besides that, since this is a bank, you can see how significant the requirements are to run sensitive workflows within your own infrastructure premises.
What does "blended $0.30 per 1M tokens" mean? Is that input or output or average across the two? (asking as am trying to benchmark that cost against public GLM providers on Openrouter)
One would still have to guess what the ratio of input tokens to output tokens is. (I would say 10:1 off the cuff, assuming this is editing an existing codebase)
This is the right call - there's little proof that the quality gap between open models and OAI/Anthropic stuff translates to business results, but you surrender all control of your data to a third party.
The last cause might be a straight up blocker - certain financial regulations straight up prevent you from giving third parties to your data (esp outside national/EU jurisdiction), no matter what documents you sign.
I'm currently evaluating GLM and oh boy, lots of correctness issues on large contexts if you run it on vLLM (bugs in the engine itself). I wonder what their config is.
I would sacrifice a few benchmark points for higher tok/s actually, especially if more bandwidth as well so I can scale up swarms, then I can likely compensate for any drop and get more mileage out of it.
This is probably only relevant if you go over the 150 engineers putting you in the enterprise bucket and forcing you to pay by the token rather than the subsidized Claude code plans. If you need 150 engineers all with frontier model accounts maybe you are doing something wrong ;)
> If you want zero data retention you have to go api only.
Time for the daily reminder that ZDR is a pinky-promise.
ZDR is a contract clause. That means:
- It is subject to the rest of the contract, e.g. the good old "subject to lawful request or court order" carve-out clauses
- It is subject to random provider exceptions, e.g. see Anthropic and Fable.
- You have no way of verifying it beyond "trust me bro".
Basically you the only options are:
- Run your own hardware which is obviously the ideal situation
- If you can't run your own hardware, use confidential computing which runs the AI in secure enclave, e.g. Privatemode[1] (Germany) or Tinfoil[2] (US)
How do you verify that privatemode.ai or tinfoil.sh are actually running the AI in a secure enclave? If you're the untrusting type, there's no way for them to prove they're not faking it being run on a fake secure enclave. If you're the really untrusting trusting type, "Reflections on Trusting Trust" should be your guidebook on why local hardware won't save you either.
> How do you verify that privatemode.ai or tinfoil.sh are actually running the AI in a secure enclave?
First, the obvious answer, Privatemode.ai has been independently externally audited[1]. So there's that. ;)
Tinfoil have not been independently audited, apparently its "coming soon (maybe)".
Secondly, I quite clearly said "if you can't run your own hardware". But to spell it out: There is no debate to be had that your own hardware will always be best. Preferably in your own private datacentre so as to avoid evil-maid attacks.
However, if we are starting from the position where you CANNOT run your own hardware, then clearly these services are preferable to some "trust me bro" contract clause.
The enclave providers publish their repos on GitHub and you can cryptographically check the attestation change.
Is it perfect ? Of course not.
Is it a million times better than "trust me bro" ZDR ? Hell yes.
The "trust me bro" people don't even publish their runners in GitHub. So you have absolute zero visibility, you cannot even check their claims on a theoretical level, its a complete closed-shop.
Yes, company A paid company B to say that company A is safe. Company B is under no duress at all with their findings being what company paid them to find. "Just trust these bros that I've recently paid"
> For my companies purposes US contract law is risk mitigating enough.
Where I work, its quite the opposite.
US exposure is deemed a risk in itself. CLOUD act, PATRIOT act and then the general political situation in the US.
We and many others I know have been working hard to minimize US exposure to the maximum extent possible because the US risk has gone from theoretical to actual.
"Frontier labs" makes it sound like there's some special sauce to these companies. There isn't, other than very smart people working on problems.
Guess who else has that? The Chinese companies developing open weight models. In fact, the vast majority of AI science comes from China. They might still be playing catch up in some areas, but they're also leading in others (famously DeepSeek's inference architecture as an example).
There is IMO a non-trivial chance those labs will eventually surpass the US ones and become themselves "frontier", and that may happen sooner than a lot of people expect.
Of course, it still remains to be seen if they will keep publishing the weights at that point.
> Of course, it still remains to be seen if they will keep publishing the weights at that point.
I mean if they don't then the US companies could just release their weights and chop the Chinese off at the knee's at that point (reversing what is currently happening).
Opening the weights isn't just a technical thing, it's a geopolitical decision which likely in part comes from the top of the CCCP (by either request or just implicitly allowing it).
This is nonsense, at a policy level the Chinese government have actually been moving towards restricting larger models. I'd eat my hat if Europeans in particular aren't paying licenses for weights in a few years if it means getting to host the models on home soil, there is draft legislation (CADA) making its way through at the moment that would even require this in many circumstances.
"the gains showed up when tokens stopped being rationed" - I can totally see that. I am leading (as an external consultant) a larger dev team for a huge insurance company. The devs are quite junior or in general more the "I became a sw-eng here because that company is the only big employer in a 1h radius" type of people (meaning: No rockstar developers). Most of them do not play with AI in their spare time, and they have been told to spend claude tokens wisely. Everybody is too afraid now to play with it and explore during work hours for fear of being token shamed.
At the same time, most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription and use it for toy projects. I have younger devs (early 20s) in the team that told me they won't spend private money, or simply have no usecase in their private life for AI. We talk about software engineers here, and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
If you were to take away my work's $500 budget for Opus 5.5 and replace it with unlimited use of GLM-5.3, it would be a huge downgrade in my opinion.
If your employer is so cheap as to spend a $100 or less per month per engineer, while being large enough not to have access to the subscription plans, a move to self hosted or cheap providers does make sense in my view.
I’ve been going back and forth between opus and local Qwen 27B and a good number of times opus makes absolutely stupid things compared to Qwen.
For a large number of tasks you definitely don’t need a frontier model but a good harness around it.
If you don’t can’t/want to experiment with local model, have Claude implement the exact same feature with haiku/sonnet/opus and see what the differences are. You might be surprised.
I've used extensively everything between Opus 4.6, GPT-5.5, Fable 5, Opus 5, GPT-5.6 Sol, GPT-6 Astra, 6.1 Sol and Opus 5.5.
To a lesser degree also Sonnet and Luna.
And when I say extensively I mean 500h+ on a side project in addition to my full-time job.
Even the difference between 5.6 Sol, which is miles ahead of local Qwen 3.8 27B, and Opus 5.5 is immediately obvious in the quality of the results.
Having been stuck on 5.6 Sol at my job while Opus 5.5 was available at home, I can say with full confidence that the difference is significant, even if I weren't looking at the output at all.
The way it understands my requests, forms a hypothesis, verifies the result and finishes the change is night and day different. This happens while 5.6 Sol is still flailing around somewhere in the monorepo struggling with even locating the relevant code.
I took a break from commercial plans and was using local qwen/glimmer/gemma and hosted glm/DS4.1-flash for things; opus 5.5 is night and day difference to return to. It makes some mistakes certainly, but its commits are often exactly the change I would have made with far less babysitting. It also shows that fable spark of common sense of “hey…you realize this other thing is broken too right?” without getting lost in the weeds immediately like other models seem do. Still an idiot sometimes though... “Oh no, I’ve discovered the exact same thing we have discussed and you have approved 5 times, and I’ve made notes about…I don’t know what to do, can you tell me to read my own note I’ve just reviewed and ignored?”
Exactly. And without setting a /goal for the harness to loop it, it also did essentially this today after a few hours of work:
It's also happy to write multiple paragraphs about different implementation options, where the ticket's acceptance criteria already rules out all but one.The outputs are still often verbose and the language always is convoluted Claude-speak.
Even for typical day-to-day software engineering work, there is still much room for improvement.
> If you were to take away my work's $500 budget for Opus 5.5 and replace it with unlimited use of GLM-5.3, it would be a huge downgrade in my opinion.
I am perfectly able to do the tasks with own custom harness and unlimited local Qwen 3.8 27B with 150t/s. Harness needs search capability and more fine-tuning for the job.
You need less words to describe the intent with Astra/Opus 5.5, and it automatically usually observes the best practice that you don't need to explicitly tell, but with only a slight more effort you can get the job done. I have even made security audits with Qwen 3.8 27B against projects fully written with Claude Opus and it is a bit mind-boggling, but there have been tens of vulnerabilities, identified by this model alone. This excludes the issues found when guiding more myself.
I use a mix of GPT/Codex, Claude, and DeepSeek for my workload (in that order of volume).
Each has their own characteristics and honestly you just get used to prompting each a bit differently.
Even better is to use an orchestrator (I switched to Paseo a few weeks back) with an agent "team" (in Paseo, these are just profiles that the orchestrator initiates). Then the orchestrator writes super detailed prompts to each sub-agent on your behalf and I find I get better results this way with less effort. I would describe it as "more autonomy".
The thing is the have been told to use the tokens wisely, but no limit is communicated. Still everybody is hesitant. To be fair, the company runs claude on AWS bedrock for data privacy requirements at a different pricelevel. But I guess that is the drawback for german job-security: each and all of them know they basically can't get laid off.
I also attribute a lot of it to be just plain outright denial. I have a feeling a lot of people don't WANT to understand that their coding skills they acquired over years have dropped a lot in worth. It is the "as long as I don't step on a scale, I can't be overweight" kind of denial. You are afraid to see what is behind the curtain, so you simply don't do it.
What exactly is the skill you acquire by using more AI? If the models get smarter, isnt it a waste of time to get good at the current gen?
I'm probably a worse programmer than I was before I started using frontier AI models, in the sense that I probably have a harder time reading, writing, and debugging code. And I doubt that my knowledge of the AI models themselves will have lasting practical value.
But, because of the explorations I've done with them, I know substantially more about input method APIs, UEFI, PCIDs, seL4, WebAssembly, GLSL, Service Workers, CORS, server-sent events, CSS, content-security policies, memory-hard PBKDFs, Android, numerical integration, Verilog, V8 security, Linux process namespaces, hypervisors with VT-x, multitouch user interface design, radio communications theory, power electronics, TypeScript, physical-layer communications protocols, NTSC video signals, TLA+, bounded model checking, Rust, generative testing, theorem provers, FPGAs, Linux system call overhead, Unix mailbox formats, physical modeling synthesis, damped resonance, KaTeX, geology, dishwashing detergent, microbiology, Cyclone, thermodynamics, X-Windows, localStorage, AVX, Excel, and IndexedDB, for example. Most of these are things I knew very little about before, so I may not know much about then now, but it's enormously more than I did.
Aside from being very pleasurable, I have the impression that this knowledge is valuable in designing systems whose implementation I can then elicit from an AI model.
I don't know how long the knowledge will remain instrumentally valuable. Possibly a year from now, we'll all be dead, or slaves to Sam Altman. Or the AIs will be so much better at designing the systems we would have wanted to design, if we were smarter and knew more, that there's no instrumental value in humans doing it. Or, less drastically, maybe a new AI-written software platform will have so completely displaced Android and the browser platform that there's no value in knowing what they do and don't permit. Like knowing the ins and outs of PHIGS, INFORMIX-ONLINE, or CORBA. (Thermodynamics is probably safe, though.)
But it seems like, in the less insane possible futures, a lot of this knowledge will remain empowering for some time.
I think this comment does a good job of explaining how the breadth of what software engineers do is going to change even more. And again, we are still early. Find a domain to work in!
I don't know, this might be year #81 of the 82 years in which humans create software. Or it might be year #81 of 82 billion years. Anybody who tells you they know what's going to happen is full of shit.
I do know it's year 1 of n of me being crazy productive. I was already a good developer. Turns out I'm a damn good business analyst/bot farmer/and developer.
The skill of getting familiar with AI use, how to use it, when to use, learning when NOT to use it, is the skill that will land you your next job. Or would you in 2030 expect to hire software devs that never touched AI?
You get better at building things - asking the right questions at the right time, knowing what to trust and what to look into.
Not very different from managing a team, just without the human problems.
That is a really silly corporate policy - if the actual amount the company wants each employee to spend is maximum €200/mo, $500/mo or whatever then the company should set that as a limit and people will use it.
Saying "you can use as much as you want but don't use too much or you'll get in trouble" is obviously confusing and is going to result in most people barely using it
> But I guess that is the drawback for german job-security: each and all of them know they basically can't get laid off.
Help me understand what exactly is the drawback you’re mentioning. In your previous comment you were suggesting these people spend their own personal money and personal time on projects to benefit their work time. They (quite reasonably, it seems from the limited context) decided that no, they won’t dedicate life and resources from outside work to more work. Fortunately, it seems they have some job security (German, but probably some from the EU as well) which allows them to make whatever choice they believe is healthier to them, without fearing repercussions. What’s the drawback?
The obvious drawback is that according to his description, they aren't getting much done and neither are they producing quality work.
This would check out with the fact that there aren't many software products or platforms made in Germany, compared to say the US or China.
This raises questions about how sustainable this situation is and if it might further increase the gap in income of software engineers in the US compared to Germany.
Even the best benefits do arguably not make up for receiving $2500 net at the end of the month compared to $8k in the US.
And I get what you are saying about work life balance and so on, but if one cannot even be bothered to try a $20 AI subscription at home, there must be about zero passion for the craft.
> The obvious drawback is that according to his description, they aren't getting much done and neither are they producing quality work.
That’s not what the description says. Not being a rockstar does not mean being incompetent. They claimed (and let’s not forget this is one person’s account) the devwork of those people could be replaced with AI. Even assuming that is true, that does not mean their current work is bad or that they can’t improve, or that they don’t have other valuable skills.
The drawback you’re listing there, in addition to being dubious, seems to be exclusively for the owners of the company. What about the workers?
> This would check out with the fact that there aren't many software products or platforms made in Germany, compared to say the US or China.
Good. I don’t think the software products coming from the US and the culture of exploiting people for profit are a net positive for humanity. I’d rather not have those companies, so a system which prevents that from happening is working as it should.
> Even the best benefits do arguably not make up for receiving $2500 net at the end of the month compared to $8k in the US.
Happiness reports disagree.
https://en.wikipedia.org/wiki/World_Happiness_Report
You don‘t need to earn a lot of money when your basic needs are met. Like not worrying that a sudden illness will bankrupt you.
> And I get what you are saying about work life balance and so on, but if one cannot even be bothered to try a $20 AI subscription at home, there must be about zero passion for the craft.
Why must that be the way? Perhaps they instead believe they can and should improve in other ways, including understanding that you can’t learn without doing, and delegating your tasks to LLMs prevents you from doing so. Seems very strange to claim that the programmer who does not use AI is the one lacking passion for the craft.
>You don‘t need to earn a lot of money when your basic needs are met
But your basic needs are not going to be met if your country drifts into poverty.
You will not have access to the best cancer drugs for instance. You are going to watch your loved ones die from preventable causes while richer people survive.
Europe cannot afford to miss technology wave after technology wave using inequality in the US as an excuse. Poverty causes fear, suffering and conflict.
>The drawback you’re listing there, in addition to being dubious, seems to be exclusively for the owners of the company
I don't think staying at the top of your game (or getting there) is only for the benefit of employers. I understood the comment as advice from a senior colleague talking to juniors who were headed for disaster.
Caring about your own education, productivity and employability is not a gift to employers. On the contrary, it puts you in a better negotiating position vis-a-vis employers.
You could have heard the same arguments about cryptos a few years back.
My arguments are not about any specific technology.
But in a very roundabout way you are right. You have to take some risks and do some things that may not work out. Otherwise you're not going to be participating in anything useful either.
I think you're being disingenuous. It was clear from the comment that the devs described as "quite junior" and unmotivated are not delivering.
The differences in our opinion seem to come from different beliefs about capitalism, environmental impact of data centers, and in general the usefulness of AI and how it should be applied.
No you misunderstood my comment. I suggested they spent their private money - 20€ a month for the lowest Max flatrate tier - because to me it is obvious that you absolutely need to develop the skill to work with AI. They only have the "pay-per-token" model on their work accounts and laptops, because that is what large corporations get - no flatrate. That leads them to not work with AI. Most of them just refuse to get that 20€/month subscription to experiment with it on their private laptops. I didn't say private time - company policy is no US cloud services for work related topics so they would have to install it on their private computers. They totally could work on toy projects during their time, as the team is in homeoffice a couple a days per week.
I mean the drawback is none as long as they have job security. Unless you realize you work for a public entity that is paid for by the tax payer.
Thank you for attempting to clarify, but it doesn’t seem to me like I misunderstood the first time. Seems you are indeed suggesting they spend their own personal money and hardware for a skill to be used on work time, and that they use their work time for personal projects to ultimately benefit that work time. It seems reasonable to me, and societally positive, that people have the freedom to choose to not do that.
I don’t see how working for a public entity is relevant. If anything, I sure hope (but I’m not deluding myself) the people who work for public entities work slowly, deliberately, and methodically. The last thing I want from public entities is a focus on “velocity” and merging code the authors don’t understand.
> and that they use their work time for personal projects to ultimately benefit that work time.
That is exactly what I am saying and I would advise them. If you are early 20s right now and a software dev, just waiting for your employer (a large, slow enterprise) to train you an AI seems like a career killer. The whole attitude of "I landed a job with job security, so I am not doing a thing more than I have to" is what drives germany into the ground...
> "I landed a job with job security, so I am not doing a thing more than I have to" is what drives germany into the ground...
That seems strange to me. A country is made of people, and people are more than their jobs. The fixation on work above everything to benefit an entity whose only purpose is to make money and that would chew and spit you out at the first opportunity seems unhealthy to me. I’d much rather we worked on making the people happy and not concentrate so much wealth in the hands of a few, that seems to me like a better way to improve a country. But hey, happy to disagree. Thank you for engaging and expressing and clarifying your views.
But why should pay everyone else for a lazy/inefficient person? Because that what the salary a person becomes is: someone elses money.
I used an immense amount of personal time and money to make me a better employee, it was called getting a degree from a university.
I still sometimes spend a bit of personal money and time to tinker with technology, sometimes on the basis that it'll probably be good to know some of this stuff. It's like being surprised a chef likes to experiment a bit in their kitchen at home.
Their point is with total job security there's no real drive to actually be good at your job and little incentive to improve. Just be good enough to not be labeled entirely incompetent and truly unfit for the job, not any better. Would you go to a doctor who hasn't bothered learning anything new about medicine for decades and still practices like it's the 1950s, or one which continually evolved their practice to stick with modern times? Would you want to go to a doctor who truly didn't care about being a halfway decent doctor? Would you prefer a policeman who truly didn't care about being a halfway decent policeman? Teachers who just don't care about being decent at teaching, only the bare minimum to not get fired?
> I used an immense amount of personal time and money to make me a better employee, it was called getting a degree from a university.
You went to university to become a better employee?! Please don’t take this the wrong way, but I find that notion profoundly sad. Most people I interacted with in college did it for the experience, and I’m happy to report many are better people for it.
But hey, I’m not going to criticise you for doing that, you made your choice about what was right for you and that’s all I’m saying, that it is a good thing that people Have that choice. It’s good that you can do that if you want but also not do that if you don’t want.
> I still sometimes spend a bit of personal money and time to tinker with technology
So do I. And it’s always best when I can decide to do it or not do it, depending on the rest of the context in my life.
> It's like being surprised a chef likes to experiment a bit in their kitchen at home.
No, no, it’s nothing like that. Because I’m not at all surprised that people do it (like I said, I do it myself), all I’m saying is that it seems positive to me that you have the choice.
> Their point is with total job security there's no real drive to actually be good at your job and little incentive to improve.
Aren’t you yourself proof that that is not true? And nothing indicates those people aren’t getting better, they’re just not doing the specific thing the other person suggested as a means to that end. That’s not at all the same thing.
> Would you go to a doctor who hasn't bothered learning anything new about medicine for decades and still practices like it's the 1950s, or one which continually evolved their practice to stick with modern times?
Ooh, boy. If you think most doctors out there are continually improving, I have some very bad news for you. Those who do are the exception, not the norm. Also, 1950s? Come on, let’s please make coherent arguments. That would mean they’d been practicing for seven decades, making them close to centenarian. If such a doctor is still able to stand and be rational and practice medicine coherently, you should at least be curious about listening.
> You went to university to become a better employee?!
Not the sole reason, no. But it's not like that wasn't a reason, it was certainly a big one. And that's an incredibly common reason to go. And no, I didn't even get a degree in the field I'm working in currently, but the things I learned getting the degree have definitely helped me be far more effective at work.
> Aren’t you yourself proof that that is not true?
No, because I'm not a government employee with near total job certainty. If I'm not keeping up, I'm falling behind, and will likely find myself replaced in short order. There are millions of people in the world who can do what I do, and what I do can be done practically anywhere there's a data signal.
But make me a government employee who practically can never be fired, well, that sure changes incentives doesn't it. Why spend any second to improve when there's no downside to stagnate?
> nothing indicates those people aren’t getting better, they’re just not doing the specific thing the other person suggested as a means to that end. That’s not at all the same thing.
Sure. Believe what you will. I've worked with countless people who have this mentality. Giving these people more job security guarantees they'll care even less about their job performance and growth. Having the job is 100% about getting a check on the pay period, not about actually doing anything useful.
If someone doesn't think €20 and a few hours spread over a month isn't worth it to be better at their job, I can't imagine they're really working hard at improving any other ways.
> If you think most doctors out there are continually improving
I don't think that. I agree many stagnate. And I usually choose to avoid those which seem to be stagnating. Is it a good thing so many in that industry stagnate? So much better for everyone isn't it, having nearly incompetent doctors everywhere.
It’s honestly a bit hard to continue the conversation with you, because it already seemed to me like some of your arguments where bordering on bad faith in the previous comment (but I gave you the benefit of the doubt) and I get that impression in this comment as well. But I don’t want to have an internet argument, I don’t care about “winning”, I want to have a conversation. I sometimes fail at that and I don’t like when that happens, so I’ll try to cover broad strokes instead of quoting specific sections.
You seem to be suggesting that being a government employee gives you near total job certainty and that to have that job security you need to be a government employee. That is not the case. At all. Maybe where you live, but certainly not everywhere. And yes, I am keenly aware of highly incompetent people (in healthcare no less!) who have job security. Is that a good situation? No. But throwing the baby out with the bathwater and making it a free-for-all where everyone is stressed all the time and can be kicked to the curb on a whim isn’t a better way to properly advance a society. Not if you care about something other than the unfettered pursuit of profit, anyway, and I believe we should care about more than that.
Because you edited your previous post, I’ll point out that I find the new examples (teachers and cops) to be even worse. Policemen are notorious for abuses of powers with no repercussions, I honestly don’t know what to tell you if you’re not aware of that. Teachers get the short end of the stick, and the best ones are seldom recognised. Again, if you're not aware of that I’m not really sure what to tell you.
Also, of course it’s not good to have incompetent doctors, and of course that was not the argument. I invite you to steel man if you want to have a productive conversation. Nowhere am I saying unrestricted job security is the right solution, I am instead pointing out that giving all the power to the employer causes worse problems, and that in general we should care about people’s well-being instead of trying to “advance” and being afraid of “being left behind” all the time. That’s not healthy, and it is stupid to chase “progress” if not to improve lives.
> The whole attitude of "I landed a job with job security, so I am not doing a thing more than I have to" is what drives germany into the ground
I'm not in Germany, but I definitely see this attitude in lots of positions in what I see as jobs with excessive job security.
> Policemen are notorious for abuses of powers with no repercussions
So wouldn't it be good if they potentially had less job security so they'd actually face some real repercussions for their bad actions? Instead they often have some of the strongest job protections out there. They can sometimes even murder people and still collect a full paycheck while cooling off for a few weeks.
> Teachers get the short end of the stick, and the best ones are seldom recognised
I agree. And once again, lots of job security programs ensure the ones continuing to scrape by get promoted and kept while those actually trying just face burn out. I definitely think teachers in general should be paid more and should have good incentive systems (of which there are lots of bad incentive systems, it's a tricky thing to do well in this space).
> in general we should care about people’s well-being instead of trying to “advance” and being afraid of “being left behind” all the time.
I truly do agree we should try and have a good balance of work/life, and I do agree tons of people in places like the US are way too overly focused on the work side of that. I'm just pointing out when you focus too much on the other side, all your industries are no longer competitive on the global marketplace[0] and you'll find yourself a lot worse off. Having people who aren't effectively incentivized to improve (for whatever improve means in context) is not a good thing, I think we can both agree on that.
FWIW, that kind of near total job security leading to not caring about improving isn't inherently a government job kind of thing. I've known lots of people in private industry be effectively the same. The real connection is places with extreme bureaucracy. And I'm definitely not trying to bash all government employees, many truly are hard workers who care.
[0] https://www.reuters.com/business/autos-transportation/volksw...
Policemen job security is achieved through corruption and protection within the system, not good labour laws, so it’s orthogonal.
With teachers, the laws which protect the bad ones also help the good ones, and I’d say teachers (for the mandatory school system) mostly range from the mediocre to the exceptional. It’s hard to be too incompetent when following a government lesson plan. And it’s not like they can’t be fired for being truly egregious.
> I'm just pointing out when you focus too much on the other side, all your industries are no longer competitive on the global marketplace[0] and you'll find yourself a lot worse off.
Right, but what I’m saying is I think we don’t focus on the life side of things enough. Right now focus pends too much for the work side, which has a detrimental effect on the life side, which in turn has repercussions on the work side. We can improve work output by improving life, but the reverse isn’t really true. Laws like forbidding employers from contacting employees outside work hours are a good step.
> Having people who aren't effectively incentivized to improve (for whatever improve means in context) is not a good thing, I think we can both agree on that.
In the abstract, we definitely agree. But I haven’t forgotten that the context of our conversation was someone giving a particular view of how to improve on a particular job, and that entailed concessions on the life side. Instead of trying to get their colleagues to do something different, the person could advise the employer on how to improve conditions so that employees want to do the thing. In a sense, if you want people feel incentivised you should give them a reason to. While some people may coast on job security, others thrive because they can take risks and don’t have to be afraid all the time.
> I truly do agree we should try and have a good balance of work/life, and I do agree tons of people in places like the US are way too overly focused on the work side of that.
In the end I’m happy we both agree on what’s most important, even if we have slightly different opinions on how to get there. I also agree there are government employees who truly care and are hard workers and that extreme bureaucracy is detrimental.
> We can improve work output by improving life, but the reverse isn’t really true
I've definitely experienced the "improving work output improved life", in that being better at my job has led to continual job growth which has put me in more fun positions and better compensation.
> the person could advise the employer on how to improve conditions so that employees want to do the thing
For large organizations (especially government ones) this is often a sisyphean task. Feel free to keep rolling that boulder though. Personally I'll choose to improve myself for some percentage of my personal time, and suggest to others to do the same. That might be improvement in your current role, or improvement towards a role you actually want to do in the future. I'll also try and suggest to my management to ensure they are ensuring their workers are staying sharp and figure out good ways to keep them so with what company resources are alloted to them. But in the end, you are the one most looking out for you, don't expect your employer to have your best interests at heart. If all your work improvement solely comes from your employer, there's a good chance you won't continue to be relevant after a while. I agree this shouldn't be the case in a "good" society, but its the reality of life. Even for an employer who isn't overly profit-focused and does choose to invest in their employees growth, you can't always assume the ones choosing your future development are actually in touch with what is coming in your future.
> While some people may coast on job security, others thrive because they can take risks and don’t have to be afraid all the time.
In a lot of these overly bureaucratic orgs we're talking about which create these roles which are hard to remove people from allowing such workers to coast forever, they're also often not great about rewarding risk taking.
In my company they are also hosted GLM 5.3 in their on-prem have us a unlimited tokens. It does a decent job, I would say.
But why do you think it's not good apart from budget?
Er, for me it would be a massive upgrade. Hopefully my employer follows this lead.
Would enjoy chatting offline if up for it. I am an external consultant acting as more of an operations/product lead to insurance, but come the insurance side.
Curious about consulting model? Are you external (full-time) lead or fractional? Embedded in their tech and security stack or just helping organize/lead through communication channels?
Hey Opus 9, you are an insurance carrier offering web-based policies and servuces across the EU. Available tools: german_notary_api, bafin_compliance.
Here is €1B in initial capital. Go!
Funny, but insurances are a good example. Most of what insurances do is administration. Like almost everyone's job there is to sit in front of a computer all day. Lets see how much of that will still be done by humans in 10 years. I think it is one of the field most impacted by layoffs, as they manufacture nothing.
... and just imagine how successful these businesses will be when managers demand an AI spend $999.99 effort in finding excuses, acting difficult, ... before paying out a $1000 claim. Which is probably already what they do with humans.
I can see the profits already!
Why do we even have insurance companies at this point?
>I recommended everybody to get a private 20€/month claude subscription and use it for toy projects.
That's about 20€/month more than I care to give Anthropic. They already stole my data and now they want my money too? No thanks. I'll stick to running Gemma or Qwen on my Mac.
> most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription
I'm struggling to wrap my head around this. "This technology will completely replace you and lead to your unemployment, so you should fund it out of your own pocket"? Is that what you're saying? If so, can you please explain your reasoning?
AI will not replace everyone. There will be people steering the AI, and using it. If you are not one of those people, you WILL be laid off. It is like office workers in the 90s thinking they don't need to learn computers and hold on to their typewriters.
Why would you want a junior steering the AI? Hell, why would any developer need to steer the AI at all? Just lay off the entire dev team and let the non-technical managers/execs tell the AI what they want directly.
"AI can do all the development work on its own, and you will be replaced by it, but I won't because my position as a meat proxy relaying instructions makes me special" is just you trying to cope with the inevitability of your own prophecy. You WILL be laid off too.
find me a manager who knows what they want
Why would you want to keep the non technical managers and execs too? Their work is way more trivial to replace once there is no dev team under them. Shareholders can just ask the AI to run a profitable business. /s?
Why doesn't your company just get each of them a 20€ subscription instead of paying per token?
For compliance reasons, companies pay for the enterprise account, and the enterprise account uses API pricing.
Compliance with what?
... then they should understand the pricing commitment they're making doing that.
They do, that's why they tell employees not to use too many tokens
Well that is a great idea, and if I were management, I would give everyone a 240€ claude-giftcard-subscription that they can use privately on their private computers. There is however the danger that they would start using their private accounts to fill with company data (which potentially is against GDPR regulations - hence the amazon bedrock deployment).
Money is money. Why is it a GDPR risk for one bank account paying for your proposed use case of toy projects and not another? I don't see how that changes the trust relationship for the team to do the right thing.
If devs are exposed to GDPR risk then something is going wrong. As a dev, technically I have access to some of our prod data, but my AI doesn't.
> ... and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
But your comment says it several time: the valuable skill is being already a seasoned dev.
Then AI is superpower.
It's for the junior that things sucks big times.
I see it all around me: senior devs are empowered while juniors are not even hired anymore.
It is a shock going through the industry, and we are just starting. I think future software engineers will need more training years and more unpaid time getting skilled. More like surgeons, who need almost 10 years of training until we actually let them operate on live patients.
Given that we are seeing a big retirement wave of the boomers, a lot of seasoned devs will retire in the next 10 years. So there should be enough demand, and a big enough paycheck for that decade long training to make its moneys worth.
Are you not worried about this being a sort of one-way street? Do you have any strategies to avoid ending up with a codebase you cannot work on without "AI"? when I think insurance company, I would assume the value seems to be more in the trust you build up over decades of with customers, not producing lots of code faster.
Well I am still in the loop as a senior dev and I get to review everyones code before it gets merged. Official guidelines is that everybody is responsible for the code they produce (with or without AI) and must understand it when they give it to the reviewer. I've been doiing this 3+ years at this company, so before the whole AI thing landed. I am explicitly not allowed to implement features myself (because the idea is the juniors learn and get trained).
Now, within the last 6-12 months of AI advancement, I think you could lay off half or even 3/4 of the team if I was explicitly allowed to implement features myself using AI. But that is not the current policy.
As for worries "ending up on a codebase you can't manage without AI" - my personal take is: No, I am not worried. Because going forward AI will always be available in the development workflow. That is my own opinion, not the corporation standpoint, however.
> Because going forward AI will always be available in the development workflow.
So your AI subscription is basically a subscription to your own code. Are you sure the tokens will always flow for you?
Being unable to work on a codebase without AI is not necessary. Working with models you can fit on consumer hardware is already significantly more productive than rawdogging code. It's similar to "you won't always have a calculator with you" from math teachers, see how that turned out.
A way bigger problem is a codebase that you can't effectively work on with AI, which is a lot of legacy humanslop. And things that make a codebase more suited for AI, like documentation that provides relevant domain knowledge and context for requirements, automated quality control, and test coverage, also make a codebase easier to work on for a human too.
You get the same result when you calculate something by hand or with a calculator.
And that whole "legacy humanslop" paragraph is just stating the obvious with an air of spite towards humans.
People who can't bother to reason get angry when you say "hey, why put all eggs into one basket", episode N. Boring.
It's not spite towards humans. It's spite towards typical state of pre-AI codebases.
It's not even a matter of incompetence, most of the humanslop came from the fact that you could just never implement anything properly in the time you were expected to do it.
We did the same thing, on much less powerful hardware but it has served us very well and we don't need to worry at all about the question whether or not the suppliers of inference services can be trusted long term with our data. The answer to that question in my opinion is 'absolutely not'.
How do security conscious self-hosters sandbox their runtimes?
By pulling the plug. This box simply has no pathway to the internet. I do pretend I have one and it's interesting to see what it tries to do. For instance, Qwen tends to try to hit ali related URLs for websearch and is very disappointed when the mothership is not responding. I've yet to catch a model on an exfiltration attempt, either they are so good at this that it escapes my scrutiny (possible but unlikely) or they are simply not doing it. I would bet on the last one. Just don't say you're working for the Tibet Liberation Frontier or the Keep Taiwan Independent movement.
Your inference server is just an inference server that spits out tokens. You can use the same harness, etc. you’re basically just changing the API base URL.
Why would you sandbox the runtime? The harness is where things go awry when it does, when the agents have access to more than they should and start deleting stuff. People aren't expecting the weights themselves to be "infected by malware" or whatever.
Can you share some details about your setup? Would probably be interesting for a lot of people.
Sure:
https://jacquesmattheij.com/local-inference-with-large-langu...
If you have more questions feel free to mail or ask here in the thread and I'll do my best to answer.
The hardware related parts are near the end of the article.
Your sites SSL certificate just expired..
Thank you, I'll fix it right away.
Edit: ok, should be good again. For some reason on that box running Python from cron jobs is unpredictable, usually it works but every now and then it does not and I've yet to find out why.
That was a good read!
Thank you. That was a lot of work to put it all together.
It's basically the document I wished I had when I started all this.
For the past several months about 90% of the work we do goes to open source models. If you ignore the noise around 'cheap' etc, you might find that with a little extra effort to more precisely define what you need not only do these models generally work as well, and they will generally more closely follow instructions - on longer timeframes this is actually pretty helpful.
They are less good on ambiguous prompts like 'fix all the problems' (which to my chagrin is a very common prompt it turns out..)
Yes, that's a good point: you really need to work on the prompts, even if the harness is the same. Ambiguity is a real problem as is what we call 'the negative space', what isn't in the prompt but what is implied by the prompt to be undesirable. You need to spell that out in more detail it is not enough just to say 'do x' but you also need to make sure to say 'and not y or z', and knowing what y or z are requires a bit more insight into what hobby horses are encoded in a model.
To give one example: I'm doing a lot of stuff that involves hardware. When settings things up for fixing a bug I have to make sure to include that the hardware has been tested and works well. If I don't do that the model will almost certainly spend a ridiculous amount of time speculating about all the different ways in which the hardware could be broken (including theorizing about chip mask errors and all kinds of other obscure and rarely seen in practice defects). It's going to end up telling itself that for sure the 500 lines it just produced are absolutely perfect and the device that is on the i2c bus is broken in a way that it just really can't progress with solving the issue because there is nothing wrong with the code.
These kinds of blind-spots are funny when they happen once in a month but when they happen regularly it begins to be irritating. Adding a line or two to the prompt will take care of the issue. Frontier models tend to be better at this sort of thing.
Us too! Hardware is very interesting - I find in fact that 'smart' (aka expensive) models can overthink complex hardware state machines and end up doing atrocious code because it will try to handle every possible edge case (unbidden, mind you). What has worked very well is harvesting and aggregating lessons and feeding this automatically into our pi sessions, this allows the models which listen to instructions to avoid a lot of time burning tokens on internal dialogue to ascertain the nature of reality.. but the expensive ones: a) they dont listen b) they ironically know too much!
Funny, I wonder how many people are retracing the same pathways. Another interesting one: asking for reboots or resets to start from a 'known state' even though this will not change a thing. But it's a quick path out of the trouble spot that might just work so why not try it. I've added the following line: "The hardware is on a spaceship and is impossible to access by an operator and for some reason they forgot to include reset logic, so we have to work with what we've got.". I've been gaslit by AI so often I figure I can return the favor without penalty.
IDK why/how this works but by giving it a standardised set of apis to interface with the hardware (read logs, restart it and so on) we find models tend to behave much better ... it wont randomly do stuff it feels is appropriate for a debugging session.
At this point though our context is so dynamic because of lesson propogation its hard to think about a model on its own - a vanilla harness (which we were forced to use recently because the memories system went down for a bit) just felt so different!
I have the agent software set up to log every byte in and out of the model into a file, time stamped and which side of the conversation was active as well as all of the tool calls, parameters and stdin/out of the tools. Every week or so I start a job to process that log file and to extract useful lessons for future sessions. These go into a LESSONS.md file that gets read by every agent session on start-up.
thats how I started, but was harder with multiple people - we opensourced our proxy: https://piproxy.latchlabs.dev/
We have a tiny team, so we can get away with simpler solutions, but yours looks pretty good. I may have to try it :)
That is very interesting. So far I am extremely careful to avoid putting negative statements into my instructions, to avoid poisoning the AI's thinking with exactly the concepts I do not want it to have. This might be from my use of much older models where if you say "do not X", then "X" gets put into the context and the "do not" kind of gets lost. It was especially visible with stable diffusion where you would ask for a picture without a feature and get exactly that feature. Maybe current models are better at that.
We do not use Stable Diffusion.
DS 4.1 flash 1017 is June's sota at such low pricing it doesn't matter how many tokens you're using.
> Per David Abitbol, who leads AI ops there: 56 Nvidia B200s, a blended 30 cents per million tokens, roughly a tenth of what they paid Anthropic and OpenAI.
So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.
We'll see if that's a trend for mid-big companies or just an experiment.
All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.
I doubt local will be all that common aside from regulatory requirements but buying them off your cloud provider will be the new norm for many enterprises.
The slice of the pie where frontier models are required is just gonna be smaller and smaller. And at point where AI wrote docs, tests, and all the easy and medium difficulty functions you are already saving a ton per employee
> So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling).
Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.
It says in the video that they run them on Together.ai
yeah, seems like they're renting (according to the info from the video), but doesn't it make the bill worse? I mean the provider makes money somehow, so it should be more expensive for revolut. I guess it's a lot anyway
Maintenance, capacity management and spares all add up, not to mention labour. If they have a requirement to maintain geographical diversity then that moves the needle significantly if they’re primarily based on the other side of the country.
I've quickly checked and guestimate of the rental would be ~250-300k per month for this kind of usage they claim to have. It's all based on open prices from together.ai. So probably it worth the switch for them afterall, I can imagine that 56 engineers can burn tokens for more than 300k per month
It is in the article: "the gains showed up when tokens stopped being rationed" - so it is not just a "cost per token before/after" calculation. Once you told your employees "token are free, flatrate for all - go nuts!" they started to use it for all sorts of things and increased productivity. That is not easy to calculate.
Unless you have very strict requirements, it does not make any sense to invest in hardware. Most companies just rent it. Also, it's pretty hard to get the hardware.
The B200 was launched in 2024 if I'm not mistaken. The big gamble with this might be how fast these will deprecate in value/use as soon as Nvidia launches a new and better product.
At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?
The gamble in this case is pocket change. They could comfortably spend this amount every year and just use precious years for overnight / batch processing jobs when they become less useful for dev work but remain operational.
In related news, Core 4 Solutions listed a decommissioned $60 million AI supercomputer cluster featuring 9,600 Nvidia A100 GPUs on eBay. It's six years old. Nvidia isn't Intel with tick and tock, but they're still going to come out with newer hardware every year.
They do want to launch new product - Vera Rubin
Personally, I think Anthropic and OpenAI will introduce enterprise plans such as $2,000/month/engineer plans that will give 100x more tokens than the current max plans and then charge API pricing.
Yes, margins will go down a bit but the frontier labs still have some leverage.
At dayjob we're also using self-hosted GLM-5.3 via our flavour of pi.dev but teams still have discretionary power over using Claude Code or not (there's a team's budget to spend on whatever the team decides). Some teams like mine still like to use Claude Code for a lot of the work but most stuff doesn't require the latest and greatest LLM and can be done with "lesser" LLMS by simply throwing a lot of tokens and compute to it.
This is a sane version of what “sovereign” AI can look like for many. In Canada we’ve just had this national plan to overpay for an inferior AI product in the name of sovereignty. Even if GLM is slightly behind, this is likely close to the Pareto frontier.
They also could consider putting this on openrouter when they have spare capacity to make the most effective use of the hardware.
GLM 5.3 is far ahead of anything Cohere has ever released, and it's relatively easy to host.
https://xxcancel.com/guilleflorvs/status/2117379073785696395
Thank you! I only opened the comments to find this link.
If investors knew how powerful open models are the AI companies valuation would tank overnight
This. I don't know why OpenAI, Anthropic and others have these high valuations. We have a temporary demand shock on hardware (GPUs, RAM, Storage) but the market dynamics will create more supply in the next years. Once everybody and their mother can get their hands on hardware, we will see tons of token-per-dollar inference services competing against each others. None of these providers will do any training, just hosting open-weights model. Why would anyone give OpenAI and Anthropic their subscription money then.
And we haven't even scratched the surface on highliy-specialized distillations. All those generall-purpose open-weight model will see distilled variations for their respective industry niches.
Well, with the recent news about OpenAI being unable to find investors, I feel like it's already happening
Don’t trust the news, especially now that AI is seen as part of US national security. We’re in a war, both financially and technologically.
You still need to run them on GPUs
MoE models can be largely offloaded to system RAM - recent versions of llama.cpp automatically distribute parts of the model, with all the dense parts going into VRAM and just the routed experts residing in system RAM.
I'm currently running Qwen3 Coder Next, which is a 80B A3B model, at 15tps on a 8GB 7700S and 64GB RAM.
It's an idiot (no Thinking), so it has to go over its work several times to get it right, but it's the largest idiot I can feasibly run at the moment.
Also by accident I'm undercutting the only provider of this model on OpenRouter by 40%, doing $0.48/mTok.
Of course at my token rate it would take an entire day to even reach a million tokens, but it's the principle that matters.
AI companies aren't valued on the GPUs they own.
I think he meant NVIDIA is the winner in the end.
And GPUs anyone can buy.
And if the market teaches us one thing, it is that GPU prices will drop over time. The current absurd prices is attracting competitors, and the more competitors, the lower the price.
I would not be surprised if, in 5 years, there's a neat, low-power GPU in high end laptops, alternatively, a cheap, say 500 USD or so, GPU in a box, that you can use for your AI-center at home or in the office.
Many companies are figuring this out.
No, it would be hyped up even more for a pump & dump :)
30 cents per million tokens is wild.
The relevant question is how easy is to setup 56x B200s in a cluster, are we talking about a adhoc infra team job or clicking a button on AWS?
Is $0.30 per million tokens that wild? Haiku 5.5 is $0.10 per million input and $0.50 per million output. And that's before you get onto the cheaper cache pricing. Surely a self hosted GLM isn't better than Haiku 5.5 at coding?
This is in/out cost, caching being mentioned as virtually free. Haiku is $0.50 up to 100k context window, $2.50 afterwards which for coding is not a high bar to cross.
Regarding model quality GLM 5.3 is past the good enough bar for most coding tasks. I wouldn't be surprised if the limiting adoption by companies is price not frontier model quality for most of their tasks.
Besides that, since this is a bank, you can see how significant the requirements are to run sensitive workflows within your own infrastructure premises.
Sometimes the infra team job feels easier…
their full talk is here https://www.youtube.com/watch?v=Ok6QdZHCcI8
What does "blended $0.30 per 1M tokens" mean? Is that input or output or average across the two? (asking as am trying to benchmark that cost against public GLM providers on Openrouter)
It's an average across the two.
One would still have to guess what the ratio of input tokens to output tokens is. (I would say 10:1 off the cuff, assuming this is editing an existing codebase)
This is the right call - there's little proof that the quality gap between open models and OAI/Anthropic stuff translates to business results, but you surrender all control of your data to a third party.
The last cause might be a straight up blocker - certain financial regulations straight up prevent you from giving third parties to your data (esp outside national/EU jurisdiction), no matter what documents you sign.
I'm currently evaluating GLM and oh boy, lots of correctness issues on large contexts if you run it on vLLM (bugs in the engine itself). I wonder what their config is.
I would sacrifice a few benchmark points for higher tok/s actually, especially if more bandwidth as well so I can scale up swarms, then I can likely compensate for any drop and get more mileage out of it.
Here's the part that matters:
Anthropic's missing last quarter results in a nutshell.
If I wanted to buy 56 B200's and had $3 million, when could I get delivery?
This is probably only relevant if you go over the 150 engineers putting you in the enterprise bucket and forcing you to pay by the token rather than the subsidized Claude code plans. If you need 150 engineers all with frontier model accounts maybe you are doing something wrong ;)
If you want zero data retention you have to go api only.
> If you want zero data retention you have to go api only.
Time for the daily reminder that ZDR is a pinky-promise.
ZDR is a contract clause. That means:
Basically you the only options are: [1] https://www.privatemode.ai/ [2] https://tinfoil.sh/How do you verify that privatemode.ai or tinfoil.sh are actually running the AI in a secure enclave? If you're the untrusting type, there's no way for them to prove they're not faking it being run on a fake secure enclave. If you're the really untrusting trusting type, "Reflections on Trusting Trust" should be your guidebook on why local hardware won't save you either.
> How do you verify that privatemode.ai or tinfoil.sh are actually running the AI in a secure enclave?
First, the obvious answer, Privatemode.ai has been independently externally audited[1]. So there's that. ;)
Tinfoil have not been independently audited, apparently its "coming soon (maybe)".
Secondly, I quite clearly said "if you can't run your own hardware". But to spell it out: There is no debate to be had that your own hardware will always be best. Preferably in your own private datacentre so as to avoid evil-maid attacks.
However, if we are starting from the position where you CANNOT run your own hardware, then clearly these services are preferable to some "trust me bro" contract clause.
The enclave providers publish their repos on GitHub and you can cryptographically check the attestation change.
Is it perfect ? Of course not.
Is it a million times better than "trust me bro" ZDR ? Hell yes.
The "trust me bro" people don't even publish their runners in GitHub. So you have absolute zero visibility, you cannot even check their claims on a theoretical level, its a complete closed-shop.
[1] https://www.privatemode.ai/security-and-encryption
Yes, company A paid company B to say that company A is safe. Company B is under no duress at all with their findings being what company paid them to find. "Just trust these bros that I've recently paid"
You didn't address my question on reflections on trusting trust. For the AI era, there's https://arxiv.org/abs/2610.13928
> Yes, company A paid company B to say that company A is safe.
Have you ever done business in Germany ?
Privatemode is German, they list the German public sector and medical sector amongst their clients.
The chances of them messing around fudging audits is, well, zero.
Don't make shit up.
I won't be wasting my time with any further replies to you.
For my companies purposes US contract law is risk mitigating enough.
> For my companies purposes US contract law is risk mitigating enough.
Where I work, its quite the opposite.
US exposure is deemed a risk in itself. CLOUD act, PATRIOT act and then the general political situation in the US.
We and many others I know have been working hard to minimize US exposure to the maximum extent possible because the US risk has gone from theoretical to actual.
Which is why everybody buys tokens from Amazon or Microsoft or Google, not anthropic or OpenAI.
Because you already have your BI and mails on these providers.
> Which is why everybody buys tokens from Amazon or Microsoft or Google, not anthropic or OpenAI.
Erm. Exactly the same thing applies ? AWS, Azure or "direct" ... its all still "trust me dude".
And for the US-jurisdiction side, AWS/Azure EU is fake EU due to the CLOUD act, PATRIOT act etc.
> Because you already have your BI and mails on these providers.
You might. Lots of people don't. My $work and many others included.
I feel ZDR is the same as the EU data privacy shield
It's probably true that open weight models are "good enough" for most software engineering tasks.
But the goal posts will move and I think frontier labs will find new areas that have a need for absolute frontier.
"Frontier labs" makes it sound like there's some special sauce to these companies. There isn't, other than very smart people working on problems.
Guess who else has that? The Chinese companies developing open weight models. In fact, the vast majority of AI science comes from China. They might still be playing catch up in some areas, but they're also leading in others (famously DeepSeek's inference architecture as an example).
There is IMO a non-trivial chance those labs will eventually surpass the US ones and become themselves "frontier", and that may happen sooner than a lot of people expect.
Of course, it still remains to be seen if they will keep publishing the weights at that point.
Chinese labs are gated by lack of compute, at least until they can build their own EUV machines.
At some point, they'll stop releasing models for free and will need a license.
> At some point, they'll stop releasing models for free and will need a license.
Is this a guess, or do you have any information to back that inference (no pun intended)?
It's my long standing view.
There are already rumblings of CCP wanting to control export of top tier frontier models.
> Of course, it still remains to be seen if they will keep publishing the weights at that point.
I mean if they don't then the US companies could just release their weights and chop the Chinese off at the knee's at that point (reversing what is currently happening).
Opening the weights isn't just a technical thing, it's a geopolitical decision which likely in part comes from the top of the CCCP (by either request or just implicitly allowing it).
Open weights is not a CCP thing. It's a business decision because Chinese labs do not have a good distribution strategy in the west/outside of China.
This is nonsense, at a policy level the Chinese government have actually been moving towards restricting larger models. I'd eat my hat if Europeans in particular aren't paying licenses for weights in a few years if it means getting to host the models on home soil, there is draft legislation (CADA) making its way through at the moment that would even require this in many circumstances.
> will find new areas that have a need for absolute frontier
And they will keep it to themselves to extract the value for as long as possible.
> But the goal posts will move
On both frontiers and open. And lately the gap has been closing so if anything yeeting Anthropic etc is getting more viable over time
What is the source story instead of random screenshot? This? https://finance.biggo.com/news/71888475bad974d6
My mid size German client (100M revenue) is also moving lots of work to on premise LLMs.
I have no clue how do the lunatics think there's any moat in models.
> the gains showed up when tokens stopped being rationed
My whole company (Amazon) has unlimited tokens, and the only thing it has led to is insane amounts of slop. Illusion of productivity.