This part sounds like AI assisted setting this up and benchmarking it:
>The fix was trivially simple: zero-pad the head count 12→16, run the fast kernel, and extract the real 12 heads from the output.
I've recently used a frontier AI (ChatGPT 5.6 Sol on ultra) to set up a much smaller local model, and the performance optimizations it introduced left the model totally incoherent. (The model just repeats a single character, etc.)
When I see a line like the one I just quoted, it leaves me wondering if the setup is still coherent like a stock install of Kimi K3 on supported hardware.
Did they run any benchmarks on it to see if it is still correct?
I have never used sol, but I regularly use Claude to set up and benchmark local models per task. It is very thorough and has always returned good setups. The only thing you have to do is make sure you point at the model card. It’s always incredulous that models exist after its training cutoff.
K3 does an ok job of setting up, but its config searching isn’t nearly as thorough and its will confidently tell you it’s found the best setup when it’s only turned a few knobs. It’s also not a good evaluator of its own output. It rates its work too highly and seems kind of defensive when benchmarking. Still a good check because it does find stuff, but open a fresh session and don’t tell it where the results came from.
What K3 does do more than any other model I’ve found is investigate folder structures. If I want to benchmark it and other models I have to move the testing methodology and any reference to other results out of the project folder bc Kimi is like an ls bloodhound. It will find them.
>and its will confidently tell you it’s found the best setup when it’s only turned a few knobs.
on one hand , yeah : a model being more aggressive towards exploration of the decision space is usually a good thing.
on the other hand : I think that it's up to the operator to set rigid test criteria to make these things actually work well in a repeatable fashion.
so in other words, i'm glad claude is doing a better job for the way you're prompting the thing, but as a spectator from afar these kind of operator complaints usually spring up from the use of weak, ambiguous, under-considered prompts.
similar with the parents' complaint; there should have been a testing criteria for legible output that got immediately flagged or failed by the larger model.
it's a very hard sell for me to think that " a a a a a a a " is accepted as a valid language output by any near-SOTA-large model without some real coercion.
It’s not a complaint, only an observation. Operator aside, to observe that the models behave differently shouldn’t be controversial, should it?
It seems like you read my comment as dissing Kimi. Kimi is in regular rotation for me. Some tasks that Claude used to own go to Kimi first now. They are just different tools with their own strengths and weaknesses.
hi I work at wafer. yes we ran benchmarks. for example our Kimi K3 is live on open router and in order to host there you have to run accuracy checks. like tau/gpqa.
and then u must pass test regarding thinking, coherency, and tool calling
How is the capex on 8 * MI354X even remotely justified at less than $10/h?
Even without the base system, power and every other expenses:
365d * 24h * $2.95 = $25842/a invoicable.
That doesn't add up within one year, that doesn't add up in three years and it is questionable that it brings in the money during the lifetime of the device?
Where do you see less than $10/h for 8 * MI354X? I can only find $2.50 for 1 * MI355X (lowest I can find for rent on other websites is $2.65, but maybe they got a better deal).
That can’t be real. Modal rents out an RTX 6000 Pro for more than that. Nvidia will rent your GPUs at a fixed cheap rate if you buy from them. Perhaps AMD has a different subsidy style program. Because those MI355X are going cheap here.
Wafer is making themselves synonymous with slop in the inference space. Exaggerated unfair comparisons in all their results, twitter hype posts with alarm emojis etc.
> $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200.
This is not an accurate price comparison for real terms.
This is a company that launched a token subscription (WaferPass), before weeks later, rugpulling the plan for being unsustainable while simultaneously claiming they achieved incredible inference efficiency gains worthy of paying them mind.
I went the gpus.io website & it’s $2.95/hr right now, this is like comparing MSRP to actual market price. B300s are in demand & hence cost more, but these lazy editors at wafer.ai can't be bothered to do TCO on actual ownership nor share code to replicate their setups. Instead just relying on current market prices to win one row, which isnt even about per/$ on actual MSRPs.
Most discourse around GPU prices is nonsense right now. Some people using Spot prices for providers who won't have Spot capacity, some people using hourly rates for instances that are never in stock, some people ignoring commitment discounts.
Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation
> Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation
The GPU price discourse is absurd, but many are serving models on single node setups when the model fits
while i'm a fan of wafer's work, 1024 input token length is not a great benchmark anymore, that number is useful only at 1-4x single node a100 / h100 at various concurrency levels
The B200 is being forcefully compared against something which is not gonna fit within it's memory in a single node & not much details about multi-node interconnectivity, disagg or not. As expected of a shoddy slop.
The only point it won is of cost per hour is one aggregation website for rentals, the premium a B300 commands against the $3/hr AMD chip which no provider has in abundance. Never bothered to do TCO of owning the hardware either.
To the B200’s defence, its numbers are somewhat deflated by the fact that it pays a cross-node all-reduce on the decode critical path (RoCE v2 at ~195 Gb/s) — it’s the only config here that spans two nodes
B300 is about 46% faster for one stream and 65% faster in aggregate. AMD wins only after Wafer divides throughput // selected cloud-rental prices: $2.50/GPU-hour for MI355X versus $6 for B300.
Benchmarks are unreproducible, power costs are missing, ROCm was patched... and on and on.
Sure, it can serve that particular model at that particular size more economically. Good for them... in particular.
I wish they wouldn't call them "open source models". They aren't open source. They didn't publish the training data. They didn't publish the tools they used to train the model.
They published the weights. It's an "open weight model", a term that it seems nearly everyone has agreed is appropriate. Why is this company not using it?
> They didn't publish the tools they used to train the model.
That's not totally true - for example Ziphu (Z.ai, developers of GLM) have published their Slime RL-training framework, developed together with Tsinghua University.
Yes, we all realize that "open weights" is more accurate than "open source", and anyways the source code would not be very interesting - it's the training data and methods that mostly define these models.
I can understand not wanting to call it open source if they don't give you the algorithm and software used for training, but wanting the training data too? That's completely different
That’s not really true. The models have source code defining their architecture and it is open-source.
People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code.
So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns.
I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch.
Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.
Your game example is a great one: If a game didn’t come with its 3D models, sound, level files, game data, or assets of any kind, but its engine was open source, I don’t think anyone would say “hey this game is open source”. They’d say the engine is, but that’s it. All of the above things are not code, but they are part of a game and yes, expected to be included in an open source game.
No, an engine is only the code that is shared between games. For instance "urban terror" by frozen sands is a closed source game on top of a (now) open source engine (the Quake3 engine). Quake3 is a (now) open source game (with closed assets) on top of its own open source engine. Quake and it's derivatives are a particularly nice example of the distinction between a game and its engine because the engine includes an interpreter for the game code, leaving a very clear boundary. Meanwhile many games end up intermingling their engine and game code (e.g. see the Eve Online devs write about this - another closed source game on top of a now open source engine).
BAR is an open source game. All the game source is open.
> Open source RTS game built on top of the Recoil RTS Engine
BAR is a game, Recoil (a different thing - not the repository I linked) is an engine it is built on top of. Recoil is - incidentally - a fork of spring, an engine many open source RTSes are built on.
Doom pioneered this model, the Doom engine was open-sourced while the assets remained proprietary. This is where projects like Freedoom exist, to create an entirely free software equivalent.
I think this misses the point I was trying to make. Both an open source game and an open weights LLM include the necessary assets. But people seem to be saying, “you need everything that was used to create the assets”. For game art, that would include sketches, prior art used as inspiration, project files that aren’t used by the game engine but were used during the design process, and so on - if you apply the same standard. But I don’t think anyone does.
Similar to this I've never seen a project request anyone license their github PR comments under any license whatsoever even though they're undoubtedly copyrighted and used in the creation of the actual source code.
My understanding was that there was a lot of non determinism in the training process because of many factors: how GPUs optimise floating point arithmetic, how the compute is distributed between the thousands of GPUs etc
I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things).
You'd likely get a model with very similar behaviour but the weights would be different.
Model training now is not a straight forwards process of input data -> run tools -> get model.
There's a whole lot of alchemy going on. We don't quite understand what works and what doesn't. Think of it like painting with water color and having to improvise very often.
The only advantage over water color is that we can revert to a working state.
It's mostly ad hoc scripts and some pretty horrible hacks being run by a hundred engineers trying to improve a thousand different things at once with a hundred thousand GPUs.
I'm sure once we understand the tech better, the training process will look like running a program.
These models are a combination of a small amount of code, a ton of training data, and a ton of expertise in how to train them (which I suspect includes how best design/curate training sets for different model improvement goals).
The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data.
You don't need the source code - just read their architecture docs and implement it yourself.
The Chinese have actually been very open about training, starting perhaps with the DeepSeek-R1 paper which told the world in detail how to train a reasoning model. The Kimi 3 paper also gives a lot of training details.
There really isn't much of a comparison to be had between building a traditional software project where all you need is the (maybe open source) source code and the Makefile that automates the build process, and building a machine learning system where it's primarily about data not source code, and even with a road map of what may be very complex training (cf build) process, you'd probably still have a hard time building it since AFAIK the training process may still involve expert knowledge and intervention - I don't think training has been reduced to a hands-off "Makefile" or build script.
So, basically lack of source code is the least of the issues in being able to build one of these models - it's mostly the training data and training processes/expertise that you would need.
> The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data.
...and because that training data is missing, they can't be replicated. Which means that you cannot assert that the Chinese are being open in their LLM development, because there's no way to verify that the techniques they describe are actually the ones being used.
The reason that the training data is missing is that they're trained on a large amount of American copyrighted data and distilled on American models, which is where a lot of their performance comes from.
You can replicate the architectural innovations, and try them for yourself with your own dataset. It seems some of them are certainly being used by western companies, such as DeepSeek Sparse Attention, now supported by NVIDIA cuDNN.
Ditto for training algorithms and procedures such as Slime or DeepSeek's details instructions on how to build a reasoning model.
This is the exact value of openly shared details - others CAN copy and try them and modify them themselves.
Yes, the training data specifically has not been released for any model, American or Chinese, but that doesn't detract from what has been shared, and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing - because they are all using data from sources they don't want you to know about, and at the end of the day the data is the closest thing any of them do have to a moat.
> You can replicate the architectural innovations, and try them for yourself with your own dataset.
That's not related to my comment. My comment was pointing out that you can't verify something that wasn't published. You have no idea what fraction of their techniques they're not publishing, and how much they contribute to their model performance, because you cannot replicate the models, because they don't publish their training data.
> and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing
This is moving the goalposts. Your claim was that "The Chinese have actually been very open about training", which is false, as discussed. Nobody ever claimed that the American labs were open.
> You have no idea what fraction of their techniques they're not publishing, and how much they contribute to their model performance, because you cannot replicate the models, because they don't publish their training data.
Why would you be concerned about THEIR model performance ?!
Surely if you are an ML researcher and read about a new technique, you are interested in how YOU may be able to use it.
If you are Ilya Sutskever sitting at OpenAI in 2017, and happen upon Google's "attention" (Transformer architecture) paper, then what you do is go and implement it for yourself, get yourself some training data, and try it.
What you are NOT going to do is whine about not being given their source code, or their training data, or their training harness, or a dump of Google's corporate secrets. You take the research that has been shared and evaluate it for yourself.
This doesn't have anything to do with Google or OpenAI. I'm not Ilya Sutskever and it's not 2017, either.
I'm not "whining" about anything. You made the claim "The Chinese have actually been very open about training" and I showed that that was false. That's all that there is to it.
You say that but for like 2 years there was a guy on huggingface releasing quants as TheBloke. No identity no nothing except he likely had a grant or a university job that let him do it. Quants aren't the same as training but it was beyond 99% of people at the time (as you say this is)
Cant stand when you guys try and force something as impossible on the rest of us simply because you could never accomplish it.
I did not say that it is impossible. I just think that we need architectural improvements, or maybe even a fundamentally different approach to get something like Kimi K3 for cheap. The point I was trying to make was that we shouldn't just laze about and hope that hardware improvements will get us there.
(Also, I know Tom Jobbins (TheBloke), and have personally contributed to increase the adoption of GGUF, e.g. in the transformers and ktransformers libraries, so I find the personal dig quite amusing.)
Unfortunately, electricity prices did not fall by the same factor, so I fear that training a frontier model will still cause a an unsustainable dent in my monthly budget.
Compiling source code also takes computing resources, only the scale is different. It's a very close analogy: source code with training data, and compiled binaries with model weights.
If we start with the same ingredients and independently make our stews they're likely still going to taste a bit different due to the non deterministic nature of the process. Isn't it similar with training LLMs at scale? Do you get identical weights if you do multiple runs?
Yeah I understand that. I mean just the methods they use to acquire the training data has already had a shit ton of drama, there's no way the western labs are ever going to be public about the ingredients.
Not sure why it matters whether weights are software or not.
One can equally argue that firmware is just settings for the CPU.
If weights aren't copyrightable then doesn't this argue against the idea of them being able to have the quality of open-sourceness? Meanwhile the training code could be copyrightable.
Well, my position is that the software that defines/runs the model is the open-source part.
The weights are the asset that makes it useful, but I don’t think 40T tokens of pre-training data should be required to call the model itself open source when you can inspect every line of code in the model without that, as well as instantiate and run the model with randomly initialized weights.
The weights are the preferred form for modifying or integrating with other models. There is no obligation in open source to transitively open source all of the documentation / tools used to create the open source project.
>a term that it seems nearly everyone has agreed is appropriate
Models being considered open source even if the original training code / data is not released also is something almost everyone has agreed to be appropriate.
> The weights are the preferred form for modifying or integrating with other models
It's the 2nd time I hear this argument and I'm already fed up with it
Is it the preferred way only because training is expensive? It's like saying binaries are the preferred way of modifying program because you can't afford to have a fast enough machine to compile it yourself.
Most people don't have resources to compile their own browser, but what makes some browsers open is access to the source.
Maybe it's the preferred way because the people sharing models are themselves working with weights and no data?
> Is it the preferred way only because training is expensive?
No, it's the preferred way because that's literally how you train it. Contrary to popular misconceptions, you don't "compile" data into weights. You initialize a model (based on architecture, config, etc) and then you modify it via training. But crucially they (i.e. model creators) modify it the same way (technically speaking) as you would. That's what the license grants you. Nothing less, nothing more. The "how" as in knowhow has never been something covered by a license.
Open-weight is something dreamt up by people misunderstanding the basics of model creation and training, and having ideological biases against AI and/or LLMs. It is what it is, but you should know that you are technically wrong. A model released under an open source license (Apache, MIT, etc) is an open source model. Because the weights are the source of the models. Training is not "compilation". Training is the "how" as in knowhow to modify the model. Training deals with values. Source deals with everything, including values.
In the past, if someone would have released a piece of software (say a PID controller algo) w/ hardcoded values, no-one would bat an eye. LLMs are just that, with billions of hardcoded values. Nothing less nothing more.
Per the OSD definition of source code, "the source code must be the preferred form in which a programmer would modify the program." which means an argument could be made (as charcircuit is making) that the weights, being the preferred form to modify, are the source.
I do prefer open weights as being more precise (like, is it even really software that has source code in the first place?) but I feel like at this point the ship has sailed somewhat (though if this is something you're willing to spend your time arguing then... moral support I guess?)
Incidentally, I do. I fucking hate the open source maximalists like Stallman who suggest that not having source code is equivalent to not having control over what runs on your computer. But you always[1] have control over what runs on your computer. Reverse engineer it if you care so much! Every form of DRM is inevitably defeated, because you do have control over program execution, regardless of access to the source code. Stallman's argument seems more lazy and entitled than principled.
[1] ...setting aside hardware attestation bullshit, which does need to die in a fire.
I think you are failing to see how the weights are the preferred form of working with a model. It's like if I shared a png with others to use as a meme template. Even if I don't share the source code to photoshop other people can use that "binary file dump" to make derivative images of their own with it.
I feel like the arguments being made about sharing the training data are from people who don’t have any experience training ML models and have never worked with weights. The biggest reason I have seen that people want the data is to complain about some aspect of it (copyright, bias, etc)
I would be very interested to hear someone explain what freedom it is they need the training data to enjoy.
The models are open source, it’s never been a requirement for someone to document the entire process of creating something to be “open source”. Just sharing the source, in this case, the weights, meets the criteria. Software engineering’s obsession with precise terms is actually one of the things I’m glad is dying with ai automation.
They’re not really even being precise. The relevant software freedom, from the FSF is []
The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
Ported to the model world, this is fulfilled by sharing the weights and implementation. There’s almost nothing that having the training data gets you (other than actually training it). The weights plus a reference implementation let you see all the states to study the behavior, and let you fine tune it to do your bidding (the abliteration etc). The freedom is satisfied.
Some might argue that without the training data you couldn’t do some classes of experiments to see how it works, say leave-one-out retraining. I’d argue things like that are not really about the model but about ML research or the class of models, which while interesting is not a free software pre-requisite.
The source is actually all training data, plus the software used for training, including some scripts or instructions to run the entire thing end to end on your own computer. That's what open source has always been about.
The weights are the output of a program, it's a binary. It's not source.
> Software engineering’s obsession with precise terms is actually one of the things I’m glad is dying with ai automation.
Yes. The next time you go to your doctor you should hope he's not being overly precise; or the engineer that builds the bridge; or the software engineer that implemented the embedded software in your insulin pump.
There is no precise thinking without precise terms.
So is a zip file of source code. There is no 'compilation process'. Model training isn't a fixed end point. You start with random weights. You train. The weights get better. You do this iteratively. At some point you say 'good enough' and release. People doing further training would start with those weights and further iterate. Demanding the original training data and training process would be equivalent to demanding a completely replay-able sequence of editing steps starting with a blank text file and allowing you to replay ever edit that led the original author to the released source code.
Now there are reasons you might want to know about the training data when you wouldn't care about the authoring process used by a traditional open-source process. And these get at the reason LLM's are different than traditional software and so maybe our existing definitions of what 'open-source' means aren't a good match for LLM's. Of course there is software associated with LLM's (beyond the weight) -- defining the structure of the particular neural net those weights fit into. In every open-weights model that I'm aware of that software is open source (though trivial).
If you do good work you should at least take the time to review the slop that details that work for slopiness. Otherwise it's hard to take it seriously. Especially the prefill section.
I'm saying everything looks like it does, with the prompt you gave Sol. Go ahead and point the same prompt to anything you don't think has serious flaws and you'll see it would tear it apart.
To begin with, as I pointed out separately, the price comparison that makes the headline is absurd. No one is easily getting the MI355X's at $2.5/hr. And then the eval is unreplicable and poorly defined.
This part sounds like AI assisted setting this up and benchmarking it:
>The fix was trivially simple: zero-pad the head count 12→16, run the fast kernel, and extract the real 12 heads from the output.
I've recently used a frontier AI (ChatGPT 5.6 Sol on ultra) to set up a much smaller local model, and the performance optimizations it introduced left the model totally incoherent. (The model just repeats a single character, etc.)
When I see a line like the one I just quoted, it leaves me wondering if the setup is still coherent like a stock install of Kimi K3 on supported hardware.
Did they run any benchmarks on it to see if it is still correct?
I have never used sol, but I regularly use Claude to set up and benchmark local models per task. It is very thorough and has always returned good setups. The only thing you have to do is make sure you point at the model card. It’s always incredulous that models exist after its training cutoff.
K3 does an ok job of setting up, but its config searching isn’t nearly as thorough and its will confidently tell you it’s found the best setup when it’s only turned a few knobs. It’s also not a good evaluator of its own output. It rates its work too highly and seems kind of defensive when benchmarking. Still a good check because it does find stuff, but open a fresh session and don’t tell it where the results came from.
What K3 does do more than any other model I’ve found is investigate folder structures. If I want to benchmark it and other models I have to move the testing methodology and any reference to other results out of the project folder bc Kimi is like an ls bloodhound. It will find them.
>and its will confidently tell you it’s found the best setup when it’s only turned a few knobs.
on one hand , yeah : a model being more aggressive towards exploration of the decision space is usually a good thing.
on the other hand : I think that it's up to the operator to set rigid test criteria to make these things actually work well in a repeatable fashion.
so in other words, i'm glad claude is doing a better job for the way you're prompting the thing, but as a spectator from afar these kind of operator complaints usually spring up from the use of weak, ambiguous, under-considered prompts.
similar with the parents' complaint; there should have been a testing criteria for legible output that got immediately flagged or failed by the larger model.
it's a very hard sell for me to think that " a a a a a a a " is accepted as a valid language output by any near-SOTA-large model without some real coercion.
It’s not a complaint, only an observation. Operator aside, to observe that the models behave differently shouldn’t be controversial, should it?
It seems like you read my comment as dissing Kimi. Kimi is in regular rotation for me. Some tasks that Claude used to own go to Kimi first now. They are just different tools with their own strengths and weaknesses.
Thanks for sharing. What kind of tasks do you give Kimi rather than Claude?
hi I work at wafer. yes we ran benchmarks. for example our Kimi K3 is live on open router and in order to host there you have to run accuracy checks. like tau/gpqa.
and then u must pass test regarding thinking, coherency, and tool calling
You can tell because instead of making an arrow like this: -> it uses a Unicode symbol instead.
How is the capex on 8 * MI354X even remotely justified at less than $10/h?
Even without the base system, power and every other expenses: 365d * 24h * $2.95 = $25842/a invoicable.
That doesn't add up within one year, that doesn't add up in three years and it is questionable that it brings in the money during the lifetime of the device?
Where do you see less than $10/h for 8 * MI354X? I can only find $2.50 for 1 * MI355X (lowest I can find for rent on other websites is $2.65, but maybe they got a better deal).
> How is the capex on 8 * MI354X even remotely justified at less than $10/h?
Is anyone actually renting them out that cheap? The very cheapest on-demand price I see online is $14, and most providers are a lot higher
That can’t be real. Modal rents out an RTX 6000 Pro for more than that. Nvidia will rent your GPUs at a fixed cheap rate if you buy from them. Perhaps AMD has a different subsidy style program. Because those MI355X are going cheap here.
Wafer is making themselves synonymous with slop in the inference space. Exaggerated unfair comparisons in all their results, twitter hype posts with alarm emojis etc.
> $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200.
This is not an accurate price comparison for real terms.
This is a company that launched a token subscription (WaferPass), before weeks later, rugpulling the plan for being unsustainable while simultaneously claiming they achieved incredible inference efficiency gains worthy of paying them mind.
I went the gpus.io website & it’s $2.95/hr right now, this is like comparing MSRP to actual market price. B300s are in demand & hence cost more, but these lazy editors at wafer.ai can't be bothered to do TCO on actual ownership nor share code to replicate their setups. Instead just relying on current market prices to win one row, which isnt even about per/$ on actual MSRPs.
gpus.io shows tensorweave pricing at $2.95/hr. Tensorweave just shows "Talk to sales".
Most discourse around GPU prices is nonsense right now. Some people using Spot prices for providers who won't have Spot capacity, some people using hourly rates for instances that are never in stock, some people ignoring commitment discounts.
Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation
> Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation
The GPU price discourse is absurd, but many are serving models on single node setups when the model fits
You can't beat current API pricing running Kimi K3 a single node, so as I said anyone serious is not doing that for Kimi K3.
Not sure how you're getting "no one serves models on a single node" out of that.
while i'm a fan of wafer's work, 1024 input token length is not a great benchmark anymore, that number is useful only at 1-4x single node a100 / h100 at various concurrency levels
Lol, such a lazily written article by wafer.ai
GPUs. 8× MI355X (TP8) B300 (TP8+DCP8)
Decode tok/s per stream 118 tok/s 172 tok/s
Peak aggregate. 952 tok/s 1,568 tok/s
Peak aggregate per GPU 119 tok/s 196 tok/s
On every row the B300 beat the MI355X
The B200 is being forcefully compared against something which is not gonna fit within it's memory in a single node & not much details about multi-node interconnectivity, disagg or not. As expected of a shoddy slop.
The only point it won is of cost per hour is one aggregation website for rentals, the premium a B300 commands against the $3/hr AMD chip which no provider has in abundance. Never bothered to do TCO of owning the hardware either.
Did you see this section?
No mention of Infiniband or SPX lol
Another ad posing as "science".
It's basically a Wafer/AMD advertisement.
B300 is about 46% faster for one stream and 65% faster in aggregate. AMD wins only after Wafer divides throughput // selected cloud-rental prices: $2.50/GPU-hour for MI355X versus $6 for B300.
Benchmarks are unreproducible, power costs are missing, ROCm was patched... and on and on.
Sure, it can serve that particular model at that particular size more economically. Good for them... in particular.
Since we're (as a society) spending tens of billions of dollars to serve models like this, it's a pretty interesting advertisement.
I wish they wouldn't call them "open source models". They aren't open source. They didn't publish the training data. They didn't publish the tools they used to train the model.
They published the weights. It's an "open weight model", a term that it seems nearly everyone has agreed is appropriate. Why is this company not using it?
> They didn't publish the tools they used to train the model.
That's not totally true - for example Ziphu (Z.ai, developers of GLM) have published their Slime RL-training framework, developed together with Tsinghua University.
https://github.com/THUDM/slime
Also, if you read Moonshot's Kimi 3 report, it does give a lot of architectural detail and details on training.
https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_repo...
Some discussion on this by HuggingFace here:
https://www.youtube.com/watch?v=MW8-kqd2SD8
Yes, we all realize that "open weights" is more accurate than "open source", and anyways the source code would not be very interesting - it's the training data and methods that mostly define these models.
Sorry, where exactly is "data" in "source"?
I can understand not wanting to call it open source if they don't give you the algorithm and software used for training, but wanting the training data too? That's completely different
For LLMs the training data is the source of their weights.
You can‘t reproduce the LLM without the same data
That’s not really true. The models have source code defining their architecture and it is open-source.
People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code.
So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns.
I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch.
Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.
Your game example is a great one: If a game didn’t come with its 3D models, sound, level files, game data, or assets of any kind, but its engine was open source, I don’t think anyone would say “hey this game is open source”. They’d say the engine is, but that’s it. All of the above things are not code, but they are part of a game and yes, expected to be included in an open source game.
People refer to games with proprietary assets as open source all the time. The source code is open - the assets are not. For one example: https://news.ycombinator.com/item?id=28201265 / https://github.com/beyond-all-reason/Beyond-All-Reason/blob/...
Source as a word has a meaning that doesn't include non-source things.
That’s an open source engine, we already have a word to define what that is.
And we have Open Weights to define what those are.
No, an engine is only the code that is shared between games. For instance "urban terror" by frozen sands is a closed source game on top of a (now) open source engine (the Quake3 engine). Quake3 is a (now) open source game (with closed assets) on top of its own open source engine. Quake and it's derivatives are a particularly nice example of the distinction between a game and its engine because the engine includes an interpreter for the game code, leaving a very clear boundary. Meanwhile many games end up intermingling their engine and game code (e.g. see the Eve Online devs write about this - another closed source game on top of a now open source engine).
BAR is an open source game. All the game source is open.
Bro, your link literally says in the read me it is an engine.
No? It says
> Open source RTS game built on top of the Recoil RTS Engine
BAR is a game, Recoil (a different thing - not the repository I linked) is an engine it is built on top of. Recoil is - incidentally - a fork of spring, an engine many open source RTSes are built on.
Doom pioneered this model, the Doom engine was open-sourced while the assets remained proprietary. This is where projects like Freedoom exist, to create an entirely free software equivalent.
I think this misses the point I was trying to make. Both an open source game and an open weights LLM include the necessary assets. But people seem to be saying, “you need everything that was used to create the assets”. For game art, that would include sketches, prior art used as inspiration, project files that aren’t used by the game engine but were used during the design process, and so on - if you apply the same standard. But I don’t think anyone does.
Similar to this I've never seen a project request anyone license their github PR comments under any license whatsoever even though they're undoubtedly copyrighted and used in the creation of the actual source code.
My understanding was that there was a lot of non determinism in the training process because of many factors: how GPUs optimise floating point arithmetic, how the compute is distributed between the thousands of GPUs etc
I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things).
You'd likely get a model with very similar behaviour but the weights would be different.
Please someone correct me if I'm wrong.
This trope was valid maybe in 2022.
Model training now is not a straight forwards process of input data -> run tools -> get model.
There's a whole lot of alchemy going on. We don't quite understand what works and what doesn't. Think of it like painting with water color and having to improvise very often.
The only advantage over water color is that we can revert to a working state.
And since none of the source is provided to make the thing, it’s not open source.
Really, it’s no more open source than a free calculator. It’s free, and you can use it to no or great effect. But, you sure as hell can’t make one.
So calling it open source is without question, wrong.
There's no program you run to 'make the thing'.
It's mostly ad hoc scripts and some pretty horrible hacks being run by a hundred engineers trying to improve a thousand different things at once with a hundred thousand GPUs.
I'm sure once we understand the tech better, the training process will look like running a program.
If they open sourced the training data and code but you had to train the network yourself, would that be open source?
Which some companies have done! Nvidia, I believe, among others.
Yes, obviously.
No, obviously!
These models are a combination of a small amount of code, a ton of training data, and a ton of expertise in how to train them (which I suspect includes how best design/curate training sets for different model improvement goals).
The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data.
You don't need the source code - just read their architecture docs and implement it yourself.
The Chinese have actually been very open about training, starting perhaps with the DeepSeek-R1 paper which told the world in detail how to train a reasoning model. The Kimi 3 paper also gives a lot of training details.
There really isn't much of a comparison to be had between building a traditional software project where all you need is the (maybe open source) source code and the Makefile that automates the build process, and building a machine learning system where it's primarily about data not source code, and even with a road map of what may be very complex training (cf build) process, you'd probably still have a hard time building it since AFAIK the training process may still involve expert knowledge and intervention - I don't think training has been reduced to a hands-off "Makefile" or build script.
So, basically lack of source code is the least of the issues in being able to build one of these models - it's mostly the training data and training processes/expertise that you would need.
> The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data.
...and because that training data is missing, they can't be replicated. Which means that you cannot assert that the Chinese are being open in their LLM development, because there's no way to verify that the techniques they describe are actually the ones being used.
The reason that the training data is missing is that they're trained on a large amount of American copyrighted data and distilled on American models, which is where a lot of their performance comes from.
You can replicate the architectural innovations, and try them for yourself with your own dataset. It seems some of them are certainly being used by western companies, such as DeepSeek Sparse Attention, now supported by NVIDIA cuDNN.
Ditto for training algorithms and procedures such as Slime or DeepSeek's details instructions on how to build a reasoning model.
This is the exact value of openly shared details - others CAN copy and try them and modify them themselves.
Yes, the training data specifically has not been released for any model, American or Chinese, but that doesn't detract from what has been shared, and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing - because they are all using data from sources they don't want you to know about, and at the end of the day the data is the closest thing any of them do have to a moat.
> You can replicate the architectural innovations, and try them for yourself with your own dataset.
That's not related to my comment. My comment was pointing out that you can't verify something that wasn't published. You have no idea what fraction of their techniques they're not publishing, and how much they contribute to their model performance, because you cannot replicate the models, because they don't publish their training data.
> and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing
This is moving the goalposts. Your claim was that "The Chinese have actually been very open about training", which is false, as discussed. Nobody ever claimed that the American labs were open.
> You have no idea what fraction of their techniques they're not publishing, and how much they contribute to their model performance, because you cannot replicate the models, because they don't publish their training data.
Why would you be concerned about THEIR model performance ?!
Surely if you are an ML researcher and read about a new technique, you are interested in how YOU may be able to use it.
If you are Ilya Sutskever sitting at OpenAI in 2017, and happen upon Google's "attention" (Transformer architecture) paper, then what you do is go and implement it for yourself, get yourself some training data, and try it.
What you are NOT going to do is whine about not being given their source code, or their training data, or their training harness, or a dump of Google's corporate secrets. You take the research that has been shared and evaluate it for yourself.
This doesn't have anything to do with Google or OpenAI. I'm not Ilya Sutskever and it's not 2017, either.
I'm not "whining" about anything. You made the claim "The Chinese have actually been very open about training" and I showed that that was false. That's all that there is to it.
The rest of the word is reading Chinese published research and benefiting from it.
Apparently you are unaware of it and not benefiting from it.
Oh well.
Even though none of us could actually afford to train it?
Some of us might be rich, or get funding, or in the future it could be useful when training is cheaper.
Open source has benefits even if you can run yourself. You can read the code for understanding/insights. Other labs could replicate/build on it.
> Some of us might be rich
I sure wish I had a few 100M of disposable income to train a frontier model.
> or in the future it could be useful when training is cheaper.
I do not think that physics will allow hardware getting that much faster. But maybe we will have different, cheaper architectures by then.
You say that but for like 2 years there was a guy on huggingface releasing quants as TheBloke. No identity no nothing except he likely had a grant or a university job that let him do it. Quants aren't the same as training but it was beyond 99% of people at the time (as you say this is)
Cant stand when you guys try and force something as impossible on the rest of us simply because you could never accomplish it.
I did not say that it is impossible. I just think that we need architectural improvements, or maybe even a fundamentally different approach to get something like Kimi K3 for cheap. The point I was trying to make was that we shouldn't just laze about and hope that hardware improvements will get us there.
(Also, I know Tom Jobbins (TheBloke), and have personally contributed to increase the adoption of GGUF, e.g. in the transformers and ktransformers libraries, so I find the personal dig quite amusing.)
If you would tell people at the start of 20th century how much energy we consume, they may not believe you or think it is wasteful.
Looks like global energy consumption has risen by an order of magnitude from 1900 to 2000: https://www.encyclopedie-energie.org/en/world-energy-consump...
Unfortunately, electricity prices did not fall by the same factor, so I fear that training a frontier model will still cause a an unsustainable dent in my monthly budget.
Prices have dropped by a factor of around 6 in real terms since 1925 for US household electricity.
https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728...
Yet, I couldn’t afford a PC to run the original Crysis when it came out, now the hardware isn’t an issue anymore
Maybe not now, but what about 10 years down the line?
Yes.
As an example, Grok 4 took $388M to train
Compiling source code also takes computing resources, only the scale is different. It's a very close analogy: source code with training data, and compiled binaries with model weights.
The weights are literally a binary blob.
The weights are the modifiable representation. You modify them with gradient updates.
They are a cooked stew; the ingredients have already been chopped and mixed together. You can add things, yes, but you can't inspect the ingredients.
If we start with the same ingredients and independently make our stews they're likely still going to taste a bit different due to the non deterministic nature of the process. Isn't it similar with training LLMs at scale? Do you get identical weights if you do multiple runs?
They don't even say what the ingredients are.
Yeah I understand that. I mean just the methods they use to acquire the training data has already had a shit ton of drama, there's no way the western labs are ever going to be public about the ingredients.
I can also open broadcom's antenna firmware in any hex editor. Why are you arguing that makes it open source?
The antenna firmware is software. Model weights are essentially learned settings, and they are not software.
Note also that software is copyrightable and model weights aren’t (in the US anyway).
Not sure why it matters whether weights are software or not.
One can equally argue that firmware is just settings for the CPU.
If weights aren't copyrightable then doesn't this argue against the idea of them being able to have the quality of open-sourceness? Meanwhile the training code could be copyrightable.
Well, my position is that the software that defines/runs the model is the open-source part.
The weights are the asset that makes it useful, but I don’t think 40T tokens of pre-training data should be required to call the model itself open source when you can inspect every line of code in the model without that, as well as instantiate and run the model with randomly initialized weights.
You can hand-patch a binary, too.
An open source Mac OS is still open source even if I can’t afford a Mac.
Yes
The weights are the preferred form for modifying or integrating with other models. There is no obligation in open source to transitively open source all of the documentation / tools used to create the open source project.
>a term that it seems nearly everyone has agreed is appropriate
Models being considered open source even if the original training code / data is not released also is something almost everyone has agreed to be appropriate.
> The weights are the preferred form for modifying or integrating with other models
It's the 2nd time I hear this argument and I'm already fed up with it
Is it the preferred way only because training is expensive? It's like saying binaries are the preferred way of modifying program because you can't afford to have a fast enough machine to compile it yourself.
Most people don't have resources to compile their own browser, but what makes some browsers open is access to the source.
Maybe it's the preferred way because the people sharing models are themselves working with weights and no data?
> Is it the preferred way only because training is expensive?
No, it's the preferred way because that's literally how you train it. Contrary to popular misconceptions, you don't "compile" data into weights. You initialize a model (based on architecture, config, etc) and then you modify it via training. But crucially they (i.e. model creators) modify it the same way (technically speaking) as you would. That's what the license grants you. Nothing less, nothing more. The "how" as in knowhow has never been something covered by a license.
Open-weight is something dreamt up by people misunderstanding the basics of model creation and training, and having ideological biases against AI and/or LLMs. It is what it is, but you should know that you are technically wrong. A model released under an open source license (Apache, MIT, etc) is an open source model. Because the weights are the source of the models. Training is not "compilation". Training is the "how" as in knowhow to modify the model. Training deals with values. Source deals with everything, including values.
In the past, if someone would have released a piece of software (say a PID controller algo) w/ hardcoded values, no-one would bat an eye. LLMs are just that, with billions of hardcoded values. Nothing less nothing more.
> There is no obligation in open source to transitively open source all of the documentation / tools used to create the open source project.
Open source means open source code. Open weight means a binary file dump, not unlike an exe file. There is nothing open source about it.
Its like having a closed source text editor that censors certain words, and an open source text editor that censors certain words.
The latter can easily be recompiled, the former requires reverse engineering. Both may give you a license to use them freely.
Per the OSD definition of source code, "the source code must be the preferred form in which a programmer would modify the program." which means an argument could be made (as charcircuit is making) that the weights, being the preferred form to modify, are the source.
I do prefer open weights as being more precise (like, is it even really software that has source code in the first place?) but I feel like at this point the ship has sailed somewhat (though if this is something you're willing to spend your time arguing then... moral support I guess?)
You can also edit/patch binary executables, but no one says about that software that is open source.
You can, but its not the first preference
Incidentally, I do. I fucking hate the open source maximalists like Stallman who suggest that not having source code is equivalent to not having control over what runs on your computer. But you always[1] have control over what runs on your computer. Reverse engineer it if you care so much! Every form of DRM is inevitably defeated, because you do have control over program execution, regardless of access to the source code. Stallman's argument seems more lazy and entitled than principled.
[1] ...setting aside hardware attestation bullshit, which does need to die in a fire.
I think you are failing to see how the weights are the preferred form of working with a model. It's like if I shared a png with others to use as a meme template. Even if I don't share the source code to photoshop other people can use that "binary file dump" to make derivative images of their own with it.
I feel like the arguments being made about sharing the training data are from people who don’t have any experience training ML models and have never worked with weights. The biggest reason I have seen that people want the data is to complain about some aspect of it (copyright, bias, etc)
I would be very interested to hear someone explain what freedom it is they need the training data to enjoy.
Agreed, it's "open weights". Open source would be to publish the entire process so you could tweak it if you wanted.
The models are open source, it’s never been a requirement for someone to document the entire process of creating something to be “open source”. Just sharing the source, in this case, the weights, meets the criteria. Software engineering’s obsession with precise terms is actually one of the things I’m glad is dying with ai automation.
> obsession with precise terms
They’re not really even being precise. The relevant software freedom, from the FSF is []
Ported to the model world, this is fulfilled by sharing the weights and implementation. There’s almost nothing that having the training data gets you (other than actually training it). The weights plus a reference implementation let you see all the states to study the behavior, and let you fine tune it to do your bidding (the abliteration etc). The freedom is satisfied.Some might argue that without the training data you couldn’t do some classes of experiments to see how it works, say leave-one-out retraining. I’d argue things like that are not really about the model but about ML research or the class of models, which while interesting is not a free software pre-requisite.
[] https://www.gnu.org/philosophy/free-sw.html#four-freedoms
Agreed, The spirit is fulfilled, as evidenced by the massive ongoing development of, effectively, new weights grown out of the sourced weights,
The source is actually all training data, plus the software used for training, including some scripts or instructions to run the entire thing end to end on your own computer. That's what open source has always been about.
The weights are the output of a program, it's a binary. It's not source.
> Software engineering’s obsession with precise terms is actually one of the things I’m glad is dying with ai automation.
Yes. The next time you go to your doctor you should hope he's not being overly precise; or the engineer that builds the bridge; or the software engineer that implemented the embedded software in your insulin pump.
There is no precise thinking without precise terms.
Model weights are literally a compressed blob of binary data. The end result of a compilation process.
So is a zip file of source code. There is no 'compilation process'. Model training isn't a fixed end point. You start with random weights. You train. The weights get better. You do this iteratively. At some point you say 'good enough' and release. People doing further training would start with those weights and further iterate. Demanding the original training data and training process would be equivalent to demanding a completely replay-able sequence of editing steps starting with a blank text file and allowing you to replay ever edit that led the original author to the released source code.
Now there are reasons you might want to know about the training data when you wouldn't care about the authoring process used by a traditional open-source process. And these get at the reason LLM's are different than traditional software and so maybe our existing definitions of what 'open-source' means aren't a good match for LLM's. Of course there is software associated with LLM's (beyond the weight) -- defining the structure of the particular neural net those weights fit into. In every open-weights model that I'm aware of that software is open source (though trivial).
How does one even purchase an AMD MI355X?
Have $300k in disposible income?
add a few grand to get a new breaker panel and a new circuit put in to power it
If you do good work you should at least take the time to review the slop that details that work for slopiness. Otherwise it's hard to take it seriously. Especially the prefill section.
People complaining about the slop - what about the atrocious text/bg contrast? Burning my eyes out faster than a B300 ever could
AI slop
this got 82% human written on pangram :(. why the ai slop allegations? how can we make this better in the future?
(i work @ wafer & witnessed ian writing this)
Yeah I think they at least put some light effort into making it readable though. Obviously slop but not quite as bad as most slop articles.
There is barely any effort, a simple GPT5.6 sol pro query rips the post apart.
if you did that in the web interface, could you share the chat? I'd be interested to read it.
https://chatgpt.com/share/6a6f09ff-2830-83ea-9578-de3016cfae...
thanks for sharing.
When I read your original comment, I was thinking you had just asked it to evaluate the article. (Like just "evaluate this article" or something.)
I don't think anything anyone (or any AI) has ever written or published (including Sol itself) wouldn't be torn apart by the prompt you gave though.
I think it's fair to expect an extensive review of an article before publishing. Not everything has a set of serious flaws.
>Not everything has a set of serious flaws.
I'm saying everything looks like it does, with the prompt you gave Sol. Go ahead and point the same prompt to anything you don't think has serious flaws and you'll see it would tear it apart.
It doesn't look like it has found a set of serious flaws to me. Just nitpicking.
To begin with, as I pointed out separately, the price comparison that makes the headline is absurd. No one is easily getting the MI355X's at $2.5/hr. And then the eval is unreplicable and poorly defined.
[flagged]
[flagged]
[flagged]