There's nothing wrong to run CUDA on non-Nvidia hardware. CUDA has an interface that is reasonably well-designed, well-documented/reverse-engineered, and battle-tested for decades. What we need is not to invent another interface just under the name of 'open standard', but to implement the same interface. ROCm is exactly doing this, and so are other hardware SDKs such as MooreThread and Alibaba T-Head.
The difference between ROCm and CUDA is that when a consumer GPU is released by nvidia it's supported for CUDA for about a decade (1xxx series cards just dropped last year). When a consumer GPU is released by AMD it's not supported by ROCm till about a year after release and then it's supported for about 3-4 years. With the RX 580 there were only 3.7 years after release before ROCm support was pulled. I bought mine a couple years after release and so only had about a year and a half of ROCm. Never again.
Things might be different in enterprise but for consumer AMD GPU ROCm is a trap. It is a mayfly. Sure, you can try to run the cards unsupported but you're just multiplying the difficulty and maintainence burden. And nothing will just work.
Someone needs to stand up a benchmark suite for ROCM, this, and everyone else attempting it to really get the ball rolling here. SemiAnalysis could have a blast with this.
That sounds nice on paper, but you’re assuming Nvidia wants to play fair.
Nvidia is never going to share future microarchitecture secrets, so the moment they drop a new chip and update the compiler, everyone playing the compatibility game has to start from scratch.
These efforts to support CUDA on non-Nvidia hardware seem to me misguided. If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that (and torch.compile uses Triton for target-specific optimization). OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUDA translation layer on non-NVidia hardware would be an even worse idea.
In some benchmarks, SCALE beats nvcc, and we have compiler optimizations in the pipeline that will improve those numbers over time.
> If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that
Somewhat true, but, CUDA is significantly larger than PyTorch and there's more to Accelerated Computing than just those types of applications supported there.
> OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUDA translation layer on non-NVidia hardware would be an even worse idea.
SOTA mlperf submissions use CUDA to achieve their high levels of performance.
It's not a "translation layer", it's a native, ahead-of-time compiler that makes full use of the native hardware features. Here's an example of a feature (Shuffles) being compiled to take advantage of native hardware instructions, resulting in speedups: https://scale-lang.com/posts/2026-01-19-optimizing-cuda-shuf...
On the contrary, it's great. Cuda is the single sane compute API and system, so I'll use it even if it means being vendor-locked. If my CUDA programs start running elsewhere without much intervention, that'd be amazing
> Ease of programming and a giant leap in performance is one of the key reasons for the CUDA platform’s widespread adoption
This, so much. Other platforms continue to ignore developer UX, but it's one of the main things that get's new users onboard and keeps old users around.
Including stuff like Fortran, Haskell, Java, .NET via PTX, Python JIT, IDE tooling integration with major IDEs, graphical GPU debugging and profiling, libraries and co?
> In fact, contrary to the competition, one can play with CUDA even on laptops, go figure.
This is the part people don't get. You can program cuda anywhere on any Nvidia card, unlike other companies' chips you don't need a data center gpu to have full programmability. It's been this way for over a decade
We haven’t yet released support for tcgen05, but we’ll deal with it the same way we deal with other inline PTX: parsing it and converting it to target-appropriate instructions together with the rest of the program.
This is something we’ve done already for the hopper-class tensorcore instructions, and the blackwell ones will map similarly, though likely with a kernel launch involved.
A guess would be some time next year — since our public launch our focus has generally been on API coverage and increasingly recently, on performance.
While performance improvements will always remain a target, we're soon at full coverage of the core CUDA APIs and will be shifting an increasing amount of effort towards developer tooling.
In this context AdaptiveCpp should also be mentioned. Started as a SYCL implementation, but recently-ish added a compiler for compiling a CUDA dialect to GPUs and CPUs from basically all vendors
SYCL is probably the most up-to-date CUDA alternative for all intents and purposes, at least if one likes modern C++ style (and lambdas inside lambdas). Expose it as C and get bindings to any other language for relatively little effort as well since it’s just C++. With AdaptiveCpp you can also compile SYCL to CUDA so both ways work with the CUDA dialect (PCUDA).
SYCL, as well as AdaptiveCpp, is a relatively active project though and has been for several years, feeding into the C++ standards committee work and is supported by several large organisations, including US national labs and several European universities. I suppose it’s worth keeping track of for people in related fields.
I suppose it’s just really hard to beat the head start and ecosystem integration NVIDIA has with CUDA.
every CUDA alternative follows the same arc: bold launch, works for 3 operations, then a Discord server where the last message is 'any updates?' from 2024
I fail to see how scale is not just another form of vendor lock in, given that their compiler is not open source. Every compiler used today except cuda's is open source. And Nvidia can get away with it because no one else cares about development experience
Alternatives exist, but little demand outside hyperscalers and special uses.
Neocloud customers just want plug-and-play CUDA. It works, it's tested, it adapts faster, and has known performance. Alternatives give no significant benefits.
i'm also interested in tenstorrent. they're building GPUs with cheap GDDR6 using a fast SRAM cache, and writing their own compiler stack (used instead of CUDA) that pipelines data to the SRAM ahead-of-time so you (in theory) never need to suffer the slow speed of GDDR6 for AI workloads. also they've got built-in SFP cages where the video ports would normally be.
Isn't the future of the industry specialized chips like those that Broadcom and Cerebras are making? I don't know how much longer I can tolerate 50 tokens per second. It feels like the dial-up era.
A couple of years ago I evaluated both Vulkan and Cuda as a choice for future projects. I couldnt get anything done after a week in Vulkan, but had the test prototype project working after just a day in Cuda.
Needless to say, I'd never ever pick Vulkan for any project after that experience. It's just way to needlessly overengineered and bloated.
I used to be big into Khronos API camp, even did my project thesis in OpenGL, up to the famous Long Peaks fail.
Vulkan ended up being the same extension spaghetti as its predecessor, and Khronos was only able to come up with something thanks to AMD offering Mantle, C++ bindings and a GLSL successor only came to be thanks to NVidia (Vulkan-hpp and Slang started at NVidia).
The "we build the specification", and then "the community builds the tools", leads to very poor experiences, and if it wasn't for LunarG own interests, there wouldn't even exist any kind of Vulkan SDK.
What they have going is naturally the vendor independence, however we can achieve the same with middleware with the benefit of much better developer experience.
I love how people say things like "extension spaghetti", as if all other non-standard APIs have the same problem: hardware gets new features that people want to use from that API, API gains extension to use that hardware feature.
CUDA is no different, in fact, often worse. Nvidia is bad at documenting which hardware does what things, and CUDA users often have to use third party tables to figure out what hardware can't do what and disappoint customers who unwisely invested into it.
Without Android and embedded, its market is mostly SteamaDeck and some universities for the most part.
Nintendo, PlayStation, Apple and Microsoft have their own APIs.
Visualisation industry is still largely on OpenGL, when not using middleware that uses each platform proprietary API, or moving into compute like CUDA as OTOY has done.
Khronos had to come up with ANARI, to convince them to even think about Vulkan in first place.
Moving the goalposts much? Linux and Microsoft is still a huge market. Ii don't know about the switch 2 but the switch 1 had vulkan support. Apple as well if you count moltenvk
Not at all, I mentioned where Vulkan actually has a market, and why using Vulkan 1.4 is not the solution you think it is.
There is hardly any commercial Vulkan market on Windows, with exception of tools like Autodesk VRED or Disney Hyperion, hardly every man tools and the reason one might use desktop Linux for 3D rendering instead, with proprietary drivers anyway. As a user, not developer.
List of commercial games on Windows using Vulkan, without having a DirectX 12 backend as option is pretty thin.
They really don't, no. Vulkan: 50 lines to allocate device memory. Cuda: One single line. What kind of extensive documentation stack do you want for functionality that is trivial in Cuda? And that exact issue continues through every little step of the way to your first usable application. I know there is VMA, it is a very poor solution to a problem that shouldn't even exist, and it only poorly addresses one of 100 parts of the API where Cuda is vastly simpler than Vulkan. Cuda also doesnt force you to use queue families but you can optionally use streams. No ridiculous descriptor management and binding in cuda, just passing pointers and handles via launch arguments. No overengineered explicit syncing mechanis in cuda, everything is nicely implicitly synced until you explicitly opt in to parallel streams. etc.
It's quite easy to set up a light abstraction layer with Vulkan where you simply use VMA, buffer device addresses and push constants for everything. No descriptor sets or bindings anything.
Alternatively you can use one of many abstraction layers that do this for you.
It absolutely isn't. After having spent 5 days not getting anything done in Vulkan, and being able to implement that same thing in a single day in Cuda (no prior experience in either API), I decided to never ever use Vulkan. It's a hopelessly overengineered API that is in dire need of a successor.
I may give it another try once it does not require a wrapper before it is remotely usable. I.e., once it has a single-line malloc without the need for third-party libs; default queues so I don't need to query and select queues; implicit sync by default and explicit sync by choice; NV-style bindless (i.e. no descriptors, just a handle); and so much more.
Of course it is a skill issue, I'm not afraid of admitting I'm not smart enough for Vulkan. That so many people have skill issues is the reason why Cuda trumps and will continue to trump Vulkan despite being vendor-locked. If you want people to actually use Vulkan, you need to remove barriers to skill-issued people like me. Poor third party wrappers like VMA that barely address one out of hundreds of issues aren't going to accomplish that, you need to resolve barriers in the core API. With a design like Cuda where there is always a default easy path, and a complex but optional path.
Weird, since the most used open source inference engine is faster on Vulkan on platforms that offer multiple options, with the sole exception being Nvidia, due to poor Nvidia driver quality (which I am forced to assume is intentional, Nvidia wishes to maintain their moat after all).
There's nothing wrong to run CUDA on non-Nvidia hardware. CUDA has an interface that is reasonably well-designed, well-documented/reverse-engineered, and battle-tested for decades. What we need is not to invent another interface just under the name of 'open standard', but to implement the same interface. ROCm is exactly doing this, and so are other hardware SDKs such as MooreThread and Alibaba T-Head.
The difference between ROCm and CUDA is that when a consumer GPU is released by nvidia it's supported for CUDA for about a decade (1xxx series cards just dropped last year). When a consumer GPU is released by AMD it's not supported by ROCm till about a year after release and then it's supported for about 3-4 years. With the RX 580 there were only 3.7 years after release before ROCm support was pulled. I bought mine a couple years after release and so only had about a year and a half of ROCm. Never again.
Things might be different in enterprise but for consumer AMD GPU ROCm is a trap. It is a mayfly. Sure, you can try to run the cards unsupported but you're just multiplying the difficulty and maintainence burden. And nothing will just work.
Someone needs to stand up a benchmark suite for ROCM, this, and everyone else attempting it to really get the ball rolling here. SemiAnalysis could have a blast with this.
So is Spectral, which is mentioned in the headline of the article! As it says there:
> SCALE delivers nearly a 6x performance boost on AMD GPUs compared to using HIPIFY to convert CUDA code to AMD’s own ROCm environment
... whilst also running CUDA.
That sounds nice on paper, but you’re assuming Nvidia wants to play fair. Nvidia is never going to share future microarchitecture secrets, so the moment they drop a new chip and update the compiler, everyone playing the compatibility game has to start from scratch.
These efforts to support CUDA on non-Nvidia hardware seem to me misguided. If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that (and torch.compile uses Triton for target-specific optimization). OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUDA translation layer on non-NVidia hardware would be an even worse idea.
We actually support NVIDIA hardware, too.
In some benchmarks, SCALE beats nvcc, and we have compiler optimizations in the pipeline that will improve those numbers over time.
> If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that
Somewhat true, but, CUDA is significantly larger than PyTorch and there's more to Accelerated Computing than just those types of applications supported there.
> OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUDA translation layer on non-NVidia hardware would be an even worse idea.
SOTA mlperf submissions use CUDA to achieve their high levels of performance.
It's not a "translation layer", it's a native, ahead-of-time compiler that makes full use of the native hardware features. Here's an example of a feature (Shuffles) being compiled to take advantage of native hardware instructions, resulting in speedups: https://scale-lang.com/posts/2026-01-19-optimizing-cuda-shuf...
On the contrary, it's great. Cuda is the single sane compute API and system, so I'll use it even if it means being vendor-locked. If my CUDA programs start running elsewhere without much intervention, that'd be amazing
Most of these "alternatives" focus on CUDA C++, and overlook what actually makes CUDA interesting.
Already in 2020,
https://developer.nvidia.com/blog/cuda-refresher-the-gpu-com...
> Ease of programming and a giant leap in performance is one of the key reasons for the CUDA platform’s widespread adoption
This, so much. Other platforms continue to ignore developer UX, but it's one of the main things that get's new users onboard and keeps old users around.
We're actually targeting all of it, and not just CUDA C++.
Including stuff like Fortran, Haskell, Java, .NET via PTX, Python JIT, IDE tooling integration with major IDEs, graphical GPU debugging and profiling, libraries and co?
Then I guess all the best.
This post has some serious peanut-gallery vibes.
Peanut-gallery is happily using CUDA, and needs actual sound reasons to move.
Then the peanut gallery has nothing to complain when Nvidia jacks up prices.
Do you see me complaining?
Here is a tip, you don't always need to suffer from FOMO and get the very latest model card.
In fact, contrary to the competition, one can play with CUDA even on laptops, go figure.
> In fact, contrary to the competition, one can play with CUDA even on laptops, go figure.
This is the part people don't get. You can program cuda anywhere on any Nvidia card, unlike other companies' chips you don't need a data center gpu to have full programmability. It's been this way for over a decade
Yes, this is how they get students hooked on CUDA.
They don't though. Money for ai is pretty much free in America to anyone who can demonstrate a modicum of competence in the field.
The only people who are without access are students or hobbyists really.
How do you deal with target-specific inline asm like tcgen05.mma?
We haven’t yet released support for tcgen05, but we’ll deal with it the same way we deal with other inline PTX: parsing it and converting it to target-appropriate instructions together with the rest of the program.
This is something we’ve done already for the hopper-class tensorcore instructions, and the blackwell ones will map similarly, though likely with a kernel launch involved.
Ambitious but neat, good luck if nothing else :)
If you were to guess, when do you think your Nsight Compute alternative might be ready with your own toolchain?
A guess would be some time next year — since our public launch our focus has generally been on API coverage and increasingly recently, on performance.
While performance improvements will always remain a target, we're soon at full coverage of the core CUDA APIs and will be shifting an increasing amount of effort towards developer tooling.
In this context AdaptiveCpp should also be mentioned. Started as a SYCL implementation, but recently-ish added a compiler for compiling a CUDA dialect to GPUs and CPUs from basically all vendors
SYCL is probably the most up-to-date CUDA alternative for all intents and purposes, at least if one likes modern C++ style (and lambdas inside lambdas). Expose it as C and get bindings to any other language for relatively little effort as well since it’s just C++. With AdaptiveCpp you can also compile SYCL to CUDA so both ways work with the CUDA dialect (PCUDA).
SYCL, as well as AdaptiveCpp, is a relatively active project though and has been for several years, feeding into the C++ standards committee work and is supported by several large organisations, including US national labs and several European universities. I suppose it’s worth keeping track of for people in related fields.
I suppose it’s just really hard to beat the head start and ecosystem integration NVIDIA has with CUDA.
every CUDA alternative follows the same arc: bold launch, works for 3 operations, then a Discord server where the last message is 'any updates?' from 2024
Actually we launched in 2024 and the last message in our discord is definitely not that: https://discord.gg/KNpgGbTc38
I fail to see how scale is not just another form of vendor lock in, given that their compiler is not open source. Every compiler used today except cuda's is open source. And Nvidia can get away with it because no one else cares about development experience
Alternatives exist, but little demand outside hyperscalers and special uses.
Neocloud customers just want plug-and-play CUDA. It works, it's tested, it adapts faster, and has known performance. Alternatives give no significant benefits.
Things can change, but they are not changing now.
There is also ZLUDA, which is open source and works on pre-compiled binaries.
https://github.com/vosen/ZLUDA
this is closest thing we have to "cuda on non-nvidia" hardware
We have a comparison page: https://docs.scale-lang.com/stable/manual/comparison/#zluda
I have been trying for cuda -> metal, to run it on mac, https://github.com/lulzx/cuda-metal
i'm also interested in tenstorrent. they're building GPUs with cheap GDDR6 using a fast SRAM cache, and writing their own compiler stack (used instead of CUDA) that pipelines data to the SRAM ahead-of-time so you (in theory) never need to suffer the slow speed of GDDR6 for AI workloads. also they've got built-in SFP cages where the video ports would normally be.
Is tenstorrent building GPUs now, not just tensor processors?
Why should I not just port my kernel to Triton? What's the appeal of Scale?
You can skip the porting part.
aren't llms smart enough to directly write custom kernels for custom hardware from cuda code?
Isn't the future of the industry specialized chips like those that Broadcom and Cerebras are making? I don't know how much longer I can tolerate 50 tokens per second. It feels like the dial-up era.
@claude add this to the graveyard of wannabees
Its easier to just get rid of your legacy code entirely and use Vulkan for compute, or have your compiler emit SPIR-V directly.
No reason to tie yourself to Nvidia's moat.
A couple of years ago I evaluated both Vulkan and Cuda as a choice for future projects. I couldnt get anything done after a week in Vulkan, but had the test prototype project working after just a day in Cuda.
Needless to say, I'd never ever pick Vulkan for any project after that experience. It's just way to needlessly overengineered and bloated.
I used to be big into Khronos API camp, even did my project thesis in OpenGL, up to the famous Long Peaks fail.
Vulkan ended up being the same extension spaghetti as its predecessor, and Khronos was only able to come up with something thanks to AMD offering Mantle, C++ bindings and a GLSL successor only came to be thanks to NVidia (Vulkan-hpp and Slang started at NVidia).
The "we build the specification", and then "the community builds the tools", leads to very poor experiences, and if it wasn't for LunarG own interests, there wouldn't even exist any kind of Vulkan SDK.
What they have going is naturally the vendor independence, however we can achieve the same with middleware with the benefit of much better developer experience.
I love how people say things like "extension spaghetti", as if all other non-standard APIs have the same problem: hardware gets new features that people want to use from that API, API gains extension to use that hardware feature.
CUDA is no different, in fact, often worse. Nvidia is bad at documenting which hardware does what things, and CUDA users often have to use third party tables to figure out what hardware can't do what and disappoint customers who unwisely invested into it.
The other platforms have better ways to deal with progress instead of "here find entries on dynamic libraries by yourself", and good luck.
Profiles and API versions are much better approaches.
It is no accident than the ongoing efforts to make Vulkan more friendly are moving away from extension spaghetti into profiles.
If you think that Vulkan is extension spaghetti you're clearly using it wrong. Set the API to 1.4 and many existing extensions get merged in.
If you think changing to Vulkan 1.4 solves all the problems, you clearly aren't writing cross platform code.
First of all, that isn't even a thing if you need to target Android, or embedded hardware, secondly there are other extensions on the horizon.
The vast majority of vulkan usecases aren't android or embedded. I indeed wouldn't recommend it there.
Without Android and embedded, its market is mostly SteamaDeck and some universities for the most part.
Nintendo, PlayStation, Apple and Microsoft have their own APIs.
Visualisation industry is still largely on OpenGL, when not using middleware that uses each platform proprietary API, or moving into compute like CUDA as OTOY has done.
Khronos had to come up with ANARI, to convince them to even think about Vulkan in first place.
Moving the goalposts much? Linux and Microsoft is still a huge market. Ii don't know about the switch 2 but the switch 1 had vulkan support. Apple as well if you count moltenvk
Not at all, I mentioned where Vulkan actually has a market, and why using Vulkan 1.4 is not the solution you think it is.
There is hardly any commercial Vulkan market on Windows, with exception of tools like Autodesk VRED or Disney Hyperion, hardly every man tools and the reason one might use desktop Linux for 3D rendering instead, with proprietary drivers anyway. As a user, not developer.
List of commercial games on Windows using Vulkan, without having a DirectX 12 backend as option is pretty thin.
Weird, most people have the exact opposite experience.
Having to deal with closed source opaque poorly documented stacks sucks.
They really don't, no. Vulkan: 50 lines to allocate device memory. Cuda: One single line. What kind of extensive documentation stack do you want for functionality that is trivial in Cuda? And that exact issue continues through every little step of the way to your first usable application. I know there is VMA, it is a very poor solution to a problem that shouldn't even exist, and it only poorly addresses one of 100 parts of the API where Cuda is vastly simpler than Vulkan. Cuda also doesnt force you to use queue families but you can optionally use streams. No ridiculous descriptor management and binding in cuda, just passing pointers and handles via launch arguments. No overengineered explicit syncing mechanis in cuda, everything is nicely implicitly synced until you explicitly opt in to parallel streams. etc.
It's quite easy to set up a light abstraction layer with Vulkan where you simply use VMA, buffer device addresses and push constants for everything. No descriptor sets or bindings anything.
Alternatively you can use one of many abstraction layers that do this for you.
It absolutely isn't. After having spent 5 days not getting anything done in Vulkan, and being able to implement that same thing in a single day in Cuda (no prior experience in either API), I decided to never ever use Vulkan. It's a hopelessly overengineered API that is in dire need of a successor.
I may give it another try once it does not require a wrapper before it is remotely usable. I.e., once it has a single-line malloc without the need for third-party libs; default queues so I don't need to query and select queues; implicit sync by default and explicit sync by choice; NV-style bindless (i.e. no descriptors, just a handle); and so much more.
Skill issue. Vulkan is intended to be unopinionated around those things. If you want defaults then use a wrapper.
P.s. devices and queues are generally ordered for simple programs you can just pick the 1st one.
Of course it is a skill issue, I'm not afraid of admitting I'm not smart enough for Vulkan. That so many people have skill issues is the reason why Cuda trumps and will continue to trump Vulkan despite being vendor-locked. If you want people to actually use Vulkan, you need to remove barriers to skill-issued people like me. Poor third party wrappers like VMA that barely address one out of hundreds of issues aren't going to accomplish that, you need to resolve barriers in the core API. With a design like Cuda where there is always a default easy path, and a complex but optional path.
Unfortunately, Vulkan Compute doesn’t to all the things that OpenCL, SYCL, HIP or CUDA do.
Yep, there are inference stacks where it just does not work without cuda in any meaningful performance
Weird, since the most used open source inference engine is faster on Vulkan on platforms that offer multiple options, with the sole exception being Nvidia, due to poor Nvidia driver quality (which I am forced to assume is intentional, Nvidia wishes to maintain their moat after all).
Being fast and being as easy to program as CUDA are two different things.
There's nothing stopping any of us from writing a better Nvidia driver btw. LLMs are very helpful with reverse engineering.
Vulkan tooling is light years behind what CUDA offers in 2026, across programming languages, IDE tooling, graphical debuggers and libraries.
Ports are very often incredibly difficult and very time consuming.
One of the biggest complaints we hear from the industry is "we tried to port to X and we could never complete it".
An established codebase can have years of refinement. It will take time to achieve the same with the port.
And with our compiler, just using cuda is no longer putting urself inside the moat :)
Ironically, this is what people claim AI can do with a snap of the fingers.
Should be real simple if the HN AI echochamber is right, right?