Broadly enjoyed my read, though I do have a nitpick here:
>The absence of side effects or stalls cannot be enforced by the type system ...
Side effects is probably correct, but stalls would imply wasm code that calls out to a long-time blocking function - that's generally quite easy to type-system-ify, and wasm stuff often does so with promise-like constructs. So there would be a need for spacetimedb-side markers for "this func might do HTTP", but that kind of marker for WASM-contact-able code is very much a normal expectation.
If they don't have that kind of marker, and do allow blocking calls in their beta API, then yeah - huge problem with that kind of internal structure (shared global lock), completely agreed.
The SpacetimeDB launch video came up on my YouTube feed - and I was surprised the video didn't go at all into how it was implemented. I assumed it was proprietary magic, but then I was surprised to find it was opensource. That confused me more - to get the kind of semantics they were talking about I had assumed that it was pretty novel and if it was open source they should be leading with that.
Kind of bummed to see its essentially 2015-era React Flux in Rust around a mutex.
There’s also an excellent paper called “Fair Benchmarking Considered Difficult:
Common Pitfalls In Database Performance Testing”.
Benchmarks are genuinely hard. Modern systems and hardware are notoriously complex and have different behaviors across runs and with minor tweaks to performance settings.
Yep, absolutely... and in the end benchmarks aren't going to sell a database anyways. Reliability and provable correctness and ease of integration are probably going to win in the end. It's a dubious marketing edge to lead with.
PostgreSQL itself is remarkably slow in many configurations and leaves a lot of performance on the table due to its architecture and assumptions. But it works and people trust it. So they use it.
Very cool watching late stage software shops undiscover n tier architecture. Of course, breaking from hard learned wisdom for niche use cases is fine… until you bury the reality in a misleading and edge lord-ish PR sweep.
One of the problems with running the application code inside the database server is that you need to make application developers use the programming language(s) you prepared for them. The other is that you need to trust them to not do crash out, or do infinite loops. Good luck moving application developers from their choice into the right choice.
This is why you need to compile your code for spacetimedb to WebAssembly and it runs inside a VM. Use the language of your choice, and assume the code is only semi-trusted with the ability to have a watchdog. That part makes sense.
"Because the system is, well, a hash table with a lock in front of it."
This actually surprises me? I was a very early employee/contractor working on this product but left after 3 months. But this wasn't the shape it had when I was there. It would have been better described as a CoW tree of various sorts, which lets reads and writes advance on separate transactions and reconciles at commit with what I recall was fairly minimal lock-holding time.
When I was there there was no global lock being held while a WASM "reducers" was executing. They operated on their own separate transaction in their own thread.
I haven't looked at their source since -- and I really don't want to because I work on similar things I don't want IP contamination as last I looked they had some sort of weird license -- but there's been .. 3 years? of development on it since and I know Tyler well enough and he hired smart-ish people.
I had a lot of criticisms of what he was making and the way it was being made... but ehh....I would want to confirm in source that this claim is accurate, it makes me a bit suspicious of the accuracy and fairness of the rest of TFA.
Ok, so I had a clanker go look. It actually sounds as bad as the article says. For some reason they apparently lock things while "reducer"s are running.
Robot brain summarized:
"Your historical description may be completely accurate.
Your request that the author substantiate the claim was warranted.
But the source substantiates it extremely clearly.
The blog’s concurrency conclusion is right.
Its “hash table,” fairness, “cannot sync WAL,” and “questionable to call it a database” language is deliberately contemptuous and sometimes technically sloppy."
I watched a few of their talks, and while I'm somewhat impressed by the results, it's not exactly surprising to me that you can get those numbers when you make the trade-offs they're making. Most applications cannot make those trade-offs and, thus, cannot ever hope to see those numbers. The lack of rigor in database benchmarking in any of the videos I saw was an obvious indicator of something to be wary of.
Broadly enjoyed my read, though I do have a nitpick here:
>The absence of side effects or stalls cannot be enforced by the type system ...
Side effects is probably correct, but stalls would imply wasm code that calls out to a long-time blocking function - that's generally quite easy to type-system-ify, and wasm stuff often does so with promise-like constructs. So there would be a need for spacetimedb-side markers for "this func might do HTTP", but that kind of marker for WASM-contact-able code is very much a normal expectation.
If they don't have that kind of marker, and do allow blocking calls in their beta API, then yeah - huge problem with that kind of internal structure (shared global lock), completely agreed.
The SpacetimeDB launch video came up on my YouTube feed - and I was surprised the video didn't go at all into how it was implemented. I assumed it was proprietary magic, but then I was surprised to find it was opensource. That confused me more - to get the kind of semantics they were talking about I had assumed that it was pretty novel and if it was open source they should be leading with that.
Kind of bummed to see its essentially 2015-era React Flux in Rust around a mutex.
Always a fun time to bring up my favorite writeups in database benchmarking: https://questdb.com/blog/lies-damn-lies-and-database-benchma...
There’s also an excellent paper called “Fair Benchmarking Considered Difficult: Common Pitfalls In Database Performance Testing”.
Benchmarks are genuinely hard. Modern systems and hardware are notoriously complex and have different behaviors across runs and with minor tweaks to performance settings.
Yep, absolutely... and in the end benchmarks aren't going to sell a database anyways. Reliability and provable correctness and ease of integration are probably going to win in the end. It's a dubious marketing edge to lead with.
PostgreSQL itself is remarkably slow in many configurations and leaves a lot of performance on the table due to its architecture and assumptions. But it works and people trust it. So they use it.
Or if you're in enterprise, whichever company's sales rep some executive has locked you into while 10 drinks in at a fancy dinner.
Very cool watching late stage software shops undiscover n tier architecture. Of course, breaking from hard learned wisdom for niche use cases is fine… until you bury the reality in a misleading and edge lord-ish PR sweep.
One of the problems with running the application code inside the database server is that you need to make application developers use the programming language(s) you prepared for them. The other is that you need to trust them to not do crash out, or do infinite loops. Good luck moving application developers from their choice into the right choice.
This is why you need to compile your code for spacetimedb to WebAssembly and it runs inside a VM. Use the language of your choice, and assume the code is only semi-trusted with the ability to have a watchdog. That part makes sense.
"Because the system is, well, a hash table with a lock in front of it."
This actually surprises me? I was a very early employee/contractor working on this product but left after 3 months. But this wasn't the shape it had when I was there. It would have been better described as a CoW tree of various sorts, which lets reads and writes advance on separate transactions and reconciles at commit with what I recall was fairly minimal lock-holding time.
When I was there there was no global lock being held while a WASM "reducers" was executing. They operated on their own separate transaction in their own thread.
I haven't looked at their source since -- and I really don't want to because I work on similar things I don't want IP contamination as last I looked they had some sort of weird license -- but there's been .. 3 years? of development on it since and I know Tyler well enough and he hired smart-ish people.
I had a lot of criticisms of what he was making and the way it was being made... but ehh....I would want to confirm in source that this claim is accurate, it makes me a bit suspicious of the accuracy and fairness of the rest of TFA.
Ok, so I had a clanker go look. It actually sounds as bad as the article says. For some reason they apparently lock things while "reducer"s are running.
Robot brain summarized:
"Your historical description may be completely accurate. Your request that the author substantiate the claim was warranted. But the source substantiates it extremely clearly. The blog’s concurrency conclusion is right. Its “hash table,” fairness, “cannot sync WAL,” and “questionable to call it a database” language is deliberately contemptuous and sometimes technically sloppy."
touche
I watched a few of their talks, and while I'm somewhat impressed by the results, it's not exactly surprising to me that you can get those numbers when you make the trade-offs they're making. Most applications cannot make those trade-offs and, thus, cannot ever hope to see those numbers. The lack of rigor in database benchmarking in any of the videos I saw was an obvious indicator of something to be wary of.
They're calling a hashmap behind a RwLock a "database"? Damnit I wish I had the capacity for grifting, I could be rich by now.