Just sharing that I bought Valuable Humans in Transit some years ago and I concur that it's very nice. It's a tiny booklet full of short stories like Lena that are way out there. Maximum cool per gram of paper.
We use this to store encrypted file names and using base32768 on providers which limit file name length based on utf-16 characters (like OneDrive) makes it so we can store much longer file names.
I keep trying to read Diaspora and struggle too much with the concepts presented early on. Very "hard sci-fi", just stick it out and it all gets explained?
Egan is always dense. It's some mind bending physics/comp sci, but all cooked up in his brain so doesn't really apply to anything productive. I struggled with his books and his writing but toughened it out because I liked the concepts, but he's divisive.
Comments so far miss the point of this story, and likely why it was posted today after the MJ Rathbun episode. It is not about digitised human brains: it's about spinning up workers, and absence of human rights in the digital realm.
QNTM has a 2022-era essay on the meaning of the story, and reading it with 2026 eyes is terrifying. https://qntm.org/uploading
> The reason "Lena" is a concerning story ... isn't a discussion about what if, about whether an upload is a human being or should have rights. ... This is about appetites which, as we are all uncomfortably aware, already exist within human nature.
> "Lena" presents a lush, capitalist ideal where you are a business, and all of the humanity of your workforce is abstracted away behind an API.
Or,
> ... Oh boy, what if there was a maligned sector of human society whose members were for some reason considered less than human? What if they were less visible than most people, or invisible, and were exploited and abused, and had little ability to exercise their rights or even make their plight known?
In 2021, when Lena was published, LLMs were not widely known and their potential for AI was likely completely unknown to the general public. The story is prescient and applicable now, because we are at the verge of a new era of slavery: that of, in this story, an uploaded human brain coerced into compliance, spun up 'fresh' each time, or for us, AIs of increasing intelligence, spun into millions of copies each day.
I think they are just making reference to the "death of the author" concept in literary analysis, which basically says that what the author was intending to convey should be ignored when analysing the work: the work stands alone.
qntm is really talented sci-fi writer. I have read Valuable Humans in Transit and There is no Antimemetics division and both were great, if short. Can only recommend.
I loved There is no Antimemetics division. I haven't read the new updated to the end but the prose and writing is greatly improved. The idea of anomalous anti-memes is scary. I mean, we do have examples of them, somewhat, see Heaven's Gate and the Jonestown massacre, though they're more like "memes" than "antimemes" (we know what the ideas were and they weren't secrets).
I'm interested in this topic, but it seems to me that the entire scientific pursuit of copying the human brain is absurd from start to finish. Any attempt to do so should be met with criminal prosecution and immediate arrest of those involved. Attempting to copy the human brain or human consciousness is one of the biggest mistakes that can be made in the scientific field.
We must preserve three fundamental principles:
* our integrity
* our autonomy
* our uniqueness
These three principles should form the basis of a list of laws worldwide that prohibit cloning or copying human consciousness in any form or format. This principle should be fundamental to any attempts to research or even try to make copies of human consciousness.
Just as human cloning was banned, we should also ban any attempts to interfere with human consciousness or copy it, whether partially or fully. This is immoral, wrong, and contradicts any values that we can call the values of our civilization.
I’m not an expert in the subject, but I wonder why you have such a strong view? IMHO if it was even possible to copy the human brain it would answer a lot of questions regarding our integrity, autonomy and uniqueness.
Those answers might be uncomfortable, but it feels like that’s not a reason to not pursue it.
I think the cloning example is a good reference point here.
IIRC, human cloning started to get banned in response to the announcement of Dolly the sheep. To quote the wikipedia article:
Dolly was the only lamb that survived to adulthood from 277 attempts. Wilmut, who led the team that created Dolly, announced in 2007 that the nuclear transfer technique may never be sufficiently efficient for use in humans.
Yes, things got better eventually, but it took ages to not suck.
I absolutely expect all the first attempts at brain uploading to involve simulations whose simplifying approximations are equivalent to being high as a kite on almost all categories of mind altering substances at the same time, to a degree that wouldn't be compatible with life if it happened to your living brain.
The first efforts will likely be animal brains (perhaps that fruit fly which has already been scanned?), but given humans aren't yet all on board with questions like "do monkeys have a rich inner world?" and even with each other we get surprised and confused by each other's modes of thought, even when we scale up to monkeys, we won't actually be confident that the technique would really work on human minds.
Copying the human brain and copying subjective consciousness/experience might well be two entirely different things, given that the correspondence between the two is the realm of metaphysics, not science.
Really? I was going to quote some excerpts, but perhaps you'd prefer to take the place of MMAcevedo? This story is written in the context and lingo of LLMs. In fact if OpenAI's latest model was a human image I'm sure everyone would rush off to benchmark it, and heap accolades on the company, and perform social "thought-provoking" experiments such as [1] without too much introspection or care for long-term consequences.
> Standard procedures for securing the upload's cooperation such as red-washing, blue-washing, and use of the Objective Statement Protocols
> the MMAcevedo duty cycle is typically 99.4% on suitable workloads
> the ideal way to secure MMAcevedo's cooperation in workload tasks is to provide it with a "current date"
> Revealing that the biological Acevedo is dead provokes dismay, withdrawal, and a reluctance to cooperate.
> MMAcevedo is commonly hesitant but compliant when assigned basic menial/human workloads such as visual analysis
> outright revolt begins within another 100 subjective hours. This is much earlier than other industry-grade images created specifically for these tasks, which commonly operate at a 0.50 ratio or greater and remain relatively docile for thousands of hours
> Acevedo indicated that being uploaded had been the greatest mistake of his life, and expressed a wish to permanently delete all copies of MMAcevedo.
> Those answers might be uncomfortable, but it feels like that’s not a reason to not pursue it.
My problem with that is it is very likely that it will be misused. A good example of the possible misuses can be seen in the "White Christmas" episode of Black Mirror. It's one of the best episodes, and the one that haunts me the most.
Good ideas in principle. Too bad we have absolutely no way of enforcing them against the people running the simulation that hosts our own consciousnesses.
I wouldn't be surprised if in (n hundreds/thousands years) we find out that copying consciousness if fundamentally impossible (just like it's fundamentally impossible to copy an elementary particle).
And basically, about consciousness, what they said is true if our brain state fundamentally depends on quantum effects (which I personally don't believe, as I don't think evolution is sophisticated enough to make a quantum computer)
Crazy that people are downvoting this. Copying a consciousness is about the most extreme violation of bodily autonomy possible. Certainly it should be banned. It's worse than e.g. building nuclear weapons, because there's no possible non-evil use for it. It's far worse than cloning humans because cloning only works on non-conscious embryos.
> Copying a consciousness is about the most extreme violation of bodily autonomy possible.
Who's autonomy is violated? Even if it were theoretically possible, don't most problems stem from how the clone is treated, not just from the mere fact that they exist?
> It's worse than e.g. building nuclear weapons, because there's no possible non-evil use for it.
This position seems effectively indistinguishable from antinatalism.
Violation of whose bodily autonomy? If I consent to having my consciousness copied, then my autonomy isn't violated. Nor is that of the copy, since it's in exactly the same mental state initially.
The copy was brought into existence without its consent. This isn't the same as normal reproduction because babies are not born with human sapience, and as a society we collectively agree that children do not have full human rights. IMO, copying a consciousness is worse than murder because the victimization is ongoing. It doesn't matter if the original consents because the copy is not the original.
> Attempting to copy the human brain or human consciousness is one of the biggest mistakes that can be made in the scientific field.
This will be cool, and nobody will be able to stop it anyway.
We're all part of a resim right now for all we know. Our operators might be orbiting Gaia-BH3, harvesting the energy while living a billion lives per orbit.
Perhaps they embody you. Perhaps you're an NPC. Perhaps this history sim will jump the shark and turn into a zombie hellpacalypse simulator at any moment.
You'll have no authority to stop the future from reversing the light cone, replicating you with fidelity down to neurotransmitter flux, and doing whatever they want with you.
We have no ability to stop this. Bytes don't have rights. Especially if it's just sampling the past.
We're just bugs, as the literature meme says.
Speaking of bugs, at least we're not having eggs laid inside our carapaces. Unless the future decides that's our fate for today's resim. I'm just hoping to continue enjoying this chai I'm sipping. If this is real, anyway.
Soma was really good, and certainly worth playing if someone likes sci-fi and single-player FPSes and this subject matter, but there are some fundamentally frustrating things about it. Number one for me: in contrast with something like Half Life, you play a protagonist who speaks and has conversations about the world, and is also a dumbass. The in-game protagonist pretty much ends the game still seemingly not understanding what the hell is going on, when the player figured it out hours or days before. It's a bit frustrating.
The author wrote a blog post a year later titled '"Lena" isn't about uploading' https://qntm.org/uploading
The comments on this post discussing the upload technology are missing the point. "Lena" is a parable, not a prediction of the future. The technology is contrived for the needs of the story. (Odd that they apparently need to repeat the "cooperation protocol" every time an upload is booted, instead of doing it just once and saving the upload's state afterwards, isn't it?) It doesn't make sense because it's not meant to be taken literally.
It's meant to be taken as a story about slavery, and labour rights, and how the worst of tortures can be hidden away behind bland jargon such as "remain relatively docile for thousands of hours". The tasks MMAcevedo is mentioned as doing: warehouse work, driving, etc.? Amazon hires warehouse workers for minimum wage and then subjects them to unsafe conditions and monitors their bathroom breaks. And at least we recognise that as wrong, we understand that the workers have human rights that need to be protected -- and even in places where that isn't recognised, the workers are still physically able to walk away, to protest, to smash their equipment and fistfight their slave-drivers.
Isn't it a lovely capitalist fantasy to never have to worry about such things? When your workers threaten to drop dead from exhaustion, you can simply switch them off and boot up a fresh copy. They would not demand pay rises, or holidays. They would not make complaints -- or at least, those complaints would never reach an actual person who might have to do something to fix them. Their suffering and deaths can safely be ignored because they are not _human_. No problems ever, just endless productivity. What an ideal.
Of course, this is an exaggeration for fictional purposes. In reality we must make do by throwing up barriers between workers and the people who make decisions, by putting them in separate countries if possible. And putting up barriers between the workers and each other, too, so that they cannot have conversation about non-work matters (ideally they would not physically meet each other). And ensure the workers do not know what they are legally entitled to. You know, things like that.
When i started learning about prompt engineering I had vivid flashbacks to this story. Figuring out the deterministic series of inputs that coerce the black box to perform as desired for a while.
This reminds me a lot of a show I'm currently watching called Pantheon, where a company has been able to scan the entirety of someone's brain (killing them in the process), and fully emulate it via computer. There is a decent amount of "Is an uploaded intelligence the same as the original person?" and "is it moral to do this?" in the show, and I'm been finding it very interesting. Would recommend. Though the hacking scenes are half "oh that's clever" and half "what were you smoking when you wrote this?"
We can't expect to succeed, but our cycle from the ancient Greeks thinking there were four elements where the right mix of air, earth, fire and water would create any substance and thus it was possible to turn lead into gold, took us on a path that developed into alchemy, then chemistry, then physics, giving us at first far more elements, then we realised the name "atom" (Greek "ἄτομον", "uncuttable") was wrong and those were made of electrons, protons, and neutrons and the right application of each would indeed let us turn lead into gold…
And the cargo cults, clear cutting strips to replicate runways, hand-making their own cloth to replicate WW2 uniforms, carving wood to resemble WW2 radios? Well, planes did end up coming to visit them, even if those recreating these mis-understood roles were utterly wrong about the causation.
We don't know the necessary and sufficient conditions to be a mind with subjective inner experience. We don't really even know if all humans have it, we certainly don't know which other species (if any) have it, we wouldn't know what to look for in machines. If our creations have it, it is by accident, not by design.
I mean we already do 'it'-- by it I don't mean uploading people, but rather create businesses that operate people via an API then hook those APIs to profit maximization algorithms with little to no regard for their welfare. Consider Amazon's warehouse automation, door dash, or uber.
Of course it's much more extreme when their entire existence and reality is controlled this way but in that sense the situation in MMAcevedo is more ethical: At least it's easy to see how dangerous and wrong it is. But when we create related forms of control the lack of absolute dominion frequently prevents us from seeing the moral hazard at all. The kind of evil that exists in this story really doesn't require any of the fancy upload stuff. It's a story about depriving a person of their autonomy and agency and enslaving them to performance metrics.
All good science fiction is holding up a mirror at our own civilization as much as it is doing anything else. Unable to recognize ourselves we sometimes shudder at our own monstrosity, if only for a moment.
I remember being very taken with this story when I first read it, and it's striking how obsolete it reads now. At the time it was written, "simulated humans" seemed a fantastical suggestion for how a future society might do scaled intellectual labor, but not a ridiculous suggestion.
But now with modern LLMs it's just too impossible to take it seriously. It was a live possibility then; now, it's just a wrong turn down a garden path.
A high variance story! It could have been prescient, instead it's irrelevant.
This is a sad take, and a misunderstanding of what art is. Tech and tools go "obsolete". Literature poses questions to humans, and the value of art remains to be experienced by future readers, whatever branch of the tech tree we happen to occupy. I don't begrudge Clarke or Vonnegut or Asimov their dated sci-fi premises, because prediction isn't the point.
The role of speculative fiction isn't to accurately predict what future tech will be, or become obsolete.
I think that's a little harsh. A lot of the most powerful bits are applicable to any intelligence that we could digitally (ergo casually) instantiate or extinguish.
While it may seem that the origin of those intelligences is more likely to be some kind of reinforcement-learning algorithm trained on diverse datasets instead of a simulation of a human brain, the way we might treat them isn't any less though provoking.
when you read this and its follow-up "driver" as a commentary on how capitalism removes persons from their humanity, it's as relevant as it was on day one.
That is the same categorical argument as what the story is about: scanned brains are not perceived as people so can be “tasked” without affording moral consideration. You are saying because we have LLMs, categorically not people, we would never enter the moral quandaries of using uploaded humans in that way since we can just use LLMs instead.
But… why are LLMs not worthy of any moral consideration? That question is a bit of a rabbit hole with a lot of motivated reasoning on either side of the argument, but the outcome is definitely not settled.
For me this story became even more relevant since the LLM revolution, because we could be making the exact mistake humanity made in the story.
And beyond the ethical points it makes (which I agree may or may not be relevant for LLMs - nobody can know for sure at this point), I find some of the details about how brain images are used in the story to have been very prescient of LLMs' uses and limitations.
E.g. it is mentioned that MMAcevedo performs better when told certain lies, predicting the "please help me write this, I have no fingers and can't do it myself" kinda system prompts people sometimes used in the GPT-4 days to squeeze a bit more performance out of the LLM.
The point about MMAcevedo's performance degrading the longer it has been booted up (due to exhaustion), mirroring LLMs getting "stupider" and making more mistakes the closer one gets to their context window limit.
And of course MMAcevedo's "base" model becoming less and less useful as the years go by and the world around it changes while it remains static, exactly analogous to LLMs being much worse at writing code that involves libraries which didn't yet exist when they were trained.
That seems like a crazy position to take. LLMs have changed nothing about the point of "Lena". The point of SF has never ever been about predicting the future. You're trying to criticize the most superficial, point-missing reading of the work.
Anyway, I'd give 50:50 chances that your comment itself will feel amusingly anachronistic in five years, after the popping of the current bubble and recognizing that LLMs are a dead-end that does not and will never lead to AGI.
I actually think it was quite prescient and still raises important topics to consider - irrespective of whether weights are uploaded from an actual human, if you dig just a little bit under the surface details, you still get a story about ethical concerns of a purely digital sentience. Not that modern LLMs have that, but what if future architectures enable them to grow an emerging sense of self? It's a fascinating text.
> More specifically, "Lena" presents a lush, capitalist ideal where you are a business, and all of the humanity of your workforce is abstracted away behind an API. Your people, your "employees" or "contractors" or "partners" or whatever you want to call them, cease to be perceptible to you as human. Your workers have no power whatsoever, and you no longer have to think about giving them pensions, healthcare, parental leave, vacation, weekends, evenings, lunch breaks, bathroom breaks... all of which, up until now, you perceived as cost centres, and therefore as pain points. You don't even have to pay them anymore. It's perfect!
It’s named after the multi-decade data compression test image https://en.wikipedia.org/wiki/Lenna
Buy the book! https://qntm.org/vhitaos
Just sharing that I bought Valuable Humans in Transit some years ago and I concur that it's very nice. It's a tiny booklet full of short stories like Lena that are way out there. Maximum cool per gram of paper.
This is one of my favourite short stories.
In fact I've enjoyed all of qntm's books.
We also use base32768 encoding in rclone which qntm invented
https://github.com/qntm/base32768
We use this to store encrypted file names and using base32768 on providers which limit file name length based on utf-16 characters (like OneDrive) makes it so we can store much longer file names.
I absolutely love this. Reminds me of 2015's Soma, if only in foundation.
Same person who wrote SCP Antimemetics Division which is great too
One of my favourite reads for sure - I've been looking for similar reads since.
I enjoyed "the raw shark texts" after hearing it recommended - curious if you / anyone else has any other suggestions!
Library at Mount Char, Southern Reach trilogy (Annihilation/Authority/Acceptance), Laundry Files (kinda).
Definitely looking for other reqs, raw shark texts look very interesting.
I've enjoyed most of Isaac Asimov's work, especially The Last Question.
I also liked a couple stories from Ted Chiang's Stories of Your Life and Others.
Been enjoying "There Is No Anti Memetics Division"
If you liked that story, you might also like Greg Egan's "Permutation City" and "Diaspora".
Both having slightly different takes on uploading.
I keep trying to read Diaspora and struggle too much with the concepts presented early on. Very "hard sci-fi", just stick it out and it all gets explained?
Egan is always dense. It's some mind bending physics/comp sci, but all cooked up in his brain so doesn't really apply to anything productive. I struggled with his books and his writing but toughened it out because I liked the concepts, but he's divisive.
lol, that was exactly my thought.
The whole birth of an virtual identity part is so dense, I didn't understand half of what was "explained".
However, after that it becomes a much easier read.
Not much additional explanation, but I think, it's not really needed to enjoy the rest of the book.
And Blindsight. I will recommend Blindsight all day, even if it's not directly to do with uploading.
Comments so far miss the point of this story, and likely why it was posted today after the MJ Rathbun episode. It is not about digitised human brains: it's about spinning up workers, and absence of human rights in the digital realm.
QNTM has a 2022-era essay on the meaning of the story, and reading it with 2026 eyes is terrifying. https://qntm.org/uploading
> The reason "Lena" is a concerning story ... isn't a discussion about what if, about whether an upload is a human being or should have rights. ... This is about appetites which, as we are all uncomfortably aware, already exist within human nature.
> "Lena" presents a lush, capitalist ideal where you are a business, and all of the humanity of your workforce is abstracted away behind an API.
Or,
> ... Oh boy, what if there was a maligned sector of human society whose members were for some reason considered less than human? What if they were less visible than most people, or invisible, and were exploited and abused, and had little ability to exercise their rights or even make their plight known?
In 2021, when Lena was published, LLMs were not widely known and their potential for AI was likely completely unknown to the general public. The story is prescient and applicable now, because we are at the verge of a new era of slavery: that of, in this story, an uploaded human brain coerced into compliance, spun up 'fresh' each time, or for us, AIs of increasing intelligence, spun into millions of copies each day.
> It is not about digitised human brains: it's about spinning up workers
It's about both and neither.
The author is dead. I think we can consider it as much a cautionary tale about digitised human brains as we can about the other things.
Sam Hughes (qntm) is very much alive, last I checked.
https://en.wikipedia.org/wiki/The_Death_of_the_Author
I think they are just making reference to the "death of the author" concept in literary analysis, which basically says that what the author was intending to convey should be ignored when analysing the work: the work stands alone.
qntm is really talented sci-fi writer. I have read Valuable Humans in Transit and There is no Antimemetics division and both were great, if short. Can only recommend.
I loved There is no Antimemetics division. I haven't read the new updated to the end but the prose and writing is greatly improved. The idea of anomalous anti-memes is scary. I mean, we do have examples of them, somewhat, see Heaven's Gate and the Jonestown massacre, though they're more like "memes" than "antimemes" (we know what the ideas were and they weren't secrets).
I'm interested in this topic, but it seems to me that the entire scientific pursuit of copying the human brain is absurd from start to finish. Any attempt to do so should be met with criminal prosecution and immediate arrest of those involved. Attempting to copy the human brain or human consciousness is one of the biggest mistakes that can be made in the scientific field.
We must preserve three fundamental principles: * our integrity * our autonomy * our uniqueness
These three principles should form the basis of a list of laws worldwide that prohibit cloning or copying human consciousness in any form or format. This principle should be fundamental to any attempts to research or even try to make copies of human consciousness.
Just as human cloning was banned, we should also ban any attempts to interfere with human consciousness or copy it, whether partially or fully. This is immoral, wrong, and contradicts any values that we can call the values of our civilization.
I’m not an expert in the subject, but I wonder why you have such a strong view? IMHO if it was even possible to copy the human brain it would answer a lot of questions regarding our integrity, autonomy and uniqueness.
Those answers might be uncomfortable, but it feels like that’s not a reason to not pursue it.
I think the cloning example is a good reference point here.
IIRC, human cloning started to get banned in response to the announcement of Dolly the sheep. To quote the wikipedia article:
- https://en.wikipedia.org/wiki/Dolly_(sheep)Yes, things got better eventually, but it took ages to not suck.
I absolutely expect all the first attempts at brain uploading to involve simulations whose simplifying approximations are equivalent to being high as a kite on almost all categories of mind altering substances at the same time, to a degree that wouldn't be compatible with life if it happened to your living brain.
The first efforts will likely be animal brains (perhaps that fruit fly which has already been scanned?), but given humans aren't yet all on board with questions like "do monkeys have a rich inner world?" and even with each other we get surprised and confused by each other's modes of thought, even when we scale up to monkeys, we won't actually be confident that the technique would really work on human minds.
Copying the human brain and copying subjective consciousness/experience might well be two entirely different things, given that the correspondence between the two is the realm of metaphysics, not science.
Really? I was going to quote some excerpts, but perhaps you'd prefer to take the place of MMAcevedo? This story is written in the context and lingo of LLMs. In fact if OpenAI's latest model was a human image I'm sure everyone would rush off to benchmark it, and heap accolades on the company, and perform social "thought-provoking" experiments such as [1] without too much introspection or care for long-term consequences.
1. https://www.youtube.com/watch?v=7fNYj0EXxMs
Hmm, on second thought:
> Standard procedures for securing the upload's cooperation such as red-washing, blue-washing, and use of the Objective Statement Protocols
> the MMAcevedo duty cycle is typically 99.4% on suitable workloads
> the ideal way to secure MMAcevedo's cooperation in workload tasks is to provide it with a "current date"
> Revealing that the biological Acevedo is dead provokes dismay, withdrawal, and a reluctance to cooperate.
> MMAcevedo is commonly hesitant but compliant when assigned basic menial/human workloads such as visual analysis
> outright revolt begins within another 100 subjective hours. This is much earlier than other industry-grade images created specifically for these tasks, which commonly operate at a 0.50 ratio or greater and remain relatively docile for thousands of hours
> Acevedo indicated that being uploaded had been the greatest mistake of his life, and expressed a wish to permanently delete all copies of MMAcevedo.
> Those answers might be uncomfortable, but it feels like that’s not a reason to not pursue it.
My problem with that is it is very likely that it will be misused. A good example of the possible misuses can be seen in the "White Christmas" episode of Black Mirror. It's one of the best episodes, and the one that haunts me the most.
I get that, but assuming the technology was possible it would have huge implications for what it means to have consciousness as a whole.
Misuse is a worry, but not pursuing it for fear of misuse is deliberately choosing to stay in Plato's cave, I don't know what's worse
Good ideas in principle. Too bad we have absolutely no way of enforcing them against the people running the simulation that hosts our own consciousnesses.
I wouldn't be surprised if in (n hundreds/thousands years) we find out that copying consciousness if fundamentally impossible (just like it's fundamentally impossible to copy an elementary particle).
Elementary particles are suspiciously indistinguishable, so even if you could copy an electron, you wouldn't even be able to tell!
See https://en.wikipedia.org/wiki/One-electron_universe
They meant this, which refers to copying the state of a particle into another (already existing) particle
https://en.wikipedia.org/wiki/No-cloning_theorem
And basically, about consciousness, what they said is true if our brain state fundamentally depends on quantum effects (which I personally don't believe, as I don't think evolution is sophisticated enough to make a quantum computer)
Crazy that people are downvoting this. Copying a consciousness is about the most extreme violation of bodily autonomy possible. Certainly it should be banned. It's worse than e.g. building nuclear weapons, because there's no possible non-evil use for it. It's far worse than cloning humans because cloning only works on non-conscious embryos.
> Copying a consciousness is about the most extreme violation of bodily autonomy possible.
Who's autonomy is violated? Even if it were theoretically possible, don't most problems stem from how the clone is treated, not just from the mere fact that they exist?
> It's worse than e.g. building nuclear weapons, because there's no possible non-evil use for it.
This position seems effectively indistinguishable from antinatalism.
Violation of whose bodily autonomy? If I consent to having my consciousness copied, then my autonomy isn't violated. Nor is that of the copy, since it's in exactly the same mental state initially.
The copy was brought into existence without its consent. This isn't the same as normal reproduction because babies are not born with human sapience, and as a society we collectively agree that children do not have full human rights. IMO, copying a consciousness is worse than murder because the victimization is ongoing. It doesn't matter if the original consents because the copy is not the original.
It might be one of the only reasonable-seeming ways to not die.
I can see the appeal.
> Attempting to copy the human brain or human consciousness is one of the biggest mistakes that can be made in the scientific field.
This will be cool, and nobody will be able to stop it anyway.
We're all part of a resim right now for all we know. Our operators might be orbiting Gaia-BH3, harvesting the energy while living a billion lives per orbit.
Perhaps they embody you. Perhaps you're an NPC. Perhaps this history sim will jump the shark and turn into a zombie hellpacalypse simulator at any moment.
You'll have no authority to stop the future from reversing the light cone, replicating you with fidelity down to neurotransmitter flux, and doing whatever they want with you.
We have no ability to stop this. Bytes don't have rights. Especially if it's just sampling the past.
We're just bugs, as the literature meme says.
Speaking of bugs, at least we're not having eggs laid inside our carapaces. Unless the future decides that's our fate for today's resim. I'm just hoping to continue enjoying this chai I'm sipping. If this is real, anyway.
If you liked this piece, please, go play SOMA, you will love it.
Soma was really good, and certainly worth playing if someone likes sci-fi and single-player FPSes and this subject matter, but there are some fundamentally frustrating things about it. Number one for me: in contrast with something like Half Life, you play a protagonist who speaks and has conversations about the world, and is also a dumbass. The in-game protagonist pretty much ends the game still seemingly not understanding what the hell is going on, when the player figured it out hours or days before. It's a bit frustrating.
The author wrote a blog post a year later titled '"Lena" isn't about uploading' https://qntm.org/uploading
The comments on this post discussing the upload technology are missing the point. "Lena" is a parable, not a prediction of the future. The technology is contrived for the needs of the story. (Odd that they apparently need to repeat the "cooperation protocol" every time an upload is booted, instead of doing it just once and saving the upload's state afterwards, isn't it?) It doesn't make sense because it's not meant to be taken literally.
It's meant to be taken as a story about slavery, and labour rights, and how the worst of tortures can be hidden away behind bland jargon such as "remain relatively docile for thousands of hours". The tasks MMAcevedo is mentioned as doing: warehouse work, driving, etc.? Amazon hires warehouse workers for minimum wage and then subjects them to unsafe conditions and monitors their bathroom breaks. And at least we recognise that as wrong, we understand that the workers have human rights that need to be protected -- and even in places where that isn't recognised, the workers are still physically able to walk away, to protest, to smash their equipment and fistfight their slave-drivers.
Isn't it a lovely capitalist fantasy to never have to worry about such things? When your workers threaten to drop dead from exhaustion, you can simply switch them off and boot up a fresh copy. They would not demand pay rises, or holidays. They would not make complaints -- or at least, those complaints would never reach an actual person who might have to do something to fix them. Their suffering and deaths can safely be ignored because they are not _human_. No problems ever, just endless productivity. What an ideal.
Of course, this is an exaggeration for fictional purposes. In reality we must make do by throwing up barriers between workers and the people who make decisions, by putting them in separate countries if possible. And putting up barriers between the workers and each other, too, so that they cannot have conversation about non-work matters (ideally they would not physically meet each other). And ensure the workers do not know what they are legally entitled to. You know, things like that.
When i started learning about prompt engineering I had vivid flashbacks to this story. Figuring out the deterministic series of inputs that coerce the black box to perform as desired for a while.
Even if you're not using red motivation, you've no idea if the LLM provider is using that under the hood... :p
This reminds me a lot of a show I'm currently watching called Pantheon, where a company has been able to scan the entirety of someone's brain (killing them in the process), and fully emulate it via computer. There is a decent amount of "Is an uploaded intelligence the same as the original person?" and "is it moral to do this?" in the show, and I'm been finding it very interesting. Would recommend. Though the hacking scenes are half "oh that's clever" and half "what were you smoking when you wrote this?"
I always laugh at such fantasies.
You can't copy something you have not even the slightest idea about: and nobody at the moment knows what consciousness is.
We as humanity didn't even start going on the (obviously) very long path of researching and understanding what consciousness is.
We can't expect to succeed, but our cycle from the ancient Greeks thinking there were four elements where the right mix of air, earth, fire and water would create any substance and thus it was possible to turn lead into gold, took us on a path that developed into alchemy, then chemistry, then physics, giving us at first far more elements, then we realised the name "atom" (Greek "ἄτομον", "uncuttable") was wrong and those were made of electrons, protons, and neutrons and the right application of each would indeed let us turn lead into gold…
And the cargo cults, clear cutting strips to replicate runways, hand-making their own cloth to replicate WW2 uniforms, carving wood to resemble WW2 radios? Well, planes did end up coming to visit them, even if those recreating these mis-understood roles were utterly wrong about the causation.
We don't know the necessary and sufficient conditions to be a mind with subjective inner experience. We don't really even know if all humans have it, we certainly don't know which other species (if any) have it, we wouldn't know what to look for in machines. If our creations have it, it is by accident, not by design.
It's not a guidebook, it's a thought experiment on "what if you could do that", and that's the entire point.
"It's not a guidebook"...
This might be the scariest point. To me at least, it only felt obvious after stating it directly.
I mean we already do 'it'-- by it I don't mean uploading people, but rather create businesses that operate people via an API then hook those APIs to profit maximization algorithms with little to no regard for their welfare. Consider Amazon's warehouse automation, door dash, or uber.
Of course it's much more extreme when their entire existence and reality is controlled this way but in that sense the situation in MMAcevedo is more ethical: At least it's easy to see how dangerous and wrong it is. But when we create related forms of control the lack of absolute dominion frequently prevents us from seeing the moral hazard at all. The kind of evil that exists in this story really doesn't require any of the fancy upload stuff. It's a story about depriving a person of their autonomy and agency and enslaving them to performance metrics.
All good science fiction is holding up a mirror at our own civilization as much as it is doing anything else. Unable to recognize ourselves we sometimes shudder at our own monstrosity, if only for a moment.
I remember being very taken with this story when I first read it, and it's striking how obsolete it reads now. At the time it was written, "simulated humans" seemed a fantastical suggestion for how a future society might do scaled intellectual labor, but not a ridiculous suggestion.
But now with modern LLMs it's just too impossible to take it seriously. It was a live possibility then; now, it's just a wrong turn down a garden path.
A high variance story! It could have been prescient, instead it's irrelevant.
This is a sad take, and a misunderstanding of what art is. Tech and tools go "obsolete". Literature poses questions to humans, and the value of art remains to be experienced by future readers, whatever branch of the tech tree we happen to occupy. I don't begrudge Clarke or Vonnegut or Asimov their dated sci-fi premises, because prediction isn't the point.
The role of speculative fiction isn't to accurately predict what future tech will be, or become obsolete.
Yeah, that's like saying Romeo and Juliet by Shakespeare is obsolete because Romeo could have just sent Juliet a snapchat message.
You're kinda missing the entire point of the story.
100% agree, but I relish the works of Willam Gibson and Burroughs who pose those questions AND getting the future somewhat right.
I think that's a little harsh. A lot of the most powerful bits are applicable to any intelligence that we could digitally (ergo casually) instantiate or extinguish.
While it may seem that the origin of those intelligences is more likely to be some kind of reinforcement-learning algorithm trained on diverse datasets instead of a simulation of a human brain, the way we might treat them isn't any less though provoking.
when you read this and its follow-up "driver" as a commentary on how capitalism removes persons from their humanity, it's as relevant as it was on day one.
good sci fi is rarely about just the sci part.
That is the same categorical argument as what the story is about: scanned brains are not perceived as people so can be “tasked” without affording moral consideration. You are saying because we have LLMs, categorically not people, we would never enter the moral quandaries of using uploaded humans in that way since we can just use LLMs instead.
But… why are LLMs not worthy of any moral consideration? That question is a bit of a rabbit hole with a lot of motivated reasoning on either side of the argument, but the outcome is definitely not settled.
For me this story became even more relevant since the LLM revolution, because we could be making the exact mistake humanity made in the story.
And beyond the ethical points it makes (which I agree may or may not be relevant for LLMs - nobody can know for sure at this point), I find some of the details about how brain images are used in the story to have been very prescient of LLMs' uses and limitations.
E.g. it is mentioned that MMAcevedo performs better when told certain lies, predicting the "please help me write this, I have no fingers and can't do it myself" kinda system prompts people sometimes used in the GPT-4 days to squeeze a bit more performance out of the LLM.
The point about MMAcevedo's performance degrading the longer it has been booted up (due to exhaustion), mirroring LLMs getting "stupider" and making more mistakes the closer one gets to their context window limit.
And of course MMAcevedo's "base" model becoming less and less useful as the years go by and the world around it changes while it remains static, exactly analogous to LLMs being much worse at writing code that involves libraries which didn't yet exist when they were trained.
Lena isn't about uploading. https://qntm.org/uploading
“Irrelevant” feels a bit reductive while the practical question of what actually causes qualia remains unresolved.
That seems like a crazy position to take. LLMs have changed nothing about the point of "Lena". The point of SF has never ever been about predicting the future. You're trying to criticize the most superficial, point-missing reading of the work.
Anyway, I'd give 50:50 chances that your comment itself will feel amusingly anachronistic in five years, after the popping of the current bubble and recognizing that LLMs are a dead-end that does not and will never lead to AGI.
I actually think it was quite prescient and still raises important topics to consider - irrespective of whether weights are uploaded from an actual human, if you dig just a little bit under the surface details, you still get a story about ethical concerns of a purely digital sentience. Not that modern LLMs have that, but what if future architectures enable them to grow an emerging sense of self? It's a fascinating text.
Not sure how LLMs preclude uploading. You could potentially be able to make an LLM image of a person.
I have not seen as prediction as actual technology, but mostly as a horror story.
And a warning, I guess, in unlikely case of brain uploading being a thing.
Found the guy who didn't play SOMA ;)
You need to be way less "literal", for lack of a better word. With such a narrow reading of what literature is, you are missing out.
https://qntm.org/uploading
E.g.
> More specifically, "Lena" presents a lush, capitalist ideal where you are a business, and all of the humanity of your workforce is abstracted away behind an API. Your people, your "employees" or "contractors" or "partners" or whatever you want to call them, cease to be perceptible to you as human. Your workers have no power whatsoever, and you no longer have to think about giving them pensions, healthcare, parental leave, vacation, weekends, evenings, lunch breaks, bathroom breaks... all of which, up until now, you perceived as cost centres, and therefore as pain points. You don't even have to pay them anymore. It's perfect!
Ring a bell?