Very interesting article. As a developer that has focused on writing more code than the infrastructure and tooling, I can definitely see the benefit to a product existing that allows everything to work together more seamlessly because on the surface everything we need to do it already exists. Something like glue code we could drop in or like you mentioned the whole integrated environment. I didn't realize how specific and difficult it was to do something like this, maybe it's not that we should have deployable option, but that technology could get to a point to where maybe implementing and deploying this could be worth it as you could get more value than effort it takes to implement on a per company basis so companies besides Google and other big tech companies can operate like this.
On the one hand, the scenarios described by "vertical integration" sound like a paradise to use.
On the other hand, in real life it's going to far too complex, far too brittle, and ultimately unworkable to maintain.
Tool boundaries exist for a good reason -- so you can swap tools in and out easily, and the toolmakers can stay sane by focusing on doing one thing well. With vertical integration, nothing is swappable, because you can only use tools built for your specific architecture. You become isolated from the broader ecosystem, and locked into one.
And then, that architecture is going to be this bespoke thing that becomes so enormously complex and hard to work with that everybody just gives up. Because every time somebody wants to change a tool, it turns out to be 10x more work to figure out how to make the change work with this vertically-integrated architecture.
Vertical integration isn't the "only thing that matters". Flexibility and swappability matter too, as does saving time and money by following common industry technologies and not inventing a bunch of custom architecture if it's not your core product and not absolutely necessary.
This article made me laugh and cry at the same time. I've spent so much time in my career trying to make things nice and seamless only to see my own team members throw out my careful work for something newer, shinier, and shittier. I was on a team that used Bazel. Like two devs took the time to figure out how it worked, everybody else just either worked around it or complained endlessly about it.
I am now thoroughly convinced that software engineers, if there is currently no snake whispering in their ear to throw away the paradise garden they're been handed on high, will find a way to do it anyway. Coders will, to the last, prefer self-inflicted misery over the heaven they've been gifted for free.
And if you don't believe me, let me tell you we already have a fully-vertically integrated tool stack, a whole family of them. It's called Smalltalk, it's been around since the 70s, and modern variants of course exist. You can build stuff in it today, and thoroughly enjoy your computing life as a result.
The second you turn your head though, your fellow teammates will conspire to replatform onto Go or Rust or NodeJS or GitHub Actions and make everything miserable again.
Don't buy the nonsense that vertical integration is hard. It's not. You just hire really sharp folks, get them excited about the idea, and they do all the hard integration work, then you release it to the community and let them build on it. Rails was like this back in the 2010s, there was this golden age where everything just worked. Then all of a sudden javascript took over the web world like cancer and the web stopped being fun.
It's not that it's hard, what it is is brittle. A vertically-integrated stack, by its very nature, cannot survive forces that jostle it in the horizontal direction. And coders are too afraid of falling behind that they end up fetishizing any new idea that comes along, no matter how daft. Javascript on the server?!?! Your solution to js's, let's call them problems, is gradual typing? That snake's never gonna run out of ears to whisper in, lemme tell ya.
Integrated toolsets can, have, and are still being done. You can use them now. But you don't want it. Even if you do, nobody you work with will trust it or keep it after you leave. And so companies have no motive anymore to sell them to you. Microsoft themselves stopped trying after 2019.
> The second you turn your head though, your fellow teammates will conspire to replatform onto Go or Rust or NodeJS or GitHub Actions and make everything miserable again.
Curious how would you use use Smalltalk in replace of GitHub Actions assuming you need a GitHub integrated CI runner?
All any build toolkit is is automations over bash. You can make your own. GitHub integration need not be any more than the most trivial thing that works. Your coworkers, naturally, won't be disciplined enough to keep the integration trivial and will build super complicated crap that's realty hard to troubleshoot, because they can.
The Rails experience of 15 years ago is still achievable, with Rails, today. It's even better if you can believe it. You just have to throw out the notion of SPA frameworks and be willing to actually learn how HTML / CSS works. I read an article here a month ago about how nobody's willing to do that.
The deficiency in web standards that SPA frameworks were invented to resolve have all been fixed, there's nothing they offer anymore that you can't do without them. But they hang over the neck of the web world like an albatross.
Meh, I think its just hubris to ignore your users (in this case the rest of the team) and tell yourself they're just dumb for not using your clearly perfect system.
There is a slight amount of NIH you have to fight, but for the most part, if a process is truly seamless, that NIH will get pushed to a different part of the stack that's more front of mind.
The real issue is that you're trivializing needs that live outside your purview and because of that you can't fathom why a js dev who wants a server to do something would want js on the server.
Ignore them? I did anything but. I know the new solution was shittier because I was the one who implemented it. Every single time. It never made my life better, only worse.
> It's not that it's hard, what it is is brittle. A vertically-integrated stack, by its very nature, cannot survive forces that jostle it in the horizontal direction.
Is this not exactly the whole problem, though? Fault tolerance.
My chisel doesn't need to integrate with my hand plane which doesn't need to integrate with my band saw. One of them breaking doesn't break the others.
This is why experienced developers gravitate toward powerful tools that they trust deeply but that have extremely hard boundaries. It's why we're still stuck with ASCII text as the primary artifact for coding, for example. The moment we try to move off of that a single fault can bring down the whole house of cards.
It is but 90% of this horizontal pressure is just nonsense. Not actual faults that need to be tolerated. Cooler heads stick around and solve the problems, but by the time that happens, the larger dev community has already moved on to something thoroughly crappier.
ASCII is perhaps a poor example, all modern tooling is UTF-8 capable.
This kind of rigid, singular view of operational workflows based on precomposed automations not only constantly break but also inevitably introduce huge inefficiences.
yes! couldn't agree more with your long post.
Especially this part:
"(This is exacerbated when components of the automation require internal-only tooling—the poor data scientist now needs to go read through a bunch of half-written, out-of-date documentation about tools they simply don't care about to do a task that is not a core responsibility for them.)"
"Vertical integration" in my experience has just been turning a group of simple tools into a complex monolith that no one understands and is extremely difficult to debug
It's also very easy to complain about how employees have résumé-focus in their approach to their work: “why should I bother to learn some internal-only tooling that I'll never use anywhere else (for a task that I don't really even care that much about…)?”
But, to borrow a line from Warren VanderBurgh's ‘Children of the Magenta’: “(in the industry) we created you like this.”
Another key flaw of precomposed automations for rigidly-defined work-flows is that they usually exist in precisely the circumstances that give rise to their own subversion. (I might even go so far as to suggest that the circumstances are the cause of both the mistake and the maladaptive behaviours that address the mistake…)
Ultimately, deep stacks of tightly-integrated components forming a precomposed automation that enacts some work-flow—“vertical integration” as the post frames it—is obvious enough that it seems every big company tries it… only to fail in basically the same ways every time.
Vertical integration matters when you are your own major customer on a process that isn’t already done at scale somewhere else with no reason to change.
Very interesting article. As a developer that has focused on writing more code than the infrastructure and tooling, I can definitely see the benefit to a product existing that allows everything to work together more seamlessly because on the surface everything we need to do it already exists. Something like glue code we could drop in or like you mentioned the whole integrated environment. I didn't realize how specific and difficult it was to do something like this, maybe it's not that we should have deployable option, but that technology could get to a point to where maybe implementing and deploying this could be worth it as you could get more value than effort it takes to implement on a per company basis so companies besides Google and other big tech companies can operate like this.
Unfortunately, I have to disagree.
On the one hand, the scenarios described by "vertical integration" sound like a paradise to use.
On the other hand, in real life it's going to far too complex, far too brittle, and ultimately unworkable to maintain.
Tool boundaries exist for a good reason -- so you can swap tools in and out easily, and the toolmakers can stay sane by focusing on doing one thing well. With vertical integration, nothing is swappable, because you can only use tools built for your specific architecture. You become isolated from the broader ecosystem, and locked into one.
And then, that architecture is going to be this bespoke thing that becomes so enormously complex and hard to work with that everybody just gives up. Because every time somebody wants to change a tool, it turns out to be 10x more work to figure out how to make the change work with this vertically-integrated architecture.
Vertical integration isn't the "only thing that matters". Flexibility and swappability matter too, as does saving time and money by following common industry technologies and not inventing a bunch of custom architecture if it's not your core product and not absolutely necessary.
Did HN just reinvent Smalltalk from first principles?
Made me lol. Nice name.
This article made me laugh and cry at the same time. I've spent so much time in my career trying to make things nice and seamless only to see my own team members throw out my careful work for something newer, shinier, and shittier. I was on a team that used Bazel. Like two devs took the time to figure out how it worked, everybody else just either worked around it or complained endlessly about it.
I am now thoroughly convinced that software engineers, if there is currently no snake whispering in their ear to throw away the paradise garden they're been handed on high, will find a way to do it anyway. Coders will, to the last, prefer self-inflicted misery over the heaven they've been gifted for free.
And if you don't believe me, let me tell you we already have a fully-vertically integrated tool stack, a whole family of them. It's called Smalltalk, it's been around since the 70s, and modern variants of course exist. You can build stuff in it today, and thoroughly enjoy your computing life as a result.
The second you turn your head though, your fellow teammates will conspire to replatform onto Go or Rust or NodeJS or GitHub Actions and make everything miserable again.
Don't buy the nonsense that vertical integration is hard. It's not. You just hire really sharp folks, get them excited about the idea, and they do all the hard integration work, then you release it to the community and let them build on it. Rails was like this back in the 2010s, there was this golden age where everything just worked. Then all of a sudden javascript took over the web world like cancer and the web stopped being fun.
It's not that it's hard, what it is is brittle. A vertically-integrated stack, by its very nature, cannot survive forces that jostle it in the horizontal direction. And coders are too afraid of falling behind that they end up fetishizing any new idea that comes along, no matter how daft. Javascript on the server?!?! Your solution to js's, let's call them problems, is gradual typing? That snake's never gonna run out of ears to whisper in, lemme tell ya.
Integrated toolsets can, have, and are still being done. You can use them now. But you don't want it. Even if you do, nobody you work with will trust it or keep it after you leave. And so companies have no motive anymore to sell them to you. Microsoft themselves stopped trying after 2019.
> The second you turn your head though, your fellow teammates will conspire to replatform onto Go or Rust or NodeJS or GitHub Actions and make everything miserable again.
Curious how would you use use Smalltalk in replace of GitHub Actions assuming you need a GitHub integrated CI runner?
All any build toolkit is is automations over bash. You can make your own. GitHub integration need not be any more than the most trivial thing that works. Your coworkers, naturally, won't be disciplined enough to keep the integration trivial and will build super complicated crap that's realty hard to troubleshoot, because they can.
Luckily I work with laravel and that's a spiritual successor to rails.
The development experience is almost always really smooth and there are more and more tools to further smoothen that experience every day.
There are definitely better tools out there but given how the web ecosystem functions, it could be much worse.
The Rails experience of 15 years ago is still achievable, with Rails, today. It's even better if you can believe it. You just have to throw out the notion of SPA frameworks and be willing to actually learn how HTML / CSS works. I read an article here a month ago about how nobody's willing to do that.
The deficiency in web standards that SPA frameworks were invented to resolve have all been fixed, there's nothing they offer anymore that you can't do without them. But they hang over the neck of the web world like an albatross.
What's a good doc or code template for SPA-free Rails?
What do you think of htmx + Rails?
Just use `rails new`. It's all built in.
Meh, I think its just hubris to ignore your users (in this case the rest of the team) and tell yourself they're just dumb for not using your clearly perfect system.
There is a slight amount of NIH you have to fight, but for the most part, if a process is truly seamless, that NIH will get pushed to a different part of the stack that's more front of mind.
The real issue is that you're trivializing needs that live outside your purview and because of that you can't fathom why a js dev who wants a server to do something would want js on the server.
Ignore them? I did anything but. I know the new solution was shittier because I was the one who implemented it. Every single time. It never made my life better, only worse.
> It's not that it's hard, what it is is brittle. A vertically-integrated stack, by its very nature, cannot survive forces that jostle it in the horizontal direction.
Is this not exactly the whole problem, though? Fault tolerance.
My chisel doesn't need to integrate with my hand plane which doesn't need to integrate with my band saw. One of them breaking doesn't break the others.
This is why experienced developers gravitate toward powerful tools that they trust deeply but that have extremely hard boundaries. It's why we're still stuck with ASCII text as the primary artifact for coding, for example. The moment we try to move off of that a single fault can bring down the whole house of cards.
It is but 90% of this horizontal pressure is just nonsense. Not actual faults that need to be tolerated. Cooler heads stick around and solve the problems, but by the time that happens, the larger dev community has already moved on to something thoroughly crappier.
ASCII is perhaps a poor example, all modern tooling is UTF-8 capable.
You can at least get (3) for low/no cost with pre-commit hooks running locally and in CI.
The problem with hooks is they’re not enforced (in git) so you need to run them in CI.
Yes, that's what I was saying. But you can do that, and really easily, and it buys you a lot for very little.
I just could not disagree more.
This kind of rigid, singular view of operational workflows based on precomposed automations not only constantly break but also inevitably introduce huge inefficiences.
I posted a longer comment on lobste.rs: https://lobste.rs/s/azpsqe/vertical_integration_is_only_thin...
yes! couldn't agree more with your long post. Especially this part: "(This is exacerbated when components of the automation require internal-only tooling—the poor data scientist now needs to go read through a bunch of half-written, out-of-date documentation about tools they simply don't care about to do a task that is not a core responsibility for them.)"
"Vertical integration" in my experience has just been turning a group of simple tools into a complex monolith that no one understands and is extremely difficult to debug
It's also very easy to complain about how employees have résumé-focus in their approach to their work: “why should I bother to learn some internal-only tooling that I'll never use anywhere else (for a task that I don't really even care that much about…)?”
But, to borrow a line from Warren VanderBurgh's ‘Children of the Magenta’: “(in the industry) we created you like this.”
Another key flaw of precomposed automations for rigidly-defined work-flows is that they usually exist in precisely the circumstances that give rise to their own subversion. (I might even go so far as to suggest that the circumstances are the cause of both the mistake and the maladaptive behaviours that address the mistake…)
Ultimately, deep stacks of tightly-integrated components forming a precomposed automation that enacts some work-flow—“vertical integration” as the post frames it—is obvious enough that it seems every big company tries it… only to fail in basically the same ways every time.
Love seeing Children of the Magenta come up!
Vertical integration matters when you are your own major customer on a process that isn’t already done at scale somewhere else with no reason to change.
I like the points made in the article. I hate this overloading of the term.