It’s worth understanding that Shin Jinse has been significantly stronger than his nearest human opponents for a while now, more so than Magnus was even at his very peak.
In go ELO like scoring he’s something like 120 points over the next strongest player. No other player has ever broken a 3800 rating let alone 3850. Ke Jie (the previous long time champion) peaked at 3755. Shin Jinseo’s strength graph is the most absurd straight line.
2 stones is historically the gap between a 9P ranked and a 1P ranked professional player (very roughly the gap between super grandmasters and an almost grandmaster)
That is to say it’s shocking that Katago (almost certainly significantly stronger than AlphaGo) is a mere 2 stones stronger than Shin Jinseo. I suspect it would be 3-4 stones vs any other human pro.
I read this comment before looking at the article and thought that the grandmaster beat the AI even giving the AI 2 stones. Too bad. But this way around is of course more realistic.
And of course you would need to take into consideration the scale of go ratings and chess ratings when making that comparison. With top chess ratings being around 2800, being 1000 less than the top go ratings, one would have to apply a factor of roughly 3/4.
If you scaled Shin Jinseo to 2800, you would have players with extremely negative ratings. This page shows ratings of European players on a roughly aligned scale: https://europeangodatabase.eu/EGD/createalleuro3.php?country.... It still has negative numbers on it, and this only contains players who have attended a tournament (though it's more common for beginners to play tournaments in the west, since it's hard to find times to play).
It's not a comparison of the worth of the games (I play both, though I'm better at Go, and prefer it), but the dynamic range of Go is larger.
That said, any cross-game/sport comparisons of this kind are pretty tough to do properly.
Depends on what chess rating you're talking about. Online at fast time controls, where many many more games get played at than OTB classical, the top super GMs are rated in the 3400-3600 range.
You're right that Shin Jinseo is a generational talent, and more dominant than anyone since Lee Changho (peaked in the 90s and was strong into the early-mid 2000s).
However, you can't compare goratings over time, the top ranks are not nearly stable enough. https://www.goratings.org/en/history/ (I think it's believable Shin Jinseo is better than Lee Changho, but not that there has been steady progress since the days of Lee Changho, so that there are now 20 players stronger than him).
The problem is ambient go knowledge. A top 100 player would easily beat time traveling Lee Changho in his first few matchups. Of course give peak Lee Changho a fortnight to prep with Katago and … well that would be something!
> Katago (almost certainly significantly stronger than AlphaGo)
Interesting KataGo is an open sourced Go program written primarily by David Wu in C++ and recently heavily vibe coded by Claude. It's running on four Nvidia RTX-3090 GPUs with 96GB VRAM. [1]
Important to note that KataGo was double-handicapped. 20 seconds per move maximum; it couldn’t read deep. Against an amateur, it doesn’t matter, but against a historically strong pro it matters a lot.
...on a 4x 3090 rig. The game ran 299 moves, giving katago 100 minutes if it exhausted time on each move (which must be the optimal strategy under that time control). Shin used about 205 minutes, over twice as much time and of course had leeway to spend more time on difficult moves.
Is 20 seconds on that hardware really overkill and well into the diminishing-returns curve, as a top-level comment suggested, or is it plausible katago could have played better if given 40 seconds per move?
For another comparison, top world class chess players will have solid odds to beat Leela Chess Zero when given a knight odds handicap (Leela Chess Zero starts with 1 fewer knight). For human vs human, I think this would be somewhere in the ballpark of the ~10,000th best chess player having fair odds against Magnus.
I wonder if this means the best Go play is closer to theoretically perfect play or if it just happened the current computer methods didn't manage to get much farther than humans. Go has vastly more valid games but also a simpler ruleset, so I'm not sure if there is really a good way to tell beyond "keep trying and find out"?
I don't think it's solely a matter of raw strength, but as Shin said, a willingness not to play to the program's strengths. I mean, one thing that rankled me about original Lee Sedol match was that Lee had no access to the program's "record" while the machine by the nature of the AI training process had effectively studied Lee's games in great detail.
I recall a while back someone came up with a set of "anti-computer" strategies that allowed even an amateur to defeat a strong go program. These moves weren't anything like ordinary go moves (and perhaps the "loophole" has been closed now) but imo, their existence suggests that a study of programs may reveal other unexpected weakness.
We saw this with AlphaStar too, but ultimately it feels like simply an exploit. I expect even a relatively simple modern LLM/model working with the custom transformer would have been able to address these exploits after a game.
On one hand, Shin Jinseo is an outlier player of this generation. On the other hand, the newest generation of new pros will have exclusively learned by using the AI to tell them what the best move is, so there's reason to believe that peak human level has yet to be reached.
Could someone sufficiently motivated invest in training Katago to be able to beat Shin Jinseo with 3 stones of handicap? Unfortunately - probably yes.
This in no way detracts from how absurd and remarkable it is that Shin Jinseo can beat KataGo (it gets a LOT of training and architecture refinements https://katagotraining.org/#eloGraphButtons) with 2 stones of handicap.
Probably a better comparison from the chess world(in reasonably modern times, though perhaps players like Capablanca and Lasker could be mentioned as well. Alas, I don't think FIDE rating existed back then) is Bobby Fischer. In the july 1972 FIDE rating list he held a rating of 2785, the highest in history at the time, with Spassky in second sitting at a "measly" 2660, and only 13 players being above 2600 even.
In Go, there are exchanges of plays called "joseki". Professionals consider the outcome of joseki to be an equal result for both players. Most joseki are only a handful of moves, but some, such as the "flying knife" joseki have variations that continue for upwards of 50 moves. A traditional 19x19 go board has 361 intersections.
Shin's genius was to play out a complex variation of the flying knife joseki that was, in essence, a one-way path to reach an equal board position that occupied about 1/4 of the board. Due to the 2-stone handicap, the position favoured black with the game ~25% complete. KataGo could not have played any other way, where a human may have tried to foil the plan by introducing further complications.
What was truly incredible was how Shin held the advantage from that point on.
I have some questions as a chess player who barely even understands the rules of go. First these josekis sound like what in chess is called a forced tactical sequence. When you say Shin played out a long complex joseki, how does he do that? Does he have to read/calculate it out over the board(50 moves seems crazy to me unless the search tree is highly constrained by geometry/deduction/very few candidate moves, which does occasionally happen in chess endgames), or is the joseki more of a fixed sequence of moves which he's memorised, only needing to read to "punish" if the opponent diverges?
Second, if it is a fixed sequence, how position independent is it? In chess, tactical sequences end up depending on the entire board state to work when they get sufficiently long. I guess what I'm asking is, could a player with some capacity for stategic thinking recognise this idea and take steps to make the flying knife impossible?
I really should spend some more time learning go, it's such a fascinating game.
You can think of joseki as “local opening”. Like, in a vacuum, this is known by study / AI to be an even result for black and white. It’s just like a chess opening, there’s no calculation up to a certain point. And it doesn’t exist in the midgame, it’s not similar to forced sequences which exist in both games; it’s much more like choosing French closed vs open or gambit/gambit declined. The one thing is (and this is huge), since Go board is very big, existing stone formations on other parts of the board influence the value of joseki and make certain ones more advantageous for black or white. To my knowledge this doesn’t really exist in chess, because the opening is already the entire board.
However, when Shin executed the 50 move flying knife, the board was pretty much empty. So there is really no need for calculation, both Shin and the AI know it’s locally optimal. But getting to play a very long locally optimal sequence is good for the weaker player, so they have less “real” moves to lose EV on. Notably Shin probably can’t open with the flying knife in one corner past a certain point in the game, even if that corner were completely empty - the rest of the board positions would change the end values of the variants.
If the AI could know this, they might play a variant that ends 30 moves sooner but is 0.01 pts worse. Then they would have more time to mess Shin up through organic new moves (which the AI will be better at of course).
The headline is a bit misleading, though perhaps not intentionally.
Shin took a 2-stone handicap from KataGo which means that Shin is the weaker of the two. But to give that more context, Shin is also the strongest human player to have ever lived in raw strength terms by a good margin, and is known as replicating AI move-for-move more closely than anyone else.
If they were to play even then there’s no chance any human could win (and pretty much all pros agree with that). Lee Sedol beating AlphaGo in game 4 of that series is largely considered the last time a human beat a modern AI in an even game, which is why it was so amazing.
RE the game, Katago was set to use the strongest available model and ran on a 3x 3090 GPU system, which is a lot for KataGo. 20 seconds might sound like a handicap, but that’s over 100,000 play out variations which is essentially infinite for modern KataGo models (anything over 10,000 is overkill).
Shin played well in all games, but his strategy was to avoid complexity. KataGo reads out complex fighting like an absolute monster, so Shin was trying to play very very solid and very very calm so as to not give KataGo an in.
The 2-stone handicap could be thought of as roughly 10-15 points of ‘buffer’. That’s massive in professional games, and that’s what Shin used to win. He played so overly solid that it sometimes cost a point or two, but it removed an opening for Katago to fight. He did this at the key opening and middle-game sections and never burnt through the full buffer of handicap points in the last two games. That’s why those games look kinda ‘boring’, it’s because Shin wanted them to be that way.
Also note that KataGo probably could have won if its ’variance’ was tuned higher (basically it taking risks). Standard KataGo won’t take risks, it just wins with brute force. For handicap games though you can tune its willingness to start fights higher to prevent people from just playing ultra solid (like Shin did).
Shin did an absolutely amazing job and he deserves all the recognition. Katago routinely beats professionals giving them 3-4 stones of handicap, so the win by Shin highlights to me how strong he is, but also just how well he understands how the AI ‘thinks’.
naive question--does " He did this at the key opening and middle-game sections and never burnt through the full buffer of handicap points in the last two games. " imply in retrospect he could have won with a smaller handicap? or was having the rest of that buffer in reserve guiding strategy?
Apparently 2 stones is a huge advantage. An estimate is that the computer is roughly 4-600 ELO stronger on an even match.
Also, the human played a strategy tailored to that huge initial advantage. He said that the AI did not handle this particularly well, and played high probability moves instead of trying to lure him into a mistake.
Also, even though this was the best Go engine, it was not running on a supercomputer, and had a relatively limited amount of time per move.
So, this was an important victory for a human, but not a sign that humans are now stronger than AIs at Go.
> Also, the human played a strategy tailored to that huge initial advantage. He said that the AI did not handle this particularly well, and played high probability moves instead of trying to lure him into a mistake.
Yeah, katago's training is not really focused at all on handicap games, because it's by nature learning from even games against similar-strength opponents.
It doesn't have specific training from playing in a way to exploit a weaker player. In a handicap game you have to give your opponent opportunities to fuck up if you want to play optimally.
If a move loses 0.0005 points if the opponent plays optimally, katago won't play it even if there's ~zero chance a weaker player would play it right.
There have been go AIs that tried to train more directly on uneven opponents, one called "sai" comes to mind, but katago has huge advantages otherwise and won out over the others (for very good reason, it's a great project).
While AlphaGo originally only had win rate as a metric, modern Go AIs have more knobs, including an evaluation of "complexity".
Just stating this off the top of my head so I could be misremembering, but I heard that the KataGo settings used were tweaked to favor complexity. This was most apparent in Game 1 which Shin Jinseo lost, where the AI had an unusual opening. However, the last game was quite plain leading me to wonder whether that setting was present in the last game (or at all).
There are tweaks to move choice like that, but it's all evaluation time. None of that happens in training. Zero times in the RL loop does katago see a two stone game against a weaker player.
You can kind of tweak towards play this metric or that, but it's not the same.
Ah this is interesting. Essentially the idea is that the compute can try and move into positions that it can evaluate but humans might have trouble evaluating because of the board state's complexity?
> So, this was an important victory for a human, but not a sign that humans are now stronger than AIs at Go.
Another way to look at this: Go's handicap system gives us a genuinely interesting metric for the distance between a human and a machine at this specific game. Instead of just "computers beat humans" we get a quantified gap.
The question of whether machines or humans are stronger is moot, isn't it?
In any intellectual contest between human and machine, all the machine winning implies is that the endeavor is algorithmic.
The machine can be given practically unlimited memory and compute; we consider it cheating if the human would use memory aids. The machine could be implemented as many agents cooperating; we'd think it's not right if thousands of humans collaborated to face the machine, etc.
So statements like "not a sign that humans are now stronger than AIs at Go" are pretty meaningless, IMO
The article is hilariously triumphalist given the AI had a handicap (which it only mentions in passing unlike the title here).
Obviously its an impressive human intellectual feat but the hubris is instructive perhaps for our wider interactions with AI as its abilities accelerate around us.
> "This series taught me that rather than trying to imitate AI, it is far more important to build the board according to my own style."
It's never made sense to me that so many go players study AI go play in the hope of emulating it in a human game. We're not machines. We can't do thousands of Monte Carlo tree searches per second.
They study it because it has done things that humans had long assumed were bad until AI proved otherwise. The conceptual knowledge has been valuable at the top level.
"The 26-year-old South Korean grandmaster became the first human to win an official series against a state-of-the-art Go engine under a two-stone handicap, a margin considered the absolute boundary for human competition against modern AI."
What a powerful story. Humans have a chance of remaining superior because emotions are the fuel for our intellect and wisdom.
I don't know what the fascination with these AI versus human tournaments is. I'm old enough to remember the whole "Deep Blue" vs Kasparov exhibition, and I didn't really understand the fascination with all that either. That humans can make sufficiently strong calculators has never been a dispute in my mind. If the human wins over the calculator, good, but if the calculator wins, okay. What does that tell us exactly? The human had a bad day? The machine had a good one?
> That humans can make sufficiently strong calculators has never been a dispute in my mind.
Within our lifetimes (unless you're _very_ young) it was doubtful if a go ai would ever beat a decent human. Same was true for chess a generation or two earlier. It's news because it's the handoff of man to machine being the best at a particular thing.
This current news is news from the other way, this human did _exceptionally_ well.
What I found interesting was that he adapted and shifted to a very unconventional strategy of play, opposed to the AI who primarily seems to play high probability moves. Does this not demonstrate the human edge against AI in novel / unconventional thinking?
I recommend "The Master of Go" by Kawabata, 1951. It's a beautiful story of the change of power at the apex of the Go ranking system. I read it the same year as I read "The Glass Bead Game" by Hesse published in 1943 and its hard not to believe a relationship but in truth I think they are completely independent stories of the same situational tension.
KataGo isn't very good at exploiting weaker opponents.
In chess people were convinced a grandmaster can never be beaten with a knight odds. It's just too easy to simplify the position and win. It was very easy (for a grandmaster) vs already super human Stockfish. It was still kinda easy (for a strong GM) vs 200+ ELO stronger NNUE Stockfish.
And then someone made a net optimized for exploiting humans. Its games are amazing and it beats GMs with knight odds with ease. It's unreal how good it is at setting traps, playing lines that don't work in theory but the refutation is based on precise tactical sequence a few moves deep. Playing vs it feels like playing vs a spider that slowly weaves a net around you till you can't move anymore.
I predict the same thing is going to happen in Go once the engines catch up.
(You can play those chess bots on Lichess for free. Challenge LeelaQueenOdds or LeelaRookOdds if you are master level or stronger)
For all four of you that are like me and understand Dota 2 a lot better than Go, and are wondering what impact a “two-stone handicap” has and what it means, ChatGPT Pro claims that to analogize this scenario to a professional team playing against OpenAI Five:
> The professional human team begins from a legal eight-to-ten-minute game state in which it has decisively won the laning stage: roughly a 6,000–8,000 team-net-worth lead, a 4,000–6,000 team-XP lead, two enemy Tier 1 towers destroyed, the third badly damaged, and all three friendly Tier 1 towers standing.
It’s worth understanding that Shin Jinse has been significantly stronger than his nearest human opponents for a while now, more so than Magnus was even at his very peak.
In go ELO like scoring he’s something like 120 points over the next strongest player. No other player has ever broken a 3800 rating let alone 3850. Ke Jie (the previous long time champion) peaked at 3755. Shin Jinseo’s strength graph is the most absurd straight line.
https://www.goratings.org/en/
2 stones is historically the gap between a 9P ranked and a 1P ranked professional player (very roughly the gap between super grandmasters and an almost grandmaster)
That is to say it’s shocking that Katago (almost certainly significantly stronger than AlphaGo) is a mere 2 stones stronger than Shin Jinseo. I suspect it would be 3-4 stones vs any other human pro.
I read this comment before looking at the article and thought that the grandmaster beat the AI even giving the AI 2 stones. Too bad. But this way around is of course more realistic.
And of course you would need to take into consideration the scale of go ratings and chess ratings when making that comparison. With top chess ratings being around 2800, being 1000 less than the top go ratings, one would have to apply a factor of roughly 3/4.
>Go grandmaster Shin defeats AI KataGo with a two-stone handicap
English is not my first language but for clarity perhaps the title should be:
Go grandmaster Shin with a two-stone handicap defeats AI KataGo
If you scaled Shin Jinseo to 2800, you would have players with extremely negative ratings. This page shows ratings of European players on a roughly aligned scale: https://europeangodatabase.eu/EGD/createalleuro3.php?country.... It still has negative numbers on it, and this only contains players who have attended a tournament (though it's more common for beginners to play tournaments in the west, since it's hard to find times to play).
It's not a comparison of the worth of the games (I play both, though I'm better at Go, and prefer it), but the dynamic range of Go is larger.
That said, any cross-game/sport comparisons of this kind are pretty tough to do properly.
Depends on what chess rating you're talking about. Online at fast time controls, where many many more games get played at than OTB classical, the top super GMs are rated in the 3400-3600 range.
You're right that Shin Jinseo is a generational talent, and more dominant than anyone since Lee Changho (peaked in the 90s and was strong into the early-mid 2000s).
However, you can't compare goratings over time, the top ranks are not nearly stable enough. https://www.goratings.org/en/history/ (I think it's believable Shin Jinseo is better than Lee Changho, but not that there has been steady progress since the days of Lee Changho, so that there are now 20 players stronger than him).
The problem is ambient go knowledge. A top 100 player would easily beat time traveling Lee Changho in his first few matchups. Of course give peak Lee Changho a fortnight to prep with Katago and … well that would be something!
Could you just have superhuman Go AI just how good humans are somewhat more objectively?
Not without flaws of course, but probably interesting
> Katago (almost certainly significantly stronger than AlphaGo)
Interesting KataGo is an open sourced Go program written primarily by David Wu in C++ and recently heavily vibe coded by Claude. It's running on four Nvidia RTX-3090 GPUs with 96GB VRAM. [1]
[1] https://github.com/lightvector/KataGo
I had to read your comment five times thinking "how it is written in C++ if it's a go program?" Do'h!
It significantly predates Claude, and has been one of, if not the best engine in the world for many years.
Important to note that KataGo was double-handicapped. 20 seconds per move maximum; it couldn’t read deep. Against an amateur, it doesn’t matter, but against a historically strong pro it matters a lot.
...on a 4x 3090 rig. The game ran 299 moves, giving katago 100 minutes if it exhausted time on each move (which must be the optimal strategy under that time control). Shin used about 205 minutes, over twice as much time and of course had leeway to spend more time on difficult moves.
Based on the youtube video, it looks like katago was only using 16 seconds per move, is that right? https://www.youtube.com/watch?v=-86zF4mTWOY
Is 20 seconds on that hardware really overkill and well into the diminishing-returns curve, as a top-level comment suggested, or is it plausible katago could have played better if given 40 seconds per move?
match details: https://gostonebase.com/blog/shin-jinseo-vs-katago-kishin-ma...
Deep link to Shin Jinseo's strength graph
https://www.goratings.org/en/players/1313.html
It would be interesting to find out what insight he discovered about the game to consistently rise like that.
It can't be just play like AI.
Any other Korean on the Korean Go program could have done the same.
In fact, many did when AlphaGo was the pinnacle of AI.
For another comparison, top world class chess players will have solid odds to beat Leela Chess Zero when given a knight odds handicap (Leela Chess Zero starts with 1 fewer knight). For human vs human, I think this would be somewhere in the ballpark of the ~10,000th best chess player having fair odds against Magnus.
I wonder if this means the best Go play is closer to theoretically perfect play or if it just happened the current computer methods didn't manage to get much farther than humans. Go has vastly more valid games but also a simpler ruleset, so I'm not sure if there is really a good way to tell beyond "keep trying and find out"?
I don't think it's solely a matter of raw strength, but as Shin said, a willingness not to play to the program's strengths. I mean, one thing that rankled me about original Lee Sedol match was that Lee had no access to the program's "record" while the machine by the nature of the AI training process had effectively studied Lee's games in great detail.
I recall a while back someone came up with a set of "anti-computer" strategies that allowed even an amateur to defeat a strong go program. These moves weren't anything like ordinary go moves (and perhaps the "loophole" has been closed now) but imo, their existence suggests that a study of programs may reveal other unexpected weakness.
I saw the same things when the OpenAI Dota bots could eviscerate humans 1v1 - even pros lost!
Until a more average player confuses the AI with an unseen behaviour (pulling creeps between the towers etc) to get an advantage.
We saw this with AlphaStar too, but ultimately it feels like simply an exploit. I expect even a relatively simple modern LLM/model working with the custom transformer would have been able to address these exploits after a game.
For a non-Go player, do you think this trend will persist, or is it more of a dead-cat/human bounce?
On one hand, Shin Jinseo is an outlier player of this generation. On the other hand, the newest generation of new pros will have exclusively learned by using the AI to tell them what the best move is, so there's reason to believe that peak human level has yet to be reached.
Distillation of our blessed models is no fair!
Could someone sufficiently motivated invest in training Katago to be able to beat Shin Jinseo with 3 stones of handicap? Unfortunately - probably yes.
This in no way detracts from how absurd and remarkable it is that Shin Jinseo can beat KataGo (it gets a LOT of training and architecture refinements https://katagotraining.org/#eloGraphButtons) with 2 stones of handicap.
Probably a better comparison from the chess world(in reasonably modern times, though perhaps players like Capablanca and Lasker could be mentioned as well. Alas, I don't think FIDE rating existed back then) is Bobby Fischer. In the july 1972 FIDE rating list he held a rating of 2785, the highest in history at the time, with Spassky in second sitting at a "measly" 2660, and only 13 players being above 2600 even.
In Go, there are exchanges of plays called "joseki". Professionals consider the outcome of joseki to be an equal result for both players. Most joseki are only a handful of moves, but some, such as the "flying knife" joseki have variations that continue for upwards of 50 moves. A traditional 19x19 go board has 361 intersections.
Shin's genius was to play out a complex variation of the flying knife joseki that was, in essence, a one-way path to reach an equal board position that occupied about 1/4 of the board. Due to the 2-stone handicap, the position favoured black with the game ~25% complete. KataGo could not have played any other way, where a human may have tried to foil the plan by introducing further complications.
What was truly incredible was how Shin held the advantage from that point on.
I have some questions as a chess player who barely even understands the rules of go. First these josekis sound like what in chess is called a forced tactical sequence. When you say Shin played out a long complex joseki, how does he do that? Does he have to read/calculate it out over the board(50 moves seems crazy to me unless the search tree is highly constrained by geometry/deduction/very few candidate moves, which does occasionally happen in chess endgames), or is the joseki more of a fixed sequence of moves which he's memorised, only needing to read to "punish" if the opponent diverges?
Second, if it is a fixed sequence, how position independent is it? In chess, tactical sequences end up depending on the entire board state to work when they get sufficiently long. I guess what I'm asking is, could a player with some capacity for stategic thinking recognise this idea and take steps to make the flying knife impossible?
I really should spend some more time learning go, it's such a fascinating game.
You can think of joseki as “local opening”. Like, in a vacuum, this is known by study / AI to be an even result for black and white. It’s just like a chess opening, there’s no calculation up to a certain point. And it doesn’t exist in the midgame, it’s not similar to forced sequences which exist in both games; it’s much more like choosing French closed vs open or gambit/gambit declined. The one thing is (and this is huge), since Go board is very big, existing stone formations on other parts of the board influence the value of joseki and make certain ones more advantageous for black or white. To my knowledge this doesn’t really exist in chess, because the opening is already the entire board.
However, when Shin executed the 50 move flying knife, the board was pretty much empty. So there is really no need for calculation, both Shin and the AI know it’s locally optimal. But getting to play a very long locally optimal sequence is good for the weaker player, so they have less “real” moves to lose EV on. Notably Shin probably can’t open with the flying knife in one corner past a certain point in the game, even if that corner were completely empty - the rest of the board positions would change the end values of the variants.
If the AI could know this, they might play a variant that ends 30 moves sooner but is 0.01 pts worse. Then they would have more time to mess Shin up through organic new moves (which the AI will be better at of course).
(disclaimer: only ranked 1 dan)
Joseki are akin to book openings in chess, eg, we routinely see players going 20+ moves entirely from AI prep.
The headline is a bit misleading, though perhaps not intentionally.
Shin took a 2-stone handicap from KataGo which means that Shin is the weaker of the two. But to give that more context, Shin is also the strongest human player to have ever lived in raw strength terms by a good margin, and is known as replicating AI move-for-move more closely than anyone else.
If they were to play even then there’s no chance any human could win (and pretty much all pros agree with that). Lee Sedol beating AlphaGo in game 4 of that series is largely considered the last time a human beat a modern AI in an even game, which is why it was so amazing.
RE the game, Katago was set to use the strongest available model and ran on a 3x 3090 GPU system, which is a lot for KataGo. 20 seconds might sound like a handicap, but that’s over 100,000 play out variations which is essentially infinite for modern KataGo models (anything over 10,000 is overkill).
Shin played well in all games, but his strategy was to avoid complexity. KataGo reads out complex fighting like an absolute monster, so Shin was trying to play very very solid and very very calm so as to not give KataGo an in.
The 2-stone handicap could be thought of as roughly 10-15 points of ‘buffer’. That’s massive in professional games, and that’s what Shin used to win. He played so overly solid that it sometimes cost a point or two, but it removed an opening for Katago to fight. He did this at the key opening and middle-game sections and never burnt through the full buffer of handicap points in the last two games. That’s why those games look kinda ‘boring’, it’s because Shin wanted them to be that way.
Also note that KataGo probably could have won if its ’variance’ was tuned higher (basically it taking risks). Standard KataGo won’t take risks, it just wins with brute force. For handicap games though you can tune its willingness to start fights higher to prevent people from just playing ultra solid (like Shin did).
Shin did an absolutely amazing job and he deserves all the recognition. Katago routinely beats professionals giving them 3-4 stones of handicap, so the win by Shin highlights to me how strong he is, but also just how well he understands how the AI ‘thinks’.
naive question--does " He did this at the key opening and middle-game sections and never burnt through the full buffer of handicap points in the last two games. " imply in retrospect he could have won with a smaller handicap? or was having the rest of that buffer in reserve guiding strategy?
Apparently 2 stones is a huge advantage. An estimate is that the computer is roughly 4-600 ELO stronger on an even match.
Also, the human played a strategy tailored to that huge initial advantage. He said that the AI did not handle this particularly well, and played high probability moves instead of trying to lure him into a mistake.
Also, even though this was the best Go engine, it was not running on a supercomputer, and had a relatively limited amount of time per move.
So, this was an important victory for a human, but not a sign that humans are now stronger than AIs at Go.
> Also, the human played a strategy tailored to that huge initial advantage. He said that the AI did not handle this particularly well, and played high probability moves instead of trying to lure him into a mistake.
Yeah, katago's training is not really focused at all on handicap games, because it's by nature learning from even games against similar-strength opponents.
It doesn't have specific training from playing in a way to exploit a weaker player. In a handicap game you have to give your opponent opportunities to fuck up if you want to play optimally.
If a move loses 0.0005 points if the opponent plays optimally, katago won't play it even if there's ~zero chance a weaker player would play it right.
There have been go AIs that tried to train more directly on uneven opponents, one called "sai" comes to mind, but katago has huge advantages otherwise and won out over the others (for very good reason, it's a great project).
While AlphaGo originally only had win rate as a metric, modern Go AIs have more knobs, including an evaluation of "complexity".
Just stating this off the top of my head so I could be misremembering, but I heard that the KataGo settings used were tweaked to favor complexity. This was most apparent in Game 1 which Shin Jinseo lost, where the AI had an unusual opening. However, the last game was quite plain leading me to wonder whether that setting was present in the last game (or at all).
There are tweaks to move choice like that, but it's all evaluation time. None of that happens in training. Zero times in the RL loop does katago see a two stone game against a weaker player.
You can kind of tweak towards play this metric or that, but it's not the same.
Ah this is interesting. Essentially the idea is that the compute can try and move into positions that it can evaluate but humans might have trouble evaluating because of the board state's complexity?
> So, this was an important victory for a human, but not a sign that humans are now stronger than AIs at Go.
Another way to look at this: Go's handicap system gives us a genuinely interesting metric for the distance between a human and a machine at this specific game. Instead of just "computers beat humans" we get a quantified gap.
The question of whether machines or humans are stronger is moot, isn't it?
In any intellectual contest between human and machine, all the machine winning implies is that the endeavor is algorithmic.
The machine can be given practically unlimited memory and compute; we consider it cheating if the human would use memory aids. The machine could be implemented as many agents cooperating; we'd think it's not right if thousands of humans collaborated to face the machine, etc.
So statements like "not a sign that humans are now stronger than AIs at Go" are pretty meaningless, IMO
> not a sign that humans are now stronger than AIs at Go
Totally wrongheaded, actually, since the computer gave the human a 2 stone advantage from the start.
The article is hilariously triumphalist given the AI had a handicap (which it only mentions in passing unlike the title here).
Obviously its an impressive human intellectual feat but the hubris is instructive perhaps for our wider interactions with AI as its abilities accelerate around us.
> "This series taught me that rather than trying to imitate AI, it is far more important to build the board according to my own style."
It's never made sense to me that so many go players study AI go play in the hope of emulating it in a human game. We're not machines. We can't do thousands of Monte Carlo tree searches per second.
They study it because it has done things that humans had long assumed were bad until AI proved otherwise. The conceptual knowledge has been valuable at the top level.
"The 26-year-old South Korean grandmaster became the first human to win an official series against a state-of-the-art Go engine under a two-stone handicap, a margin considered the absolute boundary for human competition against modern AI."
What a powerful story. Humans have a chance of remaining superior because emotions are the fuel for our intellect and wisdom.
For those who don’t follow human vs AI Go (I don’t), this is with a 2 stone handicap in favor of the human which is apparently standard.
Yes, so calling it a "defeat" is improper to me.
The headline is misleading, but this is still huge. 2 stones against katago is insane, I'd never have guessed we'd see that, ever.
Annoyingly misleading. I read it that the human player was handicapped.
I don't know what the fascination with these AI versus human tournaments is. I'm old enough to remember the whole "Deep Blue" vs Kasparov exhibition, and I didn't really understand the fascination with all that either. That humans can make sufficiently strong calculators has never been a dispute in my mind. If the human wins over the calculator, good, but if the calculator wins, okay. What does that tell us exactly? The human had a bad day? The machine had a good one?
> That humans can make sufficiently strong calculators has never been a dispute in my mind.
Within our lifetimes (unless you're _very_ young) it was doubtful if a go ai would ever beat a decent human. Same was true for chess a generation or two earlier. It's news because it's the handoff of man to machine being the best at a particular thing.
This current news is news from the other way, this human did _exceptionally_ well.
What I found interesting was that he adapted and shifted to a very unconventional strategy of play, opposed to the AI who primarily seems to play high probability moves. Does this not demonstrate the human edge against AI in novel / unconventional thinking?
> a very unconventional strategy
Correction: a very conservative strategy, so as not to lose the advantage he started with.
I recommend "The Master of Go" by Kawabata, 1951. It's a beautiful story of the change of power at the apex of the Go ranking system. I read it the same year as I read "The Glass Bead Game" by Hesse published in 1943 and its hard not to believe a relationship but in truth I think they are completely independent stories of the same situational tension.
KataGo isn't very good at exploiting weaker opponents. In chess people were convinced a grandmaster can never be beaten with a knight odds. It's just too easy to simplify the position and win. It was very easy (for a grandmaster) vs already super human Stockfish. It was still kinda easy (for a strong GM) vs 200+ ELO stronger NNUE Stockfish. And then someone made a net optimized for exploiting humans. Its games are amazing and it beats GMs with knight odds with ease. It's unreal how good it is at setting traps, playing lines that don't work in theory but the refutation is based on precise tactical sequence a few moves deep. Playing vs it feels like playing vs a spider that slowly weaves a net around you till you can't move anymore.
I predict the same thing is going to happen in Go once the engines catch up.
(You can play those chess bots on Lichess for free. Challenge LeelaQueenOdds or LeelaRookOdds if you are master level or stronger)
can the grandmasters still win with rook odds?
Is there an equivalent of LeelaKnightOdds for Go? That might be harder to tackle.
Neo...
Okay, now remove the 2-stone handicap
For all four of you that are like me and understand Dota 2 a lot better than Go, and are wondering what impact a “two-stone handicap” has and what it means, ChatGPT Pro claims that to analogize this scenario to a professional team playing against OpenAI Five:
> The professional human team begins from a legal eight-to-ten-minute game state in which it has decisively won the laning stage: roughly a 6,000–8,000 team-net-worth lead, a 4,000–6,000 team-XP lead, two enemy Tier 1 towers destroyed, the third badly damaged, and all three friendly Tier 1 towers standing.
I don't get it, can you explain in StarCraft II?
I'm struggling to believe it's not good enough to beat a mere mortal.
This was with a two-stone advantage for the strongest-ever human player, apparently.
Thinking machines were finally defeated and will now be prohibited.
We're so back
The fact that it's news that a human beat an AI, post-AlphaGo, demonstrates the current norm.