Problem #10

I found this puzzle on lichess.org, and failed to find the solution. White to play:

[Show solution][Hide]
1. Rg5!, preparing Rxf6 and Rh5#. The immediate 1. Rxf6 does not work in view of 1. ... gxf6 (1. ... Bxf6 2. Rf5 and White again mates).

Problem #9

The position below was reached after 22. ... e4 in a 90+30 game on lichess (full game is below). Black played e5 to defend against Rd3+. A few people were spectating the game, and one user found, several minutes later, the best move that could have been played in this position. It's such an unexpected move that at first I thought the user was kidding...

[Show solution][Hide]
23. h4!, an amazing move that allows White to bring the remaining rook into the attack with Rh3+, now that Rd3+ is no longer viable. This is the only move that yields a winning attack (Stockfish sees a mate in 13 instead of the +3 score for the next-best move). Stockfish suggests 23. ... Ng4 24. Rh3+ e3 25. Bxg4 and mate in 11, or 23. ... e3 Rd3+ and mate in 8 as possible continuations.

The full game is available here (White missed the best move at move 23):

My queen is invisible #3

This is the third installment of this series! Why isn't my queen taken seriously!!! I can do stuff with it.

I played a 2+0 casual game on lichess.org against a player who I had lost against previously, so I was thrilled to get a shot at revenge. I reached the position below from a Caro–Kann defense, my opponent just played 13. dxc5:

I can't immediately take back the pawn, and my opponent is threatening to play b4 to protect it, so I played 13. ... a5 14. a3 (once again threatening b4) 14. ... a5, preventing b4 once more (or so I thought). My opponent decided to play 15. b4?! anyway:

I took with 15. ... axb3 16. Qxb3, and here in an attempt to regain the pawn I played 16. ... Qa5:

It might look like the b7 pawn is hanging with Qxb7, but then I would have Qxc3+, so the pawn is poisoned. My opponent tried to hold his pawns together by playing 17. Bd4, but this does nothing to protect c4, so I took with 17. ... Bxc5, still having in mind that 18. Bxc5 Qxc5 19. Qxb7 would leave the c3 pawn hanging with check. But just when I thought the b7 pawn was poisoned, my opponent took it with 18. Qxb7?:

This is a blunder a simply loses a piece on d4, since the c3 pawn is pinned to the king by the queen. So I played 18. ... Nxd4 (18. ... Bxd4 19. Qxc6+ loses):

Stockfish recommends for white to give up the piece on d4 and castle kingside. But my opponent, as many of my opponents do, went for the double-rook sweep (or whatever), and played 19. Qxa8+?? Qxa8 0-1. Stockfish does concede that this variation wins back the piece on d4 however.

The game is available in full here:

Taking full advantage of a bad tactic

I reached the dominating position below in a casual 10 3 game on lichess after 20. Nb6 Qa7:

Here my knight on b6 is attacked, but my b2 pawn is also threatened by the bishop on g7. I could try to defend both with, for instance, 21. Nxc8 Rbxc8 22. b3, but I didn't want to give up my knight for the c8 bishop, which has very limited scope right now (it can't develop to b7 because of Rc7 or to f5 because of e4). I also didn't want to give up the control of the c-file. So instead I sacrificed the b2 pawn and played 21. Rc6 (defending the knight, and perhaps preparing Rfc1) 21. ... Bxb2:

I wasn't sure how to best play this position. Of course 22. Rfc1 is not immediately possible. I'm still trying to prevent Black from activating his pieces, and the best way to do this is to prevent Black from developing the c8 bishop, which is also preventing the rooks from being connected. So I played 22. Kh1, so that I can play e3-e4 if Black attempts Bf5 (Bb7 is still not a good idea in view of Rc7). Next I can think of playing Rf2-c2 or even e3-e4-e5. But my opponent tried a bad tactic here, he played 22. ... Ba6?:

The goal of this move was to free his position with 23. Qxa3 Rxb6, exchanging my forward knight for his dark-squared bishop. I successfully took advantage of this mistake, but it turns out there was a slightly better continuation which I had dismissed early on in my calculations. Here I played the simple 23. Nxc8 Bxd6 24. Nxa7, which won a piece (Black continued with 24. ... Ra8 25. Rxd6 Rxa7, though Stockfish recommends 24. ... Rfd8 instead).

But I had a better way to take advantage of 22. ... Ba6. I could simply play 23. Qxa3!, which I had dismissed because of the obvious 23. ... Rxb6, but here White has the move 24. Qc4!:

And Black would have been forced to play 24. ... Rb7 (safeguarding the rook), which loses the bishop after 25. Rxc8. The passed d pawn is ready to be pushed, which would also open up the attack of the f3 bishop onto the b7 rook.

For what it's worth, the difference in Stockfish evaluations between my continuation and the best continuation is about 2 pawns, ~+4.2 vs. ~+6.1.

The full game is available here:

Chess Analytics: Predicting rating from average centipawn loss

This page has moved to Chess Digits:
https://web.chessdigits.com/articles/predicting-rating-from-centipawn-loss

I have had the idea of trying to derive a player's rating "empirically", through their play rather than through their results (as is currently done). As a starting point, I thought that it might be possible to approximate a player's rating by looking at their average centipawn loss. The average centipawn loss (aCPL) is the amount by which a chess engine's evaluation of the position changes after each of the player's moves. For example, if the score is about +1.20 before White's move, and White plays a small mistake and the evaluation is +0.30 after White's move, then White's centipawn loss for that move is +1.20 - +0.30 = +0.90; the average centipawn loss is the mean CPL over the course of the game. I thought that perhaps stronger players would have a lower aCPL, such that it would be possible to approximate a player's rating from their aCPL.

The Data

To investigate this question, I downloaded one of the databases available on lichess.org. These databases include all rated games played on lichess for a given period. For the purposes of this exercise, I used a subset of the August 2014 database. From these games, I kept only the ones for which computer analysis was available, so that the engine's evaluation--here, Stockfish--is available after each half-move.

I used Python to transform the file of PGNs into a dataset amenable to statistical analysis. In this dataset, each row is a different game (i.e., a different PGN in the database downloaded on lichess). The figure below shows the first few columns and rows of the dataset:

As you can see, all the meta data for each PGN is kept, and the tags (like "WhiteElo", "Event", etc.) are used as column names. I had to perform operations on the "Moves" column in order to extract the evaluation at each ply. The resulting variables (up to 200 half-moves in this particular dataset) look like this:

Here, I've limited my attention to evaluations ranging from -3 to +3 to avoid getting statistics that are too influenced by extreme scores; any evaluations greater than +3 or less than -3 have been removed (however, the results presented below are very similar whether or not the full range is included, so this is more of a detail at present). From there, I've calculated the difference between each evaluation and the preceding one; half of those differences represents the centipawn loss for the white player, and the other half represents the centipawn loss for the black player. The resulting variables look like this:

Finally, for each game, it is possible to calculate the average centipawn loss for each player by averaging those difference variables across rows, yielding the two aCPL variables:

Descriptive Statistics

Now that the data are ready for analysis, I looked at some descriptive statistics for the variables of interest, the white and black ratings and aCPL. For the final dataset, the average rating for the white players was 1637 (SD = 238), and the average rating for the black players was 1630 (SD = 239). The average aCPL for the white and black players were also very similar, with 0.26 (SD = 0.16) for white and 0.24 (SD = 0.14) for black (again, this is only looking at positions in which the evaluation was between -3.00 and +3.00)

Here's a histogram showing the frequency at which each aCPL is observed in the sample (separated by colors), along with the corresponding kernel density estimation plot:

As can be seen in the second figure, the vast majority of players (with these transformed data) had a relatively low aCPL, below 0.50. This is to be expected, since players with even higher aCPLs would quickly lose almost every game.

Visually, it's also possible to inspect the relationship between aCPL and rating through a scatterplot:

The horror! Even before jumping into the modeling, our guess is that there is not a strong relationship between aCPL and rating! Observations are relatively tightly distributed on the x axis (aCPL) for the full range of the y axis (player rating). In other words, the varying aCPLs are observed at all rating levels in this dataset. Knowing a player's aCPL would not help us much in guessing their rating.

Predicting Rating from aCPL

I tested the relationship between aCPL and player rating more formally through linear regression, predicting the player rating from the player's aCPL, separately for white and black players (it would be possible to include both white and black players simultaneously in the same analysis, but linear regression would not be appropriate here because the scores for both players are closely related within each game, so these two sets of observations are not independent; instead, a good choice here to accommodate the dependency in the data would be a mixed-effects model).

As expected given the graphs above, it is rather difficult to predict a player's rating accurately from their aCPL. Results are slightly different for white and black players, such that it's slightly easier to predict a black's player rating from their aCPL than it is to do the same for a white player (in these data anyway). For white players, the regression coefficient for aCPL was -346 [SE = 34.6, t(1843) = -10.00, p < .001, R² = .05, adj. R² = .05], meaning that we would expect two white players who have a 1-point difference between them in terms of aCPL to have a 346-point difference in terms of rating. The corresponding coefficient for black players was -442 [SE = 37.3, t(1844) = -11.83, p < .001, R² = .07, adj. R² = .07], meaning that we would expect two black who differ by 1 point in terms of aCPL to differ by 442 in terms of rating.

Unfortunately these predictions are not very useful, as indicated by the low R²: between 5-7% of the variation in ratings can be accounted for by variation in aCPL, while the remaining 93-95% of the variation in player ratings remains to be accounted for. I've tried to include higher-order polynomials of aCPL as predictors (2nd and 3rd degree), but as you might have guessed given the scatterplot above, this was not helpful (there simply does not seem to be much of a relationship between aCPL and rating, of any form). So my overall conclusion, stated somewhat crudely: I would not use this model to predict a player's rating after having observed one of their games and analyzed it with Stockfish. The search for a better method continues.

What's Next?

There has to be ways to predict a player's rating from their gameplay, since after all their play is what dictates the outcome of the game. The newer lichess databases provide the player's clocks during the game, so it would be possible to examine whether time taken at each move has an effect on the player's rating (or on the outcome of the game). In this analysis I've also included all games without regard to their time control, but perhaps it would be best to exclude shorter time-control (i.e., bullet) games, as well as games involving players with provisional ratings. As a first step I've also combined information from all moves into one summary statistic, the aCPL, but incorporating move-level information might yield better results. We shall see in a future post.

My queen is invisible #2

After a long period of queens behaving normally, I have now encountered two more opponents who gave up their queen to recapture a piece that's defended, for no apparent reason. If you recall, I had encountered two opponents in two days who had done exactly the same thing: use their queen to capture a piece as if the piece was hanging, but which was actually defended by my queen. I had therefore concluded that my queen is invisible, as anyone would.

In the first game below (played yesterday), the same scenario occurred: My opponent captured a piece with their queen, which was protected by my own queen. But in the second game (played today), there was a new twist: My opponent used their queen to capture a piece that was defended by my rook and not my queen! So there is nothing left for me to do but to revise my theory. I now think not that my queen is invisible, but perhaps that my opponents think their queen is invisible? Or maybe invincible? But I will need more games to figure this out.

In the meantime, I've added the category Eerie for this post and the earlier post, for the inexplicable phenomena that we encounter in chess.

In the first game (a 1-minute game on lichess.org), we reached this position after 15. ... h5:

For better or worse, here I decided to play 16. Qb1 to prevent Black from playing the advance d3 (I'm now X-raying that square). My opponent continued his play on the kingside with 16. ... h4, and I responded with 17. g4, preferring to keep things closed where my king is. And then my opponent decided to push his central pawn anyway and played 17. ... d3?:

And after 18. cxd3, my opponent simply played 18. ... Qxd3?? 19. Qxd3.

In the second game (also a 1-minute game on lichess.org), my opponent just made a mistake with 10. ... Bd6?:

This allows me to win a piece with the simple tactic 11. e5 Bxe5 12. Nxe5 Nxe5 13. Bxe5:

And here, my opponent clicked on his invisible queen (or so we think) and played 13. ... Qxe5?? 14. Rxe5.

Both games are available here: 

Twice the same position at move 14, and an improvement that failed

Merry Halloween!, or whatever. It's fitting that today's the Halloween because this is a scary post--soon I'll add a category "Eerie" for posts like this one and the one about the invisible queen.

Yesterday I played two bullet games against the same guy within a few minutes, and astonishingly or not, we reached exactly the same position in both games at move 14. The move order was slightly different (starting at move 8; I tried to improve my move order the second time around). My opponent chose a different continuation at move 15 because he lost a piece in our first game, and it should have been a better continuation had he found the best move, but to no avail, he ended up losing an exchange in the worst way possible in the second game.

Here's a side-by-side comparison of the games:

Game #1:
1. e4 c6 2. Bc4 d5 3. exd5 cxd5 4. Bb3 Nf6 5. d4 Bg4 6. f3 Bf5 7. Ne2 Nc6 8. O-O e6 9. c3 Bd6 10. Ng3 Bg6 11. Bd2 Qc7 12. Be1 O-O 13. Nd2 Bf4 14. Bf2 Na5

Game #2:
1. e4 c6 2. Bc4 d5 3. exd5 cxd5 4. Bb3 Nf6 5. d4 Bg4 6. f3 Bh5 7. Ne2 e6 8. O-O Bd6 9. Bd2 O-O 10. Be1 Qc7 11. Ng3 Bg6 12. Bf2 Nc6 13. Nd2 Bf4 14. c3 Na5

Game #1: 15. Bc2

Game #2: 15. Re1

Game #1: 15. … Nc4 16. Bxg6? Nxd2!

Game #2: 15. … b5 16. Bc2 Nc4 17. Bxg6? Bxd2!

Game #1: 17. Bxf7+ Qxf7 18. Re1

Game #2: 18. Re2 hxg6

Game #1: Here I played 18. ... Nc4 and I'm up a piece.

Game #2: 

Here my opponent needed to play 19. Rxd2 Nxd2 20. Qxd2, 

still conceding the exchange (why not play 18. Bxh7+ otherwise?). 

Instead, he tried to deflect my knight from the defense of the bishop.

19. b3? Bxc3 20. bxc4 Bxa1 21. Qxa1 dxc4

And here I'm up an exchange with a protected passed pawn.

Here are the two games in full (1-minute games on lichess.org):

Game #1:

Game #2:

Lichess Founder on Cheating

I recently wrote a post on cheating in online chess. I just found a presentation by Thibault Duplessis, the founder of lichess.org, in which he address the issue of cheating on the website (from 17:45 to 24:45 in the video):

Chess Joke

Nothing beats a good joke about chess patterns:

Improving once you've stalled

I thought I'd write a quick post to point you guys to an interesting article I found on the /r/chess subreddit. The article was written in 2011 by IM Kostya Kavutskiy, and attemps to provide tips as to how to overcome a drought in chess improvement (i.e., when your rating has stopped increasing and has effectively stalled). The article is titled "... on Breaking 2366", and is about how this IM came to reach the rating 2366 after stalling around 2250. (Funnily enough though, 6 years later his FIDE profile shows a standard rating of 2390, with games played as recently as May 2017, so perhaps he's stalled again and would be due for another article...)

He separates the tips into the following categories:

Analyzing your own games

Kavutskiy recommends reviewing the opening of every serious game you play. You shouldn't make the same mistake in two different games that use the same opening. I've recently played in an OTB tournament and have definitely made (costly and annoying) mistakes in my openings. Also, if after a game you realize you had trouble finding a good plan once you reached the middlegame, then Kavutskiy says your understanding of that particular opening is lacking.

Psychological mistakes

A psychological mistake is "usually the main culprit of any chess slump". Kavutskiy gives as example the main mistake he found in his own play, namely relying too much on intuition (or rather, not relying enough on concrete calculations); he writes that he would sometimes refrain from sacrifices because (1) he didn't trust his calculations ("the hard part was convincing myself that my opponent hadn't set some nasty trap. A year ago I would have just assumed that I [had] missed something in my calculations and would have played some 'solid' move instead"), or (2) he was simply not "brave" enough. Unfortunately, I feel like this section doesn't really answer one very difficult question: how do you figure out the psychological mistakes you make during your games?!

Working harder

To get over stallation (if that's a word), Kavutskiy says that you need to work harder than you're used to, at home (by analyzing your games, reading books, etc.), but also during your games. In particular, you need to work harder than your opponent: stay more focused, calculate variations more deeply and accurately, etc. I think this advice makes sense, and for me this was one of the main takeaways from this article. It can be hard to stay focused for the whole duration of a longer game, or after you start feeling like you have a sizable advantage against your opponent.

Chess books ("Chess books are ridiculously helpful")

Finally, Kavutskiy recommends reading more chess books as a road to improvement. He recommends a few specific books, a list which you may or may not find interesting (and which I invite you to consult in his article). He does recommend Perfect Your Chess by Andrei Volokitin, which I own and would also wholeheartedly recommend; the book presents hundreds of puzzles that target more advanced players. I can also recommend Excelling at Positional Chess, which I've reviewed here almost 10 years ago. I think benefits you get from books depend on how much work you put into them (do you get invested, set up boards with the position and run through the variations in the book, or do you read passively?), and I think that was one of the main messages in Kavutskiy's article, for better or worse (I've been reading very passively these days).

Cheating in Online Chess

I was wondering: How prevalent is cheating in online chess? More specifically, it would be great to get an estimate of the percentage of games that involve cheating in the form of engine use.

I couldn't get an exact answer to this question, but I think the answer is that it's prevalent enough to be bothered by it. Chess.com even has a private forum dedicated to the discussion of online cheating in chess (you need to submit a request to be admitted into the forum, and there are some restrictions--like having been a member on the site for at least one month--but overall it's easy to gain access). Chess.com also employs two full-time employees whose only task is to research and prevent cheating on that site. Similarly, both lichess and the Internet Chess Club have systems in place to detect cheating (sometimes doing automatic checks while the game is being played).

Five Percent?

The question about cheating prevalence has been asked several times over the years on multiple forums (like here). One user estimated the prevalence at 5%; another user said that a chess.com employee told them that 70% of games involve cheating (that's just hearsay, and 70% does sound pretty implausible). Yet another user said that the prevalence is "considerably higher" than 5% if your rating is high enough (1800+). One lichess user who is a National Master mentions on reddit that looking through his history of games in the classical category (8+ minutes), “about 80% of my losses were to people that subsequently had their accounts closed for cheating”

The 5% figure is one that seems to come back in forum posts. This figure was also mentioned by one of the lichess developers on reddit, 2 years ago:

About 1.2k accounts are make on lichess each day, and each day some 50 or more get marked (this number is set to increase as our detection system has just become fully operational). So if we assume that all cheaters use new accounts, about 4-5% of new accounts are cheaters. This figure has remained about the same since I started on the site and there were half as many new accounts made each day.
It's far less likely for a cheater to be using an older account, so that figure drops dramatically once they've played a hundred or more games.

Miscellaneous Stuff about Cheating in Chess

• At least one person has proposed a hypothesis test to detect cheating in online chess, and one International Master (also an Associate Professor in the Department of Computer Science and Engineering at the University at Buffalo) used statistics on the case of Borislav Ivanov, accused of cheating at the 2012 Zadar Open.
• Wikipedia has an entry for cheating in chess. Among other events recorded there, one grandmaster was stripped of his GM title for cheating, though he was allowed to keep his IM title.
• Chessbase.com has a 5-part series on the history of cheating in chess.
• GM Vladislav Tkachiev shows on YouTube just how easy to cheat with an accomplice using an earpiece in a video called How I Became a Cheater.
• There used to be a user on lichess, who was later banned, with the nickname IMaDirtyCheater, who made it very clear that he was cheating; his profile read:

I cheat, it should be obvious!
Really obvious...

• Some users have written programs to cheat directly from the same window the game is being played on; one such example is shown in a YouTube video on a lichess bullet game (I don't provide the link to the video because the description contains a link to the program, which is freely available on github). The resulting board looks like this, with the move recommended by the engine being indicated through a dark square (starting destination) and red square (ending destination):

My Recent Encounters with Cheaters

I recently started streaming games with longer time controls on Twitch, and as luck would have it, my first 2 games were played against cheaters! (So we're 0 for 2!) Both players were later banned from lichess. I lost both games pretty quickly, and both opponents played their part very fast (adding to the misery). Here are both games, available on my YouTube channel: 

Chess without squares?!

I recently ran into an interesting post on Reddit, on the subreddit /r/dataisbeautiful. It's an image that shows the pattern of "78 million chess moves, from 1 million top-level games". The image looks like this:

There are some interesting comments on the Reddit thread, so if you're interested check it out. For instance, one comment points out that bishops & knights seem to move towards the center and very rarely towards the sides of the board-- a "large data" way of learning how to play chess like the pros, in a way.

But what I found striking with this image is that, of course, the pieces' movements are bound by the squares on the board (giving the image the jagged, linear, square-y appearance that it has, with a ton of untouched white space). "What if there were no squares?", asked John Lennon (I think). In other words, is it possible to imagine (no pun intended) a chess game where pieces move freely on the board, no longer restricted by the squares?

In truth, I don't know how we could figure out the legality of moves without the neatness of the squares. (Perhaps it would need to be more "physics-based", with a maximum distance [read: speed?] that can be travelled by the pieces at each move, which itself could depend on their previous momentum.) But those technicalities notwithstanding, it's fun to imagine...

Four-Player Chess

I don't know if you got wind of this, but Chess.com is developing what is supposed to be the first-ever variant of chess that can be played with four players simultaneously, which they call four-player chess. The variant was announced on September 1st 2017 on chess.com here. I first heard of this through IM Danny Rensch with Chess Today, who spends most of his video trying it out for the first time: https://youtu.be/f-cVupxkQpo

The board looks like this:

And prior to entering the game, you are shown this notice, which includes the gist of the rules:

I played only one game, and somehow I finished 1st out of the four players. I technically lost on time while there was one player left in the game, but the penalty for this is 20 points being awarded to my opponent, and because I had more points than my opponent even after they got their 20 points, I won the game. From one of the rules above, it sounds like I could have flat-out resigned and still win the game!

Personally, I did not enjoy the game that much. The board is huge, such that your opponents' long-range pieces (like the bishops) take a while to get used to because (1) they can come from much farther away than we're used to, and (2) they can come from 3 different players! You also don't have a choice of time controls--everyone starts with 4 minutes (with some minor increment), which I find pretty fast when you're not used to this variant.

If you want to try it out, four-player chess is accessible here: https://www.chess.com/4-player-chess

Importantly, the variant is still in beta phase, so my guess is it's bound to change in the upcoming weeks. Among other things, Danny Rensch mentions in his video that there is currently no notation system for this variant, such that developing theory is difficult.

My queen is invisible!

I have great news: My opponents don’t see my queen (at least sometimes). Maybe it’s only when my opponents are about to move their own queen, I’m not sure yet.

Yesterday I played a rated classical (standard) game on lichess with time control 10 0. This is the position after I played 17. g4. I wanted to open up Black’s kingside (where the king is uncastled) and bring a rook on the eventually-open g-file (or f-file, if lines open up). But my opponent played a move I didn’t think was possible, they played 17. … f4?!, so I responded with 18. exf4, simply winning the pawn. Perhaps Black’s plan could have been to divert my e-pawn to play 18. … d4 (as well as to keep files closed around their king), but no, instead they played 18. … Qxf4??, taking the pawn back:

And they simply resigned after 19. Qxf4+ 1-0.

Then the next day (today), I played a casual (unrated) bullet 2 1 game, and we reached this position:

Here we castled on opposite sides, and I just won a pawn. We’re both planning to attack each other’s king, although my opponent’s attack seems to be coming quite a bit faster--they’ve got both their rooks aligned with my kingside, and the three wing pawns have reached the 4th and 5th ranks. Here I was planning on playing an eventual Bc4, pinning the e6 knight and preventing the move d4, which would prevent my knight from jumping into c4. Eventually I’d also like to develop the queen and bring a rook to the c-file, and play a4, a5, and a6 or b6. But here I decided to hold back on Bc4 in case the bishop is needed for defense on the kingside, so I played 19. Qb3, also preventing d4 and making way for the rooks, with the intention of playing Bc4 next move and pin the knight. My opponent played 19. … g4 and I played 20. Bc4, attacking the knight. Or so I thought--my opponent continued with 20. … g3?, so I took the knight with 21. Bxe6:

And then evidently, my opponent recaptured with 21. … Qxe6?? and resigned after 22. Qxe6 1-0.

So I don’t know what’s going on with my queen…

Both games are available here: