Five Ideas for the Future

Five Ideas for the Future

Lists predicting future innovations are pretty common these days. There are usually a few recurring themes, like artificial intelligence, personalized medicine, and the “Internet of Things.” This list is five ideas that could make a big change in the future but haven’t really been explored yet.

  1. The Sharing Economy of Goods

The sharing economy is an economic system where an individual’s (or group’s) excess resource is provided to another individual (or group) that needs that resource. My favorite example Is Airbnb, a company that helps people with excess rooms rent them out to travelers in search of accommodations.

I think there is a huge opportunity to create the sharing economy for goods. The first modern iteration of the sharing economy for goods was Craigslist, which allowed people to sell their unused stuff to local buyers. eBay provided a similar marketplace on a national scale, with auctions and electronic payments to automate pricing.

The main issue with the existing methods of peer-to-peer commerce are that they involve too much friction (contact seller, haggle over price, arrange meetup, travel to meetup location, transfer money and goods) and there is no trust (Craigslist repeatedly warns you of scams, selling on eBay is fraught with scams as well, Craigslist murderers, etc). However, Craigslist were combined with Amazonesque logistics, the volume of goods sold locally would explode.

The other big missing piece is rentals. In my neighborhood, every home has its own lawnmower, even though no lawnmower is ever used for more than two hours a week. There are many other goods, like lawnmowers, that sit idle most of the time*. Peer-to-peer rentals would create the true sharing economy of goods. Standards of living would go up as people would consume and waste less. Lower consumption is better for the environment by reducing trash volume and reducing resources needed to produce items.

The above is why I’m working on Circa, a startup that removes the logistical barriers of local buying and selling.

  1. Personalized Education

The advent of MOOCs and flipped classrooms in the last decade has begun changing the way we learn. I think we can take it one step further, by using machine learning to personalize education.

Imagine if, for every class, a large set of problems was created, covering all the topics in the class and of varying difficulty levels. Imagine that, using a mobile app, website, or even (shudder) desktop software, every student began completing those questions. Now imagine aggregating this over hundreds or thousands of students. The results is that, using machine learning, you could quickly determine with a high level of accuracy what a student’s individual strengths and weaknesses were, and only show them questions/content in their weak areas so they’re not wasting time reviewing what they already know.

It’s possible that you could build a platform for this, where any content could be “plugged-in” and individualized for each user, ensuring they learn the most material in the shortest amount of time. I actually started building this, but had to shelve it when other projects came up.

  1. The Personal API

I think the “Internet of Things” is overhyped, but one aspect that excites me is devices that monitor biometrics. I eventually see a future where data like heart rate, serum nutrient and hormone levels, sleep/REM cycles, and even stool composition are regularly collected and analyzed. This data could be collected and accessible through one API, which developers could build apps atop of. While there would certainly need to be thought to privacy and security, creating a personal API that could be accessed in a standardized way could lead to useful analysis, such as:

  • Relationships between nutrition and sleep
  • Relationships between when meals are eaten sleep
  • Relationships between nutrition and mood
  • Relationships between exercise and mood
  • Preemptively discoveries of allergies
  • Preemptively discoveries nutrition deficiencies

The above is by no means an exhaustive list—I could write a whole article about the beneficial analysis that could come out of a Personal API. With machine learning, all of this analysis could be personalized, while at the same time being anonymized and uploaded to the crowd, where it would be aggregated and provided to researchers.

  1. Unlimited Energy

I believe, within our lifetimes, most (if not all) of our energy needs will be met by solar energy (or we figure out how to harvest lighting). What’s most exciting is that lack of clean energy is the base case for many modern problems as “[a] lot of problems—economic, environmental, war, poverty, food and water availability, bad side effects of globalization, etc.—are deeply related to the energy problem.”

There could be some revolutionary breakthrough, but I think the most likely route is simply the continual, evolutionary increase in efficiency (both cost efficiency and energy efficiency) of photovoltatic cells and decrease in cost of solar power. Solar energy is already cheaper than conventional fossil fuels when accounting for negative externalities.

Since the sun’s useful energy is limited only by our ability to harness it, I think we’ll eventually reach a point where we’ll have virtually unlimited energy. What kind of future can we build where energy isn’t a concern? Perhaps we could eliminate water scarcity or alleviate the intensity of droughts via mass desalinization, which is currently too energy intensive to do in most places. Maybe there are incredible materials currently requiring too much energy to produce on a mass scale. It’s exciting to think about.

  1. Green Roofs

Green roofs are roofs covered with vegetation—essentially, mini-parks atop buildings. The American Planning Association notes parks have all sorts of benefits, such as reducing air pollution, reducing the heat island effect, decreasing stress levels, promoting exercise, reducing crime, increasing happiness, and making cities more aesthetically pleasing. If every or nearly every urban rooftop was a green roof—that is, every resident of a city was within a 100 second walk from a park or garden—it would revolutionize urban living.

I’ll admit I don’t know quite as much about this topic, and there are certainly quite a few practical considerations (weight bearing capabilities of rooftops, altitude limits, possibilities of storms knocking trees over, etc.). Wikipedia claims, “[These issues make] it unlikely for intensive green roofs to become widely implemented due to a lack of buildings that are able to support such a large amount of added weight as well as the added cost of reinforcing buildings to be able to support such weight.” But if we went in with the attitude that this is the default normal roof, rather than as a peculiar exception to normal roofs, I think we’d figure out a way to scale it, and I think it’d be shocking just how drastically improved all aspects of urban life would become with such a simple change.

 

* My neighborhood has about 250 houses, each with a yard. Currently, each house has its own lawnmower. During the summer, there are 12 suitable hours for mowing the lawn, or 84 hours a week. Assuming each homeowner wants to mow the lawn once per week, that means there is demand for 250 hours of lawn-mowing per week. Even if we only assume 50 percent efficiency in lawnmower allocation, that means six lawnmowers could service the entire neighborhood! Lawnmowers are about $200 (on the lower end, anyway), so the neighborhood would collectively save about $50,000 on lawnmowers.

Computer Science Should Be Two Separate Majors

Computer Science Should Be Two Separate Majors

I college, I majored in Computer Science and Political Science. The poli sci kids could be divided into two camps—the people who wanted to go to law school (60 percent), and the people who wanted to get into politics (40 percent). Similarly, there was a divide in CS—the people who wanted to become software engineers (80 percent), and the people who wanted to become computer scientists in some form (20 percent). The aspiring computer scientists often double majored in CS and math.

Ultimately, both groups were left unsatisfied. The future software engineers found themselves unprepared for the job market, while the future computer scientists had to add on another major to get the theoretical, math-heavy content they enjoyed. The solution is to split CS into two majors—Computer Science and Software Engineering.

The new, purer CS major would focus on more theoretical topics—algorithm analysis, data structures, artificial intelligence, database design, automata, design of programming languages, artificial intelligence, cryptography, graph theory, etc. The newly created Software Engineering major would focus on the skills software engineers need on the job—version control, collaboration tools, environment configuration, test-driven development, basic web development, security practices, performance optimization, software development methodologies (waterfall, agile, etc.), creating and consuming APIs, etc.

There would naturally be some overlap, as every engineering discipline has a theoretical basis, and many software engineering skills (such as using version control) would prove very useful even in a CS research environment. Ultimately, the two would be distinct but related disciplines, in the similar fashion to two separate spoken languages that evolved from the same ancestor language.

While I believe this would lead to better outcomes for students, I can think of two counterarguments. The first is that such a separation is unnecessary. This argument contends that CS majors are doing fine as they are –salaries are high and the job market robust for a CS grad fresh out of college. Why fix what isn’t broken?

My counterargument is that it is broken. I have friends who studied CS but felt woefully unprepared for the software engineering jobs available, though internships alleviated this somewhat. Many of them were not only uninterested in some of the more theoretical topics they learned in college, but honestly didn’t need to know them for their jobs. On the flip side, I’ve known people hiring software engineers who’ve complained about having to spend six months training recent graduates. This is all anecdotal since there’s no hard data on any of this, but I think there is a broad consensus that there’s a wide gap between college CS and industry software engineering. I’ve heard some universities already have a software engineering major for exactly this reason.

The second argument is that such specialization goes against the philosophy of a college education, which places a broader emphasis on learning for its own sake, critical thinking and “learning how to learn”, and the broadening of the mind that occurs from a college education. College isn’t vocational school and creating a software engineering major, which is clearly designed to train people for a specific profession, pushes universities away from that philosophy.

The counterargument here is that STEM majors, by their inherent nature, tend to be much more specialized and narrow than traditional liberal arts majors. There is no fixed career path for an English major, while an Electrical Engineering major is expected to become an electrical engineer. The “college is for learning for its own sake” argument (which I saw on many college admissions pamphlets when I was applying to college) lost credibility once college became exorbitantly expensive. While I agree in principle with the tenets of broad, liberal education that concerns itself with the expansion of the mind rather than a specific vocational goal, it’s naïve in today’s abysmal job market. When students are burdened by tens or even hundreds of thousands of student debt, there must be a concrete return on that investment to justify it—and at that point, a job that pays the bills means much more than the lofty goals of a liberal education.

Two Years Ago

This post was originally published on September 21, 2013 at www.remainfree.com.

Troy Davis was executed two years ago today.

I woke up shortly after sunrise. Today was gameday in Athens, Georgia, and as I hustled out of my apartment I saw families sprawled among dozens of red UGA tents, chatting excitedly and huddling around the television in anticipation of the big game. A few children were tossing a football back and forth, while a group of teenagers cackled as they played cornhole. There wasn’t time to join the festivities. I was already running late.

My mother flew in last night from Boston. Last year, it was just the two of us. This time we were joined by my father and my younger sister. It rained for the first hour of the drive, but occasional rays of sunshine poked through. As we passed Jackson, Georgia, I remembered the last time I was there, on this very day two years ago. The chants still resonated in my mind. I was transported back in time, an observer from the future watching as the hope and tragedy of that night unfolded.

We stopped at a rest area a little past the half way point.

It wasn’t too far from Dublin, Georgia. The last time I was there was in 2010, when the staff and interns of Amnesty International and I stopped there on our way to the historic Troy Davis evidentiary hearing. Today I was clad in the same blue “ I Am Troy Davis” t-shirt that I first wore in Savannah during the hearing. I was only sixteen years old then, a bit chubby and sporting long, shaggy hair. But what I saw in the courtroom furthered my conviction that Troy Davis should not be executed.

As we continued to drive the rain cleared, and the grayness gradually gave way to blue.

We drove past a few farms, but the cotton fields stood out the most. The other crops gave way to shoots of bright, billowy cotton, as if a snow storm had delicately placed bundles of flakes upon each plant.

As we neared the city, we crossed Clarence Thomas interchange. I thought back to the spring of 2012, when Justice Thomas and I sparred over the case. He had voted against giving Troy the evidentiary hearing. He held no doubts that justice had been done with Troy Davis’s execution. And yet, he was also the one who issued the reprieve on September 21, 2011, the one that delayed Troy’s execution for another four hours.

We passed Savannah’s city hall. The building was grand and elegant. But it wasn’t by the Savannah boardwalk or park-like squares that dotted downtown. It was out here, in the Savannah hinterlands, surrounded by overgrown forest and weeds and a half empty strip mall with signs for a Subway and World’s Most Famous Asian Cuisine.

Magnolia Memorial Gardens was exactly the way I remembered it last year. The front office was a small hut, large enough to fit maybe a dozen people.

Last year we rushed to arrive before the staff left, so they could show us where Troy’s grave was. This year, September 21 fell on a Saturday, and the whole place was deserted.

As I thought about the first time I walked here, when Troy was first buried, my mother had already walked far ahead to Troy’s grave.

The three graves were all there. Troy Davis.

Buried beneath him, his mother, Virginia Davis.

Buried beside them, Troy’s sister, Martina.

We placed a note and blue flowers (Troy’s favorite color) on the grave and remained there for a few minutes.

The only flowers on Martina’s grave were dead and wilted. The only ones on Troy and his mother’s grave were beginning to wilt and must have been there a few days.

My thoughts were similar to last year’s, so I won’t repeat them.

We began walking away from the graves. There was only one exit path.

As I walked, keys in hand, I noticed a police car nearby. It had been sitting there, watching us the whole time. As we left I heard the car start up and drive off into the distance.

We began the long drive back. Less than an hour into the drive, the rain began. It bombarded the car, angrily striking the windshield and blinding my view.The sun had set and the two lane interstate had no lights. One of the lanes was closed for construction so all of the cars were crammed into one narrow, bumpy lane. The car began hydroplaning, skidding and swerving every few seconds as I desperately tried to maintain control. Other drivers were tailgating me and each other, apparently oblivious to fact that one small mistake could send us all careening into each other at 70 miles per hour. I was sleep deprived and hungry, and my eyes were straining to see the road amid the whirlwind of mist, rain, and headlight glare. We had four hours left to drive, and the rain was not expected to let up anytime soon. It was too dangerous to pull over–a skidding car could slam into us on the shoulder.

This wasn’t just about me. My father sat beside me, advising me as I drove. My mother and younger sister were in the back seat, blissfully ignorant about how much danger we were in. My life and the lives of my family were in the hands of other people and other forces. No matter how carefully I drove, I wasn’t in full control. Was this how Troy and Martina and Virginia felt, as their lives were eaten away by imprisonment and cancer? I thought back to the cemetery, where the three Davises lay, their lives all snuffed out in 2011. Would the same fate befall my family in 2013? I couldn’t shake the thought.

There’s nothing like a cemetery to make you feel stupid. Yes, stupid. This week had been a rough week for me. At least, I thought it had been a rough week. I seriously injured my eye during a game of frisbee, caught a cold that caused me to miss several important classes and miss a week at the gym, and had a stressful situation with a close friend. But seeing so many graves dug for so many young people makes you realize how petty most of your worries really are. I felt stupid for being angry at such minor things.  At the risk of being cliche, l was reminded that life is short, and I wasn’t making the best use of the time that remained.

When Troy was alive, we would talk about how to live life. Troy noted just how unhappy people outside of prison were. He looked at me sadly and said, “People out there are living to die. I’m dying to live.”

When we made it home safely, I resolved to renew my commitment to live life to the fullest. Tell people how you feel. Help others. Be adventurous. Follow your dreams. Remain free.

Imagining a Better Facebook

Imagining a Better Facebook

Today is February 4, 2014, exactly one decade after a nineteen-year-old Mark Zuckerberg launched thefacebook from his dorm room. Today, the company is worth over $150 billion and has over 1.2 billion active users. I joined Facebook in late 2007. Back then, Facebook was fun. I’d waste hours on the site writing statuses, scrolling through my newsfeed, and messaging friends. But these days, I don’t enjoy using Facebook. Many of my friends don’t like using it either. Although they’re still active Facebook users, most of them find Instagram and Snapchat more enjoyable.

The problem isn’t inherent in the concept of an online social network, but rather in Facebook itself. What would the ideal social network look like? To answer this, I dissected all of the good and bad things about Facebook.

The Good

Everybody is on Facebook

The most compelling reason to use Facebook is because everybody else is on Facebook. It’s rare to have a Facebook search come up empty. These “network effects” make it difficult for any competing social network to displace Facebook.

Chat/Messaging

Email is slow and can get lost in a crowded inbox. Texting isn’t ideal for longer back and forth conversations, has poor support for group conversations, and can be invasive if you don’t know the other person well. Facebook messaging fits nicely between the two.

Facebook groups

I keep in touch with high school friends through a Facebook group, which we use to plan events and let each other know when we’re in town. In college I joined Facebook groups for my majors and various campus organizations, and they were useful for announcements, asking questions, and connecting with like-minded people.

Events

Planning and coordinating events is easy with Facebook. The downside is that people are flaky online (only 30-50 percent of RSVP’d event guests show up), but that’s probably not a Facebook-specific issue.

Photos

Before, if you wanted to share photos online, you’d have to upload them somewhere and email the link to a bunch of people. Facebook allows you to share photos with your friends and family and have them view it in one centralized place.

Profiles

You can gain a basic understanding of a person by looking through their profile and seeing the pages they follow. I’ll browse through friends’ profiles to see if we have similar intellectual interests, or watch the same TV shows. I’ll even learn new things about people I know well from their profiles.

Facebook Pages

Facebook pages, if used correctly, can build communities around shared interests. However, I follow dozens of Facebook pages and can only think of three or four that actually update me with things I care about.

The Bad

Privacy

Facebook’s privacy issues are well documented, so I won’t repeat them here. What bothers me most is that Facebook tracks you around the internet even when you aren’t logged in. And if I forget to log out and accidentally click one of the ubiquitous “like” buttons around the internet, the action shows up on my profile and my friends’ newsfeed.

Irrelevance

The most unpleasant aspect about Facebook is how irrelevant my newsfeed is. I randomly selected posts from my newsfeed and categorized them as “relevant” or “irrelevant,” and only 20 percent fell into the first category. This is abysmally low considering Facebook’s sophisticated algorithms for newsfeed content, my selectivity in accepting friend requests, and my consistent efforts to hide posts from or unfriend people who share things I don’t care about. There’s just too much junk on Facebook, which leads to…

Sharing ad nauseum

Facebook has built up a pervasive culture of sharing. I get spammed with invites from stupid third party apps on the Facebook platform, invites to events that the sender knows I’m not interested in, and Facebook messages from people I don’t want to talk to. My newsfeed is polluted with uninteresting updates from pages I follow and a torrent of photos of pets, food, and babies1. This culture of endless sharing is why Facebook needs newsfeed curation algorithms in the first place.

I had to jump through hoops to make myself invisible to certain people on Facebook chat so they would stop spamming me with messages2. I’ve had to manually unfollow people who post annoying things. I shouldn’t have to keep fighting against my social network like this.

Friendship Expectations

I’ve learned firsthand that people get very upset when you don’t accept their friend requests. Most people I know are Facebook friends with their relatives, coworkers, bosses, and random acquaintances out of social obligation.

Bloated

Facebook is overloaded with useless stuff. Trending topics, hashtags, Facebook gifts, location check-ins, third party apps, graph search, timeline, etc.

The core features are: profiles (though these have become bloated too), newsfeed, photos, chat, events, and groups. Everything else is extraneous.

Notification Overload

These days I hesitate to comment on posts because I’ll be flooded with notifications about unrelated five-word comments from people I don’t know. Facebook has tried to fix this by consolidating notifications and allowing you to unfollow posts, but I’d rather not receive those notifications at all. I no longer stay logged in on the Facebook app because of its push notifications.

The Ideal Social Network

Using this list, we can imagine what the ideal social network (IDS) would look like.

Simple

IDS would be premised around a clean, minimal interface with only essential features, similar to Facebook in the early days. It contains only the essentials: profiles, a newsfeed, photos, chat/messaging, events, and groups. Privacy and security settings are simple and streamlined.

Structured Around Friend Circles

Google+ got it right when they created friendship “circles.” In IDS, every friend must be assigned a circle, such as close friends, family, co-workers, etc, and posts can be targeted to individual circles. Circles give better privacy controls and more accurately represent how we manage our social connections in real life. Facebook awkwardly tried to copy the circle idea with Facebook lists, but few use them since they aren’t a central aspect of the network, as in Google+.

Different Business Model

I’m not opposed to personalized ads. If I have to see ads, I’d rather they be relevant. What is unacceptable is Facebook tracking me around the internet, even when I’m logged out, and giving that data to advertisers. IDS users would be able to opt-out of personalized ads or be able to upgrade to a premium, ad-free version. I wouldn’t mind paying a few dollars a month for an ad-free privacy-respecting social network.

Discourages Extraneous Sharing

To prevent newsfeed pollution, IDS would do the following:

  1. Allow downvoting. The downvotes would be hidden from the original poster, and would be used to make newsfeed curation algorithms more accurate.
  2. Make it easy to unfollow someone or reduce their posts’ prevalence in your newsfeed.
  3. Gamify posts by showing engagement statistics. If people see their posts are consistently not getting likes/upvotes or comments, they may change their posting behavior.

Discourages Meaningless Connections

I see three possible solutions:

  1. Hard limit on the number of friends. Dunbar’s Number suggests we can only maintain 150 relationships at a time, so IDS could limit the number of friends to 300 to add some breathing room.
  2. Don’t have a suggested friends feature. Instead, require each user to make a conscious, specific effort to add someone as a friend. Most users wouldn’t make that effort for people they barely know.
  3. Have a “suggested unfriend” feature, based on frequency of interactions and downvotes.

IDS is the kind of social network I want to build and use. Facebook’s enormous userbase means IDS is probably doomed from the start, but it’s something to think about as Facebook enters its second decade.

 

1- Okay, the baby photos haven’t started yet. Give it a few years.

2- To be invisible on chat for a subset of your Facebook friends, create a separate friend list, add the appropriate people to that list, and change the list settings so they can’t see when you’re logged on.

Play Like Picasso

I was rummaging through old files on my computer and came across my primary college application essay, “Play like Picasso.” I remember staying up late one night in what must have been November 2010 and writing it in about an hour. Surprisingly, almost nothing was changed in the editing process.

Play like Picasso

I can’t draw. I really can’t draw. My drawing abilities haven’t changed since I was five years old; they consist of crudely constructed androgynous stick figures with smiling or frowning faces. My painting and sculpting abilities are equally uninspiring, and early childhood summers spent in weeklong art camps failed to improve my aptitude. Perhaps this early realization of my lack of artistic talent directed me to nontraditional avenues of creative expression, and it just so happened that one of these avenues was chess. I immediately took to the game, playing almost daily with my father when I was five years old, losing every time but entranced with its possibilities. I became more and more involved, joining my school chess club, playing on the internet and in tournaments, and excitedly bringing my chess set to sleepovers at my neighbor’s house.  What was it about pushing those pieces of plastic that excited me so much? Maybe it was the feeling that every game I played was unique, the knowledge that every game I play has never been played before and will never be played again. Maybe it was going to a chess tournament and being able to see both a homeless man intently analyzing the chess board, unrestricted by the limitations life had placed on him, and a Mercedes-driving Cuban doctor wrestle with the fact that the very hands and mind that had saved hundreds of lives in decades past could not prevent an imminent defeat by his nine-year-old opponent.

Perhaps it was because, like an author with his characters or an artist with his subjects, I could empathize with my pieces. Maybe I was that lone piece, bravely yet recklessly straying into enemy territory on an all-or-nothing gamble to prove what I’m capable of to my opponent and to myself. Maybe I was one of the two Bishops working side-by-side, perfectly complementing my counterpart on the other color complex, realizing that in our differences lay our strength. Maybe I was a Knight, indecisive, hopping between dark and light squares and awkward in my irregular and idiosyncratic movements, yet capable of great beauty if given an opportunity to flourish. Maybe I was a lowly Pawn, jeered at by the other pieces for my limited powers yet containing the hidden potential to transform into a Queen, the most powerful and majestic piece on the board, and prove my worth. Maybe it was because I knew what it was like to be in zugzwang, a chess term that describes a situation where a player wishes he could freeze time, since every move he makes worsens his position. Maybe I wished life were like chess, because even though it would still be confusing, I could find beauty in every move. Maybe it was all of these things.

I see a chess position the way I see myself: an imperfect work of art, full of flaws and failures but also of hope and potential, viewed differently by each and every person yet unambiguous in its defining characteristics. It, like me, is a peculiar work of art I will never fully understand but will always strive to improve.  Every time I sit down at the chess board, I’m creating art. And one day, maybe not too long from now, I will play like Picasso.

How to Get Good at Chess, Fast: A simple, step-by-step guide to rapid chess improvement

How to Get Good at Chess, Fast

A simple, step-by-step guide to rapid chess improvement

Last updated: March 1, 2016

Edit: This article was unexpectedly popular, reaching #2 on Hacker News and being linked to on LifeHacker. Thanks for your patience as I work through all the comments and emails I receive.

Magnus Carlsen’s meteoric rise to the top ranked player in the world (at age 19), the highest chess rating in history (age 22), and as of a few days ago, the title of World Chess Champion (age 22) has brought with it a renewed interest in chess. This is exciting, because Carlsen represents the first real hope of renewing chess’s mass appeal since the days of Bobby Fischer1.

In the context of discussions about Magnus Carlsen, many people mentioned that they enjoyed playing chess but quit because of the sheer time commitment it took to get “good.” It turns out there are many misconceptions about rapid chess improvement. In this post I’m going to lay out a simple but effective way to get good at chess, fast.

What does it mean to be “good” at chess?

I define “good” as the 90th percentile among the player pool you’re competing against. In competitive chess in the United States, that means a United States Chess Federation (USCF) Elo rating of about 18002. If you’re a casual player playing against your friends, my guess is that 90th percentile is around 900. Even though I was only rated 1100 when I first began playing competitively, I was already able to beat the vast majority of non-competitive players.

Results with this system

This system is based on lessons learned from my own chess improvement and from coaching others. The good news is that you can become better than the vast majority of other players with minimal but targeted effort.

I actively trained for a period of about 3.5 years using a (much, much less disciplined) version of this system, during which my rating increased from 1100 to 1950, a 135 fold increase I strength3. In one 12 month period I improved from 1198 to 1639. I improved even faster with my quick rating (games with less than 30 minutes per side), where I went from 1001 to 1740 in 15 months (75 fold increase in playing strength).

My first experience using these ideas with other players was in high school, when I began coaching the lowest ranked player in our chess club. Within a few months he had improved so rapidly that he represented the school in the state championships and won every single game in the tournament.

Given that I managed to do this despite my own inexperience and mistakes with studying chess and my own laziness, I’m convinced others can improve much more quickly if they follow this system strictly4.

The system

Since this article is meant for both casual and competitive players, I specify minimum rating requirements when appropriate. If you’re a casual player and this is overkill for your goals, skip to the footnotes for a much simpler system5.

Playing

To improve quickly you need to play often. If you are (or aspire to be) a competitive player, play as many over-the-board (OTB) tournaments as possible. In my heyday I played 3-4 tournaments per month. Online is not enough! Use online games (15 minutes per side or slower) to practice openings or for practice if there is no tournament for a while. If you’re a casual player, play OTB chess with your friends as much as you can, and play online if nobody wants to play with you.

Tactics

I did two types of tactics training. The first was “Chess Vision” and “Knight Sight” exercises, as described in this article. They may sound stupid, but they work. I did these exercises every day for two weeks initially, and then would do them the day of a tournament and once in a while as a refresher.

My primary method of tactics training was using Chess Tactics for Beginners, which is absolutely fantastic.

If you only buy one thing to help your chess game, this should be it. I did 50 puzzles per day, every day, and once I finished the entire CD I repeated the process six more times. Online tactics sites usually don’t cut it, because they aren’t structured so that you learn based off previous ideas and many don’t incorporate the pedagogical features of Chess Tactics for Beginners. Trust me, paying for CTB is worth it. If it becomes too easy for you (which won’t be a problem until you hit about 1600), use it as a refresher from time to time and get Chess Tactics for Intermediate Players.

If I had to recommend a book to accompany such study (which is helpful, since the above software doesn’t actually have any explanatory text), I’d recommend Chess Tactics for the Tournament Player for intermediate players, and Winning Chess Tactics for less experienced players.

 

        

I’ll admit, there is a bit of a leap between solving tactics puzzles and applying it to real games–obviously nobody’s going to tell you when a tactic is available, and you won’t be “primed” to find tactics the way you would be when solving a bunch of puzzles. To counteract this I created a binder of puzzles taken from tactics I missed in my games, and reviewed them from time to time.

Analysis

Analysis is by far the most undervalued part of chess training. As a kid I barely analyzed my games after tournaments, because I was lazy. This was a huge mistake—your games are worth their weight in gold! Learn algebraic chess notation so you can write down your moves, and analyze your games using the method outlined in this article. Use the analysis phase to brush up on your openings and endgames and practice your strategic play. If possible, have a stronger player go over your games with you after you’ve done your own analysis.

One big mistake is to rely heavily on computers for chess analysis. Too often, players use computers as a crutch to replace their own study of the game. Working through games on your own and trying to find the best moves and ideas is highly instructive. Computer analysis should be done only after you analyze the game on your own, so you can compare your analysis to the computer’s and unearth any mistakes you made in assessing critical positions in the game.

Openings

One of the biggest mistakes players make is to devote massive amounts of time to openings. This is because openings tend to be very concrete, and beginners think that simply memorizing an opening will give them an unassailable advantage over their opponents6.

Don’t bother spending any time studying openings outside of analyzing your games. Just make sure you know the basic opening principles. I teach my beginning students simple openings like the London System as white, and a kingside fianchetto system as black7. These openings are simple, solid, can be played against virtually anything.

Once you hit 1600, get a good opening book that gives you both specific moves and the ideas behind the opening. Don’t mindlessly memorize! Some good books here are Alburt’s Chess Openings for Black, Explained (I hear Chess Openings for White, Explained is pretty good but I’ve never used it), and Cox’s Starting Out: 1. d4!8.

         

Obviously this depends on your opening preferences. Even here openings should not be your main focus. I only consult these books when analyzing my games to see where I deviated from established opening theory

Strategy

Until you hit 1400-1500, you should be picking up strategic play from analyzing your games and going over annotated games. Once you hit that level, I recommend Silman’s The Amateur’s Mind and Seirawan’s Winning Chess Strategies. 

         

Once you hit 1800, Silman’s Reassess Your Chess, Fourth Edition.

Endgame

After learning the basic checkmates (King and Queen vs. King, King and Rook vs. King, etc.), Silman’s Complete Endgame Course is the only book you need. Study the appropriate section based on your rating, and only come back to it if it’s clear that you keep messing up endgames.

Annotated Games

Go over at least one annotated game a week (and more frequently if you’re a serious competitive player). A good annotated game book is Winning Chess Brilliancies by Seirawan. I hear the Mammoth Book of the World’s Greatest Chess Games is pretty good too, but I can’t personally vouch for it.

                

Psychology

Magnus Carlsen is my favorite chess player. In equal positions where many grandmasters would agree to a draw, Carlsen patiently pushes and probes, waiting until his opponent cracks and then grinding out a win. Magnus Carlsen is the world’s best player because he doesn’t give up.

When I was younger, I had an unfortunate habit of withdrawing from tournaments where I was doing badly. I made various excuses, but usually I withdrew because I had mentally given up after a few demoralizing losses. I did the same thing in chess games—after making a major mistake, I mentally gave up.

If chess is anything, it is a game of second chances. Chess, like life, rewards perseverance. I’ve turned countless losses into draws and wins because my opponents got overconfident while I dug in. I’ve also turned wins into losses because I was too intimidated by my opponent’s rating or reputation.

Chess psychology can be distilled to two simple rules:

  1. Don’t ever be afraid of your opponent
  2. Fight as hard as you can until the game is over9

Simply following these rules will add hundreds of points to your rating.

General Advice

Study broader topics, like strategy or endgame, only when you feel like that topic is causing you to lose. For instance, only open a strategy book if you keep getting outplayed positionally. Otherwise, your default state should be studying tactics and analyzing your games.

The tl;dr of this training plan is: play a lot, analyze your games, and primarily study tactics. Your knowledge of openings, endgame, middlegame, etc. will come from analyzing your games and going over grandmaster games. Only study one of those specific topics if it is clear you are specifically losing because of that topic.

Recommended Materials

This is just a compiled list of all the stuff I recommended in this article, and rating recommendations for each item.

Analysis

A hardcore guide to analyze your chess games (all levels)

Annotated Games

Winning Chess Brilliancies (1000+)

Mammoth Book of the World’s Greatest Chess Games* (1500+)

Endgame

Silman’s Complete Endgame Course (1000+)

Openings

The books will vary depending on your individual opening preferences

Starting Out: 1. d4! (1600+)

Chess Openings for Black, Explained (1600+)

Chess Openings for White, Explained* (1600+)

Strategy

The Amateur’s Mind (1400+)

Winning Chess Strategies (1400-1800)

Reassess Your Chess, Fourth Edition (1800+)

Tactics

Chess Vision and Knight Sight exercises from 400 Points in 400 Days Part I (all levels)

Chess Tactics for Beginners (all levels up to 1600)

Chess Tactics for the Tournament Player (1400+)

Winning Chess Tactics (1000+)

* I haven’t personally used these items

Tournament Materials

If you’re playing in tournaments, you’ll need three more items: a chess clock (pretty much mandatory, since tournaments don’t provide them), a tournament chess set (sometimes tournaments provide them, sometimes they don’t, and it’s useful to have a set to analyze between rounds), and a scorebook (optional, but highly recommended).

Chronos Chess Clock: This is the clock most serious players use, because it’s built to last. Mine is ten years old and still running strong, despite lots of drops and falls. Two cheaper clocks I bought before my Chronos eventually broke; in the long run, the Chronos is the cheapest clock to buy. Nonetheless, if $100 is too much, I recommend the DGT North American Clock.

Triple Weighted Tournament Chess Set: A chess set is another long-term investment; you want one that’ll last. Weighted pieces feel so much nicer than hollow, plastic pieces, and are less likely to get knocked over during time scrambles when both sides have little time on the clock. However, here’s a cheaper, unweighted set as well.

Deluxe Chess Tournament Scorebook with Lay Flat Binding: This is the scorebook I use to notate my chess games (required in most chess tournaments). The cheaper, spiral bound scorebooks with paper covers eventually rip and tear, while this holds twice as many games (100, versus 50) and lasts forever.

United States Chess Federation Official Rules of Chess, Fifth Edition: This is strictly optional, but the official USCF rulebook is useful to have in case of disputes (for US players, of course). I’ve used my copy to successfully appeal unfair rulings made by tournament directors.

What about all the chess books I already have?

If you’re like many other chess players, you’ve accumulated many chess books that you simply don’t need for rapid chess improvement. My advice: trade them in for Amazon gift cards.

Footnotes:

1 – Fischer’s appeal was that he was a sole American fighting against the Soviet machine that had dominated chess since World War II, and is 1972 World Championship match against Russia’s Boris Spassky was imbued with Cold War symbolism. Carlsen’s appeal is his incredible talent, his youth, his normalcy (compared to Fischer’s infamous egotism and antics) and yes, even his looks.

2 – I couldn’t find recent aggregate percentile data, but the USCF provides percentile data for individual active players, so I determined rating percentiles by looking up individual player ratings. The 50th percentile is around 800.

3 – I stopped playing serious competitive chess about four years ago (when I was 16, rated about 1950) because I got burned out. I still plan on someday making a return to competitive chess, and when I do I’ll pretty much be using this system to train and improve.

4 – I think the only reason I managed to improve reasonably quickly despite being so undisciplined about training was because I was young (my main competitive years were from age 13-16), I played a lot, and I had at least some natural aptitude. How quickly could I have improved if I had followed this system in a disciplined way? Probably about twice as fast.

5 – Here’s a very simplified guide for beginning players who want to improve rapidly in a month or two

  1. Learn the basic opening principles: control the center, develop your pieces, and king safety. Googling this should yield useful articles.

  2. Learn the basic checkmates: King + Queen vs King, King and two Rooks vs King, and King and one Rook vs King

  3. Get Chess Tactics for Beginners and do 50 puzzles a day

  4. Do the Chess Vision and Knight Sight exercises from 400 Points in 400 Days Part I

  5. Play as much as you can

  6. If possible, go over your games with a stronger player

6 – A lot of this is just to impress other players. It’s a common sight at chess tournaments to see players rattling off complicated sounding opening variations. At first these players intimidated me, but as I grew stronger I realized that these players were often the easiest to beat. Just get ’em out of the openings and crush ’em with tactics!

7 – This setup involves the moves Nf6, g6, d6, Bg7, and O-O, resulting in a setup as seen below. Experienced players might point out that this could lead to the King’s Indian Defense or the Pirc Defense, which turn out to be rather complicated openings. This is true, but you can play both these openings with little theoretical knowledge up to the 1600 level and still be fine.

8 – Incidentally, I own but don’t recommend the book’s counterpart, Starting Out: 1.e4!

9 – This doesn’t mean never, ever, resign. If you’re down a queen in an absolutely hopeless position against a strong opponent, it’s good etiquette to resign rather than needlessly drag on the game. A good rule of thumb is to ask yourself, “If my opponent were playing Magnus Carlsen in this position, would Carlsen be able to win?” If the answer is yes, keep playing. If it is no, then resign.


If you’d like to receive my future posts, including other posts on rapid chess improvement, in your inbox, enter your email below:

You'll receive all of my future posts in your inbox.

 

Disclaimer: The Amazon links in this post are affiliate links, which allow me to earn a commission of approximately five percent if you purchase using that link (you pay the exact same amount as you would via a non-affiliate link). Thanks for supporting this site!

FAQ About My Name

FAQ About My Name

I get a lot of questions about my name. Here’s a handy list of answers to soothe your inquisitive mind so you can finally sleep at night.

How is your name pronounced?

“Gotham,” like Gotham City from Batman.

Wow, that’s really cool!

That isn’t a question.

Is it really pronounced like that?

I often hear this question. Indians or people of Indian descent assume I’m “Americanizing” the pronunciation of my name because they’ve heard a different pronunciation. Non-Indians assume I’m not giving them the “authentic” pronunciation because it would be too difficult to pronounce.

There is actually a spectrum of valid pronunciations of my name, based on how strong an inflection you want to put on the ‘au’ sound. On one end is my pronunciation, “Gotham,” which pretty much eliminates the inflection. On the other end is “Gow-tham,” a pronunciation which is common in south India. There is a range of pronunciations in between those two, and they are all correct.

No, but is that really how it’s pronounced?

Yes.

Really?!

Yes.

You?!

Nice Simpsons reference. That was a good episode.

So did you choose that pronunciation because of its relation to Batman?

I didn’t choose anything. My parents are north Indian, and north Indians tend to put less of an inflection on the ‘au’ sound. I’ve been called “Gotham” since I was born. They did not realize that I would forever be associated with The Dark Knight when they named me.

Why isn’t your name spelled with an ‘h’?

Probably because of British imperialism. North Indians pronounce the English “th” with a really strong emphasis (I don’t know how to write it out, but try making the “th” sound and then do it much more forcefully). To avoid having people pronounce my name with the overpowered “th,” some North Indian transliterated my name from the Devanagari गौतम to the Roman script “Gautam.” They were probably transliterating the name in the first place since English became the lingua franca in India following British colonization. My mother’s name also has a ‘t’ that is pronounced ‘th’ for this reason.

Nonetheless, I have seen alternate spellings, such as “Gowtham.” Deepak Chopra’s son even changed the spelling of his name from my spelling to “Gotham” to end the mispronunciation.

Personally, I prefer my name’s spelling. Although it isn’t phonetic, I find the combination of letters aesthetically pleasing. I get a better vibe from “Gautam” than I do from “Gotham.”

Edit: Gautam Arya pointed out that the reason Gautam isn’t spelled with a ‘th’ is because  the Hindi letter त is a voiceless dental stop.

Are you sure it isn’t spelled ‘Guatam’?

Yes. Please stop sending me emails beginning with, “Dear Guatam,” especially when my correctly spelled first name is part of my email address. I’d also appreciate my elementary, middle, and high school reprinting all the certificates awarded to “Guatam Narula.” Please. Those certificates comfort me on those lonely nights at sea.

What are other misspellings/mispronunciations?

Guatham, Guaram, Garfield, Gus (Seriously. Gus.), Garfam, Gotham, Gautham, Guatham, Gaytam, Guavam, Gutam, Gautum, Gowtam, Gotem, Gautman, etc.

What nicknames emerged from your name?

Batman, Got ham?, Guantanamo Bay, Guantum Mechanics, Guatamala, Got’em, Siddhartha Guatananama, Gotham City, Goatman, and every single mispronunciation.

What does your name mean?

I was named after Siddharth Gautam (Siddhartha Gautama in the west), the founder of Buddhism. My parents almost named me Siddharth, but didn’t want me turning into “Sid.” The name means “The Enlightened One” or alternatively, “The Remover of Darkness.” In both cases the “light” symbolizes knowledge and the “darkness” symbolizes ignorance. Basically, I’m the Human Torch from the Fantastic Four, if his super power was educating others.

So does this mean you’re Buddhist?

Nope. My parents aren’t Buddhist either. Guess they just thought the dude was legit or something, man.

Is your name common in India?

Not super duper common, but common enough that I can’t get a decent Twitter username.

So there are other Gautam Narulas?

https://www.youtube.com/watch?v=lKie-vgUGdI

What was it like watching the Batman movies?

Weird. When Gordon said, “Gotham needs a hero.” I felt the need to tell him I was perfectly capable of taking care of myself. I didn’t appreciate Bane saying stuff like “I terrorize Gotham,” “We will destroy Gotham,” “We take Gotham from the corrupt!” etc. Seriously bro, what’d I ever do to you?

What are the best pickup lines related to your name?

“Hey baby, I may not be Batman but I can give you a tour of Gautam city.”

“Ay girl, you wanna see downtown Gautam?”

“Hey beautiful, what are you looking at him for? You’ve already Gautam man you need.”

Do any of those actually work?

I’ll leave that as an exercise for the reader.

What Javier Taught Me

What Javier Taught Me

In March, I traveled to Costa Rica. The purpose of the trip was to gain a better understanding of how globalized surf culture and surf tourism impacted the country and its people. We spent a week speaking to locals and foreigners, exploring towns and villages along the Pacific coast, observing conservation efforts, meeting with community leaders, and discussing what we saw. I could write a lengthy post about everything I learned.1 But there was one event that stood out from the rest of the trip. It was a conversation with a man named Javier.

Javier grew up in a wealthy family in the capital, San José, where his life consisted of country clubs and tennis lessons. When he was 18, Javier traveled to a small coastal village of 300 people. He camped under the stars with his friends, explored the surrounding forests and mangroves, and fell in love with the place. He moved there permanently, opened a small surf shop, and taught himself how to surf, and gave mangrove tours.

“What I like about this place,” Javier told me, “is the whole town is one community of friends. If someone is getting married, we all come together and celebrate. If someone dies, we all go to the funeral.” This philosophy inspired the way he ran his business. “If somebody forgets their credit card or their money, I’ll give them their surf gear anyway. I trust people.”

For many, including members of his family, Javier’s decision to forgo the material comforts of San José was puzzling. But Javier’s philosophy was that he was happier not in spite of having less, but because of it. I didn’t have to take his word for it—I could see it with my own eyes. The man radiated positive energy. He was always smiling, and his skin seemed to glow. Although he was a week shy of 43 years old, Javier looked no older than 25. This was the first time I had ever met someone who seemed truly fulfilled.

I kept thinking about our conversation long after I left Costa Rica. Javier left his old life behind to pursue what held meaning for him. What about me? What held meaning for me, and was I pursuing it? The whole grind of college—writing papers that would only be read once by my professor and then discarded, writing code that would only be run once by my TA and then deleted, taking tests that encouraged rote memorization and regurgitation, building a resumé whose sole purpose was to get me a job where “success” is determined by prestige and money—just left me unsatisfied. Where was the real learning? Where was the impact?

I knew what I wanted. I wanted to make the world a better place, and I wanted to do it in a meaningful, impactful way. I found an essay by Paul Graham where he created a list of five commands that he keeps on top of his todo list. After reflecting for a few days, I modified them to create the five commands that I keep on top of mine:

Build things that help people2

Live healthy

Be happy

Help others be happy

Do new things

What can Javier teach you?

 

1) Just a sample of the other topics I also learned about: sustainability efforts and conservation, community activism, the effects of market economics on communal societies, how to surf, “soul surfers” vs competitive surfers, local vs foreign control of land/resources/money, localism and sentiments towards foreigners, income inequality, mating habits of turtles, etc.

2) I use “build” to loosely mean anything you create. You could build a movement, or an app, or a friendship. I’m building a book.

How to Eat Dinner with Barack Obama

How to Eat Dinner with Barack Obama

When I was a kid, my favorite TV show was The Simpsons1. During the commercial breaks I often saw ads for contests associated with some product, i.e. “Mail in the box top from your bran flakes to be entered in our $10,000 sweepstakes!” While the ads avoided running afoul of any illegal lottery rules by including a “no purchase necessary” clause in the fine print at the end, the hope was that you buy more of their product to be entered in the sweepstakes.

The Obama campaign did something similar during the 2012 election. They ran several “Dinner with Barack” sweepstakes, where entrants could win a personal dinner with President Obama. For every donation you made to the campaign, you were automatically entered into the sweepstakes. At the bottom of the donation page was the familiar “no purchase necessary” fine print, along with a link to enter the competition without donating. I entered myself a few times (who wouldn’t want a chance to speak directly to the President?), tediously reentering the information and submitting the form each time. As I did so I thought, “Why don’t I write a program to do this for me?”

Once I had the idea, I immediately noticed that there was a no CAPTCHA or any other method of preventing a bot from submitting the form. This meant I could write a 20 line Python script to fill out all of the form fields and use a POST request to submit it2. The Obama campaign would obviously be suspicious if the same person or IP address submitted 20 million forms in one second, but the script could be throttled to submit less often, perhaps once every ten seconds. Since these sweepstakes were usually announced about two weeks before the deadline, I could generate:

1 submission per 10 seconds = 6 submissions per minute= 360 submissions per hour =8640 submissions per day = 120,960 submissions in two weeks.

There were several of these competitions, so it’s possible that among them all I could have entered a million times (and perhaps even more, depending on how much script throttling is necessary to avoid detection). And if I could get several computers with different IP addresses submitting entries at the same time, my total entries could be on the order of tens of millions3.

Would this have worked? Although it would have been a lot of fun to try this out, I never implemented the script because of ethical concerns4, and I kept it to myself so others wouldn’t try to use it as well. When I read through the official rules and all the fine print, I found no prohibition against using an automated system. Given that the campaign staff had far more pressing concerns, they may have overlooked this simple exploit. But that is still surprising considering the Obama campaign’s reputation for technical prowess. Perhaps I should tell the President about this over dinner.

 

1 The Simpsons isn’t designed with kids in mind, but the show’s clever wit and gentle satire really resonated with me. As a longtime fan it saddens me to see the show steadily go downhill. The best season was undoubtedly season five, which came out in 1993 (the year I was born!).

2 Writing the script would have taken no more than an hour.

3 To further avoid detection, each instance of the script enter slightly different information, like “Gautam Narula”, “Gautam R Narula” “G. Narula” and multiple (valid) email and mailing addresses.

4 It probably wasn’t illegal, but this exploit definitely exists in the gray area which hackers (in the programming sense of the word, not the steal-your-identity Hollywood usage of the word) spend much of their time in. I felt it was against the spirit of the competition to use a bot to enter myself hundreds of thousands of times in the competition. Then again, maybe that’s why I never ate dinner with President Obama.

 

Building a Rating System

Building a Rating System

Most games today have a poor ranking system. Take tennis. The ATP ranking system is based on “points”, which are awarded by participating in select tournaments in the previous 52 weeks. The system is needlessly complicated and somewhat arbitrary, and does not directly take strength of opposition into account. The upper echelons of Halo and Call of Duty online rankings are, according to a friend, often populated with mediocre players who just play a lot of games, a result of a system that rewards sheer quantity of play over quality of results1. I played high school quizbowl, and the system for ranking the teams was subjective and fairly arbitrary. The top 20 or so teams were fairly accurate because people knew who the best teams were, but beyond that it was guesswork based on unreliable metrics2.

Chess has a different rating system, one that nearly every game and sport should use. The underlying premise of the Elo rating system, named after its Hungarian inventor, is that the purpose of a rating is to be able to predict the outcome of future games. Philosophically, it was based on a big data approach before big data existed–the only thing that should determine the rating formula is the corpus of games played. With enough prior games, Elo contends, you should be able to calculate the probability of any player beating any other player. The Elo rating system is better than many other ranking systems for several reasons:

  1. It rewards the quality and consistency of results over sheer frequency of playing. Three good results will gain more points than ten mediocre results.
  1. It provides a way to accurately predict the probability of one player beating another. In a two player Elo system, a player rated 100 points higher than his or her opponent has a 64% probability of winning3.
  1. The ratings directly account for the opposition strength. Beating strong opponents gains more points. Losing to weak opponents loses more points.

Developing the Algorithm

My friends and I started getting into the board game Settlers of Catan towards the end of my senior year of high school. We played a few games every week and would get pretty competitive about it. We’d argue over who was the best but had no metric to determine that other than, “well, he seems to win all the time.” After our first semester of college, we all came back home for the break and resumed playing. Inspired by chess ratings, I sought to create a rating system to put an end to the debate once and for all. I spent a day thinking about the various algorithms to use and a night coding an app that I later called the “Game Rating Calculator.” The rest of this post explains the process I went through in creating a rating algorithm, and the results after nearly a year of using it.

When I was developing a rating algorithm for the Game Rating Calculator, I was inspired to create a system modeled after the Elo system. I couldn’t port the Elo formula exactly, since it was designed for only two player games. I looked at a few approaches before settling (pun intended) on the final formula.

There are many variations of the Elo formula4. For the sake of clarity, all references in this post will be to the following formula:

Rnew= Rpre+ K(S-E)

Where Rnew is the new rating, Rpre is the pregame rating, K is the “k factor”, an arbitrary multiplier (a higher k factor means higher rating volatility), S is the total score in a rated event, and E is the total expected score in a rated event, which is calculated by:

formula

where Ropp  is the opponent’s rating.

Assumptions:

I took my base scenario to be a four player game of Settlers of Catan, where all players were of equal strength. Intuitively, each player’s probability of winning the game is 25%. Assume that A came in first, and, if it is possible to have a second, third, and fourth place, then B came in second, C came in third, and D came in fourth. To determine a player’s opposition strength, I take the average rating of all of the opponents. All calculations using an Elo formula are done with a K factor of 16.

Approach #1 – The Pairwise Approach

The pairwise approach tries to get around Elo’s inability to work for more than two players by treating multiplayer games with one winner as a series of pairwise matches. In a four player game with players A, B, C, and D, if A won the game, you would rate the game as the result of three individual matches: A vs B, A vs C, and A vs D. But that leaves us with a problem: A has “played” three games, while B, C, and D have played one. The official online Settlers of Catan website and ranking system attempts to rectify this by taking into account relative positions in the game. That is, the second place player (determined by the number of points at the end of the game) will have “played” three games, losing against the first place player and winning against the third and fourth place players, and so on.

There are several problems with this approach. In our hypothetical case of four equally rated players playing a game, each player should achieve each possible position (first, second, third, and fourth) 25% of the time. In this case, the winning player’s rating was calculated as if he won three games in a row. Since each of his opponents is equally rated, the probability of him winning any individual game is 50% and the probability of winning three games in a row is .5*.5*.5=.125=12.5%. This means the formula acts as if the winner has an accomplished a task that is twice as difficult as it really was, and therefore rewards too many points. Similarly, the last place player will lose too many points, because even though his odds of coming in last were 25%, his rating will be calculated as if he lost three games in a row, which only has a 12.5% chance of occurring.

We can use the binomial theorem to calculate the implied odds for the second and third place players. The probability of winning two out of three games against an equally rated opponent (second place) is 3C2(.5)2(.5)1 = 3*.125= .375 =37.5%. Similarly, for winning one out of three games against an equally rated opponent: 3C1(.5)1(.5)2= 3*.125= .375 =37.5%5.

We can summarize the results in the following table:

Position Actual Probability of Achieving Position Implied Probability of Achieving Position Through Pairwise Ratings Result
First 0.25 0.125 Overrated
Second 0.25 0.375 Underrated
Third 0.25 0.375 Overrated
Fourth 0.25 0.125 Underrated

We can determine the results using a standard elo calculator with a k factor of 32. Let all four players be rated 1000. Their new ratings would be as follows:

First place 1000 → 1048

Second place 1000 → 1016

Third place 1000 → 984

Fourth Place 1000 → 952

One could argue that even though the implied expected scores are off, since the players are all equal in strength, things should “balance out”. That is, in this example, the last place player is expected to come in first as many times as he comes in last, so things should even out. And if he doesn’t then he really deserves the lower rating.

The problem is that this really messes with the predictability with elo ratings. In that one example, there is now a 96 point difference between the first place player and last place player, which is so big that, after just one game, A’s predicted probability of beating D head-to-head had jumped from 50% to 63.5%. Even with a lowered K factor dampening the rating chance, the probability would still be overly optimistic for A. And since Elo uses the predicted probability (expected score) for calculating rating changes, this can mess with subsequent rating changes. And even if all the kinks could be worked out, it seems sloppy to use incorrect probabilities.

From a practical standpoint, this system also has two major problems.

  1. In a game where there is only a winner and multiple losers, this system wouldn’t work. In Settlers of Catan, the player with the second most points at the end of the game is not necessarily the second most likely to win. Games where a player in third or even fourth “leapfrogs” to first and wins the game are not uncommon.
  1. Having ratings dependent on positioning is ripe for manipulation. For instance, if you were were in dead last in a game of Settlers, you could offer to throw the game in favor of another player in exchange for help in moving up from last to place to third. This kind of manipulation is incentivized under this system.

Approach #2 – The Proportional Rating Approach

Elo operates on differences between actual performance and expected performance. As I studied potential solutions to a multiplayer Elo algorithm, I realized that if I could develop a way to determine expected performance in a multiplayer game, I could just plug it into the two player Elo formula. One way of doing this would be to give each player an expected performance equal to the proportion of their rating to the sum of the total ratings. If there were four players with ratings R1, R2, R3, R4, each player’s individual expected performance would be their rating divided by (R1+R2+R3+R4). In the case of our hypothetical example, each player’s expected score would be 1000/4000=.25, which is what our intuition expects.

The one issue with this approach it implies fairly slow improvement. Winning five games in a row is a pretty big deal (after all, in 4 person games the odds of that happening are .255= .09%), but the rating gain would only be 65 points, which means that in a sixth game against 1000 rated opponents the expected probability of winning would only be 1065/4065= 26.2%,when it clearly should be higher.

As for evaluating its accuracy, there was really no way to test if the expected score was reasonable other than to eyeball it for completely inaccurate values. In a four player game with players rated 2000, 1000, 1000, and 1000, the first player has a 40% chance of winning, which seems reasonable for a player who is twice as good as each of his individual opponents. The only problem is that it would have taken so many wins to get to 2000 that the player is almost certainly underrated to begin with.

Approach #3 – The Modified Expected Score Approach

There was one final approach to examine, and this was the simplest of them all. Take our base scenario of four players, each rated 1000. The expected score of each player is .25, while the expected score in a one game, pairwise comparison is .5. In a five player game, the expected score is .20, while the pairwise comparison score is .5. In a six player game, the expected score is .17, while the pairwise comparison score is .5. The expected score for a game of N equally rated players can be generalized as:

2Ep/N

where Ep is the expected score in a pairwise comparison. The key question was, could I expand this formula for the expected score beyond the case of all players being equally rated? I added this multiplier to the Elo formula and tried it out with a few hypothetical games, and the rating adjustments seemed to make sense. But the only true way to establish this was to actually use the system in production. After all, Elo was revised over 40 years of real world usage (though others have found better algorithms). There were two things I really liked about this formula:

  1. It’s point neutral among established players (not counting rounding errors)6 .
  1. It gives a reasonable point adjustment for different size games.

This seemed to be the most promising approach and the algorithm I ended up using. The only issue that might arise is how the ratings would operate in the case of a player rated far above the rest. In a four player game, the theoretical maximum expected score is .5, which means that even if a player were rated one million points higher than each of his opponents, he’d still gain a sizable 16 points from winning. This issue only came up in such unrealistic scenarios, so I wasn’t too worried about it.

Provisional Ratings

The United States Chess Federation has a concept of provisional ratings. For the first 25 rated games, players’ ratings are more volatile7. The idea behind this is that it will more quickly get players to their “true rating”. It also takes into account that moving from casual chess to tournament chess is a disruptive process that has a learning curve, and that a player’s strength will be more volatile as he moves up that curve.

I didn’t include such a system in the rating calculator for a few reasons. First, it is easy to play such a large number of chess games, which makes reaching 25 fairly simple. In a game like Settlers of Catan, where you need to gather at least three (usually more) people for 90+ minutes, this isn’t practical. In the rating pool I set up, only five people had played 25 or more games after 11 months.

There is also a real risk of rating deflation, depending on the number of provisional games. For example, let’s say a provisional rating is twice as volatile as a non-provisional rating. That is, you will gain or lose twice as many points if you are provisionally rated. If there was just one provisional rated game and an average game of Settlers had four players, 75% of the players would have their ratings deflated, since 75% would lose their first game. That 75% would get only half the points they would have otherwise gotten when they win later.

The idea is to tailor the number of provisional games to the point where players are equally likely to get their wins and losses in that set of provisional games as they are to win and lose in general.

The provisional period is five games in my rating pool. At the time, the average game seemed to be around five players. We’ve played many more games since then, and the average has dropped to four players per game. In the first four games, an average player is expected to win one game. But in the fifth game, 75% of the players will lose. In this case, 75% of the incoming players have a deflated rating. Unfortunately, due to a desire for consistency in the rating system, I can’t dynamically adjust the number of provisional players each time. I may retroactively recalculate the ratings based on an average game size of four, since that has held stable for a while.

 Results

It was time to put this rating algorithm into production, and see how it held up. As of this writing, 80 rated games have been played, and a total of 28 players have played at least one rated game. Below are all the results for the players who’ve played enough games to not be provisionally rated (6+).

Player Rating Games Played Average Game Size Winning Percentage* Expected Winning Percentage Net Difference **
K.S. 1073 44 3.82 40.9 26.2 14.7
W.A. 1052 13 4.85 38.5 20.6 17.8
G.N. 1052 38 4.37 31.6 22.9 8.7
A.S. 1028 60 3.72 25 26.9 -1.9
K.S. 1012 41 3.88 31.7 25.8 5.9
I.M. 984 16 4.5 25 22.2 2.8
M.R. 977 7 5 14.3 20 -5.7
A.S. 965 10 4.9 10 20.4 -10.4
G.M. 937 9 4.44 11.1 22.5 -11.4
C.W. 934 6 4.5 0 22.2 -22.2
S.S. 889 29 3.66 13.8 27.4 -13.6
B.S. 859 12 4.25 0 23.5 -23.5

Average: 980.2

Median: 980.5

Average game size of all games played: 3.93

* The Expected Winning Percentage is based on the average game size, assuming all other opponents were equally rated

** The Net Difference may not match Winning Percentage- Expected Winning Percentage due to rounding

At first, I arbitrarily started off all players at 1000. Once there was a decent sized pool, I started off all new players at the median rating of non-provisional players. The median rating has dropped over time, probably because of the deflationary effects of the provisional ratings I mentioned earlier.

Overall, the results look pretty good. The ratings intuitively correspond with how I feel most people should be rated. Importantly, the ratings fit one of the main criteria I wanted in a rating algorithm: it awards the quality and consistency of results, and not just the sheer quantity of games. For instance, W.A. and I are rated exactly the same. He has a much better net winning percentage than I do, but that is compensated by the fact that I’ve maintained a decent percentage for far more games. His strong results are balanced by my consistency. There is one obvious outlier here, and that is A.S. A.S. used to be much lower rated (below 900) but has recently surged with a lot of strong results. One possible explanation for his high ratting and low winning percentage is that he played a lot of games with high rated players. I haven’t calculated each player’s average opponent’s rating, so I can’t say for sure.

I’ll eventually put this all in a SQL database to automate some of the data gathering. But the best way to improve the rating system would be to run data analysis algorithms on the results. How accurately do the ratings predict results? Unfortunately there is still not enough data. Heuristically, I’d think there would have to be a rating pool on the order of at least 25 people who have played at least 25 games to accurately assess the system.

But for now, this algorithm will do. If you’re interested in learning more about the Elo rating system in order to develop your own, check out the book by Elo himself, The Rating of Chess Players: Past and Present.

Tl;dr: Download the desktop version and Android version of the app.

 

1 But then again, that’s what the game makers want. These days games usually have some sort of in-game store or revenue steam, so more play usually means more money.

2 There have been some attempts to institute a chess style elo rating for quizbowl, but the ratings aren’t very accurate since data is sporadic.

3 Assuming you are using the Elo formula mentioned in this post.

4 If you’re interested in learning more about the Elo formula, here is a good treatise about it by Mark Glicko, the chief statistician in charge of the United States Chess Federation implementation of the Elo formula.

5 You could have also intuitively determined this result by noting that the probability distribution for the four possible outcomes were symmetrical, and since winning three in a row and losing three in a row are each .125, the other two possibilities must add to .75, hence .375.

6 It’s interesting to note that Elo system used by the United States Chess Federation and FIDE (the world chess federation) are not point neutral, because they have a K factor that declines as rating goes up. This was apparently used a way to stop rating inflation at the top of the rating pool. Not all chess organizations agree with this—the Internet Chess Club, the premier online chess server, does not use variable K factors outside of provisional games.

7 The volatility decreases within the set of 25 games.


If you’d like to receive my future posts in your inbox, enter your email below:

You'll receive all of my future posts in your inbox.