"More broadly, I think Valett can provide the foundation for answering other interesting questions in word games, such as how to quantify the difficulty of Boggle boards."
Well, you can already quantify the difficulty of Boggle boards pretty well just by counting up all the points you can make on them (it's a small enough problem that you can do that pretty fast). I think the topology of the board is important enough (whether those I N and G blocks are next to each other or not makes a _big_ difference) that a simple histogram of letters isn't going to be sufficient.
I was thinking that you could sum up the transition probabilities from the actual transitions available on the board as the main measure. Then you could use frequency by length to weight legal Boggle plays (3/4 letters and up, so 2-letter words wouldn't even count).
You're right that with a Boggle board you could just count up the available Boggle points by finding all the possible words, but that might miss some aspects of how hard the words are for a human to find.
If you had enough real games data you could see how often a word is found when present and use that is a weighing factor. Although the arrangement if a word might also impact the likelihood to be found.
Changing the values of tiles will change the game, so you'll have a new mismatch. I think that a simpler fix would be to ban plays of fewer than three or four tiles are placed (I'd say four since three is still too easy). You can still score off the stupid two letter gotchas, but you have to place a three or four letter word down to score anything.
The game would end when all players had two few tiles to play.
For valuing Scrabble tiles, Valett also needs to include the representation of the letter in common prefixes and suffixes that can be prepended or appended to existing words. I believe that one reason that G is so playable is the common -ING suffix.
I believe that is part of what his definition of “ingoing and outgoing entropy” is refering to. That said, it does seem weird to heavily bias for the 2-3 letter and 7-8 cases. I would think the moderate length words should not get such short shrift, since these words tend to be the ones used to determine when the various letter and words value tiles get hit in any given game. (Full disclosure: not a regular Scrabble player.)
Competitive Scrabble player here :) -- the reason the C goes down to two points despite having no two-letter words is that it is actually a very good letter - basically, it's in a lot of high-probability bingos. In contrast, the V is one of the worst tiles in the game. It's telling that the Q stays at 10 despite QI -- the Q is by far the worst tile in the game.
In any case I've always wondered what the values of tiles should be if we were to use our current word list rather than the NY Times - thanks for this analysis. The blank is missing though, it would probably have a significantly negative value (-25 to -30?).
At the end of it all I'd rather keep the values what they are. Part of the beauty of Scrabble is the fact that there is chance. Despite the Z and the Q both being worth 10 points, one realizes pretty quickly how much worse the Q is.
Yes, my distaste for C is probably because I'm not a good enough player, haha. I received an email mentioning the CLARINETS heuristic for choosing tiles to leave in one's rack, and in that context the drop in C's value makes sense, as you say.
as a competitive scrabble player, one mistake that I think he might have made is overweighting corpus-based probability versus game playability. his transition-in and transition-out weights are a good start, but there's also the fact that n+1 letter words that can be made by "hooking" (adding a single letter before or after) n letter words are far more useful than those that cannot. also the layout of the premium squares, and the letter distribution of the bag, factor into how playable certain tiles are. intuitively, at the least I'd expect one more point for the U, and for the V to catch up with the Z, though of course it's very easy to fool yourself about these things when using strategies based on the current letter values.
Maybe you and the OP should petition the Words With Friends people to give you a statistically significant data-dump of real games so you can analyze and revalue the letters.
There would be a lot of factors to consider that might make it hard to apply to re-valuing Scrabble, though:
1. Most Words With Friends play is casual, and casual players are likely to play very differently than competitive players.
2. The data would show you what words players play given the current scoring system, and wouldn't necessarily translate to another scoring system.
3. Words With Friends is a different game than Scrabble (different board layout, different bingo scoring, different number of tiles and different tile frequency).
4. Players end up playing the majority of the tiles they draw, so the frequency with which they play a letter may have a closer relationship to how many of the printed tiles have that letter than it is to the frequency of the letter in the corpus.
Regarding 1: perhaps one could segment the players based on their scores --- the people I know who play scrabble competitively generally have a significantly higher average word score.
on the granularity of an individual game there's a surprising amount of luck even in the top levels of the game. for instance, i'm at best a high B-division player, but i've won tournament games against two world champions (and, on the flip side, lost to players rated far lower than me). it's the overall performance in an (ideally 15-game+) tournament that lets the best players consistently rise to the top.
The luck element is a result of the malapportionment that the OP mentions (it's good to receive the overvalued tiles, and who receives them is down to luck).
that's part of it, but not nearly as large a part as you'd imagine. there are ways to defend against, e.g., someone getting the X both ways on a triple letter score (the most common large "tile lottery" moment). harder to overcome is someone simply drawing one bingo after another (possibly by being lucky with the blanks and Ss), getting an early 100-200 point lead, and then simply closing the board down (both players have low-scoring moves thereafter, but you already have the lead), or having a close-fought game be irretrievably lost because you get a final rack with six vowels, or none, or an unplayable Q that hits you with a 20 point penalty and let's your opponent play his final rack out letter by letter, for a large number of points.
Quite frankly, the words with friends board layout and tileset was so badly balanced that after then first couple of weeks I could no longer stand to play it. I think a better route would be to have quackle play itself millions of times and get data from there (it's a reasonably common thing for people developing scrabble heuristics to do nowadays).
I've come across words in the OSPD that Words with Friends didn't take, as well as words not in the OSPD that it did take. A casual Google suggests that Words with Friends uses a modified version of a dictionary called ENABLE[1], so while it would be interesting data to see, the dictionary difference is a little troublesome for conclusions pertaining to Scrabble.
(You can, of course, play Scrabble with any dictionary but most serious folk I know use OSPD. I'd never even heard of ENABLE until Googling this question.)
The problem, of course, is convincing us "old hat" Scrabble fanatics to change our minds on the point values. "Only 5 points for an 'X'? Nonsense!". Then again, competitive Scrabble and the original foundation of the game have little in common, methinks. Rather frequently I think about the Scrabble-likes (Words with Friends, Literati, etc) and the boards themselves. I am more curious about the designs and placements of the premium squares than the value of the tiles.
Little known is the fact that Butts actually did some manual fine-tuning of the frequencies he obtained from NYTimes. E.g., he decreased the number of S's to four in order to mitigate its affixability to almost every singular noun and enhance the gameplay.
It makes a lot of sense to reduce the number of Ss for the sake of gameplay, and it seems like Butts redistributed those extra tiles amongst the vowels.
Good analogy, actually. There is a popular perception that being good at chess indicates skill at strategy or reasoning. In reality, these are helpful, yes, but mostly what it indicates is that you know a lot about the game of chess. Similarly, there is a perception that skill at Scrabble indicates a large English vocabulary.
It's reasonable for someone with good reasoning or a large vocabulary to find it distasteful that they lose consistently simply because they haven't memorized the 1,000 most common moves.[0] In chess, this is pretty much built into the mechanics of the game, but it's easy to fix in Scrabble with a simple house rule: Don't use bullshit words.
It's only part of the game if you let it be.
[0] I'll give an illustrative example. The official Scrabble dictionary[1] lists "re" with the definition "the second tone of the diatonic musical scale". Just so for the other sounds of diatonic solfeggio that everyone knows: re, mi, fa, sol, la, and ti. All of these, according to Hasbro, are English words, with plurals.
We're all descriptivists here, that's fine. But did you ever wonder why there are only seven words for the diatonic scale, but actually twelve tones in an octave? Or why re, mi, and fa have different vowel sounds? The answer is that solfege has a way to verbalize accidentals (notes not in the key) or non-diatonic scales by modulating the vowel sound. A raised do is a di; a lowered re is a ra. So you can easily pronounce a chromatic scale beginning at do: do, di/ra, re, ri/me, mi, fa, fi/se, sol, si/le, la, li/te, ti, do. (The syllable used depends on the key.)
Go ahead, look up any of those accidental syllables in the dictionary. La and le: Both French articles, both solfeggio syllables. Only one is a Scrabble word. You can't say that's not stupid.
You've just moved the line of argumentation: instead of "is the word in the dictionary we're using", it's "is that word a bullshit word" -- but only one of those questions is answerable definitively. The only way to play without argument is to use a dictionary, at which point you might as well use the standard one.
(Like another poster in this thread, I do also play friendly games with an open two-letter-word list, to level the playing field for people who don't care to memorize trivia.)
I gave the example to illustrate what I mean by "bullshit word", but the question is more or less this: Are you using this word because it's a word you know, or because it's a word you found in a list of Scrabble words?
I'm not saying that the idea of a canonical word list doesn't make game-mechanical sense, just that it turns Scrabble from a word game into a list of symbols game in a way that seems stupid to me.
It's not meant to be more than an intuitive house rule to encourage people to play with "real" words.
It's a demonstration that scrabble's dictionary uses completely arbitrary and unpredictable methods to choose which bullshit words it has. It can't even use the excuse it's trying to be 'complete'.
That, and stalemate ("what do you mean, a tie? You can't move, so you lose!"), but that is a rarer occurrence, as it requires the neophyte to be on the winning side (he won't complain if he is the one without a legal move, and you announce it a tie)
I had never even heard of that (although I'm by no means an avid chess player). I think I'll have a hard time convincing my friends that's a real rule.
The Wikipedia article is good at explaining the rationale. According to it, it was added around the time the two-space opening move for a pawn was added to balance the game.
Oddly, though Wikipedia deletes programming languages like Factor due to non-notability[1], editors get to go nuts on everything chess or Star Trek without recourse. If you want to know more about why things in chess are the way they are, Wikipedia is your pal.
It's a little different than chess. Chess rules can be laid out in a few thousand words, including castling.
Right off the bat, you need to pick a set of words that are allowed. Tournaments play, of course, allow a small set of those odd two letter words. However, both sides have to agree on a word set up front. it's perfectly reasonable to pick a dictionary that doesn't recognize "za" as a real word for pizza. However, i think if transfire really wants to use "fu", he'd have to resort to urban dictionary for the word list. That also admits za, so it's kind of a lose lose there.
As a house rule (since none of us have any plans to ever competitively play Scrabble), the list of all legal two-letter words which came in our "deluxe Scrabble" instruction manual is available to all players at all times.
It has really helped cut down on the number of games where we get stuck or frustrated because there's "nowhere to move", both by making more moves legal, and helping us keep the board configuration more compact, while at the same time eliminating what we see as trivia (what 2-letter words are legal) vs. "regular English" in the games we play.
one of the world's largest tournaments, the annual king's cup tournament in bangkok, actually supplies 2, 3 and selected 4 letter word cheat sheets to all the players, to compensate for the fact that thai players use the north american rather than the world dictionary.
Aside; It's funny: I never though anyone actually used the word "za" seriously (for "pizza") until I saw it used in an episode of "The Office." When we saw that, my wife and I burst out laughing. Now we use it all the time ironically. In any event, I love the two letter words. For me, they give the game strategic depth.
Been playing Scrabble casually for about a year, and I've found two letter words are where you get the most points. If there's a vowel beside a triple letter, often you can stick an X on there and get it 6 times. And X's have a ton of two letter words, at least in the dictionary I've been using. So maybe they're "offensive", but they seem to be a great way to score points.
That's what makes them so offensive. If you just got a tiny advantage from these obscure words it would be fine, but with the tournament scrabble ruleset, any player with higher strategic skill but who hasn't bothered to rote-memorize the wordlists will lose. Which makes it an unfun game.
One advantage of using the NY Times over the full OSPD for determining letter scores is that it gives precedence to words that people know and use. Casual players don't memorise lists of two letter words. Having imperfect letter scores adds an element of luck to the game which can make it more fun.
Well, you can already quantify the difficulty of Boggle boards pretty well just by counting up all the points you can make on them (it's a small enough problem that you can do that pretty fast). I think the topology of the board is important enough (whether those I N and G blocks are next to each other or not makes a _big_ difference) that a simple histogram of letters isn't going to be sufficient.