git poem
This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.
Dan Q
This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.
As you may know, I’ve lately found an excuse to play with some new web technologies, and I’ve also taken the opportunity to try to gain a deeper understanding of some less bleeding-edge technologies that I think have some interesting potential. And so it was that, while I was staffing the Three Rings stall at last week’s NCVO conference, I made use of the time that the conference delegates were all off listening to a presentation to throw together a tech demo I call Steer!
As you can see from the GIF above, Steer! is a driving game. The track and your car are displayed in a web browser on a large screen, for example a desktop or laptop computer, television, or tablet, and your mobile phone is used to steer the car by tilting it to swerve around a gradually-narrowing weaving road. It’s pretty fun, but what really makes it interesting to me is the combination of moderately-new technologies I’ve woven together to make it possible, specifically:
The desktop browser does all of the real work: it takes the orientation of the device and uses that, and the car’s current speed, to determine how it’s position changes over the time
that’s elapsed since the screen was last refreshed: we’re aiming for 60 frames a second, of course, but we don’t want the car to travel slower when the game is played on a
slower computer, so we use requestAnimationFrame
to get the fastest rate possible and calculate the time between renderings to work out how much of a change has
occurred this ‘tick’. We leave the car’s sprite close to the bottom of the screen at all times but change how much it rotates from side to side, and we use it’s rotated to decide how
much of its motion is lateral versus the amount that’s “along the track”. The latter value determines how much track we move down the screen “behind” it.
The track is generated very simply by the addition of three sine waves of different offset and frequency – a form of very basic procedural generation. Despite the predictability of mathematical curves, this results in a moderately organic-feeling road because the player only sees a fraction of the resulting curve at any given time: the illustration below shows how these three curves combine to make the resulting road. The difficulty is ramped up the further the player has travelled by increasing the amplitude of the resulting wave (i.e. making the curves gradually more-agressive) and by making the road itself gradually narrower. The same mathematics are used to determine whether the car is mostly on the tarmac or mostly on the grass and adjust its maximum speed accordingly.
In order to help provide a visual sense of the player’s speed, I added dashed lines down the road (dividing it into three lanes to begin with and two later on) which zip past the car and provide a sense of acceleration, deceleration, overall speed, and the impact of turning ‘sideways’ (which of course reduces the forward momentum to nothing).
This isn’t meant to be a finished game: it’s an experimental prototype to help explore some technologies that I’d not had time to look seriously at before now. However, you’re welcome to take a copy – it’s all open source – and adapt or expand it. Particular ways in which it’d be fun to improve it might include:
Those who know me well know that I’m a bit of a data nerd. Even when I don’t yet know what I’m going to do with some data yet, it feels sensible to start collecting it in a nice machine-readable format from the word go. Because you never know, right? That’s how I’m able to tell you how much gas and electricity our house used on average on any day in the last two and a half years (and how much off that was offset by our solar panels).
So it should perhaps come as no huge surprise that for the last six months I’ve been recording the identity of every piece of music played by my favourite local radio station, Jack FM (don’t worry: I didn’t do this by hand – I wrote a program to do it). At the time, I wasn’t sure whether there was any point to the exercise… in fact, I’m still not sure. But hey: I’ve got a log of the last 45,000 songs that the radio station played: I might as well do something with it. The Discogs API proved invaluable in automating the discovery of metadata relating to each song, such as the year of its release (I wasn’t going to do that by hand either!), and that gave me enough data to, for example, do this (click on any image to see a bigger version):
I almost expected a bigger variance by hour-of-day, but I guess that Jack isn’t in the habit of pandering to its demographics too heavily. I spotted the post-midnight point at which you get almost a plurality of music from 1990 or later, though: perhaps that’s when the young ‘uns who can still stay up that late are mostly listening to the radio? What about by day-of-week, then:
The chunks of “bonus 80s” shouldn’t be surprising, I suppose, given that the radio station advertises that that’s exactly what it does at those times. But still: it’s reassuring to know that when a radio station claims to play 80s music, you don’t just have to take their word for it (so long as their listeners include somebody as geeky as me).
It feels to me like every time I tune in they’re playing an INXS song. That can’t be a coincidence, right? Let’s find out:
Yup, there’s a heavy bias towards Guns ‘n’ Roses, Michael Jackson, Prince, Oasis, Bryan Adams, Madonna, INXS, Bon Jovi, Queen, and U2 (who collectively are responsible for over a tenth of all music played on Jack FM), and – to a lesser extent – towards Robert Palmer, Meatloaf, Blondie, Green Day, Texas, Whitesnake, the Pet Shop Boys, Billy Idol, Madness, Rainbow, Elton John, Bruce Springsteen, Aerosmith, Fleetwood Mac, Phil Collins, ZZ Top, AC/DC, Duran Duran, the Police, Simple Minds, Blur, David Bowie, Def Leppard, and REM: taken together, one in every four songs played on Jack FM is by one of these 34 artists.
I was interested to see that the “top 20 songs” played on Jack FM these last six months include several songs by artists who otherwise aren’t represented at all on the station. The most-played song is Alice Cooper’s Poison, but I’ve never recorded them playing any other Alice Cooper songs (boo!). The fifth-most-played song is Fight For Your Right, by the Beastie Boys, but that’s the only Beastie Boys song I’ve caught them playing. And the seventh-most-played – Roachford’s Cuddly Toy – is similarly the only Roachford song they ever put on.
Next I tried a Markov chain analysis. Markov chains are a mathematical tool that examines a sequence (in this case, a sequence of songs) and builds a map of “chains” of sequential songs, recording the frequency with which they follow one another – here’s a great explanation and playground. The same technique is used by “predictive text” features on your smartphone: it knows what word to suggest you type next based on the patterns of words you most-often type in sequence. And running some Markov chain analysis helped me find some really… interesting patterns in the playlists. For example, look at the similarities between what was played early in the afternoon of Wednesday 19 October and what was played 12 hours later, early in the morning of Thursday 20 October:
19 October 2016 | 20 October 2016 | ||
12:06:33 | Kool & The Gang – Fresh | Kool & The Gang – Fresh | 00:13:56 |
12:10:35 | Bruce Springsteen – Dancing In The Dark | Bruce Springsteen – Dancing In The Dark | 00:17:57 |
12:14:36 | Maxi Priest – Close To You | Maxi Priest – Close To You | 00:21:59 |
12:22:38 | Van Halen – Why Can’t This Be Love | Van Halen – Why Can’t This Be Love | 00:25:00 |
12:25:39 | Beats International / Lindy – Dub Be Good To Me | Beats International / Lindy – Dub Be Good To Me | 00:29:01 |
12:29:40 | Kasabian – Fire | Kasabian – Fire | 00:33:02 |
12:33:42 | Talk Talk – It’s My Life | Talk Talk – It’s My Life | 00:38:04 |
12:41:44 | Lenny Kravitz – Are You Gonna Go My Way | Lenny Kravitz – Are You Gonna Go My Way | 00:42:05 |
12:45:45 | Shalamar – I Can Make You Feel Good | Shalamar – I Can Make You Feel Good | 00:45:06 |
12:49:47 | 4 Non Blondes – What’s Up | 4 Non Blondes – What’s Up | 00:50:07 |
12:55:49 | Madness – Baggy Trousers | Madness – Baggy Trousers | 00:54:09 |
Eagle Eye Cherry – Save Tonight | 00:56:09 | ||
Feeling – Love It When You Call | 01:04:12 | ||
13:02:51 | Fine Young Cannibals – Good Thing | Fine Young Cannibals – Good Thing | 01:10:14 |
13:06:54 | Blur – There’s No Other Way | Blur – There’s No Other Way | 01:14:15 |
13:09:55 | Pet Shop Boys – It’s A Sin | Pet Shop Boys – It’s A Sin | 01:17:16 |
13:14:56 | Zutons – Valerie | Zutons – Valerie | 01:22:18 |
13:22:59 | Cure – The Love Cats | Cure – The Love Cats | 01:26:19 |
13:27:01 | Bryan Adams / Mel C – When You’re Gone | Bryan Adams / Mel C – When You’re Gone | 01:30:20 |
13:30:02 | Depeche Mode – Personal Jesus | Depeche Mode – Personal Jesus | 01:33:21 |
13:34:03 | Queen – Another One Bites The Dust | Queen – Another One Bites The Dust | 01:38:22 |
13:42:06 | Shania Twain – That Don’t Impress Me Much | Shania Twain – That Don’t Impress Me Much | 01:42:23 |
13:45:07 | ZZ Top – Gimme All Your Lovin’ | ZZ Top – Gimme All Your Lovin’ | 01:46:25 |
13:49:09 | Abba – Mamma Mia | Abba – Mamma Mia | 01:50:26 |
13:53:10 | Survivor – Eye Of The Tiger | Survivor – Eye Of The Tiger | 01:53:27 |
Scouting For Girls – Elvis Aint Dead | 01:57:28 | ||
Verve – Lucky Man | 02:00:29 | ||
Fleetwood Mac – Say You Love Me | 02:05:30 | ||
14:03:13 | Kiss – Crazy Crazy Nights | Kiss – Crazy Crazy Nights | 02:10:31 |
14:07:15 | Lightning Seeds – Sense | Lightning Seeds – Sense | 02:14:33 |
14:11:16 | Pretenders – Brass In Pocket | Pretenders – Brass In Pocket | 02:18:34 |
14:14:17 | Elvis Presley / JXL – A Little Less Conversation | Elvis Presley / JXL – A Little Less Conversation | 02:21:35 |
14:22:19 | U2 – Angel Of Harlem | U2 – Angel Of Harlem | 02:24:36 |
14:25:20 | Trammps – Disco Inferno | Trammps – Disco Inferno | 02:28:37 |
14:29:22 | Cast – Guiding Star | Cast – Guiding Star | 02:31:38 |
14:33:23 | New Order – Blue Monday | New Order – Blue Monday | 02:36:39 |
14:41:26 | Def Leppard – Let’s Get Rocked | Def Leppard – Let’s Get Rocked | 02:40:41 |
14:46:28 | Phil Collins – Sussudio | Phil Collins – Sussudio | 02:45:42 |
14:50:30 | Shawn Mullins – Lullaby | Shawn Mullins – Lullaby | 02:49:43 |
14:55:31 | Stars On 45 – Stars On 45 | Stars On 45 – Stars On 45 | 02:53:45 |
16:06:35 | Dead Or Alive – You Spin Me Round Like A Record | Dead Or Alive – You Spin Me Round Like A Record | 03:00:47 |
16:09:36 | Dire Straits – Walk Of Life | Dire Straits – Walk Of Life | 03:03:48 |
16:13:37 | Keane – Everybody’s Changing | Keane – Everybody’s Changing | 03:07:49 |
16:17:39 | Billy Idol – Rebel Yell | Billy Idol – Rebel Yell | 03:10:50 |
16:25:41 | Stealers Wheel – Stuck In The Middle | Stealers Wheel – Stuck In The Middle | 03:14:51 |
16:28:42 | Green Day – American Idiot | Green Day – American Idiot | 03:18:52 |
16:33:44 | A-Ha – Take On Me | A-Ha – Take On Me | 03:21:53 |
16:36:45 | Cranberries – Dreams | Cranberries – Dreams | 03:26:54 |
Elton John – Philadelphia Freedom | 03:30:56 | ||
Inxs – Disappear | 03:36:57 | ||
Kim Wilde – You Keep Me Hanging On | 03:40:59 | ||
16:44:47 | Living In A Box – Living In A Box | ||
16:47:48 | Status Quo – Rockin’ All Over The World | Status Quo – Rockin’ All Over The World | 03:45:00 |
The similarities between those playlists (which include a 20-songs-in-a-row streak!) surely can’t be coincidence… but they do go some way to explaining why listening to Jack FM sometimes gives me a feeling of déjà vu (along with, perhaps, the no-talk, all-jukebox format). Looking elsewhere in the data I found dozens of other similar occurances, though none that were both such long chains and in such close proximity to one another. What does it mean?
There are several possible explanations, including:
But the question remains: why reuse playlists in close proximity at all? Even when the station operates autonomously, as it clearly does most of the time, it’d surely be easy enough to set up an auto-DJ using “smart random” (because truly random shuffles don’t sound random to humans) to get the same or a better effect.
Which leads to another interesting observation: Jack FM’s sister stations in Surrey and Hampshire also maintain a similar playlist most of the time… which means that they’re either synchronising their ad breaks (including their duration – I suspect this is the case) or else using filler jingles to line-up content with the beginnings and ends of songs. It’s a clever operation, clearly, but it’s not beyond black-box comprehension. More research is clearly needed. (And yes, I’m sure I could just call up and ask – they call me “Newcastle Dan” on the breakfast show – but that wouldn’t be even half as fun as the data mining is…)
This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.
An annual tradition at Three Rings is DevCamp, an event that borrows from the “hackathon” concept and expands it to a week-long code-producing factory for the volunteers of the Three Rings development team. Motivating volunteers is a very different game to motivating paid employees: you can’t offer to pay them more for working harder nor threaten to stop paying them if they don’t work hard enough, so it’s necessary to tap in to whatever it is that drives them to be a volunteer, and help them get more of that out of their volunteering.
At least part of what appeals to all of our developers is a sense of achievement – of producing something that has practical value – as well as of learning new things, applying what they’ve learned, and having a degree of control over the parts of the project they contribute most-directly to. Incidentally, these are the same things that motivate paid developers, too, if a Google search for studies on the subject is to believed. It’s just that employers are rarely able to willing to offer all of those things (and even if they can, you can’t use them to pay your mortgage), so they have to put money on the table too. With my team at Three Rings, I don’t have money to give them, so I have to make up for it with a surplus of those things that developers actually want.
It seems strange to me in hindsight that for the last seven years I’ve spent a week of my year taking leave from my day job in order to work longer, harder, and unpaid for a voluntary project… but that I haven’t yet blogged about it. Over the same timescale I’ve spent about twice as long at DevCamp than I have, for example, skiing, yet I’ve managed to knock out several blog posts on that subject. Part of that might be borne out of the secretive nature of Three Rings, especially in its early days (when involvement with Three Rings pretty-much fingered you as being a Nightline volunteer, which was frowned upon), but nowadays we’ve got a couple of dozen volunteers with backgrounds in a variety of organisations: and many of those of us that ever were Nightliner volunteers have long since graduated and moved-on to other volunteering work besides.
Part of the motivation – one of the perks of being a Three Rings developer – for me at least, is DevCamp itself. Because it’s an opportunity to drop all of my “day job” stuff for a week, go to some beatiful far-flung corner of the country, and (between early-morning geocaching/hiking expeditions and late night drinking tomfoolery) get to spend long days contributing to something awesome. And hanging out with like-minded people while I do so. I like I good hackathon of any variety, but I love me some Three Rings DevCamp!
So yeah: DevCamp is awesome. It’s more than a little different than those days back in 2003 when I wrote all the code and Kit worked hard at distracting me with facts about the laws of Hawaii – for the majority of DevCamp 2016 we had half a dozen developers plus two documentation writers in attendance! – but it’s still fundamentally about the same thing: producing a piece of software that helps about 25,000 volunteers do amazing things and make the world a better place. We’ve collectively given tens, maybe hundreds of thousands of hours of time in developing and supporting it, but that in turn has helped to streamline the organisation of about 16 million person-hours of other volunteering.
So that’s nice.
Oh, and I was delighted that one of my contributions this DevCamp was that I’ve finally gotten around to expanding the functionality of the “gender” property so that there are now more than three options. That’s almost more-exciting than the geocaches. Almost.
Edit: added a missing word in the sentence about how much time our volunteers had given, making it both more-believable and more-impressive.
As you’re no-doubt aware, Home Secretary Theresa May is probably going to get her way with her “snooper’s charter” by capitalising on events in Paris (even though that makes no sense), and before long, people working for law enforcement will be able to read your Internet usage history without so much as a warrant (or, to put it as the UN’s privacy chief put it, it’s “worse than scary”).
In a revelation that we should be thankful of as much as we’re terrified by, our government does not understand how the Internet works. And that’s why it’s really easy for somebody with only a modicum of geekery to almost-completely hide their online activities from observation by their government and simultaneously from hackers. Here’s a device that I built the other weekend, and below I’ll tell you how to do it yourself (and how it keeps you safe online from a variety of threats, as well as potentially giving you certain other advantages online):
I call it “Iceland”, for reasons that will become clear later. But a more-descriptive name would be a “Raspberry Pi VPN Hotspot”. Here’s what you’ll need if you want to build one:
From here on, this post gets pretty geeky. Unless you plan on building your own little box to encrypt all of your home’s WiFi traffic until it’s well out of the UK and close-to-impossible to link to you personally (which you should!), then you probably ought to come back to it another time.
Here’s how it’s done:
1. Plug in, boot, and install some prerequisites
Plug the WiFi dongle into a USB port and connect the Ethernet port to your Internet router. Boot your Raspberry Pi into Raspbian (as described in the helpsheet that comes with it), and run:
sudo apt-get install bridge-utils hostapd udhcpd bind9 openvpn
2. Make HostAPD support your Edimax dongle
If, like me, you’re using an Edimax dongle, you need to do an extra couple of steps to make it work as an access point. Skip this bit if you’re using one of the other dongles I listed or if you know better.
wget http://dl.dropbox.com/u/1663660/hostapd/hostapd.zip unzip hostapd.zip sudo mv /usr/sbin/hostapd /usr/sbin/hostapd.original sudo mv hostapd /usr/sbin/hostapd.edimax sudo ln -sf /usr/sbin/hostapd.edimax /usr/sbin/hostapd sudo chown root.root /usr/sbin/hostapd sudo chmod 755 /usr/sbin/hostapd
3. Set up OpenVPN
Get OpenVPN configuration files from your VPN provider: often these will be available under the iOS downloads. There’ll probably be one for each available endpoint. I chose the one for Reyjkavik, because Iceland’s got moderately sensible privacy laws and I’m pretty confident that it would take judicial oversight for British law enforcement to collaborate with Icelandic authorities on getting a wiretap in place, which is the kind of level of privacy I’m happy with. Copy your file to /etc/openvpn/openvpn.conf and edit it: you may find that you need to put your VPN username and password into it to make it work.
sudo service openvpn start
You can now test your VPN’s working, if you like. I suggest connecting to the awesome icanhazip.com and asking it where you are (you can use your favourite GeoIP website to tell you what country it thinks you’re in, based on that):
curl -4 icanhazip.com
Another option would be to check with a GeoIP service directly:
curl freegeoip.net/json/
4. Set up your firewall and restart the VPN connection
Unless your VPN provider gives you DNAT (and even if they do, if you’re paranoid), you should set up a firewall to allow only outgoing connections to be established, and then restart your VPN connection:
sudo iptables -A INPUT -i tun0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -i tun0 -j DROP sudo sh -c "iptables-save > /etc/iptables.nat.vpn.secure" sudo sh -c "echo 'up iptables-restore < /etc/iptables.nat.vpn.secure' >> /etc/network/interfaces" sudo service openvpn restart
5. Configure your WiFi hotspot
Configure bind as your DNS server, caching responses on behalf of Google’s DNS servers, or another DNS server that you trust. Alternatively, you can just configure your DHCP clients to use Google’s DNS servers directly, but caching will probably improve your performance overall. To do this, add a forwarder to /etc/bind/named.conf.options:
forwarders { 8.8.8.8; 8.8.4.4; };
Restart bind, and make sure it loads on boot:
sudo service bind9 restart sudo update-rc.d bind9 enable
Edit /etc/udhcpd.conf. As a minimum, you should have a configuration along these lines (you might need to tweak your IP address assignments to fit with your local network – the “router” and “dns” settings should be set to the IP address you’ll give to your Raspberry Pi):
start 192.168.0.2 end 192.168.0.254 interface wlan0 remaining yes opt dns 192.168.0.1 option subnet 255.255.255.0 opt router 192.168.0.1 option lease 864000 # 10 days
Enable DHCP by uncommenting (remove the hash!) the following line in /etc/default/udhcpd:
#DHCPD_ENABLED="yes"
Set a static IP address on your Raspberry Pi in the same subnet as you configured above (but not between the start and end of the DHCP list):
sudo ifconfig wlan0 192.168.0.1
And edit your /etc/network/interfaces file to configure it to retain this on reboot (you’ll need to use tabs, not spaces, for indentation):
iface wlan0 inet static
address 192.168.0.1
netmask 255.255.255.0
And comment out the lines relating to hot-plugging of WiFi adapters/network hopping:
#allow-hotplug wlan0 #wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf #iface default inet manual
Right – onto hostapd, the fiddliest of the tools you’ll have to configure. Create or edit /etc/hostapd/hostapd.conf as follows, but substitute in your own SSID, hotspot password, and channel (to minimise interference, which can slow your network down, I recommend using WiFi scanner tool on your mobile to find which channels your neighbours aren’t using, and use one of those – you should probably avoid the channel your normal WiFi uses, too, so you don’t slow your own connection down with crosstalk):
interface=wlan0 driver=nl80211 ssid=your network name hw_mode=g channel=6 macaddr_acl=0 auth_algs=1 ignore_broadcast_ssid=0 wpa=2 wpa_passphrase=your network password wpa_key_mgmt=WPA-PSK wpa_pairwise=TKIP rsn_pairwise=CCMP
Hook up this configuration by editing /etc/default/hostapd:
DAEMON_CONF="/etc/hostapd/hostapd.conf"
Fire up the hotspot, and make sure it runs on reboot:
sudo service hostapd start sudo service udhcpd start sudo update-rc.d hostapd enable sudo update-rc.d udhcpd enable
Finally, set up NAT so that people connecting to your new hotspot are fowarded through the IP tunnel of your VPN connection:
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward" sudo sh -c "echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf" sudo iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE sudo sh -c "iptables-save > /etc/iptables.nat.vpn.secure"
6. Give it a go!
Connect to your new WiFi hotspot, and go to your favourite GeoIP service. Or, if your VPN endpoint gives you access to geographically-limited services, give those a go (you’d be amazed how different the Netflix catalogues are in different parts of the world). And give me a shout if you need any help or if you have any clever ideas about how this magic little box can be improved.
Further reading:
There’s a wonderful tool for making web-based “choose your own adventure”-type games, called Twine. One of the best things about it is that it’s so accessible: if you wanted to, you could be underway writing your first ever story with it in about 5 minutes from now, without installing anything at all, and when it was done you could publish it on the web and it would just work.
But the problem with Twine is that, in its latest and best versions, you’re trapped into using the Twine IDE. The Twine IDE is an easy-to-use, highly visual, ‘drag-and-drop’ interface for making interactive stories. Which is probably great if you’re into IDEs or if you don’t “know better”… but for those of us who prefer to do our writing in a nice clean, empty text editor like Sublime or TextMate or to script/automate our builds, it’s just frustrating to lose access to the tools we love. Plus, highly-visual IDEs make it notoriously hard to collaborate with other authors on the same work without simply passing it back and forwards between you: unless they’ve been built with this goal in mind, you generally can’t have two people working in the same file at the same time.
Earlier versions of Twine had a command-line tool called Twee that perfectly filled this gap. But the shiny new versions don’t. That’s where I came in.
In that way that people who know me are probably used to by now, I was very-slightly unsatisfied with one aspect of an otherwise fantastic product and decided that the correct course of action was to reimplement it myself. So that’s how, a few weeks ago, I came to release Twee2.
If you’re interested in writing your own “Choose Your Own Adventure”-type interactive fiction, whether for the world or just for friends, but you find user-friendly IDEs like Twine limiting (or you just prefer a good old-fashioned text editor), then give Twee2 a go. I’ve written a simple 2-minute tutorial to get you started, it works on Windows, MacOS, Linux, and just-about everything else, and it’s completely open-source if you’d like to expand or change it yourself.
(there are further discussions about the concept and my tool on Reddit here, here, here and here, and on the Twinery forums here, here and here)
Remember Minesweeper? It’s probably been forever since you played, so go have a game online now. And there went your afternoon.
My geek-crush Ben Foxall posted on Twitter on Monday morning to share that he’d had a moment of fun nostalgia when he’d come into the office to discover that somebody in his team had covered his monitor with two layers of Post-It notes. The bottom layer contained numbers – and bombs! – to represent the result of a Minesweeper board, and the upper layer ‘covered’ them so that individual Post-Its could be removed to reveal what lay beneath. Awesome.
Not to be outdone, I hunted around my office and found some mini-Post-Its. Being smaller meant that I could fit more of them onto a monitor and thus make a more-sophisticated (and more-challenging!) play space. But how to generate the board? Sure: I could do it by hand, but that doesn’t seem very elegant at all – plus, humans make really bad random number generators! I didn’t need quantum-tunnelling-seeded Minesweeper (yes, that’s a thing) levels of entropy, sure, but it’d still be nice to outsource the heavy lifting to a computer, right?
So naturally, I wrote a program to do it for me. Want to see? It’s at danq.me/minesweeper. Just line up some Post-Its on a co-worker’s monitor to work out
how many you can fit across it in each dimension (I found that I could get 6 × 4 standard-sized Post-Its but 7 × 5 or even 8 × 5 mini-sized Post-Its very comfortably onto one of the typical widescreen monitors in my office), decide how many mines you want, and click
Generate. Don’t like the board you get? Click it again!
And because I was looking for a fresh excuse to play with Periscope, I broadcast the first game I set up live to the Internet. In the end, 66 people ended up watching some or all of a paper-based game of Minesweeper played by my colleague Liz, including moments of cheering her on and, in one weird moment, despair at the revelation that she was married. The internet’s strange, yo.
Anyway: in case you missed the Periscope broadcast, I’ve put it on YouTube. Sorry about the portrait-orientation filming: I think it’s awful, too, but it’s a Periscope thing and I haven’t installed the new update that fixes it yet.
Now go set up a game of Post-It Minesweeper for a friend or co-worker.
What’s the hardest word to guess, when playing hangman? I’ll come back to that.
Last year, Nick Berry wrote a fantastic blog post about the optimal strategy for Hangman. He showed that the best guesses to make to get your first “hit” in a game of hangman are not the most-commonly occurring letters in written English, because these aren’t the most commonly-occurring letters in individual words. He also showed that the first guesses should be adjusted based on the length of the word (the most common letter in 5-letter words is ‘S’, but the most common letter in 6-letter words is ‘E’). In short: hangman’s a more-complex game than you probably thought it was! I’d like to take his work a step further, and work out which word is the hardest word: that is – assuming you’re playing an optimal strategy, what word takes the most-guesses?
First, though, we need to understand how hangman is perfectly played. Based on the assumption that the “executioner” player is choosing words randomly, and that no clue is given as to the nature of the word, we can determine the best possible move for all possible states of the game by using a data structure known as a tree. Suppose our opponent has chosen a three-letter word, and has drawn three dashes to indicate this. We know from Nick’s article that the best letter to guess is A. And then, if our guess is wrong, the next best letter to guess is E. But what if our first guess is right? Well, then we’ve got an “A” in one or more positions on the board, and we need to work out the next best move: it’s unlikely to be “E” – very few three-letter words have both an “A” and an “E” – and of course what letter we should guess next depends entirely on what positions the letters are in.
What we’re actually doing here is a filtering exercise: of all of the possible letters we could choose, we’re considering what possible results that could have. Then for each of those results, we’re considering what guesses we could make next, and so on. At each stage, we compare all of the possible moves to a dictionary of all possible words, and filter out all of the words it can’t be: after our first guess in the diagram above, if we guess “A” and the board now shows “_ A _”, then we know that of the 600+ three-letter words in the English language, we’re dealing with one of only about 134. We further refine our guess by playing the odds: of those words, more of them have a “C” in than any other letter, so that’s our second guess. If it has a C in, that limits the options further, and we can plan the next guess accordingly. If it doesn’t have a C in, that still provides us with valuable information: we’re now looking for a three-letter word with an A in the second position and no letter C: that cuts it down to 124 words (and our next guess should be ‘T’). This tree-based mechanism for working out the best moves is comparable to that used by other game-playing computers. Hangman is simple enough that it can be “solved” by contemporary computers (like draughts – solved in 2007 – but unlike chess: while modern chess-playing computers can beat humans, it’s still theoretically possible to build future computers that will beat today’s computers).
Now that we can simulate the way that a perfect player would play against a truly-random executioner, we can use this to simulate games of hangman for every possible word (I’m using version 0.7 of this British-English dictionary). In other words, we set up two computer players: the first chooses a word from the dictionary, the second plays “perfectly” to try to guess the word, and we record how many guesses it took. So that’s what I did. Here’s the Ruby code I used. It’s heavily-commented and probably pretty understandable/good learning material, if you’re into that kind of thing. Or if you fancy optimising it, there’s plenty of scope for that too (I knocked it out on a lunch break; don’t expect too much!). Or you could use it as the basis to make a playable hangman game. Go wild.
Running the program, we can see that the hardest three-letter word is “xxv”, which would take 22 guesses (20 of them wrong!) to get. But aside from the roman numeral for 25, I don’t think that “xxv” is actually a word. Perhaps my dictionary’s not very good. “Oak”, though, is definitely a word, and at 20 guesses (17 wrong), it’s easily enough to hang your opponent no matter how many strokes it takes to complete the gallows.
There are more tougher words in the four-letter set, like the devious “quiz”, “jazz”, “zinc”, and “faux”. Pick one of those and your opponent – unless they’ve seen this blog post! – is incredibly unlikely to guess it before they’re swinging from a rope.
As we get into the 5, 6, and 7-letter words you’ll begin to notice a pattern: that the hardest words with any given number of letters get easier the longer they are. That’s kind of what you’d expect, I suppose: if there were a hypothetical word that contained every letter in the alphabet, then nobody would ever fail to (eventually) get it.
When we make a graph of each word length, showing which proportion of the words require a given number of “wrong” guesses (by an optimised player), we discover a “sweet spot” window in which we’ll find all of the words that an optimised player will always fail to guess (assuming that we permit up to 10 incorrect guesses before they’re disqualified). The window seems small for the number of times I remember seeing people actually lose at hangman, which implies to me that human players consistently play sub-optimally, and do not adequately counteract that failing by applying an equal level of “smart”, intuitive play (knowing one’s opponent and their vocabulary, looking for hints in the way the game is presented, etc.).
In case you’re interested, then, here are the theoretically-hardest words to throw at your hangman opponent. While many of the words there feel like they would quite-rightly be difficult, others feel like they’d be easier than their ranking would imply: this is probably because they contain unusual numbers of vowels or vowels in unusual-but-telling positions, which humans (with their habit, inefficient under normal circumstances, of guessing an extended series of vowels to begin with) might be faster to guess than a computer.
Word | Guesses taken | “Wrong” guesses needed |
---|---|---|
quiz | 24 | 20 |
jazz | 22 | 19 |
jazzy | 22 | 18 |
quaff | 22 | 18 |
zinc | 21 | 17 |
oak | 20 | 17 |
vex | 20 | 17 |
vox | 20 | 17 |
foxing | 22 | 16 |
foxed | 21 | 16 |
queued | 20 | 16 |
fuzzy | 20 | 16 |
quay | 20 | 16 |
pinup | 20 | 16 |
fox | 19 | 16 |
yuk | 19 | 16 |
vaquero | 22 | 15 |
jazzier | 21 | 15 |
quizzed | 21 | 15 |
hazing | 21 | 15 |
favour | 21 | 15 |
yoking | 21 | 15 |
quays | 20 | 15 |
quark | 20 | 15 |
joked | 20 | 15 |
guyed | 20 | 15 |
foyer | 20 | 15 |
bumph | 20 | 15 |
huge | 19 | 15 |
quip | 19 | 15 |
gibe | 19 | 15 |
rump | 19 | 15 |
guan | 19 | 15 |
quizzed | 19 | 15 |
oaks | 19 | 15 |
murk | 19 | 15 |
fezzes | 19 | 15 |
yuck | 19 | 15 |
keno | 19 | 15 |
kazoo | 19 | 15 |
Download a longer list
(there’s plenty more which you’d expect to “win” with) |
If you use this to give you an edge in your next game, let me know how it works out for you!
Update – 8 March 2019: fixed a broken link and improved the layout of the page.
This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.
How to create perfect secret messages that can be decoded using just your eyes.
The explosion of smartphone ownership over the last decade has put powerful multi-function computers into the pockets of almost half of us. But despite the fact that the average smartphone contains at least as much personally-identifiable information as its owner keeps on their home computer (or in dead-tree form) at their house – and is significantly more-prone to opportunistic theft – many users put significantly less effort into protecting their mobile’s data than they do the data they keep at home.
I have friends who religiously protect their laptops and pendrives with TrueCrypt, axCrypt, or similar, but still carry around an unencrypted mobile phone. What we’re talking about here is a device that contains all of the contact details for you and everybody you know, as well as potentially copies of all of your emails and text messages, call histories, magic cookies for social networks and other services, saved passwords, your browsing history (some people would say that’s the most-incriminating thing on their phone!), authentication apps, photos, videos… more than enough information for an attacker to pursue a highly-targeted identity theft or phishing attack.
“Pattern lock” is popular because it’s fast and convenient. It might be good enough to stop your kids from using your phone without your permission (unless they’re smart enough to do some reverse smudge engineering: looking for the smear-marks made by your fingers as you unlock the device; and let’s face it, they probably are), but it doesn’t stand up to much more than that. Furthermore, gesture unlock solutions dramatically reduce the number of permutations, because you can’t repeat a digit: so much so, that you can easily perform a rainbow table attack on the SHA1 hash to reverse-engineer somebody’s gesture. Even if Android applied a per-device psuedorandom salt to the gesture pattern (they don’t, so you can download a prefab table), it doesn’t take long to generate an SHA1 lookup of just 895,824 codes (maybe Android should have listened to Coda Hale’s advice and used BCrypt, or else something better still).
These attacks, though (and the iPhone isn’t bulletproof, either), are all rather academic, because they are trumped by the universal rule that once an attacker has physical access to your device, it is compromised. This is fundamentally the way in which mobile security should be considered to be equivalent to computer security. All of the characteristics distinct to mobile devices (portability, ubiquity, processing power, etc.) are weaknesses, and that’s why smartphones deserve at least as much protection as desktop computers protecting the same data. Mobile-specific features like “remote wipe” are worth having, but can’t be relied upon alone – a wily attacker could easily keep your phone in a lead box or otherwise disable its connectivity features until it’s cracked.
The only answer is to encrypt your device (with a good password). Having to tap in a PIN or password may be less-convenient than just “swipe to unlock”, but it gives you a system that will resist even the most-thorough efforts to break it, given physical access (last year’s iPhone 4 vulnerability notwithstanding).
It’s still not perfect – especially here in the UK, where the RIPA can be used (and has been used) to force key surrender. What we really need is meaningful, usable “whole system” mobile encryption with plausible deniability. But so long as you’re only afraid of identity thieves and phishing scammers, and not being forced to give up your password by law or under duress, then it’s “good enough”.
Of course, it’s only any use if it’s enabled before your phone gets stolen! Like backups, security is one of those things that everybody should make a habit of thinking about. Go encrypt your smartphone; it’s remarkably easy –
As web developers, we’re used to working around the bugs in Microsoft Internet Explorer. The older versions are worst, and I’m certainly glad to not have to write code that works in Internet Explorer 6 (or, increasingly, Internet Explorer 7) any more: even Microsoft are glad to see Internet Explorer 6 dying out, but even IE8 is pretty ropey too. And despite what Microsoft claim, I’m afraid IE9 isn’t really a “modern” browser either (although it is a huge step forwards over its predecessors).
But imagine my surprise when I this week found what I suspect might be a previously undiscovered bug in Internet Explorer 8 and below. Surely they’ve all been found (and some of them even fixed), but now? But no. It takes a very specific set of circumstances for the bug to manifest itself, but it’s not completely unbelievable – I ran into it by accident while refactoring parts of Three Rings.
Here’s the crux of it: if you’re –
Then you’ll see a dialog box like the one shown above. Switching any of the prerequisites in that list out makes the problem go away: even switching the header from a strict “no-cache” to a more-permissive “private” makes all the difference.
I’ve set up a test environment where you can see this for yourself: HTTP version; HTTPS version. The source code of my experiment (PHP) is also available. Of course, if you try it in a functional, normal web browser, it’ll all work fine. But if you’ve got access to a copy of Internet Explorer 8 on some old Windows XP box somewhere (IE8 is the last version of the browser made available for XP), then try it in that and see for yourself what a strange error you get.
Last week I was talking to Alexander Dutton about an idea that we had to implement cookie-like behaviour using browser caching. As I first mentioned last year, new laws are coming into force across Europe that will require websites to ask for your consent before they store cookies on your computer. Regardless of their necessity, these laws are badly-defined and ill thought-out, and there’s been a significant lack of information to support web managers in understanding and implementing the required changes.
To illustrate one of the ambiguities in the law, I’ve implemented a tool which tracks site visitors almost as effectively as cookies (or similar technologies such as Flash Objects or Local Storage), but which must necessarily fall into one of the larger grey areas. My tool abuses the way that “permanent” (301) HTTP redirects are cached by web browsers.
[callout][button link=”http://c301.scatmania.org/” align=”right” size=”medium” color=”green”]See Demo Site[/button]You can try out my implementation for yourself. Click on the button to see the sample site, then close down all of your browser windows (or even restart your computer) and come back and try again: the site will recognise you and show you the same random number as it did the first time around, as well as identifying when your first visit was.[/callout]
Here’s how it works, in brief:
Compared to conventional cookie-based tracking (e.g. Google Analytics), this approach:
Moreover, this technique falls into a slight legal grey area. It would certainly be against the spirit of the law to use this technique for tracking purposes (although it would be trivial to implement even an advanced solution which “proxied” requests, using a database to associate conventional cookies with unique IDs, through to Google Analytics or a similar solution). However, it’s hard to legislate against the use of HTTP 301s, which are an even more-fundamental and required part of the web than cookies are. Also, and for the same reasons, it’s significantly harder to detect and block this technique than it is conventional tracking cookies. However, the technique is somewhat brittle and it would be necessary to put up with a reduced “cookie lifespan” if you used it for real.
[callout][button link=”http://c301.scatmania.org/” align=”right” size=”medium” color=”green”]See Demo Site[/button] [button link=”https://gist.github.com/avapoet/5318224″ align=”right” size=”medium” color=”orange”]Download Code[/button] Please try out the demo, or download the source code (Ruby/Sinatra) and see for yourself how this technique works.[/callout]
Note that I am not a lawyer, so I can’t make a statement about the legality (or not) of this approach to tracking. I would suspect that if you were somehow caught doing it without the consent of your users, you’d be just as guilty as if you used a conventional approach. However, it’s certainly a technically-interesting approach that might have applications in areas of legitimate tracking, too.
Update: The demo site is down, but I’ve update the download code link so that it still works.
As I indicated in my last blog post, my new blog theme has a “pop up” Dan in the upper-left corner. Assuming that you’re not using Internet Explorer, then when you move your mouse cursor over it, my head will “duck” back behind the bar below it.
This is all done without any Javascript whatsoever: it’s pure CSS. Here’s how it’s done:
<div class="sixteen columns">
<div id="dans-creepy-head"></div>
<h1 id="site-title" class="graphic">
<a href="/" title="Scatmania">Scatmania</a>
</h1>
<span class="site-desc graphic">
The adventures and thoughts of "Scatman" Dan Q
</span>
</div>
The HTML for the header itself is pretty simple: there’s a container (the big blue bar) which contains, among other things, a <div> with the id "dans-creepy-head". That’s what we’ll be working with. Here’s the main CSS:
#dans-creepy-head {
position: absolute;
top: -24px;
left: 15px;
width: 123px;
height: 133px;
background: url(/dans-creepy-head.png) top left no-repeat;
transition: all 800ms;
-o-transition: all 800ms;
-webkit-transition: all 800ms;
-moz-transition: all 800ms;
}
#dans-creepy-head:hover {
top: 100px;
height: 60px;
}
The CSS sets a size, position, and background image to the <div>, in what is probably a familiar way. A :hover selector changes the style to increase the distance from the top of the container (from -24px to 100px) and to decrease the height, cropping the image (from 133px to 60px – this was necessary in this case to prevent the bottom of the image from escaping out from underneath the masking bar that it’s supposed to be “hiding behind”). With just that code, you’d have a perfectly workable “duck”, but with a jerky, one-step animation.
The transition directive (and browser-specific prefix versions -o-transition, -webkit-transition, and -moz-transition, for compatability) are what makes the magic happen. This element specifies that any ("all") style is changed on this element (whether via CSS directives, as in this case, or by a change of class or properties by a Javascript function), that a transition effect will be applied to those changes. My use of "all" is a lazy catch-all – I could have specified the individual properties ( top and height) that I was interested in changing, and even put different periods on each, but I’ll leave it to you to learn about CSS3 transition options for yourself. The 800ms is the duration of the transition: in my case, 0.8 seconds.
html.ie #dans-creepy-head:hover {
top: -24px;
height: 133px;
}
@media (max-width: 780px) {
#dans-creepy-head {
display: none;
}
}
I apply some CSS to prevent the :hover effect from taking place in Internet Explorer, which doesn’t support transitions. The "ie" class is applied to the <html> tag using Paul Irish’s technique, so it’s easy to detect and handle IE users without loading separate stylesheet files for them. And finally, in order to fit with my newly-responsive design, I make the pop-up head disappear when the window is under 780px wide (at which point there’d be a risk of it colliding with the title).
That’s all there is to it! A few lines of CSS, and you’ve got an animation that degrades gracefully. You could equally-well apply transformations to links (how about making them fade in or out, or change the position of their background image?) or, with a little Javascript, to your tabstrips and drop-down menus.
So I saw this HDMI cable online:
Somehow, this triggered a transformation in me. You know how when Eric eats a banana, an amazing transformation occurs? A similar thing happened to me: this horrendously-worded advertisement turned me into an old person. I wanted to write a letter to them.
There were so many unanswered questions in my mind: what is a “virus noise” (is it a bit like the sound of somebody sneezing?)? How a polyester coating protects against them? And what kind of viruses are transmitted down video cables, anyway?
It took them five days but, fair play to them, they – despite Reddit’s expectations – wrote back.
Their explanation? The ‘Virus’ was transcribed from French terminology for interference. It’s not a computer virus or anything like that.
The world is full of examples of cables being over-sold, especially HDMI cables and things like “gold-plated optical cables” (do photons care about the conductivity of gold, now?).
Does anybody have enough of a familiarity with the French language to let me know if their explanation is believable?