Blog
Follow this blog
Collaborative zombies
20.12.08
Most recommendation systems that one sees on the internet are based on collaborative filtering. The idea here is that if Johnny, Jenny, Joseph, Julian and Gilly all like A and B, there is probably a connection between those items. Thus, if someone likes A, you can recommend them B.
There are a couple of problems with this technique.
First, it skews itself towards popularity and consensus. If A is a really popular thing, it is more likely to appear in people’s likes and, thus, more likely to be recommended.
Second, the calculations are computationally intensive. At the absolute worst, you could have (items owned ^ users) links to explore. With a modest 100,000 users, each with a tiny collection of 10 items, this would equate to a maximum of 1.0 × 10^50 connections.
Both these problems can be somewhat mitigated. If you want to err towards the eccentric and the niche, you can lower the threshold of people required to deem two items connected. So, maybe you would only require Johnny, Jenny and Julian to like both A and B in order to connect those items.
If you want to make collaborative filtering less computationally expensive, you can do a couple of things. First, cut out low-frequency items. Second, process data in chunks. Doing 50,000 users in one chunk and 50,000 in another will lose you some connections, but it will also massively reduce the computational load.
Recently, I found a way to extract user music library data from Emusic. First, I found that if you add an Emusic user username onto “http://www.emusic.com/profile/ajax/downloads/artists.html?nickname=”, you get back a list of artists by whom the user has downloaded songs. I compiled a corpus of common words, first names and last names. I wrote a script to fuse these two things and ended up with 382,808 music library artist entries for 15,155 users.
I wrote another script to extract 588,425 artist connections from this data. To deem two artists connected, the script required that they occur together in at least twenty people’s music libraries.
As some of you may know, the perceptron uses a number of music recommendation data sources. Each has a weight that is based on the approval that users have shown the data source’s recommendations. At the moment, the Emusic data source’s weight is hovering around 12. The top two sources each have weights of 34. However, Emusic’s weight will increase as its recommendations are exposed to more users.
One of my favourite things about the perceptron is that the vast majority of its data sources are based on conscious human opinions. The data comes from gig line-ups, mixtapes, human recommendations, artist admiration, record labels and mp3 blogs. I find collaborative filtering a little depressing: it’s just SO mechanical. One way to a) further improve its efficiency and b) make it a little more human would be to cluster users based upon demographic indicators, and then do the filtering. How ace would it be to be able to do collaborative filtering on the music libraries of everyone who went to the same gig? Or everyone who is the same age? Or groups of people who are friends with each other? Or who all like the same film? If last.fm aren’t doing this, they are fucking crazy.
The icon
18.12.08
For me, every idea and dream and activity has a tableau associated with it. Like, when I think of making music with my band, I think the records I’ve already made. Or, when I think of having children, I think of lying in bed with my baby and my baby.
When I think of the central idea of the perceptron, I think of a giant web of connections between artists and the tendrils of recommendation spidering out from Des Ark to other bands I love.
However, these are idealisations and simplifications. More accurate versions would be simultaneously rushing my daughter to school and wiping her runny nose, sitting on a chair in my bedroom fiddling around on my guitar, a mess of pre-calculation and map-reading and planned routes.
Which is to say, things are beautiful and theoretical in the mind and then get ugly and practical.
So, in order to calculate recommendations in reasonable time, the perceptron does a few things. Every link between artists has a grade. Fine. But it has to be precalculated. In most cases, spidering does not go past direct links from the starting point. That tenuous, five-leap link between two bands that turns out to be brilliant: bullshit.
However, sometimes, that practicality can bring beauty. The moment of hitting upon a beautiful new melody. Your daughter saying, “I love you, Mummy”, as she gets out of the car. Exploring indirect links that go via artists that the user has said they like and getting recommended Converge from Des Ark.
New API calls: artist information and searches
17.12.08
You can now get artist data (website, Myspace, summary, WIkipedia link, sample track URL) via the API.
You can also see all the searches that have ever been done on the site, along with who did them.
Find out more on the API page.
last.fm recommendations
16.12.08
There is a new data source: last.fm recommendations. Woo.
Lab
16.12.08
I’ve created a new section: the Lab. This is where the experimental stuff I’m working on will be. At the moment, there is just the mini hack I did for the last.fm hack day, Where Should I Live?
Waxy
10.12.08
the perceptron got linked on Waxy. Thanks so much, Andy.
Welcome, Waxalicious horde.
*blush*
01.12.08
the perceptron has been getting a little bit of interest from blog people.
Decision Factory mentioned the site in a piece about using music recommendation services as gift suggesters. Paul Lamere, a dev at Sun, dedicated a whole article to the perceptron and how it works. And a blog called Fine Tuning posted a brief review of the site in French (English translation).
I am very flattered that people like the site enough to write about it. Thank you.
Latest data source weights
28.11.08
The table below gives you a rough idea of the quality of suggestions from each of the perceptron’s data sources. Please note, the User suggestions source is being kept artificially low for now.
It is interesting to compare the weights above with those from two months ago. The Epitonic similar artists source is slightly less highly rated, and the Tiny Mix Tapes similar artists source is slightly better rated. More noticeable is that the Myspace top friends and Mixtapes sources have swapped places, both in position on the leaderboard and numeric weight.
One possible reason for this is that, in the intervening time, the perceptron has been being used by many more people. Before, my music taste was probably a massive influence on it. Now, it is probably becoming more diverse.
Dead dead dead dead deeeaaaaaaaad
31.10.08
Some things have happened lately: I’ve started my day job again after a three month sabbatical, I’ve been writing playmary.com. However, development on the perceptron continues unabated.
Sample tracks will get much more reliable over the next few days. I’m also about to add a new source of human recommendations.
Update: recommendations are now coming in from the absolutepunk.net reviewers.
How is this magic possible?
14.10.08
I’ve just written up a non-technical post on my blog about how the perceptron’s recommendation system works.
20.12.08
Most recommendation systems that one sees on the internet are based on collaborative filtering. The idea here is that if Johnny, Jenny, Joseph, Julian and Gilly all like A and B, there is probably a connection between those items. Thus, if someone likes A, you can recommend them B.
There are a couple of problems with this technique.
First, it skews itself towards popularity and consensus. If A is a really popular thing, it is more likely to appear in people’s likes and, thus, more likely to be recommended.
Second, the calculations are computationally intensive. At the absolute worst, you could have (items owned ^ users) links to explore. With a modest 100,000 users, each with a tiny collection of 10 items, this would equate to a maximum of 1.0 × 10^50 connections.
Both these problems can be somewhat mitigated. If you want to err towards the eccentric and the niche, you can lower the threshold of people required to deem two items connected. So, maybe you would only require Johnny, Jenny and Julian to like both A and B in order to connect those items.
If you want to make collaborative filtering less computationally expensive, you can do a couple of things. First, cut out low-frequency items. Second, process data in chunks. Doing 50,000 users in one chunk and 50,000 in another will lose you some connections, but it will also massively reduce the computational load.
Recently, I found a way to extract user music library data from Emusic. First, I found that if you add an Emusic user username onto “http://www.emusic.com/profile/ajax/downloads/artists.html?nickname=”, you get back a list of artists by whom the user has downloaded songs. I compiled a corpus of common words, first names and last names. I wrote a script to fuse these two things and ended up with 382,808 music library artist entries for 15,155 users.
I wrote another script to extract 588,425 artist connections from this data. To deem two artists connected, the script required that they occur together in at least twenty people’s music libraries.
As some of you may know, the perceptron uses a number of music recommendation data sources. Each has a weight that is based on the approval that users have shown the data source’s recommendations. At the moment, the Emusic data source’s weight is hovering around 12. The top two sources each have weights of 34. However, Emusic’s weight will increase as its recommendations are exposed to more users.
One of my favourite things about the perceptron is that the vast majority of its data sources are based on conscious human opinions. The data comes from gig line-ups, mixtapes, human recommendations, artist admiration, record labels and mp3 blogs. I find collaborative filtering a little depressing: it’s just SO mechanical. One way to a) further improve its efficiency and b) make it a little more human would be to cluster users based upon demographic indicators, and then do the filtering. How ace would it be to be able to do collaborative filtering on the music libraries of everyone who went to the same gig? Or everyone who is the same age? Or groups of people who are friends with each other? Or who all like the same film? If last.fm aren’t doing this, they are fucking crazy.
The icon
18.12.08
For me, every idea and dream and activity has a tableau associated with it. Like, when I think of making music with my band, I think the records I’ve already made. Or, when I think of having children, I think of lying in bed with my baby and my baby.
When I think of the central idea of the perceptron, I think of a giant web of connections between artists and the tendrils of recommendation spidering out from Des Ark to other bands I love.
However, these are idealisations and simplifications. More accurate versions would be simultaneously rushing my daughter to school and wiping her runny nose, sitting on a chair in my bedroom fiddling around on my guitar, a mess of pre-calculation and map-reading and planned routes.
Which is to say, things are beautiful and theoretical in the mind and then get ugly and practical.
So, in order to calculate recommendations in reasonable time, the perceptron does a few things. Every link between artists has a grade. Fine. But it has to be precalculated. In most cases, spidering does not go past direct links from the starting point. That tenuous, five-leap link between two bands that turns out to be brilliant: bullshit.
However, sometimes, that practicality can bring beauty. The moment of hitting upon a beautiful new melody. Your daughter saying, “I love you, Mummy”, as she gets out of the car. Exploring indirect links that go via artists that the user has said they like and getting recommended Converge from Des Ark.
New API calls: artist information and searches
17.12.08
You can now get artist data (website, Myspace, summary, WIkipedia link, sample track URL) via the API.
You can also see all the searches that have ever been done on the site, along with who did them.
Find out more on the API page.
last.fm recommendations
16.12.08
There is a new data source: last.fm recommendations. Woo.
Lab
16.12.08
I’ve created a new section: the Lab. This is where the experimental stuff I’m working on will be. At the moment, there is just the mini hack I did for the last.fm hack day, Where Should I Live?
Waxy
10.12.08
the perceptron got linked on Waxy. Thanks so much, Andy.
Welcome, Waxalicious horde.
*blush*
01.12.08
the perceptron has been getting a little bit of interest from blog people.
Decision Factory mentioned the site in a piece about using music recommendation services as gift suggesters. Paul Lamere, a dev at Sun, dedicated a whole article to the perceptron and how it works. And a blog called Fine Tuning posted a brief review of the site in French (English translation).
I am very flattered that people like the site enough to write about it. Thank you.
Latest data source weights
28.11.08
The table below gives you a rough idea of the quality of suggestions from each of the perceptron’s data sources. Please note, the User suggestions source is being kept artificially low for now.
| Source | Weight |
| Epitonic similar artists | 0.41474 |
| Tiny Mix Tapes similar artists | 0.332759 |
| Mixtapes | 0.137909 |
| User suggestions | 0.111111 |
| Myspace top friends | 0.0788359 |
| Record labels | 0.0181446 |
| Epitonic other artists | 0.0157806 |
| MP3 blogs | 0.00183043 |
| Gigs (very little data) | 0.0 |
| Absolute Punk similar artists (very little data) | 0.0 |
It is interesting to compare the weights above with those from two months ago. The Epitonic similar artists source is slightly less highly rated, and the Tiny Mix Tapes similar artists source is slightly better rated. More noticeable is that the Myspace top friends and Mixtapes sources have swapped places, both in position on the leaderboard and numeric weight.
One possible reason for this is that, in the intervening time, the perceptron has been being used by many more people. Before, my music taste was probably a massive influence on it. Now, it is probably becoming more diverse.
Dead dead dead dead deeeaaaaaaaad
31.10.08
Some things have happened lately: I’ve started my day job again after a three month sabbatical, I’ve been writing playmary.com. However, development on the perceptron continues unabated.
Sample tracks will get much more reliable over the next few days. I’m also about to add a new source of human recommendations.
Update: recommendations are now coming in from the absolutepunk.net reviewers.
How is this magic possible?
14.10.08
I’ve just written up a non-technical post on my blog about how the perceptron’s recommendation system works.