New Approaches

The leader board has shown some great progress as of late, to get on it you now need a 0.96 RMSE or lower and the top of the leader board is 0.907, an almost 5% improvement over Cinematch. The push to the .8563 RMSE and $1 million dollar prize will be a tough one.

So Far:
The best we've come up with on the entire probe set so far is 0.98. And that is just using the basic statistics forula's, nothing fancy like content or collaborative filtering yet. We are still struggline with the immensity of the dataset and how to come up with an efficient way to do the collaborative filtering part.

Many have said that the content filtering will not yield any better results however, after a number of preliminary tests using small groups for collaborative filtering our results suggest more data points will be required when predicting for users and/or movies with few ratings or disparate ratings. But hey, we aren't scientists so we could be wasting our time. We might take a little side track here :).

On ward we go, but slower now..

Oh the fun we are having ;)

Yes, this is truely an enjoyable challenge. In the process of working with hundreds of millions of records, data base tuning, mathematical paradoxes, and other interesting things, we are learning a lot. And the bonus is these are things that can be applied to our daily challenges at our regular jobs.
Since we are now well into the leader board with a few algorithms, we won't be spoiling things by divulging any 'secrets' here for our competitors. Not that we are even a competitor in this challenge, but hey, you never know who will come up with the magic algorithm in the end. I suspect that like many great inventions, this one may very well come from a group not necessarily specializing in this area, otherwise why would Netflix go 'out-of-the-box' to try and solve an old problem.
Alot of folks think computing power is a big constrainer but I think otherwise. I think the dataset provided is well suited to something like MySQL running on an average home computer. Time will tell.
We should be attempting a submission sometime next week as we are getting probe set results that would put us there.

Data Geeks!..........

Crunch those numbers!

D has been furiously, and I say furiously because this stuff can be frustrating as heck, working on the cohort tables. We think we know how these will be formatted but are constrained by CPU power and time to process them in their final format so we are looking at different ways to compress all the data into a model that we can load into around 1GB of RAM come crunch time.

Like some others we are now using a suite of tools for all sorts of tasks, the main one being MySQL. So far we are using C# to do some analysis, Perl for data transforming and user-user testing on smaller subsets.

Hooyaaa Data.....

0.9685 on 50% of the probe set!

Wow, this is fun. Numbers numbers and more numbers. Applying some simple standard diviation based algorithms had us at the score above for the middle 50% of the probe data. Unfortunately the upper and lower percentiles are proving nearly impossible to predict accurately without training our algorithms to the probe set only. So it seems we must do much better than a RMSE of 0.9685 for the stuff we can predict. Here are some numbers:

For the middle 50% we are at 0.9685, for the upper and lower percentile remainder we so far cannot get below 1.011, for an overall overage of 0.9968 which though a respectable RMSE it is not on the leader board though we may submit something soon just to see how our submission files work and maybe we just might get lucky? Wishful thinking I guess.

Anyhow, so far our Ubuntu Linux box (spider) is holding up well, our MySQL catalog for this project is now up near 15-20Gb, almost half of which are indexes to speed access to the data. I expect once we complete our cohort groupings we may need alot more disk space so I may need to swap that 40G for an 80G or larger, and maybe even getter more RAM, a faster CPU, and a faster hard drive while I'm digging around in there, heck I'll just get a near system to boot with one of them fancy SATA drives.

Our newest member D is trying more radical approaches, drilling into the data beyond the numbers as well working directly on the Linux box using Perl and other ~ix stuff. We downloaded the IMDB genre and other tables and may at some point do some joining and analysis using this data but we'll see.


Eat Fresh!!!

Getting to know MySQL

Ok, so dealing with over 200 million records takes a bit of work, yes it does. Thank fully Linux and MySQL have both come along way in user-friendliness. I was able to load the standard 4.0.x version of MySQL that came with my old Ubuntu Breezy Badger CD and I later upgraded to 4.1, without going crazy, or wasting an entire weekend, wooohooo..!

Getting all the data into what I refer to as indexed base tables from which I can build layers of data to support my future algorithms was not fun. For some reason my original download was off on its MD5 and since I didn't bother to check it I was chasing my tail for a day or so trying to figure out why my record numbers weren't jiving. Note to self-check the damn checksums!

Anyhow, as of tonight base data and indexes are all in place and the real work begins. I probably won't be sharing any 'trade' secrets here but will continue to post our progress as we start running the probe set against our algorithms. Most everyone is going to use some sort of collaborative predictor along with some brewed up statistical tweaks I'm sure. We will be heading that way. Our goal is a sub 1.0 subission by December.

I like MySQL, why? Check this out:

select avg(r) from r where m = 12654
>2.75
1 row fetched in 0.0019s

And thats from 120 records out of a table with 100,480,507 records.
Ok, so I indexed it, but still, running Ubuntu Linux on AMD Athlon 2GHz with 512Mb RAM the box makes a decent database server for doing minor calculations and data transformations.

Only thing I don't like about Ubuntu is no root account. Which helps to keep me out of trouble but sudo'ing gets old.


Laters All,
Marc

Now that's alot of data!

Ok, so we move forward, a little bit anyway. Since we don't have any extra cash sitting around we needed to make use of some free stuff and other stuff sitting around.

A database server was needed to house the 100 million plus records as well as perform the large array of statistical queries we plan on running. Linux has definitely come a long way. We chose an olrder Ubuntu Distro since I already had a CD burned. We used an older 2GHZ AMD Athlon box with 512MB of RAM, one that usually serves as my test box for various projects.

So for $0 and about 2 hours we had a nice little MySQL database server ready to accept a ton of data! Tonight I wrote a simple program in c# to convert and load all the test data into our new database server. So far we are getting about 1 Million records loaded per hour so this will take a few days to complete. In the meantime we'll start working on some recipes for our predictor algorithm and start testing it on the live data as it is loaded. I probably should have transformed the data then direct loaded into MySQL instead of from my XP client via ODBC but oh well, the thing about this whole project is for it to be fun!

Onward we go...

Hey, whats this all about..

I decided to hop on the Netflix contest band wagon. I even managed to recruit my daughter, afterall she does have an amazingly creative and intuitive mind. So, whats this contest all about? Well, mostly about numbers, and lots of them. Check the links out on the right for more info and check back here for our updates. We'll be building a database server tonight, not sure yet if it will be Linux or Windows as its only job will be to serve the data. I'm thinking Linux with MySQL or something similar. We have an extra older PC lying around to will fit perfectly into this role. Better check wit the wife first though. I expect the database needs to exceed 1 billion records once everything is loaded so some research in is order. See ya later...