Getting to know MySQL

Ok, so dealing with over 200 million records takes a bit of work, yes it does. Thank fully Linux and MySQL have both come along way in user-friendliness. I was able to load the standard 4.0.x version of MySQL that came with my old Ubuntu Breezy Badger CD and I later upgraded to 4.1, without going crazy, or wasting an entire weekend, wooohooo..!

Getting all the data into what I refer to as indexed base tables from which I can build layers of data to support my future algorithms was not fun. For some reason my original download was off on its MD5 and since I didn't bother to check it I was chasing my tail for a day or so trying to figure out why my record numbers weren't jiving. Note to self-check the damn checksums!

Anyhow, as of tonight base data and indexes are all in place and the real work begins. I probably won't be sharing any 'trade' secrets here but will continue to post our progress as we start running the probe set against our algorithms. Most everyone is going to use some sort of collaborative predictor along with some brewed up statistical tweaks I'm sure. We will be heading that way. Our goal is a sub 1.0 subission by December.

I like MySQL, why? Check this out:

select avg(r) from r where m = 12654
>2.75
1 row fetched in 0.0019s

And thats from 120 records out of a table with 100,480,507 records.
Ok, so I indexed it, but still, running Ubuntu Linux on AMD Athlon 2GHz with 512Mb RAM the box makes a decent database server for doing minor calculations and data transformations.

Only thing I don't like about Ubuntu is no root account. Which helps to keep me out of trouble but sudo'ing gets old.


Laters All,
Marc