Language

I was having a discussion the other night with a friend about computer programming languages. She was trying to explain what a programming language is to a non-tech person. Something she said triggered a memory about a talk I saw at OOPSLA '98 in Vancouver 10 years ago. Out of all the computer conferences and talks I've heard over the years, this one particular talk stands out and I point people to it whenever appropriate.

It was the keynote talk by Guy Steele, one of the authors of the Java language. He did something very unique in the talk that kept the audience engaged for almost the entire hour. It was a technique I haven't seen before or since, and while a bit gimicky, it was definitely memorable.

Fortunately, someone has posted the talk to Google Video with some enhanced graphics.

http://video.google.com/videoplay?docid=-8860158196198824415

It might be surprising to some younger folks I know to see that he used an actual overhead projector with handwritten slides. Projectors and powerpoint for presentations were rare even just 10 years ago. The content of the talk is just as relevant today to computer science as it was 10 years ago.

Compiling Firefox with tcmalloc

Here's how I compile firefox with tcmalloc to experiment if it helps with the fragmentation problem.

Follow the standard firefox build instructions here: http://developer.mozilla.org/en/docs/Build_Documentation

My .mozconfig:

. $topsrcdir/browser/config/mozconfig

# Options for client.mk.
mk_add_options MOZ_CO_PROJECT=browser
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-@CONFIG_GUESS@

# Options for 'configure' (same as command-line options).
ac_add_options --with-pthreads
ac_add_options --enable-application=browser

Patch to disable zone allocator because I want it to allocate everything, even small objects, directly via tcmalloc. There's probably a better way, but this was quick and dirty.

Index: nsprpub/pr/include/private/primpl.h
===================================================================
RCS file: /cvsroot/mozilla/nsprpub/pr/include/private/primpl.h,v
retrieving revision 3.87
diff -r3.87 primpl.h
1867c1867
< #define _PR_ZONE_ALLOCATOR
---
> /* # define _PR_ZONE_ALLOCATOR */

Build command to link tcalloc in:

LDFLAGS=-ltcmalloc make -f client.mk build

To run with the heap profiler I do:

HEAPPROFILE=/tmp/firefox ./firefox

This creates heap dumps in /tmp like firefox.0001.heap. By default it does it whenever the usage increases by 100MB and when a total of 1G of objects have been alloced (regardless how much is still in use). See http://google-perftools.googlecode.com/svn/trunk/doc/heapprofile.html for which env vars to set to change this, as well as how to examine the heap in pprof.

I don't really have any results to report yet. It would be cool to figure out how to instrument something like the images Pavlov has created showing the fragmentation. It shouldn't be that hard to do with tcmalloc or a malloc wrapper. A malloc wrapper would make it easier to compare different implementations, but might be harder to implement without adding a lot of overhead.

Building JDK 7 from source on ubuntu feisty

1. Download sources from http://jdk7.dev.java.net.


2. Several additional ubuntu packages are required:
   • apt-get install sun-java6-jdk libasound-dev libcupsys2-dev xutils-dev m4 libxt-dev x11proto-xext-dev libxext-dev libxtst-dev libxi-dev libxp-dev libxmu-dev
   • Note: if it breaks because of a missing system header or lib, go to packages.ubuntu.com and search for the missing file to find out what package to install.


3. I had to edit one Makefile to fix a compile error (will submit this patch to Sun).
   • vi j2se/make/sun/jdbc/Makefile
   • add -fPIC to the compile commands under the make_libs target (after $(CC))


4. Copy resources from a previous JDK binary release because they aren't included with the source:

   cd jdk1.7.0-b07/jre/lib
   mkdir resources ; cd resources ; jar -xf ../resources.jar
   ( cd com/sun/java/swing/plaf/motif/icons/ ; cp *.gif ~/src/jdk/jdk7/trunk/j2se/src/share/classes/com/sun/java/swing/plaf/motif/icons )
   ( cd com/sun/java/swing/plaf/windows/icons ; cp *.gif ~/src/jdk/jdk7/trunk/j2se/src/share/classes/com/sun/java/swing/plaf/windows/icons )
   ( cd javax/swing/plaf/metal/icons ; cp *.gif ~/src/jdk/jdk7/trunk/j2se/src/share/classes/javax/swing/plaf/metal/icons )
   ( cd javax/swing/plaf/metal ; cp -r sounds ~/src/jdk/jdk7/trunk/j2se/src/share/classes/javax/swing/plaf/metal )
   ( cd ../lib ; cp -r audio ~/src/jdk/jdk7/trunk/j2se/src/share/lib )
   

5. Create build-linux.sh script:

   #!/bin/sh
   export ALT_BOOTDIR=/usr/lib/jvm/java-6-sun
   export ALT_DEVTOOLS_PATH=/usr/bin
   export FULL_VERSION="$(basename $(pwd))-$(date +'%Y''%m''%d''.''%H''%M')"
   export HOTSPOT_BUILD_JOBS=2
   export PARALLEL_COMPILE_JOBS=2
   cd control/make && /usr/bin/make MAKE_VERBOSE="" ECHO="echo" dev
   

6. time ./build-linux.sh


7. Go get lunch


8. If it works, you should have a jdk in control/build/linux-amd64 (or linux-i386 depending on your platform)


9. Run bin/java -version:

   java version "1.7.0"
   Java(TM) SE Runtime Environment (1.7.0-internal-fastdebug-kellyc_06_mar_2007_22_32-b00-fastdebug)
   Java HotSpot(TM) 64-Bit Server VM (build 1.7.0-internal-fastdebug-kellyc_06_mar_2007_22_32-b00-fastdebug, mixed mode)

   
   

my iPhone thoughts

Several other folks have been blogging about the iPhone, but so far it seems to be all gushing praise, and yes, alot of that is deserved. It's a beautiful product, as Apple almost always produces. As for some of the lacking features (3G), yes, Apple should have added those, but remember this is a first gen product. Does anyone remember the first gen iPod? You have to make tradeoffs and pick features in a first gen product.

There are a few things I think they missed though, and some nagging questions which their marketing materials don't answer yet:

1. One handed operation? One of the key advantages I see for blackberry users is the scroll wheel. I do not have a crackberry, but when I have checked them out, that is really the killer feature, not the full qwerty keyboard. Most people read alot more email than they compose on it.
2. Replaceable battery? C'mon Apple. You've dealt with enough iPod battery problems that you should have a clear answer for that issue on this product. Who is going to want to send in their primary means of communication to get a battery fixed for a couple weeks?
3. Charging, especially mobile. I have an iPod nano, and I don't think the connector is particularly easy to connect normally (figuring out if it's right side up from a screen printed icon is not the best, especially if the icon has worn off). With my motorola phone (non-Razr) I can pretty much plug it into the charger in my car single handed without looking, something very nice if I'm on a call and it does the low power beep. I suppose they expect you to dock it in your car before you start driving. Similar for unplugging it. When I'm trying to catch a train to work, I want to be able to unplug my phone single handedly without looking.
   
4. What kind of CPU? I know it runs macos X, but what kind of horsepower does it have?
   
5. I'm sure the screen will get smudged up especially holding it up to your ear.
   
6. How much non-talk time battery life is there? Especially if you use it for video playback?

I expect the 2nd gen iPhone to already be in the works and probably ready for release by the end of the year or Q1 2008. I don't remember how long it took for the 2G iPod to come out, but like the iPod this is a revolutionary product that will continue to evolve into something better. I love the nano so much more than my old 2G iPod, so I'm looking forward to the 3rd and 4th gen iPhones (and what nokia, etc. do to answer it)

Linux might as well not have fonts

So I installed Fedora Core 5 on my laptop tonight because I installed it on my 64bit box last weekend and it looked kinda nice. I don't use the 64bit box's UI much. My laptop had a ubuntu 6.0x beta or something on it and I was quite please with ubuntu as far as it supported everything on my laptop out of the box but I'm so used to Redhats locations of files and settings that it was a little bit of a learning curve.

So to my surprise, FC5 really sucked out of the box on my laptop. Things that didn't work:
1. Wireless (needed to go get a proprietary firmware for Intel)
2. NTP server synch with no network basically halts the startup process
3. Media buttons (volume up/down/mute)
4. NFS mounting of my terrastation volume (RPC Timeout and then Segmentation Fault)
5. Fonts

So I fixed 1 pretty quick by hooking up an ethernet cable.

But #2 required booting into single-user mode (add -s to the grub boot line) and turning off the option to sync the clock (rm /etc/ntp/step-tickers)

Media buttons required setting the keyboard shortcuts in System->Preferences->Keyboard Shortcuts.

Fixing NFS took alot of effort including looking at the packets in ethereal. The RPC Timeout was the most frustrating. Finally I figured out that terastation was doing dns lookups and I didn't have it setup to route to any dns servers. It just happened that the timeout of DNS and the response packet from the mount command were almost always timed such that the client timed out 1/2 a second before the dns timed out and the response was sent. Once I got it so it could see a dns server the mount command core dumped. ulimit -c unlimited and gdb mount core showed nothing useful (symbols were stripped). So I went looking for the mount source. I found it at kernel.org (util-linux). After compilation running mount just worked. yay. stupid friggin redhat with their custom patches.

Ok, the last thing (so far...) Fonts. My Google personalized page looked like total crap and I had to scroll alot more than I used to. And Google Finance's charts are all but worthless cause Flash doesn't display any text at all. I solved that problem in ubuntu by installing ghostscript-fonts. Unfortunately they were already installed on fedora.

So I went hunting for the standard MS web core fonts... found several different RPMS of them. None worked. Lots and lots of research and editing of /etc/X11/fs/config later and still nothing... Going to fonts:/// in nautilus showed like 10 really basic fonts still. None of the standard ones I installed. Bummer. So I kept hunting. Finally I found that there's this fontconfig package installed. It has it's config in /etc/fonts/ but it doesn't include even some of the standard packages like the ghostscript or Type1 URW fonts that were installed with Fedora. What a crock. Somehow it finds the Bitstream ones in a subdir of /usr/share/fonts, but nothing else. So I added a local.conf with the following dirs:

/usr/share/fonts/webcore
/usr/share/fonts/default/Type1
/usr/share/fonts/default/ghostscript

Woohoo! Firefox now has a much better look. But still no text in flash on Google finance.

Oh well... it's 5am and I've been at this for like 10 hours now. I think I need to go to sleep. I hate Fedora now. I'll probably switch back to ubuntu on June 1 when the final badger comes out.

Beat the clock

The aforementioned clock problem was noticed early on while running 2.6.11 x86_64 kernels on this new machine. The clock ran 3x faster than it should have. So a minute only took 20 seconds in earth time. I searched all over and found several other people having similar issues, all of them with some type of Athlon. Usually the 3000+ just like mine. Most of them had laptop systems. I'm using a microATX board, the MSI RS480M2-IL. It will take a while to get to the solution, but I did find one.

Nobody had any solutions posted. I started out trying some different bios settings. One of these settings gave me a no-boot situation where the machine stopped mid-POST (after CPU, but before memory tests). It was the A20 setting which had options of Fast or Off. I didn't know what A20 was, but I thought, "hey, if it's fast now, maybe turning it off would slow the clock down."

So now the machine doesn't boot.
Step 1: clear the CMOS. I tried using the jumper for this purpose. No dice. Took the battery out and unplugged power. Nope.
Step 2: Remove stuff... pci cards, memory, disks. I even reseated the CPU. Nope
Step 3: What a pain... I'm actually considering the possibility that I need a new BIOS or a new board. I investigate the BIOS route, including how to build an EEPROM flasher. Or buying a new BIOS online. I bet someone at work somewhere has an EEPROM flasher, afterall, I do work in Silicon Valley now.
Step 4: Try the BIOS recovery procedure from MSI's tech support site. This requires finding a floppy drive and even more rare, a floppy disk. I disect one of my Dell's for the floppy drive, and wipe one of the driver disks that came with the mobo. After all that, the BIOS recovery doesn't even kick off like it was supposed to.
Step 5: It stops at the memory test... maybe if I try completely new memory. I pull the memory out of my Dell, and as I am putting it in, I realize the light on the optical mouse I've been using is still lit up.

Eureka! All the power is disconnected from the machine, however it's connected it to a KVM, which is also connected to another Dell which is powered up. So I unplug it from the KVM, pull out the battery, and wallah! the next boot worked. So that CMOS clearing jumper either doesn't work at all, or doesn't work when there's a powersource other than the battery on the motherboard.

Ok, after alllllll that.... I'm back to a bootable machine which runs 3x faster by the clock.

I tried booting an i386 kernel from the Fedora cd, and the clock speed was normal there, so I start looking into the kernel timer routines. I spent a few hours studying the code and undestanding more of it than I cared to. Nothing obvious stuck out. I wondered if maybe the timesource was different than what they expected. I learned all kinds of stuff about 8253 PIT's and HPET's and stuff no Java programmer should ever care about. It was fun.

I noticed how similar the x86_64 and i386 timer code was, but there were some differences. It was obvious the 64bit code had been forked from the 386 code some time back. So I started looking around on the linux kernel mailing list. I stumbled upon this patch by John Stultz, which he just happened to post a new version of hours before I started this kernel investigation: http://lkml.org/lkml/2005/3/11/309

I started trying to apply John's patches. They required some minor tweaking to get it to compile, but in the end I had a kernel rpm with the patch, and booted it up and
WoooHOOOO! A minute takes a minute now. Pretty important thing for a video recorder that schedules recording tv shows that come on at certain times. :-)

Step 2: Install linux

This has by far, been the most frustrating step so far, and I will probably make more than one post about it cause it's not done yet. Many issues have cropped up because of my hardware choices.

Here's what I've done so far:

1. Downloaded Fedora Core 3 x86_64 cd iso's.
2. The installer couldn't see my SATA drive. I tried loading different drivers. I started with the Silicon Image driver because I knew that's what the motherboard used from various sources online and the driver disk included with the mobo.
3. None of the drivers worked, so I tried changing the setting in BIOS to use the SATA RAID feature instead of IDE emulation. That didn't work either.
4. Punt: installed an old IDE disk from my old machine that was dead figuring I could install a newer kernel version (FC3 ships with 2.6.9)
5. Installed on the old IDE drive. This was a major PITA too. The install was taking forever. It seemed like it was having trouble reading the cd's. I switched into the other virtual terminals on the installer with ctrl-alt-f2, f3, etc. to read the logs. There were errors there about timeouts while reading from hda (the dvd/cd drive).
6. Since the DVD player was brand new and a cheapo OEM version, I thought maybe it was bad or the cable was bad. I tried replacing each with known working cables/players from other machines. Still no luck.
7. So I downloaded and burned the dvd iso instead of the cd iso's and just let the install run until it completed. This took half a dozen hours.
8. Finally, the machine is running linux. But it's really really slow, just like the install was, and my SATA drive is still not visible. So I searched and found that the SATA driver was updated by someone from ATI in kernel 2.6.11. So I installed the 2.6.11 rpm from Fedora rawhide.
9. Upon reboot, the SATA drive was visible, and the machine ran much much faster. In fact, the clock ran at 3x the normal speed. Like it gained 1 minute every 20 seconds. Even NTP wouldn't be able to correct for that amount of drift. This 3x clock problem deserves it's own post.
10. I wanted to try installing only on the SATA drive cause the goal is to not have the old drives in this machine (they're noisy, and I don't want a noisy machine in my living room). So I rsynced all the Fedora FC4 rawhide rpms and built an iso dvd.
11. Installed on the SATA drive from the FC4 iso. However it couldn't boot from the SATA drive. So I have to use the boot partition on the old IDE drive.

Linux is installed and running, but the 3x problem has to be fixed. MythTV's recording schedule will be way out of whack if I can't fix it. And I still have a noisy old disk in there.

My linux pvr project: step 1, build the box

I was getting bored at home (new in town, not too many friends here to hang out with yet), so I decided to build a new linux personal video recorder machine. It's turned out to be a fun and challenging project. More challenging than I expected.

I picked out the components at MicroCenter in Santa Clara last week. Here's what I got:

Motherboard: MSI RS480M2-IL
I picked this one cause it was inexpensive and had onboard video including tv-out and svideo. It's a socket 939 AMD board with SATA and all kinds of other good stuff. More on this later when I discuss getting linux installed and working.

CPU: AMD Athlon 3000+ socket 939

PVR card: Hauppage WinTV PVR-250

HDD: Maxtor 300GB SATA 7200RPM

Case: Antec Sonata

RAM: 512MB

All together, the components cost about $1000. Pretty good for a 64 bit socket 939 box (128-bit memory access path instead of 64 in socket 754)

It took about an hour to assemble the box. The hardest part there were getting the front panel usb, firewire, and sound connectors hooked up to the right jumpers on the motherboard.

Up next: installing linux