George V. Reilly's Technical Blog

StackOverflow DevDays Seattle 2009

2009-10-27T07:17:00Z

I spent last Wednesday at Benaroya Hall, attending the Seattle edition of StackOverflow's traveling DevDays conference. It was well worth $99.

Joel Spolsky, owner of FogCreek Software and co-founder of StackOverflow, opened the conference with a keynote about the dichotomy of power and simplicity. People are happier when not overwhelmed with choices. Many of the choices that software forces users to make are essentially meaningless to the users. However, even though people want simplicity, they also want features and different people use different features. Powerful software sells more copies.

He argues that developers and designers should put in the extra work to make good choices on behalf of the users: don't make users feel bad about themselves. Undo is better than a confirmation dialog. You are not in charge of what your users do.

Scott Hanselman spoke about ASP.NET MVC. We're moving away from ASP.NET to Python, but if we were to use ASP.NET again, MVC would be a compelling feature. His presentation was entertaining, if gimmicky.

Rory Blyth introduced iPhone development, in a tone of snarky ambivalence. He mentioned the Stockholm Syndrome. He stressed that Apple's Design "Guidelines" are effectively laws: violate them and you won't make it into the App Store. Looks like there's a lot of tedious messing around to hook things up in Objective-C. At the very end, he briefly demoed MonoTouch, which seemeed a little less tedious.

Cody Lindley introduced jQuery. I've done a lot of work with jQuery, but I still learned a few things. He worked through five facets of jQuery: Find something, do something; Create something, do something; Chaining; Implicit iteration; and jQuery parameters. He has an ebook at jqueryenlightenment.com, which I just picked up.

Daniel Rocha of Nokia talked about the cross-platform Qt (/cute/) toolkit, which runs on Windows, Mac, and Linux. More importantly from Nokia's point of view, it runs on their smartphones. Nokia has changed the licensing of Qt—once very expensive for closed-source apps, it's now free for apps that don't modify the Qt source. Qt is for C++, but there are bindings for other languages, such as Python.

Joel Spolsky came back and treated us to a half-hour demonstration of Fogbugz 7, Evidence-Based Scheduling, and Kiln, their new hosted Mercurial repository. Not terribly interesting to me, but the conference was only $99.

Ted Leung gave us a rather dry Hacker's Introduction to Python from slides rendered unreadable by a poor choice of colors. I've done a lot of Python, so I didn't learn much new. pip is an easy_install replacement that uninstalls; zc.buildout assembles apps from multiple parts; bpython is a fancy REPL.

Dan Sanderson talked about Google App Engine and demoed building apps with Java and with Python. Looked pretty cool and straightforward. We probably won't go that route, since we're pushing data to Amazon's S3, so EC2 makes more sense for us.

Finally, Steve Seitz from the University of Washington gave a cool talk on Modeling the World from Internet Photos. Some of this technology ended up in Photosynth. See Building Rome in a Day for some demos.

Review: Pragmatic Version Control Using Git

2009-10-19T05:44:00Z

Title: Pragmatic Version Control Using Git
Author: Travis Swicegood
Rating: $stars(4)
Publisher: Pragmatic Bookshelf
Copyright: 2008
ISBN: 1934356158
Pages: 179
Keywords: computers
Reading period: 10–18 October, 2009

As part of my personal conversion to Git, I read Swicegood's Git book. It's a decent introduction to Git and you learn how to do all the basic tasks as well as some more advanced topics. The examples are clear and well-paced.

I would have liked to see more about collaboration and workflow in a DVCS world, perhaps a few case studies: how is Git used in the Linux kernel development process; how a small, distributed team uses Git and GitHub; how a collocated team migrates from more traditional tools.

The book avoids discussing the lower levels of the Git object model, which is a reasonable choice for a pragmatic guide.

Recommended.

Third-Party Cookies

2009-10-17T07:15:00Z

Over the last few weeks, I built a PHP application that overlays Approve 71 banners on profile pictures. The actual application is hosted in an iframe and lives on a server in a different domain, eq.dm, than the main server at approvereferendum71.org.

This works fine in most browsers. Then we started getting reports that it wasn't working in IE8 on Win7 RC1. The iframe content was blank.

Poking around, I found the problem with the Fiddler proxy. The landing page on eq.dm was supposed to stick some information into the PHP session, then redirect to a second page at the same site. The second page was in an endless loop, redirecting to itself. In Fiddler, I saw a different PHPSESSID cookie on each response, and no cookie in the requests.

After reading IE 8 only has access to session cookies, I told IE8 to Accept All Cookies and the iframe content appeared. That fixed it for me, but we could hardly ask people to lower their security sessions.

I created a P3P file for the second domain, using the IBM P3P Policy Editor. (KB 323752 has more background on P3P and third-party cookies.)

IE now worked at its default security level. Problem solved! Or so I thought.

A day later, we got reports of similar problems with Safari 4 on Mac OS X.

I sniffed the traffic with Wireshark. Same problem: the “third-party“ cookie wasn't being accepted by Safari.

Unfortunately, Setting cross-domain cookies in Safari indicated that there was no reasonable workaround.

We overcame the issue up playing some DNS games, which was only possible because we control both servers. The second server is now also acting as a subdomain of the first, at dev.approvereferendum71.org. We used ini_set("session.cookie_domain",".approvereferendum71.org") to scope the iframe cookies. I've tried it in a variety of Windows, Mac, and Linux browsers, and it works in all of them.

Gitting Along

2009-10-15T15:24:00Z

In the last few weeks, I've switched over to Git for most of my version-control needs, at home and at work, after putting it on the long finger for months.

We continue to use Subversion at work, but I've recently followed Pavel and Eric's lead in using git-svn. I work locally on my own private branches and git svn dcommit and git svn rebase occasionally. I'm primarily on Windows at work, but I have a Linux box and a Mac Mini too, while at home, I have a MacBook, a Linux netbook, and a Vista desktop. I'm using msysGit, occasionally supplemented by TortoiseGit and QGit. Pavel's on a Mac and Eric's mostly on Ubuntu, so git adoption was easy for them.

When I first tried git-svn under msysGit about a year ago, it didn't work worth a damn. Git-svn works fine now, but it's slow compared to the *nix implementation. The developers say that's due to the fork() emulation of the MSys/Cygwin layer. The rest of msysGit is much faster.

For my home needs, I've had private Subversion repositories at DevjaVu.com and OpenSvn.csie.org. DevjaVu has gone out of business and OpenSvn has been unavailable too often for my liking. It was time to find some new hosting.

I've experimented with private Git repositories at GitHub and ProjectLocker. GitHub is very nice, but charges for private repositories. ProjectLocker provides free private repositories, but is comparatively clunky.

ProjectLocker lets you set up a fresh repository on their server. They tell you how to clone from that, which is great for a new repository. But they don't tell you how to hook it up to an existing local repository. Since I had some difficulty in figuring it out, here's the recipe:

git remote add origin git-foobar@freeN.projectlocker.com:foobar.git
git pull origin master
... merge, local edits and commits ...
git push origin master

I found Git, Xcode and ProjectLocker and Cygwin, SSH and ProjectLocker useful in figuring this out.

Launching 32-bit applications from batchfiles on Win64

2009-09-12T06:16:00Z

I've been running the 64-bit version of Windows 7 RC since June. It's been quite painless on the whole.

One wrinkle that I ran into was with some batchfiles which launch applications in %ProgramFiles% (normally C:\Program Files). Due to the magic WOW64 redirector, 32-bit applications are actually installed into %ProgramFiles(x86)%—normally C:\Program Files (x86)—instead of %ProgramFiles%. This is transparent to the 32-bit applications, which think they're running in %ProgramFiles% (C:\Program Files).

However, the cmd.exe shell is 64-bit (unless you make a special effort to run the 32-bit cmd.exe in SysWOW64), so batchfiles see the 64-bit %ProgramFiles% which contains 64-bit applications.

Hence, a batchfile that launches an installed 32-bit application on Win64 must use %ProgramFiles(x86)%, not %ProgramFiles%.

It sounds trivial to have a batchfile detect which flavor of %ProgramFiles% it should use, but the parentheses in the environment variable name make it tricky to parse. On earlier versions of Win64, the environment variable was called %ProgramFilesx86%. Presumably they added the strange parentheses into the variable name because the directory name always included them.

Here's a tiny batchfile that will launch the 32-bit DiffMerge correctly on both Win64 and Win32 platforms.

@setlocal
@set _pf=%ProgramFiles%
@if not "[%ProgramFiles(x86)%]"=="[]" set _pf=%ProgramFiles(x86)%
@start "" /b "%_pf%\SourceGear\DiffMerge\DiffMerge.exe" %*

I long ago found that the safest way to test environment variables whose values may include spaces, is to surround them with both double quotes and square brackets.

Recovering photos from a corrupted card

2009-09-05T07:45:00Z

I had about 60 apparently corrupted photos on a CompactFlash card the other evening. It might have been due to Lightroom going berserk, but it was more likely from my pulling the card reader out of the computer without ejecting it first.

The photos wouldn't show up under Mac, Linux, or Windows. I tried to chkdsk the card under Windows, which complained about a “raw” disk. That led me to ZAR, the Zero Assumption Recovery toolkit. The evaluation copy retrieved the photos very nicely. Whew!

XKCD's Tech Support Cheat Sheet

2009-08-25T06:57:00Z

To my tech supportees:

XKCD's Tech Support Cheat Sheet nails it! This is essentially what I'm doing when I sit down at your computer and dig you out of your latest hole.

If only you'd make some more intelligent guesses for yourself, you might be able to solve more of your problems.

But it's not quite that simple.

You probably don't understand very much about what you're doing. I have enormous depth and breadth of experience which informs my investigation. I am—no false modesty here—a master of debugging. The extra context helps me hone in on a solution more often than not.

Still, do give the flowchart a shot. Being able to solve your own problems is very empowering.

Moving Photos in Lightroom

2009-08-18T05:42:00Z

When using Lightroom before, I was never able to figure out how to move photos from one folder to another. You'd think that you could just click on a photo and drag it. I just spent twenty minutes figuring out what I've been doing wrong. After you've selected multiple photos, click on the photo thumbnail and not the surrounding gray frame, and then you can drag the photos to the target folder.

I had become accustomed to clicking on the frames to multi-select photos, so naturally I assumed that was also how you dragged a set of photos. But clicking on a frame of a selected photo merely deselects all the other photos.

64-bit Windows 7

2009-07-11T06:12:00Z

I mentioned three weeks ago that I had just repaved my work dev box and installed the 64-bit version of the Windows 7 RC. Nine or ten years after I first ported parts of IIS to Win64, I am finally running my main desktop on 64-bit Windows. With one exception, it's been painless. Programs have just worked, devices have just worked. There are relatively few native x64 applications, but for the most part it doesn't matter. The cases where it does matter—e.g., shell extensions such as TortoiseSVN—are available as 64-bit binaries.

I briefly flirted with using the 64-bit build of Python, but realized that I would have to recompile several eggs as 64-bit binaries. That was too painful and the 32-bit binary did everything I needed.

Building in Visual Studio 2005 is noticeably faster. I'm not sure how much of it was due to accumulated cruft after 18 months on Vista, but builds there were very slow.

The one exception was a major problem for the first week and a half. Whenever I ran our ASP.NET web application, it would go berserk, eat up all 4GB of my physical RAM, push the working set of IIS's w3wp.exe to 12GB, and max out one of my 4 cores! The only way to maintain any sanity was to run iisreset every 20 minutes to gently kill the process.

WinDbg and Process Explorer showed that the rogue thread was stuck in a loop in mscorjit!LifetimesListInteriorBlocksHelperIterative<GCInfoLiveRecordManipulator>. I passed a minidump on to my former colleagues in IIS, who sent it to the CLR team. They said:

The only thing I can tell is that it is Regex, and some regex expression compiled down to a method with 456KB of IL. That is huge, and yes 12GB of RAM consumed for something like that is expected.

With that clue, I was able to track down the problem, a particularly foul regex, built from a 10KB string, with 32 alternating expressions, each of which contains dozens of alternated subexpressions. The string is built from many smaller strings, so it's not obvious in the source just how ugly it is. I commented out the new Regex() and my problems went away.

Regardless of how ugly the regex is, this is a major regression in the CLR. This code has been working without blatant problems for two years on the 32-bit flavors of XP, Server 2003, Vista, and Server 2008. I've been meaning to try this code on 32-bit Windows 7, but have been too busy.

(The original, long-gone author was apparently aware that the regex is expensive to create because he runs a background thread to new the regex, which should have told him something. We'll fix the code that uses the regex to do something saner, soon.)

All that aside, I've been happy with the 64-bit version of Windows 7.

When Video Cards Go Bad

2009-06-20T04:50:00Z

I complained a week ago about my display driver going berserk. I blamed Windows Update, since it happened within hours of a pile of updates being installed. I upgraded to the latest beta NVidia drivers on Monday and it helped for a while, but by Wednesday, it was almost as bad again as it had been last Friday. It was infuriating and I was both entertaining and alarming my neighbors with my cursing.

Today was the last day of a very busy sprint for me and at last I had the time to dig into it. I opened up the case and took a look at both video cards—I have two dual-head cards connected to three monitors—and one of them had partially blown capacitors like those in the picture. I removed the bad card and did some graphics-intensive things for an hour, and the other card behaved flawlessly.

Oddly, until someone mentioned that it might be a hardware problem yesterday, it didn't occur to me, even though a video card blew in this machine last year. I came in one morning to find a black monitor, and when I pulled out that card, I found that some of the capacitors had popped right open with stuffing protruding.

On general principles, I had been meaning to repave this machine for a while. I've had it since December 2007 and it was still running the original installation of Vista. I booted from a DVD, reformatted my C: drive, and installed Windows 7 x64 RC1.

I finally have a 64-bit OS as my primary Windows desktop, so I'll actually be using the Win64 build of Vim that I maintain. My first impressions of Windows 7 on this machine are very favorable, but there's plenty more that I need to install before the machine has everything that I need.

Display driver nvlddmkm stopped responding and has successfully recovered

2009-06-13T07:34:00Z

This morning, the video adapters on my Vista dev box were resetting 2–3 times per minute.

After a pile of Windows Updates landed on my machine at 3am yesterday, it would occasionally freeze solid for a few seconds. Once in a while, all the monitors would go black briefly, then restore. Each time, I would see a status update pop up from the system tray, "Display driver nvlddmkm stopped responding and has successfully recovered."

This was irritating enough that I downloaded the latest NVidia drivers this morning, 185.85_desktop_winvista_32bit_english_whql.exe. That really screwed me. The video adapters started resetting 2–3 times per minute, rendering the machine almost unusable. I have two video adapters, NVidia GeForce 8600 GT and NVidia GeForce 7600 GT.

The eventlog was full of Event ID 4101 - Display Driver Timeout Detection and Recovery.

I reverted to the 178.24 drivers and that helped. When I'm not touching the machine, the adapters only get reset every few minutes instead of several times a minute. When I am using it, something as simple as clicking a window to bring it to the foreground can trigger a reset.

It's very irritating but I can live with it for a little while, unlike the other. I don't want to repave my box: apart from the time loss, I'm not convinced that it would help if I got the same driver config all over again.

I contacted a friend at Microsoft who tried to hook me up with a driver guy, who is unfortunately out of office. I'm hoping that it can be fixed early next week or my temper is going to fray rapidly.

Update: June 19th: See When Video Cards Go Bad.

Win64 Vim 7.2.182

2009-05-22T06:11:00Z

I updated the Win64 binaries of Vim at vim-win3264 from Vim 7.2.000 to 7.2.182.

I'm amazed that the original binaries were downloaded over 11,000 times since last August.

CrossLoop for Mac

2009-05-18T04:05:00Z

I mentioned CrossLoop before, as a tool for remotely helping someone out. It uses VNC to share desktops.

The last time I looked, it was Windows only. Now there's a Mac client too.

I had to use it to help my father out in Dublin. Somehow he had managed to delete both Adobe Reader and Adobe Flash—I haven't figured out how.

It was painful, painful, painful. The connection was dropped repeatedly and the link couldn't begin to keep up with the amount of graphical data being transferred. Even though CrossLoop reduces the color depth, actions like switching tabs in Firefox cause huge amounts of data to be sent. I couldn't tell why the connection was being dropped. There are so many places where things could go wrong: my client, my connection, the CrossLoop server, his connection, his client, some random router.

All in all, it took about 90 minutes, but it would likely have been even longer and more confusing without a shared desktop.

Safari 4 Revisited

2009-05-16T01:17:00Z

I tried Safari 4 on my MacBook back in February when it first came out in beta. It crashed immediately, every time, so I uninstalled it.

I upgraded to OS X 10.5.7 earlier in the week and new Safari bits were available, so it seemed like a good time to retry it. After all, it had been faster than any other browser on my Vista box at work.

Again, it crashed immediately. This time, however, I took a closer look at the details of the error report that was being sent to Apple. A little Googling suggested that the Glims plugin was at fault. Indeed it was. I replaced beta 8 from September 2008 with the new beta 16, and it's working again. Glims adds search engines, thumbnails in search results, favicons in tabs, etc., so it's useful.

I haven't used Safari 4 much yet on the Mac, but it seems like an improvement.

Windows 7 x64 running in Mac VirtualBox 2.2.2

2009-05-08T07:03:00Z

I ported Vim to Win64 but I don't have a convenient Win64 system to test it on.

I decided to install the Win64 build of the Windows 7 RC on VirtualBox, which has supported 64-bit guest operating systems since version 2.0.

It worked without problems on my MacBook Pro. I used VirtualBox's Virtual Media Manager to mount the Windows 7 ISO and installed from that. See also the handy guide. (Why does Windows 7 offer a choice of upgrading from a previous version of Windows on a virgin disk?) After completing the installation of the operating system, I installed the Guest Additions for mouse pointer integration and other goodies.

As always with VirtualBox VMs on my MacBook, I had to fix the Network settings to work over WiFi. When the VM is turned off, go to Settings, choose the Network tab. Change “Attached to” from “NAT” to “Bridged Adapter” and “Name” from “en0: Ethernet” to “en1: AirPort”. Tip: to get a right-click without a mouse, place two fingers on the trackpad and click the trackpad button, or Shift+F10.

I tried installing the Win64 build of Win 7 on my Win32 Vista desktop box at work. The host system bluescreened within seconds of starting the installer! I filed ticket 3963.

I had inadvertently installed the Win32 build first on my work system. That worked fine. It also seemed to have snappy disk I/O. When I unzipped the Win64 Vim binaries (not having realized yet that I had the Win32 Win 7), it was slower than unzipping in the host operating system, but not unreasonably so. On my MacBook, the details pane from the Win 7 zip extractor said that it was running at a mere 260KB per second, which is pitiful. It certainly wasn't that slow when installing the OS onto the virtual disk.