XCode 4.1 is now free in the App Store. I wanted to completely remove 3.0 version I had. Luckily there is a simple command line option for doing an uninstall:

sudo /Developer/Library/uninstall-devtools –mode=all