PostGres – PostGIS on OSX Lion

I recently just wasted a lot of time trying to get PostGIS up and running on OSX Lion. I wanted to do this so I could pull in Open Street Map data for the DC region and then play with it in QGIS and import it into Tilemill. I will be the first to admit that this is a rather complex toolchain, but it shouldn’t have been as difficult as it was.

On the off chance I can save someone time, I will recap what I did… and some of the misstep.

My big misstep was trying to use the precompiled binaries from KyngChaos. While they work fine, the OSM importer for PostGIS always SegFaulted when I tried running it against these versions. I even tried compiling the importer from source code and couldn’t get it work.

Here is what to do:

  1. Use Homebrew. It makes it easy to download and compile source code. You may need to get rid of old cruft you have previously downloaded. I had to remove MacPython and some old Frameworks. If you run “brew doctor” it will give you hints.
  2. Install Homebrew
  3. Install PostGres, PostGIS and all dependancies: ‘brew install postgis’
  4. Initialize the database. This will create a Database structure and a database user (“role”) that is the same as user Mac user name: ‘initdb /usr/local/var/postgres/’
  5. Create a database for GIS stuff (Instead of Luke, use your OSX username): ‘createdb -U Luke -E utf8 gis’
  6. Start the PostGres server – for some reason I needed to specify that it is running on localhost: ‘postgres -D /usr/local/var/postgres/ -h localhost’
  7. Add the PostGIS stuff to the DB: ‘psql -d gis -U Luke -f /usr/local/share/postgis/postgis.sql’ and also ‘psql -d gis -U Luke -f /usr/local/share/postgis/spatial_ref_sys.sql’
  8. Awesome! Now you should have a PostGIS DB loaded up. Now time to install the importer: ‘brew install osm2pgsql’
  9. It is going to fail. This is because you need to apply a patch to fix some inconsistency. Big pain in the ass. To do that, download the patch from here:
  10. To apply the patch, use the patch command: ‘cd ~/Library/Caches/Homebrew/osm2pgsql–svn/’ and then ‘patch -p0 < ~/Downloads/patch-build_geometry.cpp’
  11. Now try it again and it should work: ‘brew install osm2pgsql’
  12. Awesome, almost there. Next step is to download some OSM data. Cloudmade makes it pretty easy. I grabbed the DC data from here:
  13. Goto where ever you downloaded your files to and run (replacing it with your OSM file): ‘osm2pgsql -U Luke -H localhost -d gis  district_of_columbia.osm’
  14. TADA! All that wonderful data should be loaded into your PostGIS DB. Now open up QGIS and add in a PostGIS layer. Make sure you use your OSX Username as the username (Luke) and the PostGIS DB you just created (DB).

JSDOM Memory leaks

JSDOM is a great little module for NodeJS which lets you parse a DOM on the server. The only problem is that it has a memory leak. Not a big deal if you are only going to instantiate a couple times. A little trickier if you are screen scraping and need to call it 1000s of times. I luckily found a work around. Instead of creating a new window every time you want to parse some code, simply keep the same window around and switch what it is displaying:

From the google groups:

I’m basically doing this:

var jsdom = require(‘jsdom’);
var document = jsdom.jsdom(‘<html><body>’);
var window = document.createWindow();

var JQUERY_URL = ‘http://localhost/js/jquery-1.4.4.min.js’;

function foo(text) {
document.innerHTML = ‘<html><body>’ + text;
jsdom.jQueryify(window, JQUERY_URL, function() {
var $ = window.$;


setInterval(function() {

}, 1000);

And the process’ memory increases and eventually terminates. How do
you suggest I should reuse the DOM?

I am trying this for now:

function foo(text) {
document.innerHTML = ‘<html><body>’ + text;
if (!window.$) {
jsdom.jQueryify(window, JQUERY_URL, function() {
var $ = window.$;
} else {
var $ = window.$;


For Looping jQuery Elements

When you get a jQuery object back from a selector, you general loop through all of the elements it contains using .each(). However I wanted to use a For Loop because I wanted to access a bunch of elements at once. I am scrapping a table and I might want to access multiple elements to pull data from.

Figuring out how to do this was a little tricky. Luckily it is very easy to do:





Simply use .eq(index) to access a particular element in the jQuery object.

Dynamic DIV IDs in Jade

Jade is an HTML templating language that works with Node.js and a bunch of other languages. It takes structured templates and converts them into HTML. Jade uses indentation to show what code should be within an HTML tag.

It is also possible to use variables to generate text, they call it Interpolation and it is a very handy feature. However, it is sometimes a little tricky to figure out how to use variables for attributes in tags.

For example if you want the variable local to serve a link in an Anchor tag, this would work:

a(href=local) Test Link

You can also combine variables and text:

a(href=’http://localhost:3000/page/’ + next_page ) Next

Doing the same for DIV IDs was a little trickier though. Since there is short had for assigning IDs, I thought I could just paste the variable in the shorthand. That didn’t work. However, using an approach similar to the Anchor tag does. Try this for using variables to assign the ID for a DIV:


Cuckoo’s Egg – Some Books Age Well

I am re-reading The Cuckoo’s Egg. It is a great book by Cliff Stoll, an astronomer turned sys admin who tracks a hacker through his system. There is a ton of insight in this book on the challenges of securing a system and various techniques. The crazy thing is that this took place in 1984. A lot of the techniques he describes he has to do manually, but they still exist today. His intrusion detection system is simply making his terminal beep when a new person logs in so he can check to see if they are the hacker and exploits a particular hole. Today, we have automated IDS tools that monitor activity. His logging was to print out traffic on a particular modem line using a dot matrix printer. Things have gotten better.

The challenges he brings up and attack techniques are still very relevant today. Password hacking, privilege escalation still are core problems. The attack tools of course have gotten more sophisticated too. All of the attacking done was manual. These days they have a lot more tools in their bag too.

This is a great read and I recommend it to anyone who is interested in security. Because there is not a lot of technology to get in the way, it is really easy to see the fundamentals of attack and defense.

There are PDF versions available online but I would recommend ordering a copy directly from Cliff himself. He as first edition, hardcover editions which he is selling for $20. That is what I did and the delightful shipping email he sent out was worth it for that alone.

I will update this with observations as I read through it.


Wireless speed: AT&T vs Verizon vs Starbucks Wifi

I just ran the FCC broadband speed app on my iPad 2 using the built-in Verizon 3G, and then ran the same app on my iPhone 3G, which is on the AT&T network. Verizon gave 1.04mbs & .84mbs in download and .6mbs in two test runs. AT&T clocked in at .19mbs & .61mbs download and .06mbs upload. More surprising was the speed using the Starbucks wifi network. It only gave .94mbs download but delivered 1.16mbs up.

Pretty interesting that Verizon is competitive with crowded in store wifi…

iPad 2 – White vs. Black

So I managed to get myself an iPad 2. Not an easy feat. It took me several trips to the Apple store and a couple to target and the Verizon store. I actually managed to get two iPads… The first a white one and them finally a black one from the apple store. Managing to get an iPad during the opening weekend was a feat of luck, some one returned the white one and I got it. So why did I return it? The truth is that I like the white one, but I like the black one even more. With the white bezel some things like photos and videos don’t like quite right and I think that white balance is to blame.

Your eye automatically determines what “white” is on the fly. White can range from yellowish to blueish, but when your eye views it and it appears to be pu re white. However when you view two different hues of white your mind has to pick one. I had that problem with the white bezel of the iPad. Either the bezel looks sort of off or white is on the screen does. I watched one of the sample movies in the Apple store and the movie had been color cast sort of yellowish like many movies are. On the black iPad you didn’t really notice because your eye doesn’t have any other reference points. However, with the white iPad it made seem more pronounced because the bezel was numeral and served as a reference point.

It is a great device and I am sure if I used it for a bit I would get over it. However the Verizon store has a restocking fee if you open something. The fact that at&t and Verizon are the only people selling the iPad that have a restocking fee highlights the big divide between carriers and the rest of the tech market. Anyhow since I couldn’t open up the white iPad I sent it back… Which was really tough to do!

After a bit of getting up early and waiting in line at the Georgetown Apple a couple days I. Was able to get the exact iPad I wanted, a black 32 gb Verizon iPad 2. I plan on using it for blogging and twittering on the fly. I will try to be good and keep things updated on how it is working.