Ugh! Turns out if you don’t use a new Parser for each request you end up exiting with “Error: Can’t set headers after they are sent.”
Very annoying!
Installing NodePie on Ubuntu
I was transferring my latest creation over to my hosted Ubuntu server and came across the most annoying error. I had to reinstall the NodePie module because it relies on compiled C code. I kept getting an error though about something called lib-expat.
Anyhow, turns out you need to apt-get a new library:
sudo apt-get install libexpat1-dev
GitHub to the rescue:https://github.com/astro/node-xmpp/issues/33
Transferring from MongoDB to CouchDB
So I wanted to move a larger number of records from MongoDB to CouchDB. Here are the basic steps I took:
- I just wanted a selected number of fields (name, address, longitude, latitude) so I used mongoexport to put it into a JSON Array:
./mongoexport -d cookography -c places -f name,address,longitude,latitude -o test.json –jsonArray - Both CouchDB & MongoDB use _ID and MongoExport insists on adding it. However it does it in a way the CouchDB hates so I munge it:
sed ‘s/_id/id/g’ test.json > tested.json - Now you have to format your JSON Array so you can feed it the the REST interface of CouchDB. First add this to the beginning:
(echo ‘{“docs”:’; cat tested.json) > output.json
And then this to the end:
echo “}” >> output.json - Now send this to REST API for CouchDB:DB=”http://127.0.0.1:5984/places”
curl -d @output.json -X POST $DB/_bulk_docs -H “Content-Type: application/json”
a DIV inside a LI
Suppose you want to have a bunches of DIVs contained inside of a List Item… AND you want them to not mess with the height of the List Item and appear as a line of text?
Well then, take this sample HTML:
<li> <div class="inspection"> <div class="date"><a href="/report/1430/9115/">9-16-2010</a></div> <div class="inspection-type">License Renewal </div> <div class="violations">0 Violations</div> </div> </li>
And apply this CSS:
.inspection {
overflow: hidden;
vertical-align: middle;
display: inline-block;
}
li {
margin-left: 15px;
list-style-type: disc;
list-style-position: inside;
}
PostGres – PostGIS on OSX Lion
I recently just wasted a lot of time trying to get PostGIS up and running on OSX Lion. I wanted to do this so I could pull in Open Street Map data for the DC region and then play with it in QGIS and import it into Tilemill. I will be the first to admit that this is a rather complex toolchain, but it shouldn’t have been as difficult as it was.
On the off chance I can save someone time, I will recap what I did… and some of the misstep.
My big misstep was trying to use the precompiled binaries from KyngChaos. While they work fine, the OSM importer for PostGIS always SegFaulted when I tried running it against these versions. I even tried compiling the importer from source code and couldn’t get it work.
Here is what to do:
- Use Homebrew. It makes it easy to download and compile source code. You may need to get rid of old cruft you have previously downloaded. I had to remove MacPython and some old Frameworks. If you run “brew doctor” it will give you hints.
- Install Homebrew
- Install PostGres, PostGIS and all dependancies: ‘brew install postgis’
- Initialize the database. This will create a Database structure and a database user (“role”) that is the same as user Mac user name: ‘initdb /usr/local/var/postgres/’
- Create a database for GIS stuff (Instead of Luke, use your OSX username): ‘createdb -U Luke -E utf8 gis’
- Start the PostGres server – for some reason I needed to specify that it is running on localhost: ‘postgres -D /usr/local/var/postgres/ -h localhost’
- Add the PostGIS stuff to the DB: ‘psql -d gis -U Luke -f /usr/local/share/postgis/postgis.sql’ and also ‘psql -d gis -U Luke -f /usr/local/share/postgis/spatial_ref_sys.sql’
- Awesome! Now you should have a PostGIS DB loaded up. Now time to install the importer: ‘brew install osm2pgsql’
- It is going to fail. This is because you need to apply a patch to fix some inconsistency. Big pain in the ass. To do that, download the patch from here: http://trac.openstreetmap.org/raw-attachment/ticket/3299/patch-build_geometry.cpp
- To apply the patch, use the patch command: ‘cd ~/Library/Caches/Homebrew/osm2pgsql–svn/’ and then ‘patch -p0 < ~/Downloads/patch-build_geometry.cpp’
- Now try it again and it should work: ’brew install osm2pgsql’
- Awesome, almost there. Next step is to download some OSM data. Cloudmade makes it pretty easy. I grabbed the DC data from here: http://downloads.cloudmade.com/americas/northern_america/united_states/district_of_columbia#downloads_breadcrumbs
- Goto where ever you downloaded your files to and run (replacing it with your OSM file): ‘osm2pgsql -U Luke -H localhost -d gis district_of_columbia.osm’
- TADA! All that wonderful data should be loaded into your PostGIS DB. Now open up QGIS and add in a PostGIS layer. Make sure you use your OSX Username as the username (Luke) and the PostGIS DB you just created (DB).
JSDOM Memory leaks
JSDOM is a great little module for NodeJS which lets you parse a DOM on the server. The only problem is that it has a memory leak. Not a big deal if you are only going to instantiate a couple times. A little trickier if you are screen scraping and need to call it 1000s of times. I luckily found a work around. Instead of creating a new window every time you want to parse some code, simply keep the same window around and switch what it is displaying:
From the google groups:
I’m basically doing this:
var jsdom = require(‘jsdom’);
var document = jsdom.jsdom(‘<html><body>’);
var window = document.createWindow();
var JQUERY_URL = ‘http://localhost/js/jquery-1.4.4.min.js’;
function foo(text) {
document.innerHTML = ‘<html><body>’ + text;
jsdom.jQueryify(window, JQUERY_URL, function() {
var $ = window.$;
console.log(document.innerHTML);
});
}
setInterval(function() {
foo(“Text”);
}, 1000);
And the process’ memory increases and eventually terminates. How do
you suggest I should reuse the DOM?
I am trying this for now:
function foo(text) {
document.innerHTML = ‘<html><body>’ + text;
if (!window.$) {
jsdom.jQueryify(window, JQUERY_URL, function() {
var $ = window.$;
console.log(document.innerHTML);
});
} else {
var $ = window.$;
console.log(document.innerHTML);
}
}
For Looping jQuery Elements
When you get a jQuery object back from a selector, you general loop through all of the elements it contains using .each(). However I wanted to use a For Loop because I wanted to access a bunch of elements at once. I am scrapping a table and I might want to access multiple elements to pull data from.
Figuring out how to do this was a little tricky. Luckily it is very easy to do:
<body>
<div></div>
<div></div>
<div></div>
<div></div>
<div></div>
<div></div>
<script>
$(“body”).find(“div”).eq(2).addClass(“blue”);
</script>
Simply use .eq(index) to access a particular element in the jQuery object.
Completely removing XCode
XCode 4.1 is now free in the App Store. I wanted to completely remove 3.0 version I had. Luckily there is a simple command line option for doing an uninstall:
sudo /Developer/Library/uninstall-devtools –mode=all
Dynamic DIV IDs in Jade
Jade is an HTML templating language that works with Node.js and a bunch of other languages. It takes structured templates and converts them into HTML. Jade uses indentation to show what code should be within an HTML tag.
It is also possible to use variables to generate text, they call it Interpolation and it is a very handy feature. However, it is sometimes a little tricky to figure out how to use variables for attributes in tags.
For example if you want the variable local to serve a link in an Anchor tag, this would work:
a(href=local) Test Link
You can also combine variables and text:
a(href=’http://localhost:3000/page/’ + next_page ) Next
Doing the same for DIV IDs was a little trickier though. Since there is short had for assigning IDs, I thought I could just paste the variable in the shorthand. That didn’t work. However, using an approach similar to the Anchor tag does. Try this for using variables to assign the ID for a DIV:
div(id=permit_id)
Plugin Hybrids
It is always fun to head out to lunch and find the latest in Hybrid technology.
