Cuckoo’s Egg – Some Books Age Well

I am re-reading The Cuckoo’s Egg. It is a great book by Cliff Stoll, an astronomer turned sys admin who tracks a hacker through his system. There is a ton of insight in this book on the challenges of securing a system and various techniques. The crazy thing is that this took place in 1984. A lot of the techniques he describes he has to do manually, but they still exist today. His intrusion detection system is simply making his terminal beep when a new person logs in so he can check to see if they are the hacker and exploits a particular hole. Today, we have automated IDS tools that monitor activity. His logging was to print out traffic on a particular modem line using a dot matrix printer. Things have gotten better.

The challenges he brings up and attack techniques are still very relevant today. Password hacking, privilege escalation still are core problems. The attack tools of course have gotten more sophisticated too. All of the attacking done was manual. These days they have a lot more tools in their bag too.

This is a great read and I recommend it to anyone who is interested in security. Because there is not a lot of technology to get in the way, it is really easy to see the fundamentals of attack and defense.

There are PDF versions available online but I would recommend ordering a copy directly from Cliff himself. He as first edition, hardcover editions which he is selling for $20. That is what I did and the delightful shipping email he sent out was worth it for that alone.

I will update this with observations as I read through it.

20110519-094546.jpg

Wireless speed: AT&T vs Verizon vs Starbucks Wifi

I just ran the FCC broadband speed app on my iPad 2 using the built-in Verizon 3G, and then ran the same app on my iPhone 3G, which is on the AT&T network. Verizon gave 1.04mbs & .84mbs in download and .6mbs in two test runs. AT&T clocked in at .19mbs & .61mbs download and .06mbs upload. More surprising was the speed using the Starbucks wifi network. It only gave .94mbs download but delivered 1.16mbs up.

Pretty interesting that Verizon is competitive with crowded in store wifi…

iPad 2 – White vs. Black

So I managed to get myself an iPad 2. Not an easy feat. It took me several trips to the Apple store and a couple to target and the Verizon store. I actually managed to get two iPads… The first a white one and them finally a black one from the apple store. Managing to get an iPad during the opening weekend was a feat of luck, some one returned the white one and I got it. So why did I return it? The truth is that I like the white one, but I like the black one even more. With the white bezel some things like photos and videos don’t like quite right and I think that white balance is to blame.

Your eye automatically determines what “white” is on the fly. White can range from yellowish to blueish, but when your eye views it and it appears to be pu re white. However when you view two different hues of white your mind has to pick one. I had that problem with the white bezel of the iPad. Either the bezel looks sort of off or white is on the screen does. I watched one of the sample movies in the Apple store and the movie had been color cast sort of yellowish like many movies are. On the black iPad you didn’t really notice because your eye doesn’t have any other reference points. However, with the white iPad it made seem more pronounced because the bezel was numeral and served as a reference point.

It is a great device and I am sure if I used it for a bit I would get over it. However the Verizon store has a restocking fee if you open something. The fact that at&t and Verizon are the only people selling the iPad that have a restocking fee highlights the big divide between carriers and the rest of the tech market. Anyhow since I couldn’t open up the white iPad I sent it back… Which was really tough to do!

After a bit of getting up early and waiting in line at the Georgetown Apple a couple days I. Was able to get the exact iPad I wanted, a black 32 gb Verizon iPad 2. I plan on using it for blogging and twittering on the fly. I will try to be good and keep things updated on how it is working.

Porting Sprint Number to Google Voice

We have a shared Sprint family plan for both of our Cell phones. The thing is that I also have a Blackberry and iPhone from work. That is 3 phones… and I don’t like carrying them all around. So instead I have decided to port my cell phone number to Google Voice. Since the Sprint plan is a family plan with two lines, I can switch to a standard plan without breaking our contract or having to start a new one. We double checked this with Sprint.

The first step is to sign onto you Google Voice account and start the number port. You will need a pin for your Sprint account that you use to authorize the Port. Once everything goes through, call up Sprint and change your plans. The number I transferred was for the secondary line. If it is for your primary line, it might be worth giving Sprint or your cell carry a call first to make sure everything will work first.

I will check back once I have things running for a bit.

 

Getting Zend / PEAR to work on Ubuntu

So I recently switched over from a shared host (1&1) to a Virtual Private Server ( Linode ). With the VPS you still share a server, but you are in complete control of everything. You have to get things up in running from a blank image. This means installing Apache, PHP and all the other stuff you need. Luckily, Linode has a great library explaining how to do all of this.

It is pretty easy to add different frameworks like Zend & PEAR using the apt-get command. However when I tried porting over a PHP site I wrote that used Zend & PEAR, I got the following error:

[Sun Sep 12 19:27:56 2010] [error] [client 65.55.3.140] PHP Fatal error: require_once(): Failed opening required ‘Zend/Loader.php’ (include_path=’.:/usr/share/php:/usr/share/pear’)

I looked at phpinfo() and it looked like it was reading in all the php.ini and Zend was working. Unfortuantly, as you can see in error message, the include path wasn’t being set right and wasn’t pulling the libraries.
Turns out that the include_path in the main php.ini has to be updated and point to the location of the libraries.
To fix this I edit /etc/php5/apache2/php.ini and uncommented the include_path line and pointed to the libraries:
;;;;;;;;;;;;;;;;;;;;;;;;; ; Paths and Directories ; ;;;;;;;;;;;;;;;;;;;;;;;;;
; UNIX: “/path1:/path2” include_path = “.:/usr/share/php:/usr/share/php/libzend-framework-php:/usr/share/php/PEAR”

Creating a GeoCMS in Drupal with Custom Map Tiles

So somehow I stumbled across the idea that I would like to display different nodes on map. Instead of listing different post sequentially in a list like a blog, I want to plot them on a map based upon some sort of geographic tagging. The whole reason I wanted to do that is so I could map businesses opening and closing in my neighborhood. The easy thing to do would be to simply use a custom Google map… but what fun would that be! On top of that I want to come up with custom map tile so that it doesn’t look like a generic Open Street Map.

The guys over at Development Seed are the leaders in custom cartography and integration into Drupal. Their Stumble Safely app is a great example and the tile set they generated for it is what motivated me to craft my own. Luckily for the community, they have their process for building the tileset petty well documented. Enough rambling, lets get started!

The first thing I did was start looking at the tutorial Tillmill. It is a tool the Development Seed put out that lets you use the Amazon Cloud to build your tileset. I never quite got it working and ended up rendering everything locally. The process for getting everything setup is the and there is great background reading on everything you need to do. Lucky for me, Dev Seed is based in DC so the sample they have for DC was really helpful.

Step 1: Find some Geo Data

Files used:

The sample has most of the Shapefiles you need for DC. If you want to do everything from scratch, here are some to get you started:

DC Government has put out lots of GIS files for a variety of things. We are going to use this data in building a custom map of DC. If you search the catalog for “DC GIS” you should get a list of the different data sources that provide an ESRI Shapefile. Later on you will be layering these shapefiles to create the map. Here are some of the basic shapefiles you might need to get started, you can layer additional stuff on top depending on what you are trying to show:

Strangely enough it looks like the DC layers do not provide the names of streets in the Roads shapefile. Luckily you can get them from Open Street Maps, an crowd sourced effort to create open map data. CloudMade is a cloud based app for creating a custom tile set. It may actually provide you with everything you need to build a custom tile set… if you are a wimp and want to use a web interface instead of the scripts, the command line and a bunch of wonky XML. They are nice enough to provide easily downloadable Open Street Map data. Grab the DC Shapefiles. You want to use the highway shapefiles from inside the zip, note that a shapefile is actually a bunch of files with the same filename and some random extensions. If you use OSM data in your map, you are supposed to be a good Doobie and include attribution.
Once you have all the shape files you need, it would be a good time to learn about projections. Projections are the method used to take something round and turn it into something flat. Each layer could have been created with a different project, so it is important remember what each is. The DC data is in Maryland State Plane coordinates NAD 83 meters, which is a rather wonky projection. The OSM data is in a more standard  Google Mercator projection, also known as Spherical Mercator. Right now none of this should be making sense. Don’t worry I show you where to fill it in later.
Step 2: Viewing the Layers
Files Used:

Now that you have a nice collection of layers you are going to want to take a look at them. Quantum GIS (QGIS) is a nice open source package that provides a lot of the GIS features that the more expensive ArcGIS provides. Most importantly it lets you open and view ESRI Shapefiles. Get QGIS setup on your machine and launch it. If you haven’t done so yet, un-zip all of the shapefiles you downloaded.
Now start adding the layers you download. Begin with the base layers like the Boundary file and work your way up.
To add a layer in QGIS:

  1. Layer -> Add Vector Layer

  2. A window should pop-up, hit the Browse button to find your shapefile.
  3. A Layers pane should be open on the left-hand side and the layer you just added should appear there. Right-click on the layer and open its Properties.
  4. In the Properties window that opened, click on General and then click the Specify CRS button.
  5. This brings up the Coordinate Reference System for the layer.

    1. For the DC GIS Layers search for an ESPG of 2804 – it should return NAD83(HARN) / Maryland. For the OSM Layer search for an ESPG of 4326 – it should return WGS 84.

Now repeat this for all for layers. The order I went in was:

  1. Boundary
  2. Water
  3. Building
  4. Sidewalk
  5. Road
  6. Street names

Now that you have everything load into QGIS, it is time to set the Projection you would like to have.

  1. File – > Project Properties
  2. A window should open up, click on the Coordinate Reference System tab
  3. Search for an EPSG of 3785 – it should return Popular Visualization CRS / Mercator.

Ok, now you are all setup to view the different layers you have. Make sure you save it! Now it is time to style your map.

Step 3: Styling your Map

So here is where things start to get interesting.This is also where still have lots to learn. Luckily the smart guys at Development Seed have some good stuff:

Here are the basics; Mapnik is what is used to generate custom made maps. You can specify what layers you want, what portion of the map you want to generate, what levels of zoom should be included and how you want to color and draw each layer. Mapnik takes in an XML file which describes the different Layers to take in and a Style file which describes.

If you are using Mac, here is an easy installer.

If you know your way around code and you know exactly how you want to style your map, you can simply hand code both of these files. Quantumnik is a plugin for Quantum GIS that lets you style your map in realtime, preview how Mapnik would render and then spit out the Mapnik XML and style file. The one thing Quantumnik can’t handle is having different map styles depending on how far you have zoomed in. If you are creating a map where people are going to be zooming in and out a lot, this can be a problem. If someone is viewing an entire city, you don’t want to be showing the location of every single trash can.

What Quantumnik is great at is letting you play around with different colors, fonts and font sizes. Some of this is tough to visualize and it can take a while to render tiles. Quantumnik lets you experiment with all of this. If you plan on having a limited range of zooms, you might be able to simply use the Mapnik XML file Quantumnik spits out.

Cascadenik is another tool. It lets you use Cascading Style Sheet like style files and a simplified XML format to more easily handcode Mapnik XML files.

I ended up using Quantumnik to come up with the colors, layers and labels I wanted. I took the XML file it generated and then used that to create a custom Mapnik XML file by hand using Cascadenik. Of course, check out the good info on Mapnik files at Dev Seed. I will do a quick run through on the files I came up with.

First up is the .mss style file. In this file you define styles to go along with different objects in layers. Here are some of the interesting bits:

dc-light.mss

.building[zoom=14] {

  polygon-fill: #999999;

}

.building[zoom>=15] {

  line-color: #737373;

  line-width: 0.2;

  polygon-fill: #999999;

}

.building[zoom=16] {

  line-width: 0.3;

  polygon-fill: #999999;

}

The interesting thing in this snippet is that you can specify different styles at different zoom levels for a single element. Another interesting thing is that when you get zoomed in, pervious styles still apply for that element unless you override them. That is why at zoom level 16 you do not see line-color being set.

.road.fill[DESCRIPTIO=”Intersection”],

.road.fill[DESCRIPTIO=”Road”] {

  polygon-fill: #B3B3B3;

}

.road[DESCRIPTIO=”Hidden Median”],

.road[DESCRIPTIO=”Hidden Road”] {

/*polygon-fill: #0F0F0F;*/

line-dasharray: 4.7, 2.7;

line-width: 1.5;

outline-color: #fff;

outline-width: 0;

}

.road.fill[zoom>=14][DESCRIPTIO=”Parking Lot”]{

polygon-fill: #BFBFBF;

}

.road.fill[zoom>=14][DESCRIPTIO=”Alley”],

.road.fill[zoom>=14][DESCRIPTIO=”Paved Drive”] {

  polygon-fill: #CCCCCC;  

}

So here you can see where things get real interesting. You will notice, [DESCRIPTIO=”Alley”] this lets you define different styles for a single layer based upon fields in that layer data. This is great! It lets you take a complex layer with a lot of different objects and give each of them different style. It also lets you chose which portions of a layer should be visible based upon zoom. If a layer is zoomed out, you may not want all of the features to be visible. You may also note that you don’t have to define every zoom level, you can just do greater than or less than.

Here is one more example:

.water.major[DESC_!=”POND”] {

  line-color: #9CA1DB;

  line-width: 1;

  polygon-fill: #B3B3EC;

}

.water.minor[DESC_=”POND”] {

  polygon-fill: #B3B3EC;

}

dc-light.mml

This file defines the projection for the overall map projection and the different layers to include in the map. Here are some snippets:

<!DOCTYPE Map[

  <!ENTITY srs900913 “+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs”>

  <!ENTITY srsWGS84 “+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs”>

  <!ENTITY srsDC “+proj=lcc +lat_1=38.3 +lat_2=39.45 +lat_0=37.66666666666666 +lon_0=-77 +x_0=400000 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m +no_defs”>

]>

Here is the start of the mml file. You first define the different projections that you will be using for your map and for the layers.

<Map srs=”&srs900913;”>

  <Stylesheet src=”dc-light.mss” />

  <Layer class=”building outline outer” srs=”&srsDC;”>   <Datasource>   <Parameter name=”file”>./Layers/BldgPly/BldgPly</Parameter>   <Parameter name=”type”>shape</Parameter>   </Datasource>  </Layer>    <Layer class=”building outline inner” srs=”&srsDC;”>   <Datasource>   <Parameter name=”file”>./Layers/BldgPly/BldgPly</Parameter>   <Parameter name=”type”>shape</Parameter>   </Datasource>  </Layer>    <Layer class=”building fill” srs=”&srsDC;”>   <Datasource>   <Parameter name=”file”>./Layers/BldgPly/BldgPly</Parameter>   <Parameter name=”type”>shape</Parameter>   </Datasource>  </Layer>

Here is where you actually start to define the Map and Layer. The srs attribute defines the projection to be used for the map or layer. The ampersand means that it is embedded in the document.

Both of these files get combined by Cascadenik into a final Mapnik XML file. Now that you have a Mapnik XML file, it is time to make some tiles! It is pretty easy to do, but it requires you customize a Python file to match you setup.

To get started, download the generate_tiles.py file from open street maps. You can use svn, if you have it and know how to use it. I simple browsed over to:

http://svn.openstreetmap.org/applications/rendering/mapnik/generate_tiles.py

Save the file to the same directory you have you Mapnik XML file in and open it in a text editor.

First stop, start by correcting the number of threads to match how many cores you have in you machine, probably 2 or 4

NUM_THREADS = 2

Next the file paths, take this complex statement and point directly to the files:
Original:try:   mapfile = os.environ[‘MAPNIK_MAP_FILE’]   except KeyError:   mapfile = home + “/svn.openstreetmap.org/applications/rendering/mapnik/osm-local.xml”   try:   tile_dir = os.environ[‘MAPNIK_TILE_DIR’]   except KeyError:   tile_dir = home + “/osm/tiles/”
Updated:   home = os.environ[‘HOME’]
   mapfile = home + “/Programming/Maps/Custom Map/dc-light.xml”    tile_dir = home + “/Programming/Maps/Tiles/”

Finally, describe the bounds for the map. Clear out the old stuff and describe what it should be for your map. Getting the bounding box correct was a huge pain in the ass and pretty un-intutitive. You want to use Lat – Long to describe the box. The ever capable script will convert the Lat Long project into the Merc projection we selected for our map. I wasted a lot of time trying to figure out how to specify the bounding box. I either got stuff that was out there, or nothing at all. The problem turns out that you specify things differently than you would for other graphics programming. Instead of using the upper-left corner and bottom-right corner, you use the bottom-left corner and the upper-right corner. Bounding Box on OSM Wiki

To find these points the easiest thing to do is pop into Quantum GIS and look up the points for your map.

Here is what I used for DC. My map has a bunch of zoomed in zoom levels:

   minZoom = 10

   maxZoom = 18

   bbox = (-77.065,38.89,-77.01,38.94) #(-8577985,4708655,-8574353,4710851)

   render_tiles(bbox, mapfile, tile_dir, minZoom, maxZoom,”DC”)

Once you have made all these changes you should be able to start making tiles. Try running generate_tiles.py from your command line and see what happens. It should start spitting out tile PNGs into the directory you specified. Depending on the size of the map you are producing and the number of zoom levels, it could be running for a bit.

After a while you should be left with a ton of tiles. The next step is to get them online… and this will be covered in the next post because this is getting really long.

Color in KML is all wrong!

I am working on building a Widget for the ESRI Flex Viewer that will import KML files from Google Earth. It is pretty easy to do, they already have a widget which imports GeoRSS feeds and Google has sample Flex Code for importing KML files into a Flex Google Maps viewer. The real work is turning the structures the Google parser builds into Graphic Objects which the ESRI Flex Viewer can take. Not exactly rocket science, but it has taken a lot of detective work because some aspects are less than obvious.

A real head scratcher for me was getting the colors right. All of the KML files I imported would have some of their colors off. I took me a while to figure out that color are not encoded in a standard way in KML files. Color in KML files are encoded using Hex, similar to HTML, except they also include an Alpha channel. Each Channel (red, green, blue & alpha) are given 32 bits (0-255), which is converted into two Hexadecimal characters (0 – F). All this is pretty standard. The difference is that the Channels in KML colors are written down in a different order than almost every other standard. The order of the channels in KML is Alpha-Blue-Green-Red (ABGR) as opposed to the standard Alpha-Red-Green-Blue (ARGB). Flash takes in an integer conversion from the hex format, which is easy to do. The problem is that it expects the hex channels to be encoded in the RGB order. Once I figured this out I was able to put together a pretty simple encoder/converter. It may not be pretty but it gets the job done!

private function setKmlColor(kmlColor:String)

{

var abgr:uint = parseInt(kmlColor,16);

var alphaUint:uint =(abgr>> 24) & 0xFF;

var blueUint:uint = (abgr >> 16) & 0xFF;

var greenUint:uint = (abgr >> 8) & 0xFF;

var redUint:uint = abgr & 0xFF;

var colorUint = (redUint << 16);

colorUint += (greenUint << 8);

colorUint += (blueUint);

this._color = colorUint;

this._alpha = alphaUint / 255;

}

Zend Framework, Gdata, Blogger and AuthSub

So I am working on adding the ability to post to Blogger on the current project I am working on. It has been a tricky road getting there, so I thought I would “document” my learning trail.

  1. First place to start out is this getting started page for posting to Blogger using Gdata.
  2. There is an extension to the Zend Framework that makes it easy to work with Gdata API in PHP. You can either use entire Zend Framework, or just the Gdata Client.
  3. There is an explanation for installing and configuring the Gdata Client and the Zend Framework, here.
  4. I am installing this on MAMP, (a self-contained PHP, MySQL package) and was having trouble getting the php include_path to work correctly. I tried this method out and it works fine. Basically you create a symbolic link to the /library/Zend folder, call the link Zend and place it next to the php folder at: /Applications/MAMP/bin/php5/lib. You then have to change the php.ini file located at /Apps/MAMP/conf/php5 and change the include_path to include_path = “.:/Applications/MAMP/bin/php5/lib”.
  5. The build for just the Gdata Client instead of the entire framework is missing a file. Just comment out line 50 of client.php. Supposedly future builds will fix this problem.
  6. The sample code for doing AuthSub uses a function called getCurrentUrl(); I couldn’t find this function anywhere so I just hardcoded it.
  7. The sample code gives an include block:
    require_once 'Zend/Loader.php';
    Zend_Loader::loadClass('Zend_Gdata');
    Zend_Loader::loadClass('Zend_Gdata_Query');
    

    Zend_Loader::loadClass(‘Zend_Gdata_ClientLogin’);
    This is for doing a client login instead of AuthSub, which is used for connecting to Google via another website. In order to do this you need to include the class for this. Do this by adding this line:
    Zend_Loader::loadClass(‘Zend_Gdata_AuthSub’);   

  8. Another error in the AuthSub sample code. Here is the block:
    function getAuthSubUrl()
    {
    $next = getCurrentUrl();
    $scope = 'http://www.google.com/blogger/feeds/';
    $secure = false;
    $session = true;
    return Zend_Gdata_AuthSub::getAuthSubTokenUri($next, $scope, $secure, $session);
    }$authSubUrl = getAuthSubUrl();
    

    echo ‘<a href=\”$authSubUrl\”>login to your Google account</a>’;
    The scope is set incorrectly. It should not goto google.com. In the text below and the sample URL, they have it correct. It should be:
    $scope = ‘http://www.blogger.com/feeds/’;

  9. Turns out that they forgot to load another class needed for the blogger. Add this line following the other Zend_Loaders. You need it in order to view all of the Blogs a user has.
    Zend_Loader :: loadClass(‘Zend_Gdata_Feed’);

Yea… so not what I would call out of the box functionality. I will add more to this as I work with the API and find other fun things to work around.