BLOG

standard

Maps all parking signs in NYC.

30.07.2015 Posted in Knowledges No Comments
NYC parking sign

This post might be too easy for any Data-viz people but as a beginner tapping in this area. It took me quite a long time to figure it out so I just want to share this hoping that it might save people’s time.

Parking in NYC is really a pain, especially street parking where there’re lots of signs and regulations. There are some apps on Android and iPhone that you can download and see signs but I haven’t been able to find the one that suits my needs. I want an application that can tell me where to circle arounds on which day and time to find parking spot. For example, I parked my car on a Tuesday which has street cleaning on Wednesday and Friday from 8:30 to 9:30. That means I have to move the car on Wednesday morning to find a spot for Friday. The problem is I don’t know what’s around me. The closest application I find is http://www.nycparklife.com/streetparker/ which it doesn’t have Manhattan.

Getting the data

So, I googled and thank NYC.gov that provides the data for us. First, I played around with the csv files but they don’t have coordinates that I can place location on the map. And I have no experience dealing with shapefiles.

Extracting the data

After a few hours of mangling and munching the csv with Pandas and R. I would not be able to get the exact locations of all the signs. So I turned to shapefile hoping that I might get lucky. And I found this http://www.shpescape.com/ which promised to transform shapefile to Google Fusion Table. Awesome! I went ahead and tried it. It works great! but it only gives you the first 100,000 rows. So dug deeper. After another few hours of googling I found qGis which is a opensource project that you can use to open and view shapefile.

Here’s the example.

Parking signs on Qgis

One thing I learnt was that you cannot just click Open and choose the file. What you need to do is to Add Vector Layer. You can use shortcut Ctrl+Shift+v

Add Vector Layer

Export to Google Fusion

Now what we need is to import what we have to Google Fusion Table. How are we going to turn this beautiful layer to Google Maps. We need CSV…

In QGis you can download the file as CSV. It’s in Layer > Save As. You just need to make sure that Geometry has to be set so you get the coordinates.

Save to CSV

Import to Google Fusion Table

Now we can use that CSV to import to Google Fusion. It would look something like this.

Google Fusion Table

what you need to make sure is that you need to specify which fields are Lat, Long so Google can plot that for you.

Here’s how you do it.

Screenshot 2015-07-30 13.05.41

The click change You will see something like this and you can choose which field you want to be Longitude or Latitude.

Set Location

Once the coordinates have set we can go to Map tab and see beautiful little dots that show where all the signs are in NYC.

Parking signs in Google Fusion Table.

Walla! Now you have something you can build an application on top of it. The next post, I will create an application on top of this map to make use of our data.

Things I have tried and failed

I’ve tried using Proj4 on both R and Python to convert X,Y WGS84 to Lat, Long. Here’s my little snippet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
data = read.csv('./parking_regulation.csv')

library(proj4)
proj4string <- "+proj=lcc +lat_1=40.66666666666666 +lat_2=41.03333333333333 +lat_0=40.16666666666666 +lon_0=-74 +x_0=300000 +y_0=0 +ellps=GRS80 +datum=NAD83 +to_meter=0.3048006096012192 +no_defs"
latlong_list<-list()
calculate_long_lat <- function(x,y) {
 
  # Source data
  xy <- data.frame(x=x, y=y)
 
  # Transformed data
  pj <- project(xy, proj4string, inverse=TRUE)
  latlon <- data.frame(lat=pj$y, lon=pj$x)
  latlong_list[['latlong']] <- latlon
}

apply(data[,c('x','y')], 1, function(y) calculate_long_lat(y['x'], y['y']))

The result is not quite accurate which I think it’s because I need to find a correct proj4string.

Python has the same wrapper which is quite what I want as well.

1
2
3
4
5
6
7
8
9
10
11
12
from pyproj import Proj
import pandas as pd

data = pd.read_csv('parking_regulation.csv')

p = Proj(r'+proj=lcc +lat_1=40.66666666666666 +lat_2=41.03333333333333 +lat_0=40.16666666666666 +lon_0=-74 +x_0=300000 +y_0=0 +ellps=GRS80 +datum=NAD83 +to_meter=0.3048006096012192 +no_defs')

def cal_long_lat(row):
    return p(row['x'], row['y'],inverse=True)

data['lon'], data['lat'] = zip(data.apply (lambda row: cal_long_lat (row),axis=1))
data.head()

I will need to learn more about State Pane and what is the correct format.

This article takes 2 pomodoros to complete.

standard

I’m never going back to modern editor again.

17.07.2015 Posted in Knowledges No Comments
Vim Cheatsheet

Ok, the title might be a little bit exaggerating. But let me clear this up first. I still use modern editors, TextMate, Atom.io or IntelliJ. They are pros and cons. But here’s my real answer if people ask me. I want to be the cool kid!.

I don’t know if you’re like me but I get asked almost all the time I go to a meetup or conference, “What’s your favourite editor”?. People almost always say Vim|Emacs. I tried both of them in the past and gave up more than I can remember because of all the shortcuts and plugins and I just get too frustrated to use either of them. Finally, I have made a decision that I’m going to stick with Vim for a month and let’s see if at the end of the month I still can’t use it. It’s been almost a month and I’d say I’m never going back to modern editors again (at least when I’m coding scripting languages). Here’s why.

It’s just easier

I know it’s not really easier than TextMate or Sublime. You still have to learn a lot of shortcuts and commands. Also, bare-bone Vim just doesn’t have when you get in TextMate or Sublime (e.g, Find file, Command+T, Directory structure or code completion). However, after you powered through that learning phrase everything is just natural to you.

If you want to clear any trailing whitespaces you just have to type %s/\s\+$// and hit enter. If you want to run some external command you can always do in Vim. For example, if you just want to git status you can install Git wrapper or you can just run :!git status. You don’t even need to go to your terminal which I think it’s faster.

If you’re a good engineer, you will always do TDD. Going back and fourth between the editor and terminal, it’s just too annoying. I remember when I was coding Java I can just hit Cmd+t in IntelliJ or .NET and you see green bars. Isn’t it just awesome? I realised that TextMate and Sublime have plugins to do that as well or you can just write your own plugin or package. I tried a couple of plugins and they never work for me. In Vim, it just works!.

Community

If Vim|Emacs doesn’t do what you want, you will almost always find the plugin that does it for you. Currently, my standard plugins are. NERDTree, CommandT, vim-fugitive, vim-rooter, vim-virtualenv. All these plugins just make my Vim functions the same as Atom.io or Sublime.

Not for everyone

I have to admit that the first few weeks I got really frustrated. I almost banged my keyboard against the monitor. I found myself opening Atom.io every time I couldn’t do some basic editing in Vim. For example, using vimgrep wasn’t what I expected and I ended up googling a lot. Now I get the hang of it and I don’t use Atom anymore. The worst case is I will just use command line instead.

standard

How to start doing TDD for jQuery plugin.

9.07.2015 Posted in Knowledges No Comments
jquery

I’m a big fan of TDD. I get nervous every time when I put some code it without having tests. I’m developing a simple jQuery plugin and I think hey we can TDD this.

What is it?

The plugin is really simple. It turns ul tag to be taggable field. It’s similar to tag-it but with a lot less functionalities and doesn’t depend on jquery-ui

What you need

I decided to use Karma because I’m going to test a lot of behaviours and Karma seems like a good fit as it run on real browser. Here’s how I setup my project.

I chose jasmine-jquery because it’s easier to create some element to test and it’s easy to setup.

This is my gulpfile.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
var gulp = require('gulp');
var karma = require('gulp-karma');

var testFiles = [
  'tests/vendors/jquery-1.11.3.min.js',
  'tests/vendors/jasmine-jquery.js',
  'src/**/*.js',
  'tests/spec/**/*.js'
];
/**
 * Run test once and exit
 */

gulp.task('test', function (done) {
  return gulp.src(testFiles)
    .pipe(karma({
      configFile: 'karma.conf.js',
      action: 'run'
    }))
  .on('error', function(err) {
    // Make sure failed tests cause gulp to exit non-zero
    throw err;
  });
});

gulp.task('default', function() {
  gulp.src(testFiles)
    .pipe(karma({
      configFile: 'karma.conf.js',
      action: 'watch'
    }));
});

This is my karma.conf.js

1
2
3
4
5
6
module.exports = function(config) {
  config.set({
    browsers: ['PhantomJS'],
    frameworks: ['jasmine']
  });
};

Here’s my first test

1
2
3
4
5
6
7
8
9
10
11
12
13
describe('Taggify', function() {
  var fixture;
  beforeEach(function() {
    fixture = setFixtures('<ul id="tag"></ul>');
    jQuery('#tag').taggify();
  });
 
  it('should initialize text box', function() {
    var input = fixture.find('input');
    expect(input.length > 0).toBeTruthy();
  });

});

You will see that the test failed now we implement some code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(function($) {
  $.fn.taggify = function(options) {
    create(this);
    return this;
  };

  function create($theElement) {
    var $input = $('<input class="tag-input"></input>')
      .attr('type', 'text')
      .attr('autocomplete', 'off')
      .wrap('<li></li>');

    $theElement.append($input.parent());
  }
})(jQuery);

Now the test passed.

Now let’s add some event so when you hit enter the tag is added. So, I added one more test

1
2
3
4
5
6
7
8
9
10
11
  it('should add a tag', function() {
    var input = fixture.find('input');
    input.val('tag');
    input.trigger(jQuery.Event('keyup', { keyCode: 13 }));

    var tags = fixture.find('.tag-label');
    var tag = jQuery(tags[0]);
   
    expect(tag.html()).toBe('tag');
    expect(tags.length > 0).toBeTruthy();
  });

Now the test failed.

I’ll fix the test by doing this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
(function($) {
  $.fn.taggify = function(options) {
    create(this);
    return this;
  };

  function create($theElement) {
    var $input = $('<input class="tag-input"></input>')
      .attr('type', 'text')
      .attr('autocomplete', 'off')
      .wrap('<li></li>');

    $input.on('keyup', function(e) {
      if (e.keyCode === 13) {
        var tagText = $input.val();

        var $span = $('<span class="tag-label"></span>');

        $span.text(tagText).wrap('<li class="tag-choice"></li>');
        $theElement.prepend($span.parent());
        $input.val('');
      }
    });

    $theElement.append($input.parent());
  }
})(jQuery);

Now I want to add some negative test case.

1
2
3
4
5
6
7
  it('should not add a tag', function() {
    var input = fixture.find('input');
    input.trigger(jQuery.Event('keyup', { keyCode: 13 }));
    var tags = fixture.find('.tag-label');
   
    expect(tags.length > 0).toBeFalsy();
  });

Oops the test failed, looks like I missed something

I will fix the test by

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
(function($) {
  $.fn.taggify = function(options) {
    create(this);
    return this;
  };

  function create($theElement) {
    var $input = $('<input class="tag-input"></input>')
      .attr('type', 'text')
      .attr('autocomplete', 'off')
      .wrap('<li></li>');

    $input.on('keyup', function(e) {
      if (e.keyCode === 13) {
        var tagText = $input.val();
        if(tagText !== '') {
          var $span = $('<span class="tag-label"></span>');

          $span.text(tagText).wrap('<li class="tag-choice"></li>');
          $theElement.prepend($span.parent());
          $input.val('');
        }
      }
    });

    $theElement.append($input.parent());
  }
})(jQuery);

That’s it. I hope you enjoy and love TDD more. And here’s the github repo