Designing a New Map Portal for Karnataka Learning Partnership.

Wrote a rather detail post about the new maps for Karnataka Learning Partnership on the blog.

The map is an important part of our project, action and process because it serves as the pivot point of navigation. I will quickly talk about the data and tools before we discuss the design aspects.

We have a fairly large dataset of schools in Karnataka. The name of the school, location, number of girls and boys etc. in a database. Fortunately, the data was clean and properly stored in a PostgreSQL database with PostGIS extensions. Most of my task was to modify the API to throw GeoJSON to the client using the ST_AsGeoJSON function and export the data.

read more…

Visualizing SSLC results over seven years.

I spent most of the last one month building a dashboard for the SSLC results at the Karnataka Learning Partnership. We released it in beta yesterday and here’s the blog post I wrote detailing how we went about it.

Patterns in examination results are something which we are always interested at the Karnataka Learning Partnership. After the design jam in June 2012, where we tried to understand the SSLC data – it’s content and structure, and visualized performance of Government and Private schools in contrast to each other, we decided to take a step deep and find patterns from the past seven years. Results of this effort is what you find here, in beta.

The Karnataka Secondary Education Examination Board shared the data over the last seven years in a combination of several Microsoft Access Database. It came with very little meta data and Megha did all the hard work of making sense of this and pulled it into a PostgreSQL database. Inconsistencies are everywhere, and the quality always depends on how you handle each exception in isolation. Among other things, we decided to look at three aspects of the data to begin with – performance of Government and Private schools, performance in Mathematics, Kannada and English, and performance of each gender. All three across seven years (from 2004-2005 to 2010-2011) for each district in Karnataka.

One of the important data wrangling that we did this time was to aggregate this data at the district level. The raw data came at the educational district level and unfortunately, we did not have geographic boundary shapes for this classification. What we have instead is the geographic boundary at the political level. We massaged the shapefiles, geocoded the data and converted it to GeoJSON in QGIS. We wrote a bunch of Python scripts to perform the aggregation and generate JSON (JavaScript Object Notation) required for the visualization. Every bit of code that we wrote for this project is on Github.

A dashboard of this sort is something which we have never attempted, and honestly it took a while for us to get around it. I had tried D3.js sometime last year and found it to be amazing. D3 is Data Driven Documents, a brilliant JavaScript library to make infographics on the web driven completely by the data. What makes D3.js awesome for me is that everything is an SVG (Scalable Vector Graphic), and there are barely any limits to the representation and interaction that you can bring into the browser with it. I’ve had good experiences with Twitter’s Bootstrap to quickly design and be consistent on the page layout and aesthetics. There are some issues while you work with D3.js and Bootstrap together, especially the way bootstrap manages events. The best way is to trust D3.js and use Bootstrap features of scaffolding and layout.

We found few interesting facts from this exercise. As you may guess, private schools perform better than government schools consistently. Western districts like Udupi, Uttar Kannada and Belgaum performs better than rest of the state. North Karnataka, especially Bidar performs terribly across the last seven years. Something which we are very curious to know why. Bangalore Rural performs better than Bangalore Urban. Government schools does much better and comparable to private schools in Bangalore Rural than Bangalore Urban. Private schools grab the cap in all the three subjects across the last seven years. Girls performs way better than boys in private schools consistently across seven years in every district. Boys does a better job in Bangalore Urban while girls dominate in Bangalore Rural.

This research will continue while we churn few more aspects from the data as the dashboard gets out of beta.


Zoom level based marker interaction using Leaflet.js

With Arky I was creating a map of the Mozilla l10n teams world wide and decided to use Leaflet.js. He send across a CSV data sheet which contained the geo-tagged teams and other information. Wrote a Python script to generate markers corresponding to Leaflet.js (Yes, my JS skills suck). When we laid the markers on the map, we found that it looked all crowded and difficult to locate one team quickly. I took this discussion to the HasGeek channel and Nigel showed me the Ubuntu Loco Global Events map. The markers in the map changed based on the zoom level of the map. That is pretty neat idea to handle crowded map markers. But they were using Google Maps.

I wanted to make this happen with Leaflet.js and OpenStreetMap data. Leaflet.js is perhaps the best, easy and intuitive interactive mapping library that I’ve come across. I spent some time in the documentation and came up with this idea and voila! – it worked.

Leaflet.js provides several events, the one which we will use is zoomend. This event gets triggered whenever the zoom level of the map changes. Exactly what we need. This is how I did it.

If you have a bunch of markers, use LayerGroup, that will give you better control. You can now add/remove the entire marker set by the LayerGroup handler.

The very special Pycon 2011

Download the talk and watch the video.

This edition of Pycon was in fact very special in different perspectives. I came to know that my talk was selected from Kartik. It has been a while since I hacked on Python with Android but I did follow the updates and new releases. 4 days before the conference, I managed to buy a Samsung Galaxy S 2 which looked to me as a brilliant phone and couldn’t wait to start hacking.

Since Praveen was in Pune, I didn’t have to look for other option to crash in. Took a bus overnight to Pune reached there on Friday (16th September), went to Pravi’s place. That was my first longest bus journey. Bangalore – Pune 846 KM. Reached the venue late in the afternoon, met couple of friends. Felt good to have given Noufal, Pradeepto and bunch of other guys, a face, with whom I interacted online.

My talk was on day 2. Didn’t have much to talk either. I was planning to scratch the surface and tell the people what to look for and where to find them – to write apps in Python for Android. The talk was scheduled in Track 2 which was apparently a hall with less capacity. 5 minutes after starting the talk, someone came in and said we need to move to the main auditorium given that the crowd who wanted to attend my talk won’t fit into this small hall. We wrapped up and moved to the auditorium.

Now, the pressure was on me. I really didn’t have any idea what these people might be expecting and whether my talk would help them get even closer to it. I started off with this disclaimer and showed them how sweet and beautiful Python code can be when compared to Java on Android – a pause – loud applause. That made my day. Couple of guys were really interested and they interrupted me to ask questions. I hope that I’ve answered them all. Few people came to me and I just gave them a better idea of the architecture and how things work. All went well. Phew! I should thank Nijad for keeping track of what examples and text I should be sharing with them.

Praveen, Labeeb and Vishnu wanted me to cook ghee rice. Labeeb said he can make chicken curry. I was a bit nervous and called up mom to ask how much rice should I cook. Met Nikhil and Sandy in Pune. Hung out with them and then crashed to Pravi’s place, where we ended up cooking!

ALPR using Free Software: Paper at MESTECH 2010

Bagged the first place for the technical paper presentation in MESTECH 2010. Pitched in ALPR using Free Software. You can take a look at the paper here.

Thanking Nijad, Shehzad, and Saalim for all the support. And I don’t have words to thank Sajith sir, who always pushed forward to get things done and mentored us through out the process. Thank you sir!

Experimenting PyS60 on my Nokia 6600

Another lazy day, and did not take very long for me realize this was the 5th day of the study holidays. Had enough fun, sleep and hacks. This would add to it. Was exchanging few email conversations with Arky and don’t know from where in the world, the thought about his Nokia N70 struck me. I remember, how difficult it was to get him over phone during the NIT-C Mapping Party, because the phone was not interested in answering calls anymore, rather, run Python scripts!

And should I say more? I’m the proud owner of a six year old Nokia 6600, and I’m not gonna change it. Googled for Python S60 and came up to the PyS60 wiki run by Nokia and the project page. Figured out the suitable latest PyS60 1.4.5 release for Symbian 2nd edition phones, downloaded two files: 1. PythonForS60, 2. PythonScriptShell, installed both on the phone and whao! I’m running Python on my phone. I was too happy to resist tweeting. Downloaded the documentation and found many useful libraries for messaging, GPS etc. Wrote a simple script for SMS flooding.

import messaging
for i in range(0,10):
messaging.sms_send(“9846492504”, “Poda!”, name=”Nijad”)
import messaging
for i in range(0,10):
	messaging.sms_send("xxxxxxxxxx", "Poda!", name="abc")
Spoke to Nijad over phone, and confirmed that the study holidays are going to be hacking holidays.