Visualizing SSLC results over seven years.

I spent most of the last one month building a dashboard for the SSLC results at the Karnataka Learning Partnership. We released it in beta yesterday and here’s the blog post I wrote detailing how we went about it.

Patterns in examination results are something which we are always interested at the Karnataka Learning Partnership. After the design jam in June 2012, where we tried to understand the SSLC data – it’s content and structure, and visualized performance of Government and Private schools in contrast to each other, we decided to take a step deep and find patterns from the past seven years. Results of this effort is what you find here, in beta.

The Karnataka Secondary Education Examination Board shared the data over the last seven years in a combination of several Microsoft Access Database. It came with very little meta data and Megha did all the hard work of making sense of this and pulled it into a PostgreSQL database. Inconsistencies are everywhere, and the quality always depends on how you handle each exception in isolation. Among other things, we decided to look at three aspects of the data to begin with – performance of Government and Private schools, performance in Mathematics, Kannada and English, and performance of each gender. All three across seven years (from 2004-2005 to 2010-2011) for each district in Karnataka.

One of the important data wrangling that we did this time was to aggregate this data at the district level. The raw data came at the educational district level and unfortunately, we did not have geographic boundary shapes for this classification. What we have instead is the geographic boundary at the political level. We massaged the shapefiles, geocoded the data and converted it to GeoJSON in QGIS. We wrote a bunch of Python scripts to perform the aggregation and generate JSON (JavaScript Object Notation) required for the visualization. Every bit of code that we wrote for this project is on Github.

A dashboard of this sort is something which we have never attempted, and honestly it took a while for us to get around it. I had tried D3.js sometime last year and found it to be amazing. D3 is Data Driven Documents, a brilliant JavaScript library to make infographics on the web driven completely by the data. What makes D3.js awesome for me is that everything is an SVG (Scalable Vector Graphic), and there are barely any limits to the representation and interaction that you can bring into the browser with it. I’ve had good experiences with Twitter’s Bootstrap to quickly design and be consistent on the page layout and aesthetics. There are some issues while you work with D3.js and Bootstrap together, especially the way bootstrap manages events. The best way is to trust D3.js and use Bootstrap features of scaffolding and layout.

We found few interesting facts from this exercise. As you may guess, private schools perform better than government schools consistently. Western districts like Udupi, Uttar Kannada and Belgaum performs better than rest of the state. North Karnataka, especially Bidar performs terribly across the last seven years. Something which we are very curious to know why. Bangalore Rural performs better than Bangalore Urban. Government schools does much better and comparable to private schools in Bangalore Rural than Bangalore Urban. Private schools grab the cap in all the three subjects across the last seven years. Girls performs way better than boys in private schools consistently across seven years in every district. Boys does a better job in Bangalore Urban while girls dominate in Bangalore Rural.

This research will continue while we churn few more aspects from the data as the dashboard gets out of beta.

 

Zoom level based marker interaction using Leaflet.js

With Arky I was creating a map of the Mozilla l10n teams world wide and decided to use Leaflet.js. He send across a CSV data sheet which contained the geo-tagged teams and other information. Wrote a Python script to generate markers corresponding to Leaflet.js (Yes, my JS skills suck). When we laid the markers on the map, we found that it looked all crowded and difficult to locate one team quickly. I took this discussion to the HasGeek channel and Nigel showed me the Ubuntu Loco Global Events map. The markers in the map changed based on the zoom level of the map. That is pretty neat idea to handle crowded map markers. But they were using Google Maps.

I wanted to make this happen with Leaflet.js and OpenStreetMap data. Leaflet.js is perhaps the best, easy and intuitive interactive mapping library that I’ve come across. I spent some time in the documentation and came up with this idea and voila! – it worked.

Leaflet.js provides several events, the one which we will use is zoomend. This event gets triggered whenever the zoom level of the map changes. Exactly what we need. This is how I did it.

If you have a bunch of markers, use LayerGroup, that will give you better control. You can now add/remove the entire marker set by the LayerGroup handler.

The very special Pycon 2011

Download the talk and watch the video.

This edition of Pycon was in fact very special in different perspectives. I came to know that my talk was selected from Kartik. It has been a while since I hacked on Python with Android but I did follow the updates and new releases. 4 days before the conference, I managed to buy a Samsung Galaxy S 2 which looked to me as a brilliant phone and couldn’t wait to start hacking.

Since Praveen was in Pune, I didn’t have to look for other option to crash in. Took a bus overnight to Pune reached there on Friday (16th September), went to Pravi’s place. That was my first longest bus journey. Bangalore – Pune 846 KM. Reached the venue late in the afternoon, met couple of friends. Felt good to have given Noufal, Pradeepto and bunch of other guys, a face, with whom I interacted online.

My talk was on day 2. Didn’t have much to talk either. I was planning to scratch the surface and tell the people what to look for and where to find them – to write apps in Python for Android. The talk was scheduled in Track 2 which was apparently a hall with less capacity. 5 minutes after starting the talk, someone came in and said we need to move to the main auditorium given that the crowd who wanted to attend my talk won’t fit into this small hall. We wrapped up and moved to the auditorium.

Now, the pressure was on me. I really didn’t have any idea what these people might be expecting and whether my talk would help them get even closer to it. I started off with this disclaimer and showed them how sweet and beautiful Python code can be when compared to Java on Android – a pause – loud applause. That made my day. Couple of guys were really interested and they interrupted me to ask questions. I hope that I’ve answered them all. Few people came to me and I just gave them a better idea of the architecture and how things work. All went well. Phew! I should thank Nijad for keeping track of what examples and text I should be sharing with them.

Praveen, Labeeb and Vishnu wanted me to cook ghee rice. Labeeb said he can make chicken curry. I was a bit nervous and called up mom to ask how much rice should I cook. Met Nikhil and Sandy in Pune. Hung out with them and then crashed to Pravi’s place, where we ended up cooking!

AndroidCamp Bangalore

I’m sleep deprived and tired to write down all the awesome time I had at the AndroidCamp.
Jobless in the bus, so I should attempt to write something.

AndroidCamp was one of the best unconference that I’ve been to in Bangalore. I was so excited when Kiran and Kesava came up with this idea. I remember the discussions on the list that there were more than 200 sign ups within 48 hours of announcement.
Soon I found the proposals website proposals.androidcamp.in a wonderful idea that all unconferences should adopt.
Met a bunch of interesting people who do 100 other things around Android. I was too happy to find that there are people who hate Java and love Android :)

So copying from Manish Sinha. :)
Read it all here. http://milky.manishsinha.net/2011/04/02/android-camp-bangalore/

Experimenting PyS60 on my Nokia 6600

Another lazy day, and did not take very long for me realize this was the 5th day of the study holidays. Had enough fun, sleep and hacks. This would add to it. Was exchanging few email conversations with Arky and don’t know from where in the world, the thought about his Nokia N70 struck me. I remember, how difficult it was to get him over phone during the NIT-C Mapping Party, because the phone was not interested in answering calls anymore, rather, run Python scripts!

And should I say more? I’m the proud owner of a six year old Nokia 6600, and I’m not gonna change it. Googled for Python S60 and came up to the PyS60 wiki run by Nokia and the sourceforge.net project page. Figured out the suitable latest PyS60 1.4.5 release for Symbian 2nd edition phones, downloaded two files: 1. PythonForS60, 2. PythonScriptShell, installed both on the phone and whao! I’m running Python on my phone. I was too happy to resist tweeting. Downloaded the documentation and found many useful libraries for messaging, GPS etc. Wrote a simple script for SMS flooding.

import messaging
for i in range(0,10):
messaging.sms_send(“9846492504”, “Poda!”, name=”Nijad”)
import messaging
for i in range(0,10):
	messaging.sms_send("xxxxxxxxxx", "Poda!", name="abc")
Spoke to Nijad over phone, and confirmed that the study holidays are going to be hacking holidays. 

Thinking In Python Session ; #TIP

The Last days SciPy.in was Sprints. Few the guys in our team did not find it interested in the way sprints was happening. So we thought to sit along, and develop our skills in Python. The Thinking in Python Sessions was born. It was awesome to get some brains working behind some cool problems.

I would pull the problems here.

  1. Lexicographic sort
  2. Using docstrings for function description.
  3. Read in a file, change the case, and write to new file.
  4. Read a string from the user and print it as many number of times, he want.
  5. Parse a file, evaluate means of rows and columns.

SciPy.in India 2009

The first version of SciPy.in India Conference is being held at Technopark, Trivandrum. Thank God, despite of the minor accident, I was able to make to it. Not only me, we have a team of 14 guys!

The Team!
The Team!

The conference began with the inauguration by Travis. He is a wonderful guy.
Travis Inaugurating the conference.
Travis Inaugurating the conference.

And we had a lot of good sessions in two days!
A Session by Travis on Traits and Chaco! Really enjoyed!
A Session by Travis on Traits and Chaco! Really enjoyed!

A bit of mapping inside the Technopark Campus.
Mapping!
Mapping!

And now, getting ready for the tutorials!