Visualizing SSLC results over seven years.

I spent most of the last one month building a dashboard for the SSLC results at the Karnataka Learning Partnership. We released it in beta yesterday and here’s the blog post I wrote detailing how we went about it.

Patterns in examination results are something which we are always interested at the Karnataka Learning Partnership. After the design jam in June 2012, where we tried to understand the SSLC data – it’s content and structure, and visualized performance of Government and Private schools in contrast to each other, we decided to take a step deep and find patterns from the past seven years. Results of this effort is what you find here, in beta.

The Karnataka Secondary Education Examination Board shared the data over the last seven years in a combination of several Microsoft Access Database. It came with very little meta data and Megha did all the hard work of making sense of this and pulled it into a PostgreSQL database. Inconsistencies are everywhere, and the quality always depends on how you handle each exception in isolation. Among other things, we decided to look at three aspects of the data to begin with – performance of Government and Private schools, performance in Mathematics, Kannada and English, and performance of each gender. All three across seven years (from 2004-2005 to 2010-2011) for each district in Karnataka.

One of the important data wrangling that we did this time was to aggregate this data at the district level. The raw data came at the educational district level and unfortunately, we did not have geographic boundary shapes for this classification. What we have instead is the geographic boundary at the political level. We massaged the shapefiles, geocoded the data and converted it to GeoJSON in QGIS. We wrote a bunch of Python scripts to perform the aggregation and generate JSON (JavaScript Object Notation) required for the visualization. Every bit of code that we wrote for this project is on Github.

A dashboard of this sort is something which we have never attempted, and honestly it took a while for us to get around it. I had tried D3.js sometime last year and found it to be amazing. D3 is Data Driven Documents, a brilliant JavaScript library to make infographics on the web driven completely by the data. What makes D3.js awesome for me is that everything is an SVG (Scalable Vector Graphic), and there are barely any limits to the representation and interaction that you can bring into the browser with it. I’ve had good experiences with Twitter’s Bootstrap to quickly design and be consistent on the page layout and aesthetics. There are some issues while you work with D3.js and Bootstrap together, especially the way bootstrap manages events. The best way is to trust D3.js and use Bootstrap features of scaffolding and layout.

We found few interesting facts from this exercise. As you may guess, private schools perform better than government schools consistently. Western districts like Udupi, Uttar Kannada and Belgaum performs better than rest of the state. North Karnataka, especially Bidar performs terribly across the last seven years. Something which we are very curious to know why. Bangalore Rural performs better than Bangalore Urban. Government schools does much better and comparable to private schools in Bangalore Rural than Bangalore Urban. Private schools grab the cap in all the three subjects across the last seven years. Girls performs way better than boys in private schools consistently across seven years in every district. Boys does a better job in Bangalore Urban while girls dominate in Bangalore Rural.

This research will continue while we churn few more aspects from the data as the dashboard gets out of beta.

 

Zoom level based marker interaction using Leaflet.js

With Arky I was creating a map of the Mozilla l10n teams world wide and decided to use Leaflet.js. He send across a CSV data sheet which contained the geo-tagged teams and other information. Wrote a Python script to generate markers corresponding to Leaflet.js (Yes, my JS skills suck). When we laid the markers on the map, we found that it looked all crowded and difficult to locate one team quickly. I took this discussion to the HasGeek channel and Nigel showed me the Ubuntu Loco Global Events map. The markers in the map changed based on the zoom level of the map. That is pretty neat idea to handle crowded map markers. But they were using Google Maps.

I wanted to make this happen with Leaflet.js and OpenStreetMap data. Leaflet.js is perhaps the best, easy and intuitive interactive mapping library that I’ve come across. I spent some time in the documentation and came up with this idea and voila! – it worked.

Leaflet.js provides several events, the one which we will use is zoomend. This event gets triggered whenever the zoom level of the map changes. Exactly what we need. This is how I did it.

If you have a bunch of markers, use LayerGroup, that will give you better control. You can now add/remove the entire marker set by the LayerGroup handler.

Getting a Huawei EC159 USB Modem working on Ubuntu 11.04

The EC159 just like any USB modem gets detected as a USB mass storage or virtual CD-ROM when connected to Linux boxes. Yes, usb-modeswitch should take care of this and make it available as a GSM Modem. Unfortunately, the usb-modeswitch configuration skips this particular device, and that’s what I felt.

I decided to run usb-modeswitch to manually configure the modem by the following steps:

1. Get the latest release of usb-modeswitch

2. Build and install it.

3. Plug in the modem and run lsusb. Note down the vendor ID and Product ID.

For example:  Bus 002 Device 002: ID 8087:0020 Huawei

The vendor ID is 0x8087 and Product ID 0x0020

4. Run usb_modeswitch -H -v <vendor ID> -p <product ID>

5. Now modeprobe using the same vendor and product ID

sudo modprobe usbserial vendor=<your’s here> product=<your’s here>

6. Your device will start working like a charm!

Do let me know in the comments if you are stuck.

Data, Visualizations and Evidences.

Yesterday marked a month since I moved to Bangalore. Flooded with interesting assignments from both work places. The first two weeks went into hard-core scraping and data conversion. Taking down/mirroring couple of National Informatics Center’s servers etc. Incidentally, we started doing some analytics and couple of them are showcased here.

Third week of my life in the big city started with associating with the startup called Boost Tech. My role with them mostly involved designing a data dashboard, identifying data variables, visualizing them and bringing evidences from the research. Brij, the CEO, handed to me two of Edward Tufte’s awesome books on visualization of quantitative information. I’m enjoying my weekends and some hours of the weekdays, cruising them and exploring new ways of infographics.

Experimenting PyS60 on my Nokia 6600

Another lazy day, and did not take very long for me realize this was the 5th day of the study holidays. Had enough fun, sleep and hacks. This would add to it. Was exchanging few email conversations with Arky and don’t know from where in the world, the thought about his Nokia N70 struck me. I remember, how difficult it was to get him over phone during the NIT-C Mapping Party, because the phone was not interested in answering calls anymore, rather, run Python scripts!

And should I say more? I’m the proud owner of a six year old Nokia 6600, and I’m not gonna change it. Googled for Python S60 and came up to the PyS60 wiki run by Nokia and the sourceforge.net project page. Figured out the suitable latest PyS60 1.4.5 release for Symbian 2nd edition phones, downloaded two files: 1. PythonForS60, 2. PythonScriptShell, installed both on the phone and whao! I’m running Python on my phone. I was too happy to resist tweeting. Downloaded the documentation and found many useful libraries for messaging, GPS etc. Wrote a simple script for SMS flooding.

import messaging
for i in range(0,10):
messaging.sms_send(“9846492504”, “Poda!”, name=”Nijad”)
import messaging
for i in range(0,10):
	messaging.sms_send("xxxxxxxxxx", "Poda!", name="abc")
Spoke to Nijad over phone, and confirmed that the study holidays are going to be hacking holidays. 

Report Class for LaTeX: Highly modified

Finally, posting it here. Went through lot troubles to get the stuff in place. As I really did not understand what each clause in the class file meant, most of the time was spend messing up the page layout and features. Thankful to Department of Communication Technology, Aalborg University, for sharing the IES report class, which I obtained from here.

Download the modified report class from here and the TeX file from here.

The TeX file contains references to a bibliography too. You could easily try and hack.

ICS Ubuntu, Simplified.

We have a good WiFi enabled Internet campus. Unfortunately, one of the Ubuntu Lucid installations in a Dell Inspiron did not detect the WiFi controller. The last resort was to get an ethernet cable with Internet connection to activate the driver. We asked google and found out perhaps the best solutions to do ICS (Internet Connection Sharing) with Ubuntu here.

I would brief the steps taken so that anybody can get this done very quickly.

Our situation is different from that described in the above document.

Server connected to Internet via wlan0.

Client connected to server via eth0.

You have to make sure no changes to networking has been done since turning on both the computers. If any, you would have to restart both of them.

Server side configuration.

  • Connect the computer to wlan0 and make sure the Internet access is available.

Start a terminal and do the following to configure your network card and NAT.

  • sudo ifconfig eth0 192.168.0.1
  • sudo iptables -A FORWARD -i wlan0 -o eth0 -s 192.168.0.0/24 -m conntrack –ctstate NEW -j ACCEPT
  • sudo iptables -A FORWARD -m conntrack –ctstate ESTABLISHED,RELATED -j ACCEPT
  • sudo iptables -A POSTROUTING -t nat -j MASQUERADE

These settings will be cleared when you reboot the system. If you want to set it up permanently then:

  • sudo iptables-save | sudo tee /etc/iptables.sav

Edit /etc/rc.local and add the following lines before the “exit 0” line:

  • iptables-restore < /etc/iptables.sav

Enable routing:

  • sudo sh -c “echo 1 > /proc/sys/net/ipv4/ip_forward”

Edit /etc/sysctl.conf and add these lines:

  • net.ipv4.conf.default.forwarding=1
  • net.ipv4.conf.all.forwarding=1

Client setup:

Disable networking

  • sudo /etc/init.d/networking stop

Give static IP address.

  • sudo ifconfig eth0 192.168.0.100

Configure routing.

  • sudo route add default gw 192.168.0.1

Configure DNS Servers.

  • Open /etc/resolv.conf in the server and add the contents to resolv.conf of the client.

Open /etc/dhcp3/dhclient.conf and add:

  • prepend domain-name-servers 208.67.222.222,208.67.220.220;

Restart Networking:

  • sudo /etc/init.d/networking restart

You are good to go!

Note: If you are in a proxy network make sure to apply the proxy settings to the client also.

My first bug report on Ubuntu Lucid

Last day while me and Arky were fixing Ubuntu Lucid into some computers to be used by blind people, it came into our notice a very simple bug. These people depend on Orca Screen reader to make their way into computing. Unfortunately, at times, Orca crashes and they require to restart the X server to get in place. This was easy for them to press Ctrl + Alt + Backspace and then come back to the working environment.  But, in Lucid, after killing and restarting X server, gdm prompts for password to be keyed in, which is not desirable for these differently abled people.

The bug report is here