Visualizing SSLC results over seven years.

I spent most of the last one month building a dashboard for the SSLC results at the Karnataka Learning Partnership. We released it in beta yesterday and here’s the blog post I wrote detailing how we went about it.

Patterns in examination results are something which we are always interested at the Karnataka Learning Partnership. After the design jam in June 2012, where we tried to understand the SSLC data – it’s content and structure, and visualized performance of Government and Private schools in contrast to each other, we decided to take a step deep and find patterns from the past seven years. Results of this effort is what you find here, in beta.

The Karnataka Secondary Education Examination Board shared the data over the last seven years in a combination of several Microsoft Access Database. It came with very little meta data and Megha did all the hard work of making sense of this and pulled it into a PostgreSQL database. Inconsistencies are everywhere, and the quality always depends on how you handle each exception in isolation. Among other things, we decided to look at three aspects of the data to begin with – performance of Government and Private schools, performance in Mathematics, Kannada and English, and performance of each gender. All three across seven years (from 2004-2005 to 2010-2011) for each district in Karnataka.

One of the important data wrangling that we did this time was to aggregate this data at the district level. The raw data came at the educational district level and unfortunately, we did not have geographic boundary shapes for this classification. What we have instead is the geographic boundary at the political level. We massaged the shapefiles, geocoded the data and converted it to GeoJSON in QGIS. We wrote a bunch of Python scripts to perform the aggregation and generate JSON (JavaScript Object Notation) required for the visualization. Every bit of code that we wrote for this project is on Github.

A dashboard of this sort is something which we have never attempted, and honestly it took a while for us to get around it. I had tried D3.js sometime last year and found it to be amazing. D3 is Data Driven Documents, a brilliant JavaScript library to make infographics on the web driven completely by the data. What makes D3.js awesome for me is that everything is an SVG (Scalable Vector Graphic), and there are barely any limits to the representation and interaction that you can bring into the browser with it. I’ve had good experiences with Twitter’s Bootstrap to quickly design and be consistent on the page layout and aesthetics. There are some issues while you work with D3.js and Bootstrap together, especially the way bootstrap manages events. The best way is to trust D3.js and use Bootstrap features of scaffolding and layout.

We found few interesting facts from this exercise. As you may guess, private schools perform better than government schools consistently. Western districts like Udupi, Uttar Kannada and Belgaum performs better than rest of the state. North Karnataka, especially Bidar performs terribly across the last seven years. Something which we are very curious to know why. Bangalore Rural performs better than Bangalore Urban. Government schools does much better and comparable to private schools in Bangalore Rural than Bangalore Urban. Private schools grab the cap in all the three subjects across the last seven years. Girls performs way better than boys in private schools consistently across seven years in every district. Boys does a better job in Bangalore Urban while girls dominate in Bangalore Rural.

This research will continue while we churn few more aspects from the data as the dashboard gets out of beta.

 

Beyond just a ‘Map’

This morning, I came across another dose of my Apple maps debate. We’ve been continuously going back and forth about how bad the data on iOS6 maps are and that it’s a shame that Apple is doing this to their customers. To be fair, making a map, a base layer, is not a tiny task. Nor it is something that a team of 50 can pull together in 3 months. It takes rigorous learning and efficient processes to build a map which ‘delivers’ what people want. It’s not about building a map that looks good on the phone. It’s about building the geospatial data repository where you find what you are looking for.

The whole new ‘Social Map’ phenomenon is making me think that we are attempting the Borgesian map. In 1946, Jorge Luis Borges published a short story on how the map of an empire got too accurate and exact enough that anything smaller than the representation equivalent to the size of the empire will be considered incorrect – ‘a scale of a mile to the mile’.

May be we are not attempting it to be of that size, which quite clearly is impossible, but where we are heading is a bit scary – the map where you will find anything you want – you will see where your friends are – you will see what you would want to do next. Trust me, I’ve seen my friends in trouble after they started using Google Latitude.

The amount of detail that people want on the map makes me cringe at times. I leave it here for everyone else to think.

And yes, the Curiosity Rover checked in from Mars!

The Working with Data Workshop, Chennai.

Riju, Shashank and I pulled off a three day workshop at the Institute for Financial Research and Management, Chennai during 3-5 August. Francesca was around to support us. The key idea behind the workshop was to introduce ways of collecting, processing and analysing the data. Right after the Fifth Elephant, we got together in Bangalore and laid out the plans. The outline of the workshop was quite clear – From Data to Maps. The outline is here. All slides and related data is here.

“It’s called growing up, young man!”

I don’t know whether I want to write this post, but yeah may be. The last two weeks were super stressful. So much shit I was dealing with. Unwanted guests, dad being unwell, stuff at work – between jobs and what not. It was hard. And even when I tried to sit down and code, thinking that would make me feel better, I got bogged down with thoughts. Kingsly, Zainab and I spent a lot of time together than usual. And when I tweeted, I meant that. Kingsly is the island of calmness. His words are selected out of the best. It’s been a blessing to have him around. We would talk about everything, let alone bitching about relatives :P

Zainab was desperately trying to realize the fact that her PhD is delayed and she’s now working hard to put together The Fifth Elephant. And I realized that when she’s not able to write, she gets super stressed. When we are together, we are like kids. Heh. <clicks>

(/me can hear Z saying “I see” :P)

I went home without knowing when I could return. Dad was diagnosed for Retinal Occlusion. That was hard. And I learned that he’s going to be subjected to 4 weeks of laser treatment. I had no choice other than to go home and be with them, I guess they were quite relieved.

While telling this whole story to Francesca, I was like “I don’t want to deal with so many stuff at the same time”. I was also missing RootConf. Parag and Anu were also not around and I could see Nigel slightly freaking out, I felt really bad that I won’t be around.

Francesca said just this – “It’s called growing up, young man! ”

Wait, what?

The life changing lesson.

I’ve a bunch of friends coming home tonight to celebrate the success of the last couple of days, which were the most intense, delightful and tiring days of my life so far. The house is a mess, with clothes thrown here and there, random papers and gadgets all over the place. Yes, I’m just cleaning but I don’t want to push off writing this blog post any more. I should have done this yesterday, but my mind and body just didn’t allow, even now I can feel the warmth in my eyes.

Eight months passed since I moved to Bangalore. At the beginning of this year, I made one of the best decisions in my life, to join HasGeek. I might not be able to put this into flowery words but yes, HasGeek is a bunch of exciting, super exciting and fascinating people. The way each of us work is in itself a new big story.

My first assignment at HasGeek was to join the trip to Goa (yes, that was a work trip), and the first official task which I executed was to drive the car. That was an exciting trip. While we were planning the year ahead, I figured out that Nigel and I might be writing code while rest of the team will be running events. But right now, it feels like the world and words have suddenly changed in front of me. Even though I’m spending just few hours a week for HasGeek, I got the chance to run an event.  The Cartonama Workshop. Yes, we did it.

In 2011, right after Android Camp, bunch of us spent the whole night hacking on one thing – build a community around open geographic data and showcase the projects in India – Cartonama.com. Nowhere in the wilderness of my thoughts I could imagine that we would do events around Cartonama.

A week before JSFoo Pune, Kiran asked whether I would be able to take charge of the Cartonama Workshop. Maps are something that fascinates me all the time and I totally wanted to take this up. I remember that day quite clearly. That day marked exactly a month plus a week to the event. The news that Mikel and Schuyler are coming to Bangalore for the workshop was even more exciting.

We wanted to get the website out by the end of that week. Fortunately or unfortunately, I had to spend that weekend in Calicut. And now it’s time to tell you about one of my best friends, Ashima. We spent the days together when she figured out that I’m stressing out so much. We went for long walks on the beach talking about Cartonama and what I wanted to do about it. She’s amazing.

Kiran had couple of ideas how the website should be and I thought it would be best if he takes care of that. Kiran was just busy and didn’t have any bandwidth to get this going. Nothing happened and no sign of the website getting live. Zainab was the biggest pillar of support – and an entirely different woman that I’ve seen all my life. By the end of the week, I realized, getting the website live was just my task, no matter how I can execute it. Nigel, Riju, Aditya and I created the first iteration and we went live. There was something quite different from rest of the events that we have done – the ticket price for the workshop being INR 10,000. That amount will just help us break even with the expenses, only if we sell 30 tickets. So the problem is not to create a good looking website, but to create a website that will sell tickets. Parag did the second iteration and that was fabulous.

Unlike conferences, chances of attracting sponsorship was very weak. The task at hand was quite tricky – to convince people and sell tickets. I was slightly bothered because, two days after going live with the website we couldn’t sell any tickets. I tried to sell tickets all day, let alone near the barber shop. Zainab would say “Take it easy boy, we shall prevail!”. We did prevail. And my heart is beating fast with joy.

Mikel was in the US and Schuyler in Afghanistan. That was some serious distance which was filled by flood of emails between us. We wanted the workshop to be covering the entire technology stack that would help someone to run a location based service. The funnel was put into test again and we discovered many interesting ways to improve it.

Two weeks later, we have sold two tickets. I was getting very nervous, and there were moments when I would just close my eyes trying to stay sane. Patience.

Today, after the event I just don’t want to think about those days anymore. We sold a total of 22 tickets. Not bad.

Some people are just sloppy and intense to deal with. The most inconsistent person I’ve ever seen in my life was that new caterer with whom we signed up. He screwed up everything at the last moment and we switched. When people are sloppy, there is nothing that you can do. Patience.

Billy is another interesting character that I met recently. I can tell you that he is quite a character. Billy is prompt and perfect. His thoughts are sharp and ideas are simple and sensible. Throughout the two days of the workshop, it was Billy, I and Nigel managing the logistics at the venue, while Zainab and Kiran were busy backing us up.

I understood that technology and infrastructure is inconsistent. They wont wait or keep promises. We bumped into issues after issues on day one of the workshop and I felt terrible, but Billy and Nigel was dealing with endless crisis. Patience is the word again.

Internet is something that people care about. They get frustrated every moment when the Internet goes down. There would be sudden chaos. Every nerve in my body was hurting and there were moments when I just wanted to scream out to the people to be patient.

In the middle of the sessions, my eyes would just lock between Mikel’s and Schuyler’s. I just don’t know why, but that gives me a pulse of how everything was going.

At the end of day 1, Billy and I reached home quite late. I was tired and sleepy, but we sat down and started talking about everything that happened that day. I realized that there were too many things that we could’ve done better.

Day 2 was interesting. I was just worried by the thought that whether the participants are enjoying the sessions. Interacting with many of them over the breaks told me that things are going just alright. Sigh of relief. People just loved the way Mikel and Schuyler was taking them through the sessions. They just loved it.

And today, I can proudly say that the workshop wasn’t too bad. I guess it was intense as we promised. The amount and depth of things I learned over the last couple of days cannot be just described. Patience is the key. It is. Always.

 

 

MapQuest tiles through Leaflet.js

Cross posted from geohackers.in.

Leaflet.js by default supports Cloudmade’s awesome map tiles. The Open MapQuest project also provides beautiful tiles, which are now used at OpenStreetMap.org. To use these tiles all you have to do is to change the map preamble as follows.

The Cloudmade maps layer looks like this.

Change the preamble to.

Zoom level based marker interaction using Leaflet.js

With Arky I was creating a map of the Mozilla l10n teams world wide and decided to use Leaflet.js. He send across a CSV data sheet which contained the geo-tagged teams and other information. Wrote a Python script to generate markers corresponding to Leaflet.js (Yes, my JS skills suck). When we laid the markers on the map, we found that it looked all crowded and difficult to locate one team quickly. I took this discussion to the HasGeek channel and Nigel showed me the Ubuntu Loco Global Events map. The markers in the map changed based on the zoom level of the map. That is pretty neat idea to handle crowded map markers. But they were using Google Maps.

I wanted to make this happen with Leaflet.js and OpenStreetMap data. Leaflet.js is perhaps the best, easy and intuitive interactive mapping library that I’ve come across. I spent some time in the documentation and came up with this idea and voila! – it worked.

Leaflet.js provides several events, the one which we will use is zoomend. This event gets triggered whenever the zoom level of the map changes. Exactly what we need. This is how I did it.

If you have a bunch of markers, use LayerGroup, that will give you better control. You can now add/remove the entire marker set by the LayerGroup handler.