2.5 years.

I joined the Karnataka Learning Partnership in May 2012. We just launched version 11.0 of our web platform.

I’ve been meaning to write this post for a while now. Recently, it marked 2.5 years of my association with the Karnataka Learning Partnership. This is not my going away post and I’m not going away. I want to talk about how amazing the environment is for someone entirely new to the public education space. I want to talk about how extraordinary the team is. I want to talk about how different our processes are. And I want to talk about how patience, understanding and teamwork will go a long way.

If you haven’t heard about the Akshara Foundation or the Karnataka Learning Partnership, KLP is incubated at Akshara Foundation and they both work closely to improve schooling in Karnataka. KLP does great amounts of work on streamlining data collection processes, building dashboards and acting as an interface between several organisations in the education space in Karnataka and India.

When I joined KLP, it was a team of three – me, Megha Vishwanath, and Shivangi Desai, headed by the inimitable Gautam John. My role was largely around organising the wealth of spatial data that we just collected and collated from various sources. Wrangling dirty data and organising things makes me happy. KLP has a pretty strong software infrastructure history. Even though Python and PostgreSQL made it to the mix, there were other unmaintained open source software and a lot of unseen complexity in the code base. To be true to my open map heart, I rewrote the maps infrastructure. It was mostly a matter of throwing in couple of new end-points to the API and writing the front-end in JavaScript. Pretty basic. I wasn’t quite ready to dive into the madness of our existing API, so I made sure I touched little of the legacy code base. Couple of months down the line, I built a dashboard for showing SSLC performance. We worked on an array of projects right after – DISE dashboard, reports, status and a lot more. We were quite sure about an overhaul of our infrastructure and soon started talking about it. I was fairly open to the idea, but concerned, considering that there are only 3 of us and a rewrite is a lot of work.

Today, we are seven people, people I must mention because I owe them so much for understanding how diverse a team could be and accepting the fact that everybody is trying – Gautam John, Bibhas Debnath, Sanjay Bhangar, Vinayak Hegde, Brijesh Poojary, Deviprasad Adiga. Megha and Shivangi did much of the tough work of getting all the things in place for us to start. Everything we have built is based on what we have had. If there is a new feature, we have made sure that all of us thoroughly understand that it is required and how it should work. Our team is geographically disconnected – between Bombay, Calcutta, Pune and Bangalore. This is not the only thing we are doing. Each of us are involved in several other exciting projects and have weird schedules that barely overlap. To me, it is amazing to see how we have pulled this new platform in under five months. We have met twice in the process and rest of it happened over video conferences and telephone calls. I personally have had a very rough time over the last two months, which was when we were gearing up towards the crunch period. I was moved by the way the team managed to make sure that I was involved in a very limited capacity, but made sure that what needs to be done was taken care of.

To be honest, rewrites are hard. I try to avoid them as much as possible but when software becomes a liability, you have to make sure it works and continues to do so. Rewrites are usually a fairly intense process, because you are never starting from a clean slate. For the most part, you have a team which doesn’t quite understand most of how the legacy system works. Disconnect within the team is not uncommon, but it’s difficult. I have seen how the seven of us accommodated each of the ideas on the table. It’s about listening and understanding each other. It’s about respecting and realising that the other person is also committed to building the best product. It’s about being polite and being patient. And finally it’s about embracing the fact that there are only three certainties – life, death and fucked up data.

Update: We also work with some of the best technologists on and off, most recently Vamsee Kanakala and Rahul Gonsalves along with his team at Uncommon.

Visualizing SSLC results over seven years.

I spent most of the last one month building a dashboard for the SSLC results at the Karnataka Learning Partnership. We released it in beta yesterday and here’s the blog post I wrote detailing how we went about it.

Patterns in examination results are something which we are always interested at the Karnataka Learning Partnership. After the design jam in June 2012, where we tried to understand the SSLC data – it’s content and structure, and visualized performance of Government and Private schools in contrast to each other, we decided to take a step deep and find patterns from the past seven years. Results of this effort is what you find here, in beta.

The Karnataka Secondary Education Examination Board shared the data over the last seven years in a combination of several Microsoft Access Database. It came with very little meta data and Megha did all the hard work of making sense of this and pulled it into a PostgreSQL database. Inconsistencies are everywhere, and the quality always depends on how you handle each exception in isolation. Among other things, we decided to look at three aspects of the data to begin with – performance of Government and Private schools, performance in Mathematics, Kannada and English, and performance of each gender. All three across seven years (from 2004-2005 to 2010-2011) for each district in Karnataka.

One of the important data wrangling that we did this time was to aggregate this data at the district level. The raw data came at the educational district level and unfortunately, we did not have geographic boundary shapes for this classification. What we have instead is the geographic boundary at the political level. We massaged the shapefiles, geocoded the data and converted it to GeoJSON in QGIS. We wrote a bunch of Python scripts to perform the aggregation and generate JSON (JavaScript Object Notation) required for the visualization. Every bit of code that we wrote for this project is on Github.

A dashboard of this sort is something which we have never attempted, and honestly it took a while for us to get around it. I had tried D3.js sometime last year and found it to be amazing. D3 is Data Driven Documents, a brilliant JavaScript library to make infographics on the web driven completely by the data. What makes D3.js awesome for me is that everything is an SVG (Scalable Vector Graphic), and there are barely any limits to the representation and interaction that you can bring into the browser with it. I’ve had good experiences with Twitter’s Bootstrap to quickly design and be consistent on the page layout and aesthetics. There are some issues while you work with D3.js and Bootstrap together, especially the way bootstrap manages events. The best way is to trust D3.js and use Bootstrap features of scaffolding and layout.

We found few interesting facts from this exercise. As you may guess, private schools perform better than government schools consistently. Western districts like Udupi, Uttar Kannada and Belgaum performs better than rest of the state. North Karnataka, especially Bidar performs terribly across the last seven years. Something which we are very curious to know why. Bangalore Rural performs better than Bangalore Urban. Government schools does much better and comparable to private schools in Bangalore Rural than Bangalore Urban. Private schools grab the cap in all the three subjects across the last seven years. Girls performs way better than boys in private schools consistently across seven years in every district. Boys does a better job in Bangalore Urban while girls dominate in Bangalore Rural.

This research will continue while we churn few more aspects from the data as the dashboard gets out of beta.