In October, my team has won the Meteor Global Distributed Hackathon for the best app written with less than 100 lines of JavaScript. I would like to share the story leading to winning the hackathon, and introduce the project we built.

I participated in the hackathon out of Sydney, Australia. The hackathon was global and cities around the world kicked it off one after another. We were the second one to start just after New Zealand. Thanks to University of Western Sydney for hosting the venue.

Our Project: Meteor Icon

In the regular user meet-up prior to the hackathon, I brought up the idea of making an embeddable icon for Meteor packages. The icon would display various useful information about the package in a manner that is easy to consume.

I had recently finished building nomad-icon back then, and I had some first-hand experience in dynamically generating and serving an svg. I was positive that, if done right, such icon for Meteor packages can be built with less than 100 lines of code.

(You can read my article about building nomad-icon here.)

For the Better Package Ecosystem

NPM module authors can embed node icon in their projects’ README as an introduction to their package.

NPM

I think initiatives such as this can have a far-reaching positive effect on the package ecosystem. It is because package authors and users can establish a clear, customary way of communicating the information about the packages.

Currently, the popular places to search for Meteor packages are Atmosphere and GitHub.

Atmosphere has all the useful information about packages including the latest verions, last published date, download count. It even has internal scoring system for packages.

But GitHub is where most packages host their source code, and many people discover Meteor packages through GitHub. Here, a problem arises: users need to make unnecessary back-and-forth trips between GitHub and Atmosphere to understand the package.

That users have to do so is bad for two reasons:

  1. Users who do not make efforts to do their due research can make poor choices for the packages they use.

  2. Such process is inefficient and there is a better way. It would be better if the README files self-contained all the latest meta information about the packages.

Meteor Icon can be embedded in a project’s README to solve the problem of communication between package authors and users:

App Internal

Meteor Icon was built with 58 lines of JavaScript code. For the curious among the readers, here is a link to browse the source code at the time of submission. Below, I will provide a glimpse of the internal app structure.

It is using Meteor’s webapp to mount a middleware on /package path. webapp does not allow you to access url parameters in an Express-like syntax, so we had to parse the incoming request’s url to get the package’s name. Despite this slight inconvenience, we chose webapp because it was a very lean solution.

When it receives a request, the middleware collects the information about the requested package. It first reaches out to Atmosphere’s API endpoint. Additionally, it fetches the package scores through an DDP connection to Atmosphere. Their DDP endpoints were undocumented and we had to experiment a bit to find a proper way.

With all the information in its hand, the app dynamically renders an svg file on the server side. Then, it simply serves it to the client in an xml/svg response.

At the time of the submission, all this logic was contained in one routes.js file. You can check it out here. It was only 41 lines of JavaScript, but did a lot. But since then, we refactored the code and made some improvements.

Post-Hackathon Improvements

The users from Meteor Sydney gave a few pointers about how to improve the project. Here are some of those that I so far implemented.

  • Caching based on timestamp

Making an API call and fetching collection from a DDP connection every time serving an icon is not cheap. The delay is not noticeable on GitHub because GitHub caches external assets, but such design is still a nonsensical one.

An obvious solution is to maintain a local collection of package information. But this means that the server will have an extra overhead of synchronizing the local data with the source periodically.

Instead of performing cron jobs for some 8000 packages, we decided to let users do the synchronization for us.

Each document contains a timestamp for the last time the package icon was requested. Every time a request for icon comes in, the timestamp is compared to the current time. If it is older by more than our threshold, the app refreshes the document by making API and DDP calls. Otherwise, the app uses local data to generate the icon.

In a way, this is a lot like the concept of a reverse proxy that caches the database query.

By this design, users will see information that is a few hours old. However, the design makes sense on the account that GitHub caches the icon anyway. In addition, a few hours of delay is not a big deal for package information. We are not dealing with a stock market.

  • Avoid sleeping in default Meteor hosting

Currently, Meteor Icon is hosted on the default Meteor cloud, using the domain icon.meteor.com. The problem is that, after 15 minutes of inactivity, the app hibernates. This means some users can see a blank box instead of icon, even though the server is up.

This problem was solved by making an HTTP GET call to itself periodically to prevent prolonged inactivity.

  • Put logic out of the routes.js file and use methods.

We put logic out of routes file to make the code more modular and more maintainable. This is what it looks like now. The core logic now lives in Meteor methods. It is taking up far more lines of code, but we need not worry about that now as the Hackathon is over.

  • Simplify the icon.

The icon was kind of complicated at the time of submission, and we decided to simplify the icon.

The above is what Meteor Icon used to look like by default. It has graph in the background, and 5-star based rating followed by the score. Although such features had ample hack value and was fun to make, we discovered from feedback that their meaning was not that clear.

In addition to the vague meaning, I thought that those features might be preventing package authors from adopting Meteor Icon because the statistics were less than stellar for majority of packages. It was not uncommon to see one star rating, single digit score, and a flat graph.

I have made a proposal to simplify the icon, and some of the uninformative features are now turned off by default. Users can still opt to turn them on. The long term goal is to make the icon components as modular as possible, enabling users to customize the information.

It was hard to make the decision because I was proud of my team’s ingenuity for making all those possible with only few lines of code. But the features are neither deprecated nor removed; they are still available and will be of great use once the icon is modularized.

How many people are using it?

Maintaining a local collection for package info allowed us to see clearly which packages are using Meteor Icon, and how popular they are.

Inside a package info document, we maintain a property called requestCount and increment it every time a package’s icon is requested. A rough estimate for the number of packages using Meteor Icon is 41.

production-a:PRIMARY> db.packageInfo.count()
41

Here are 10 most popular packages using Meteor Icon at the moment:

production-a:PRIMARY> db.packageInfo.find().sort({requestCount: -1}).limit(10).map(function(info){ return {name: info.name, requestCount: info.requestCount}; });

[
	{
		"name" : "useraccounts:flow-routing",
		"requestCount" : 530
	},
	{
		"name" : "tomi:upload-jquery",
		"requestCount" : 259
	},
	{
		"name" : "useraccounts:bootstrap",
		"requestCount" : 131
	},
	{
		"name" : "useraccounts:core",
		"requestCount" : 116
	},
	{
		"name" : "mquandalle:jade",
		"requestCount" : 111
	},
	{
		"name" : "ryw:blog",
		"requestCount" : 100
	},
	{
		"name" : "ccorcos:any-db",
		"requestCount" : 96
	},
	{
		"name" : "tomi:upload-server",
		"requestCount" : 95
	},
	{
		"name" : "easy:search",
		"requestCount" : 94
	},
	{
		"name" : "useraccounts:iron-routing",
		"requestCount" : 89
	}
]

The team

Here are the collaborators from Meteor Sydney who built Meteor Icon together.

  • tomitrescak - worked on backend, connected to Atmosphere, designed the component in SVG.
  • woody1990 - helped the backend setup, integration with Atmosphere, validation and some of the front end work.

Conclusion

Meteor Icon would not have turned out this great, had I not worked with a team. Yet, working with a team felt slow at times, and I felt frustrated when I could not do the things the way I wanted to.

In this light, I think there is a subtle balance all makers need to find: a balance between one-man play and team play. It is great to work alone, but we must realize and adopt quickly if working with a team will produce better results.

I also see, from Meteor Icon, a distinction between a process and an event. People tend to see only the event, and neglect the process that makes the event possible.

Winning the hackathon is a great event.

But that event is made possible by the (almost) all-nighter my team pulled off. And in many ways, the process dawned far before the hackathon even began, when I was building nomad-icon. It seems that every moment is a process leading to possible great events in the future, and we might be delusional if we do not make them all count.