Brian E. Granger

Personal Reflections on Features and Scope

Brian Granger — Wed, 09 Jan 2013 18:44:01 +0000

In a previous blog post, I talked about the problem of ever expanding feature sets in open source software projects. I discussed the hidden costs and liabilities of adding new features and proposed practical ways of tackling these problems. However, I think that story is incomplete. Of course open source projects will want to add features, even some with massive costs and liabilities, but how should that happen? How should projects decide which features to add? What criteria should be used? To answer these questions, I want to describe a few of my own experiences in dealing with features and scope in IPython and PyZMQ.

First, IPython. Anyone who is familiar with IPython will probably read my previous blog post and say, “wait, hasn’t IPython experienced a massive amount of feature creep over the last few years. It started out as a simple terminal based interactive shell, but has grown to include parallel computing, a web application and a GUI application. The IPython developers, including yourself, have shown little, if any restraint in adding features.”

Back in 2004, IPython was the 3 year old child of Fernando Perez. At that point, it was already a popular, enhanced interactive Python shell. I had started to work on parallel computing libraries for Python, when Fernando visited me at Santa Clara University (we had been classmates in graduate school starting in 1996). We stayed up until 3 am talking about Python’s rapidly evolving ecosystem of scientific computing libraries, interactive computing environments and the future of IPython and my parallel computing libraries. IPython was a fantastic tool, but both of us wanted more. We wanted a web based IPython Notebook. We wanted integrated parallel computing tools. We began to see that all of these things required the same architecture: a stateful computational engine, the IPython Kernel, that could run user code and send back results over a network. Once that existed, everything else could be layered on top, even the terminal based IPython. The vision that emerged that night was specific and, while ambitious, was ultimately finite. As Fernando describes, in this blog post, it took us ~6 years and multiple attempts to build this architecture. Now, we have a terminal, QtConsole, web-based notebook and parallel computing library all built on top of a common IPython Kernel and message specification. While these things may look like feature creep from the outside, they have been part of a deliberate and calculated plan.

What I have learned from this experience is that it is absolutely critical for the core developers of an open source project to consciously and deliberately set the scope and vision for the project. This vision can even be ambitious. Once that scope is set, choosing which features to add becomes easy: to first order, you add features within that scope. That doesn’t mean you don’t count the cost of those features or that you add all possible features within the scope. The scope provides an upper bound on the feature set. So why does IPython have parallel computing tools? Because we set the scope early on to include that.

Second, PyZMQ. As we developed IPython’s architecture, we realized that we needed better networking/messaging libraries. This led to the creation of PyZMQ in 2010, which is a set of Python bindings for ZeroMQ. Notice the scope of PyZMQ: Python bindings for ZeroMQ. Fast forward to Spring of 2012. In the course of working on the IPython Notebook, I had written some additional code for building PyZMQ based web applications: a module for simple, PyZMQ-based RPC and a module for running Tornado event handlers in a scalable and distributed manner using PyZMQ. I submitted two pull requests to include these modules in PyZMQ. One of them was quickly merged and the other languished in code review for almost a year. Min began to sense that these modules, while useful, were not a good match for inclusion in PyZMQ. Their API was very unstable and they hadn’t been well tested. They would require a much faster release cycle compared to the rest of PyZMQ, which was very stable and well tested. This Christmas break, I finally woke up to the real problem. These modules were outside the scope of PyZMQ. They involved an implicit and undiscussed increase in scope from “Python bindings for ZeroMQ” to “A general place for ZeroMQ based tools in Python.” Once I realized this, Min and I pulled them out of PyZMQ and created separate projects for them. Again, the important thing is to set the scope of a project.

Features and Scope in Open Source Software

Brian Granger — Wed, 09 Jan 2013 03:49:20 +0000

Follow up: I have created another blog post to clarify some of these issues here.

The IPython project, through UC Berkeley and Cal Poly San Luis Obispo, just received a $1.15 million dollar grant from the Alfred P. Sloan Foundation to develop the IPython Notebook over a two year period. More details about this grant can be found on the IPython website. This is really exciting for us because, so far, we have mostly developed IPython in our spare time. But I think there is also a potential danger here. The danger is that we will add lots of new features. What, you say, lots of features will endanger IPython? What else are you going to do with a million dollars if you are not going to add lots of new features? The answer is simple: we are going to add as few features as possible and knock each of them out of the park. The future of the project depends on this.

This is a topic that I have been thinking about a lot lately: how do open source projects decide which features to implement. Most active open source projects I am involved in see a continual stream of new features. Just hop onto the GitHub pages for SymPy or IPython and watch the activity. Anyone with the right skill set can fork a project on GitHub and submit a pull request within a few hours. Amazingly, this is happening all the time; apparently people love to code. While each new feature is an asset for the project, it also brings a cost, or liability, with it. If a project ignores those costs, it can have long term, detrimental effects on the project. What are these liabilities and costs associated with new features?

Each new feature adds complexity to the code base. Complexity makes a code base less hackable, maintainable, extensible.
Each new feature increases the “bug surface” of the project. When a feature also adds complexity, those bugs become harder to find and fix.
Each new feature requires documentation to be written and maintained.
Each new feature requires support over email or IRC.
Endless feature expansion, or feature creep, requires developers to specialize. They can’t follow the entire project, so they have to focus on a subset that can fit into their brain and schedule.
Each new feature has to be tested on a wide variety on platforms (Linux, Mac, Windows) and environments (PyPy, Python 2, Python 3).
Each new feature adds complexity to the user experience. Sometimes it’s the documentation, other times the UI or configuration options.
When you spend on one feature, another feature or bug fix didn’t get worked on. If you didn’t prioritize things beforehand, you just spent time on something less important to your users. Either that or you did shoddy work while trying to do it all.
Features multiply like bunnies. How many times have you heard, “wow, that new feature is really cool, could you make it do X as well?”
Features are easy to add, difficult to remove. Once you add a feature, you are stuck with the costs and liabilities.

For a typical open source project, what percentage of features get used regularly by most users? 15%? 40%?. Let’s be really optimistic and say that number is 60%. That means that developers are spending 40% of their time and effort on features that don’t really get used. Ouch. Then why do open source projects keep adding features without counting the cost? I think there are a number of factors that lead to this, but one in particular comes to my mind. It is hard to say no. When an end user submits a feature request over email or on GitHub issues, it is hard to tell them, “great idea, but we are not doing that.” It is even more difficult to say no after someone submits a pull request that implements a new feature. It is difficult to build a vibrant project community if you are always saying no in these contexts.

Clearly, we need a better way of limiting feature expansion in open source projects. What can we do to better evaluate the hidden costs of adding new features so we can make informed, strategic decisions about which features to add? Here are some ideas that have emerged out of my recent reading and conversations.

Create a wiki page for your project, where you list all of the features you are not going to implement. Publicize this list, discuss it and make it an important part of the development workflow. Another way of phrasing this is to decide on a finite scope for the project. When you are going through this exercise, come up with an initial scope and then force yourself to reduce it.
Make features fight hard to be accepted and implemented. Communicate to the community and developers that the default answer to feature requests is no (it’s not personal!) and don’t even consider implementation until the much of the community is crying “we absolutely must have this.” Even then, you don’t have to say yes.
Create a workflow that separates feature requests from other tickets/issues. When people submit new feature requests, encourage discussion, but don’t automatically promote the feature to the project’s todo list. Then you can promote them, as needed, to the project’s todo list in an informed and prioritized manner.
When new feature requests appear, discuss the specific costs and liabilities associated with the feature. Build this thinking into your development DNA.
Communicate to the community and its developers why it is important to fight against feature expansion. Focus on the benefits of waging this war: smaller, simpler code base, fewer bugs, more time to focus on important features, easier to support, etc.
Remove features that have too great a cost or that few users actually use. Maybe even create a special exception you can raise (FeatureReductionWarning?) to help people transition away from them.
Refactor the codebase to reduce complexity. While this doesn’t directly reduce the number of features, it can mitigate the cost of existing and future features. Extra bonus points if you can implement a new feature while dramatically reducing the complexity of the code base.
Improve testing. Again, this is mitigation.

As you discuss and evaluate features, here are some questions you can ask yourself and the community:

What fraction of your user base will use the feature? How often will they use it? If it won’t be used by most of your users, just say no.
Can the feature be added as a third party plugin or library? This is especially useful if the new feature would increase the overall scope of the project, but make a great standalone project.
How difficult will it be to test, debug, document, and maintain the feature? What fraction of your development team is capable or interested in doing this work? If the maintanence is huge and only one person is willing to do it, it is time to rethink it.
Can you implement the functionality in a more limited, but much simpler manner? Developers absolutely love to implement features in the most general way possible. It requires developer discipline and focus to resist this temptation.
One way that developers over engineer things is by making every conceivable thing configurable. Can you simplify the feature by removing configurability and just choosing great defaults?

One thoughtful discussion of these issues is in the book Getting Real, by some of the folks at 37signals. They propose something quite radical for handling feature requests in commercial web applications. Here is what they say: “How do you manage them? You don’t. Just read them and then throw them away.” Their experience is that the important features will keep coming up. When this happens, you know they are important and you don’t have to write them down to keep track of them. This is definitely my experience in developing the IPython Notebook. The most important features, the ones that I am going to spend time on this year, are probably not written down anywhere, but everyone in the community is discussing them and couldn’t possibly forget them. So why on earth do we currently have 177 open new feature issues (we call them “enhancements”) on IPython’s GitHub site?

In an open source project, I don’t think it makes sense to literally throw away feature requests; sometimes the ensuing discussion is valuable and worth preserving. But what about allowing the discussion to occur, for example in the context of a GitHub issue, but then closing the issue. If someone wants to re-open the feature request at a later time to voice their support, they should be encouraged to do that. But again, once discussion stops, the issue should be re-closed. As this process repeats itself, the community is essentially voting for the features they want. This would also dramatically reduce the number of open issues, which helps developers to better manage the active work on the project.

I don’t think this is the only way to manage feature requests intelligently in an open source project. I would love to hear other ideas. How are you managing these things in your own projects? I realize that I am far from the first person to write about these things. What other resources do you know of that address these problems?

PyData NY

Brian Granger — Thu, 29 Nov 2012 05:36:31 +0000

This past month, I attended PyData NY in midtown Manhattan. This is the second PyData conference and the folks at Continuum Analytics have done a spectacular job in promoting and running this conference. In less than a year PyData has grown to over 200 attendees. The timing (right after Strata) and location (NY and Bay Area) of the PyData conferences are definitely a strong part of the success.

The conference had a wide range of talks from a fantastic set of speakers. It is very clear that the community is growing in leaps and bounds and that there are a lot of bright people writing great code. It is always fun to think back on where the community and various projects were in the past. Just think, it wasn’t that long ago, that Pandas didn’t exist! Where will we be in another 3 years?

The representation of IPython at the conference was amazing. Not only did I present two talks on the IPython Notebook and IPython.parallel, but many of the other presenters used the IPython Notebook in their talks. Ultimately, in developing the IPython Notebook we are scratching our own itch. We are working scientists who need better tools for our research. But it is seriously gratifying to see that how useful other people find it. That makes developing it one of the most enjoyable things I have ever done.

Continuum has put all of the talks on Vimeo here, but I wanted to point out a few related to IPython:

First, I gave a talk about the IPython Notebook. With 45 minutes, I was able to give a talk that was almost tutorial in nature. In preparing this talk, I went through the IPython example Notebooks and organized/updated all of them. This should provide a much nicer set of materials for people wanting to learn about the Notebook. We are currently in the process of reviewing the pull request for these updates.

Second, I gave a talk about IPython.parallel. For this talk, I mostly followed a set of materials that Min Ragan-Kelley prepared for previous SciPy/PyCon tutorials. Min has done a fantastic job with IPython.parallel and this set of materials really goes into great depth on all of the details. This material is hosted here.

Third, Michael Selik gave a nice talk entitled IPython as a teaching tool, which explores using the IPython Notebook in a teaching context. It is becoming clear that the Notebook is particularly useful in this context and it was very helpful to have someone reflect on how that was going for them.

Finally, Massimo di Stefano is doing some amazing things with the Notebook and GIS. He is really leveraging the ability of the Notebook to embed arbitrary HTML. He even has a “gearth” function that embeds Google Earth in an output cell! Here is one of his Notebooks on nbviewer.ipython.org.

Unfortunately, many people left early because of hurricane Sandy, so the sprints were sparsely attended. I was glad to catch my flight back to the west coast only hours before JFK closed. Can’t wait for PyData in Silicon Valley after PyCon.

Blogging with the IPython Notebook

Brian Granger — Fri, 05 Oct 2012 02:54:59 +0000

The IPython dev team gets a lot of questions about how IPython Notebooks can be used on various blogging platforms. There have been a number of different attempts to use nbconvert to export a notebook to HTML and then embedding that HTML in the blogging platform. With the introduction of nbviewer it is trivial to embed a Notebook in any web page using iframes. Here is a simple example:

Which produces the embedded Notebook shown below. The only subtle part is that you have to set the width and height attributes manually to avoid having scroll bars on the iframe.

Cal Poly Physics department cluster

Brian Granger — Tue, 08 Mar 2011 20:42:49 +0000

Here are some photos of the new cluster, which we are calling “dirac”, in the Physics department. This cluster will be used for research in a broad range of fields in the department including:

Quantum information
Atomic and molecular physics
Parallel and distributed computing
Astrophysics and cosmology
Particle and nuclear physics
Particle beam physics
Seismology

The system specs are as follows:

4x Dell R710 servers
Dual Xeon 5650 CPUs (6 core)
48 GB RAM
5x 1 TB hot plug, 6Gb/s SAS drives
4x Gb ethernet

This works out to a total of 48 cores, 192 GB RAM and 20 TB of storage.

Why Python?

Brian Granger — Wed, 13 May 2009 21:48:45 +0000

I began playing with computers back in the early 80′s when my elementary school got an Apple II. I was immediately drawn to it and began coming in whenever I could and playing around. I began with LOGO (remember the turtle) and later moved on to BASIC. Eventually, I convinced my parents that we really needed a computer and they purchased an Apple IIc “for the family.” We had this computer until I went to college in 1990. Thus began my enjoyment of computers and programming.

While in graduate school in Physics, I began to use Unix and Linux and learned C, C++ and Perl. During this time, my research in theoretical Physics involved lots of computational work. At the time most physicists around me were working in Fortran. But, I figured that learning C and C++ would be more useful in the long run. As I began working in C++, however, I realized that I had a dilemma: the major computing libraries for Linear Algebra (BLAS and LAPACK) were written in Fortran. Furthermore, C and C++ did not have a robust, built in-multidimensional array type like Fortran (for Linear Algebra). So C++ gave me all the niceties of an object oriented language, but none of of basic tools that I needed for my research. Surely, I was not the only person to have dealt with this problem when moving to a more modern language than Fortran.

My solution at the time, while not efficient in terms of human effort, was to write my own object oriented C++ library for performing matrix and array manipulations. This took a lot of time as I really wanted to get things right. I used C++ templates to define matrices for the different basic types of data (int, float, double). To keeps things computationally efficient, I had my classes call LAPACK for all of the time consuming things. I ended up using this library for all of my scientific codes during graduate school and my postdoc. Once I started using my library to write scientific code the benefits were obvious. I could code complex algorithms rapidly, with few bugs. While this was wonderful, the cost was also huge. I probably spent 6 months of full time work in graduate school writing and maintaining my C++ linear algebra library. I didn’t know about open source at the time, so I did this all on my own and kept it to myself.

At the beginning of my postdoc, a friend of mine introduced me to Python and open source software. My friend claimed that Python and open source software opened up completely new ways of approaching scientific computing. Python was object oriented, dynamically typed, could be used interactively (like Mathematica or Matlab), and there were high quality, open source libraries for scientific computation (NumPy, Scipy, IPython, Matplotlib, SymPy) and interfacing with other languages (Swig, Weave, f2py, Cython, Ctypes).

I first tried a few smaller projects using Python. I was amazed at how quickly I could write and debug code. Working interactively was a dream – no more endless edit, compile, debug, recomplie, execute cycles! Within a short period of time, I had made the decision to start all of my new projects in Python. Since then I have used Python on a number of scientific codes and I have been extremely happy with it. Most importantly, I am having more fun with computing than I ever have before and I have found that Python has freed me up to think differently about computing, and consequentially about Physics. Looking back, I can’t believe how painful everything used to be with my C++/bash/perl/Grace toolchain.