Testing a new pitch format for shaped work
Inspired by Christopher Alexander's Battle (see the previous newsletter), I'm experimenting with a new format for pitches at Basecamp.
When closely studying Battle, I noticed the pattern language didn't take scope or budget into account. The compromises and scope hammering came in a later step, when the construction budget was defined relative to different structures specified by the language. The real design trade-offs were made when the team went on site and started to generate a particular layout from the interaction of site, budget, and pattern language.
I haven't figured out how to articulate this yet, but that process resonates strongly with the original Getting Real principles from Basecamp's early days. We simply have to encounter the realities of the project before we can know what the actual design will be. This motivates me even more to try and use a pattern language to do the shaping at the right level of abstraction — where the interrelationships are well-defined but the specific structure is still open for the teams to resolve.
Here's my first attempt at that. It's a project idea for BC4.
|
|
Measuring usage with a Taguchi signal/noise ratio
Bob's been slowly schooling me in Taguchi methods over the last couple years. I'm starting to make some small steps towards applying them to software projects.
My favorite so far is the signal/noise metric. Here's how I understand it so far.
How do we know if a product is working "as it should" for people? For example, we could write automated tests that prove the mechanism for inviting people to Basecamp projects works as intended. But what if somebody gets added to a bunch of projects that don't matter to them? Now their Basecamp account is full of irrelevant projects. Is that "working"?
Supposed we wanted to try and measure this. What's the "right" number of projects a person should have? No idea! There's no "right answer" to that question.
We can get a new perspective by rethinking how we define the system to be measured. Every function can be thought of as having an input signal and an output response. In the case of inviting someone to a project in Basecamp, the desired output response isn't "it shows up on their account." Basecamp isn't a tool for making things appear on a screen! A better response is "when I share something with them, they look at it."
|
|
Having reframed the function like this, we can define a signal/noise ratio. The output of the function of course includes showing the project on the home screen. But the meaningful response is more than that. It's the fact that the person granted access actually looks at the project. This means we can say that just populating their account with another project is noise when they don't also look at it.
This metric doesn't say anything about how many projects are good or bad. It tells us whether the feature is being used as intended.
"Intent" is the special ingredient when defining a system with Taguchi methods. By better understanding the intent, we can look at the variation of the functionality to say when the output should be considered signal or noise.
This metric is striking when graphed. The images below show data from Basecamp's own Basecamp account. The X axis is S/N, from 0 to 100% (0 to 1). The first graph shows that nearly all of us in the company only look at half of our projects (in a 30 day period).
|
|
The second graph adds another dimension, showing the S/N per person and per number of projects.
What this tell us is, the single action of "granting access" is very imprecise — even for us, the Basecamp experts. In reaction to this, I've shaped a new take on project access for BC4 that distinguishes between different kinds of expectations for participation. This will allow us to give the function a more specific "signal" for the intent of adding someone who is a participant vs. adding someone who should just have access for reasons of transparency. This metric will allow us to concretely test to see if we've improved the ratio if/when we start using the new feature internally.
|
|
Plus:
- News on Ergodicity Economics — I've been following Ole Peters' work on Ergodicity Economics because it has a lot of overlap with how I've learned to think about decision making from the demand side. Bloomberg just posted a nice intro article called Everything We Learned About Modern Economic Theory Is Wrong. The first EE conference, EE2021, has an exciting lineup of talks and over 500 attendees registered. I contributed to one of the papers that will be presented, on "Microfoundations of Discounting."
|
|
|
React
I'm using this newsletter both to share and to learn.
Have you tried similar things? Are you doing something different? Struggled with the same challenges? What's your experience?
Replies to this email go straight to my inbox.
|
|
|
|
|
|
