Write the Docs: Beyond metrics: Using maturity models to develop a docs strategy – Sarah R. Rodlund

Catherine Heath | April 27, 2021

This talk at Write the Docs Portland was given by Sarah R. Rodlund. Sarah works as a technical writer at the Wikimedia Foundation. 

Sarah talked about metrics technical writers use to measure the impact of their work and guide their docs strategy and why they fall short. She went on to introduce the concept of the maturity model and how to apply that to your docs. 

Docs at the Wikimedia Foundation

Sarah began her talk by mentioning what it has been like working at the Wikimedia Foundation. 

When Sarah came on board, there was no unifying docs strategy; efforts were piecemeal. The team complained that the docs needed work, but she couldn’t get a clear articulation about why they felt this way. The documentation was written by staff and volunteer developers all over the world who did not communicate with each other.

Since Wikimedia is supported by a huge community of staff and volunteer developers, they couldn’t lean too hard on traditional metrics. 

Sarah had the perfect solution to Wikimedia's docs problem. She decided to build a maturity model for the tech docs, that gives you an overview of the documentation and help set some achievable goals.

How to measure your docs

According to Sarah, tech docs resist quantification. They don’t produce flashy numbers, so traditional metrics can consist of a range of methods to measure the impact and success of your tech docs strategy. 

Sarah discussed three measures for your docs: quality, performance and quantity.

Quality 

Quality of the content and reading experienced can be measured. Readability scores can measure understandability, conciseness and effectiveness. You can look at the timeliness and accuracy of the docs. Are your docs up-to-date and accurate? What percentage of the docs meet the content guidelines defined by your team or organization? 

Performance

Measuring performance can provide a picture of how your audience uses or sees the docs. How often have pages been viewed? You can use data from helpfulness ratings like stars, happy/sad faces, or thumbs up/thumbs down. Satisfaction surveys can give you more information about your audiences. 

Quantity

The quantity of your docs can be used as a metric – how many words that were written, or how many tech docs were created. More documentation isn’t necessarily better documentation, so don't rely too much on this metric. 

Maturity models

However, as Sarah noted (and many of us know), traditional metrics can fall short of the mark for a number of reasons:

• They can be hard to interpret and act on.

• They can misrepresent the success of a strategy.

• They reveal little about the overall quality of tech docs.

• Taken alone, they can’t show the big picture. 

If you have a high level view of where you are and where you want to be, you’ll be in a better position to know which metrics will be useful to you at any given time. 

Sarah proposed that maturity models can help us gain a more honest and holistic understanding of our work and its impact. They are used across a large number of industries and have been adopted by a variety of fields and processes. 

Maturity models take the form of images, written text, or formal or loose frameworks, and their creation can be a formal exercise or something more casual. As Sarah observed, they all share the similar trait of moving from an ad hoc phase, through a strategic phase, and then into an optimizing phase. Maturity models can be visual but they don’t have to be. 

Sarah then shared this example of a maturity model from Joann T. Hackos:

There are five levels of maturity in this model:

• Ad hoc

• Rudimentary

• Organized and repeatable

• Managed and sustainable

• Superior

Why maturity models are useful

Sarah talked about a number of benefits to using maturity models with your documentation:

• They’re good for giving you a starting place with your documentation.

• You think critically about the state of your docs and where you are, and you can create an honest picture of the state of your docs.

• Once you’ve defined the levels of maturity for your project, you can start goal-setting and see the areas to focus on improvement. 

• Maturity models give you a clear and common language for the documentation and where you’re going.

• You can tangibly demonstrate change, growth and success or even the failings of your strategy.

“An effective and mature process produces results” – Joann T. Hackos

The limitations of maturity models 

Sarah admits that there are some limitations to using maturity models:

• They’re not a perfect tool, since they can be inflexible or too flexible.

• Their reliability depends on your ability to see them for what they are.

• They can dupe you into thinking that progression has to be linear. 

• They are also never complete, and you will always be iterating on them.

• They can be limited by a lack of diverse perspectives so be open to bringing in other people to help you define your maturity model.

So use them with a grain of salt.

Creating your own maturity models

Sarah closed by offering some guidance on creating your own maturity model:

• Since maturity models are flexible and take different forms, explore other models for ideas/inspiration.

• Work from the highest maturity back to the lowest maturity as you describe the levels of your model.

• Adjust your expectations based on the type of organization you’re in and resources available to you.

• Bring in diverse perspectives to evaluate the accuracy of your model.  

Her parting advice felt particularly relevant:

Maturity models are about progress, not perfection. They exist for the purpose of guiding you towards your goals. Use your maturity to get you oriented to the point where you feel confident about your docs strategy. 

Final remarks

We don't currently use a maturity model at KnowledgeOwl for our own documentation, and have struggled around finding effective measures. But Sarah's presentation makes us think a maturity model might be a fantastic approach; we'd encourage other writers facing similar struggles to check it out!

Watch the full talk here. 

{{snippet.authorCatherineHeath}}

{{snippet.Disqus}}