Improving Documentation
with User Tests at Bosch BT

Problem

Bosch BT sells a range of surveillance cameras and software to manage them, called BVMS (Bosch Video Management System). The BVMS on-screen documentation was published in the CHM format (remember these purple books?). The problem with CHM was that it looked outdated, but what's worse, it also had some security issues. Time to switch to a more modern approach: an HTML5-based help.
Prior to the user test, I spotted UX and content-related issues in the HTML5 prototype. Together with our system administrator, we improved the HTML5 prototype until we felt it was good enough for the user test.

Goals

Since I would be running a user test for the HTML5 prototype, I also included the newly introduced how-to video.
The test aims to answer these questions:

• How do our users gather information when something in BVMS is unclear to them?
• What kind of documentation do they mostly use?
• How often do they use the documentation?
• Which help functionalities do users use to find the desired information?
• How easily do the users find information in the on-screen help?
• How helpful is the information that they find?
• When and in which contexts are instructional videos useful?

Process

The process consisted of three major phases: Planning, testing, and evaluation.

Planning phase

1. Creating a pre-test survey: Together with the UX-Intern supported, I created a questionnaire to collect data on demographics, knowledge of BVMS, and use of documentation.
2. Finding test tasks: My supervisor and I scoured over 900 slides of training material for suitable test tasks. We identified eight tasks that were appropriate for different levels of BVMS proficiency.
3. Acquisition: I acquired test subjects by publishing a call for participants on Bosch's internal social network. As a result, I received ten registrations, from eight different countries. I gained three additional participants through networking.
4. Test setup: Since my participants were sitting all over the world, I designed a remote user test using Skype. Fig. 2 shows the setup in detail. Apart from reaching a wider range of participants, the remote user test also posed technical advantages:
• The user does not have to install the system and help separately.
• In addition, many workstations running BVMS do not have a connection to the Internet.
• The test administrator can prepare the software so that the test task is processed immediately.
• The session can be recorded so that interesting passages can be viewed again later.

The only disadvantage was that you can not observe body language remotely. You will miss out on emotions and subtleties that are not apparent from the conversation.

Testing Phase

Nine of the 13 registered participants took part in the user test. All participants were Bosch employees and already familiar with BVMS.
The tests were following the scheme in fig. 3. First, the participants had to solve the tasks in BVMS using the HTML5 help. Afterward, the moderator asked questions about the help, such as "Were you finding the information you expected?" In the second part of the test, the participants should follow the video and answer questions afterward. Most of the tests would be conducted by the UX intern and me.
We took turns throughout the test, so I switched from being the moderator and to being the observer.

Evaluation Phase

I consolidated the survey and the notes to find recurring remarks. After that, I put together a report which I presented to my manager and the BVMS product team.

Challenges

Finding suitable test tasks

When I was planning the user test, I had been with Bosch BT for about six weeks. Since BVMS is rather complex software, I was not an expert myself - so I turned to the experts. I had asked our training department if they could put together some tasks. They did not have the time but were kind enough to provide me with their slide deck, which my supervisor and I searched for tasks.

Finding suitable participants

Initially, I planned on testing real users on-site, face to face. However, it was not possible to recruit real customers due to a lack of resources to provide compensation and pay for travel expenses. Therefore, we resorted to Bosch employees who could handle BVMS. Fortunately, many participants were in close contact with customers, so their feedback was bundled and passed on to us.

Objectivity

One of my colleagues was focusing a lot on writing down real statements that we could include in the report later on. However, sometimes the discussion moved on so quickly, that they could not capture everything, resulting in incomplete sentences. This made it really hard to evaluate the notes objectively and not interpret something that the participants had not said. I learned that the quality and structure of the transcript are important for a quick and clean evaluation.

Research Insights

The first three questions were focusing on the usage pattern of the BVMS documentation, see fig. 4. While the survey showed, that most participants use the documentation fairly often, they also rely on colleagues to help them. Since we were interviewing Bosch employees, they have access to knowledgeable colleagues. However, the average user does not have this luxury.

Next, the participants shared their opinion on the HTML5 help.
Which help functionalities do users use to find the desired information? Most users went for the search. They also used the new history feature to go back to entries they had viewed before.
How easily do the users find information in the on-screen help? The search revealed what the users were looking for most of the time. One suggestion was to enable some kind of advanced search to narrow down search results. Another issue was that the users did not know when a query was unsuccessful.
How helpful is the information that they find? The general opinion was that the documentation was focusing too much on describing functionalities, instead of guiding through a use case.

Lastly, we collected feedback regarding the how-to video to answer the question: "When and in which contexts are instructional videos useful?"
Most of the participants said they would expect the videos to be either embedded in the on-screen help or be able to click a link that leads them to YouTube or a Bosch website.
Since all the test subjects work at Bosch BT themselves, we received a lot of feedback on how learning videos could improve their everyday work. Apart from documentation, potential applications for how-to videos include:

• Customer support
• Part of a training
• Preparation for on-site visits
• Technical Marketing

Impact

The user test unveiled that the HTML5 help prototype was a step in the right direction but still had some minor UX issues to be fixed. I provided action items for the HTML5 help and recommendations on implementing them.
Furthermore, the test made clear that video documentation is in demand, as it focuses on actual use cases. Since BVMS has a steep learning curve, the videos can potentially speed up the learning process and make it easier to configure the software. The demand is so high, that a sales manager had started publishing how-to videos himself in Spanish. I believe that the responsibility to provide user-centric documentation lies within the technical writing department, therefore I provided a collection of the most wanted topics for how-to videos.
Lastly, the user test did not only show where the documentation can be improved but also revealed pain points in the actual product. Also, I raised the interest within the technical writing community at Bosch to run their own user test.
I showed that is possible to gather valuable insights with a little support. Since the technical writers provide content for multiple stages in the customer journey, improving the content will increase the overall user experience.

Lessons learned

Data-driven documentation. I believe collecting data about our users and their usage patterns needs to be a cornerstone of content creation. When technical writers and user researchers work together, the customer experience as a whole will improve.

Nature abhors a vacuum. This principle holds true for content as well. The test revealed that a sales manager in Spain already provided how-to videos to satisfy the demand for simple, use case-oriented working aids. As technical writers, we regularly need to check if the content we create still fits the needs of our audience.

The art of user testing. Conducting a user test in itself was a new but fun experience for me. I found it difficult to watch users struggle and felt the urge to help them. However, their struggle is where the insights happen, so I had to stay silent. Another learning was that notes are the foundation for the evaluation afterward, so making enough time for note-taking is crucial.