Improving Documentation
with User Tests at Bosch BT

About the Project

During my internship at Bosch Building Technologies (BT), I conducted a user test to find weak spots in the online help. The company was publishing their software documentation as a PDF and a deprecated CHM format, which should be replaced with an HTML5-based help. Since I was also introducing how-to videos at the time I decided to include them in the test. The project took place over the course of 12 weeks from start to finish. I led the project and my supervisor and a UX intern supported me.


A representative screenshot from the on-screen help in the old CHM format.
Fig. 1 - CHM help
Source: Geeksengine

Bosch BT sells a range of surveillance cameras and software to manage them, called BVMS (Bosch Video Management System). The BVMS on-screen documentation was published in the CHM format (remember these purple books?). The problem with CHM was that it looked outdated, but what's worse, it also had some security issues. Time to switch to a more modern approach: an HTML5-based help.
Prior to the user test, I spotted UX and content-related issues in the HTML5 prototype. Together with our system administrator, we improved the HTML5 prototype until we felt it was good enough for the user test.


Since I would be running a user test for the HTML5 prototype, I also included the newly introduced how-to video.
The test aims to answer these questions:


The process consisted of three major phases: Planning, testing, and evaluation.

Planning phase

  1. Creating a pre-test survey: Together with the UX-Intern supported, I created a questionnaire to collect data on demographics, knowledge of BVMS, and use of documentation.
  2. Finding test tasks: My supervisor and I scoured over 900 slides of training material for suitable test tasks. We identified eight tasks that were appropriate for different levels of BVMS proficiency.
  3. Acquisition: I acquired test subjects by publishing a call for participants on Bosch's internal social network. As a result, I received ten registrations, from eight different countries. I gained three additional participants through networking.
  4. Test setup: Since my participants were sitting all over the world, I designed a remote user test using Skype. Fig. 2 shows the setup in detail. Apart from reaching a wider range of participants, the remote user test also posed technical advantages:

The only disadvantage was that you can not observe body language remotely. You will miss out on emotions and subtleties that are not apparent from the conversation.

The picture describes the online setup in detail.
Fig. 2 - Technical setup of the remote user test

Testing Phase

Nine of the 13 registered participants took part in the user test. All participants were Bosch employees and already familiar with BVMS.
The tests were following the scheme in fig. 3. First, the participants had to solve the tasks in BVMS using the HTML5 help. Afterward, the moderator asked questions about the help, such as "Were you finding the information you expected?" In the second part of the test, the participants should follow the video and answer questions afterward. Most of the tests would be conducted by the UX intern and me.
We took turns throughout the test, so I switched from being the moderator and to being the observer.

The picture shows how the test was structured and what the participants needed to do.
Fig. 3 - What a test session looked like

Evaluation Phase

I consolidated the survey and the notes to find recurring remarks. After that, I put together a report which I presented to my manager and the BVMS product team.


Finding suitable test tasks

When I was planning the user test, I had been with Bosch BT for about six weeks. Since BVMS is rather complex software, I was not an expert myself - so I turned to the experts. I had asked our training department if they could put together some tasks. They did not have the time but were kind enough to provide me with their slide deck, which my supervisor and I searched for tasks.

Finding suitable participants

Initially, I planned on testing real users on-site, face to face. However, it was not possible to recruit real customers due to a lack of resources to provide compensation and pay for travel expenses. Therefore, we resorted to Bosch employees who could handle BVMS. Fortunately, many participants were in close contact with customers, so their feedback was bundled and passed on to us.


One of my colleagues was focusing a lot on writing down real statements that we could include in the report later on. However, sometimes the discussion moved on so quickly, that they could not capture everything, resulting in incomplete sentences. This made it really hard to evaluate the notes objectively and not interpret something that the participants had not said. I learned that the quality and structure of the transcript are important for a quick and clean evaluation.

Research Insights

The first three questions were focusing on the usage pattern of the BVMS documentation, see fig. 4. While the survey showed, that most participants use the documentation fairly often, they also rely on colleagues to help them. Since we were interviewing Bosch employees, they have access to knowledgeable colleagues. However, the average user does not have this luxury.

The results of the survey show that users rely on the documentation and colleagues for help.
Fig. 4 - The test results show that the participants rely on the documentation and colleagues for help
Screenshot of the HTML5 prototype.
Fig. 5 - UI of the HTML5 help prototype
Source: BVMS on-screen help, Bosch BT

Next, the participants shared their opinion on the HTML5 help.
Which help functionalities do users use to find the desired information? Most users went for the search. They also used the new history feature to go back to entries they had viewed before.
How easily do the users find information in the on-screen help? The search revealed what the users were looking for most of the time. One suggestion was to enable some kind of advanced search to narrow down search results. Another issue was that the users did not know when a query was unsuccessful.
How helpful is the information that they find? The general opinion was that the documentation was focusing too much on describing functionalities, instead of guiding through a use case.

Graphic shows that
				most users prefer self-help videos as part of the docs.
Fig. 6 - Participants think that the how-to videos should be embedded in the help, or at least a link should be provided

Lastly, we collected feedback regarding the how-to video to answer the question: "When and in which contexts are instructional videos useful?"
Most of the participants said they would expect the videos to be either embedded in the on-screen help or be able to click a link that leads them to YouTube or a Bosch website.
Since all the test subjects work at Bosch BT themselves, we received a lot of feedback on how learning videos could improve their everyday work. Apart from documentation, potential applications for how-to videos include:


The user test unveiled that the HTML5 help prototype was a step in the right direction but still had some minor UX issues to be fixed. I provided action items for the HTML5 help and recommendations on implementing them.
Furthermore, the test made clear that video documentation is in demand, as it focuses on actual use cases. Since BVMS has a steep learning curve, the videos can potentially speed up the learning process and make it easier to configure the software. The demand is so high, that a sales manager had started publishing how-to videos himself in Spanish. I believe that the responsibility to provide user-centric documentation lies within the technical writing department, therefore I provided a collection of the most wanted topics for how-to videos.
Lastly, the user test did not only show where the documentation can be improved but also revealed pain points in the actual product. Also, I raised the interest within the technical writing community at Bosch to run their own user test.
I showed that is possible to gather valuable insights with a little support. Since the technical writers provide content for multiple stages in the customer journey, improving the content will increase the overall user experience.

Technical communication has impact throughout the customer journey
Fig. 7 - Technical writers are involved in multiple stages of the customer journey

Lessons learned

Data-driven documentation. I believe collecting data about our users and their usage patterns needs to be a cornerstone of content creation. When technical writers and user researchers work together, the customer experience as a whole will improve.

Nature abhors a vacuum. This principle holds true for content as well. The test revealed that a sales manager in Spain already provided how-to videos to satisfy the demand for simple, use case-oriented working aids. As technical writers, we regularly need to check if the content we create still fits the needs of our audience.

The art of user testing. Conducting a user test in itself was a new but fun experience for me. I found it difficult to watch users struggle and felt the urge to help them. However, their struggle is where the insights happen, so I had to stay silent. Another learning was that notes are the foundation for the evaluation afterward, so making enough time for note-taking is crucial.

Return to home page