with User Tests at Bosch BT
About the Project
During my internship at Bosch Building Technologies (BT), I conducted
a user test to find weak spots in the online help. The company was
publishing their software documentation as a PDF and a deprecated CHM
format, which should be replaced with an HTML5-based help. Since I was
also introducing how-to videos at the time I decided to include them
in the test. The project took place over the course of 12 weeks from
start to finish. I led the project and my supervisor and a UX intern
Bosch BT sells a range of surveillance cameras and software to manage
them, called BVMS (Bosch Video
Management System). The BVMS
on-screen documentation was published in the CHM format (remember
these purple books?). The problem with CHM was that it looked
outdated, but what's worse, it also had some security issues. Time to
switch to a more modern approach: an HTML5-based help.
Prior to the user test, I spotted UX and content-related issues in the
HTML5 prototype. Together with our system administrator, we improved
the HTML5 prototype until we felt it was good enough for the user
Since I would be running a user test for the HTML5 prototype, I also included the
newly introduced how-to video.
The test aims to answer these questions:
How do our users gather information when something in BVMS is unclear
What kind of documentation do they mostly use?
How often do they use the documentation?
Which help functionalities do users use to find the desired
How easily do the users find information in the on-screen help?
How helpful is the information that they find?
When and in which contexts are instructional videos useful?
The process consisted of three major phases: Planning, testing, and
Creating a pre-test survey: Together with the UX-Intern supported, I
created a questionnaire to collect data on demographics, knowledge of
BVMS, and use of documentation.
Finding test tasks: My supervisor and I scoured over 900 slides of
training material for suitable test tasks. We identified eight tasks
that were appropriate for different levels of BVMS proficiency.
Acquisition: I acquired test subjects by publishing a call for
participants on Bosch's internal social network. As a result, I
received ten registrations, from eight different countries. I gained
three additional participants through networking.
Test setup: Since my participants were sitting all over the world, I
designed a remote user test using Skype. Fig. 2 shows the setup in
detail. Apart from reaching a wider range of participants, the remote
user test also posed technical advantages:
The user does not have to install the system and help separately.
In addition, many workstations running BVMS do not have a connection
to the Internet.
The test administrator can prepare the software so that the test
task is processed immediately.
The session can be recorded so that interesting passages can be
viewed again later.
The only disadvantage was that you can not observe body language
remotely. You will miss out on emotions and subtleties that are not
apparent from the conversation.
Nine of the 13 registered participants took part in the user test. All
participants were Bosch employees and already familiar with BVMS.
The tests were following the scheme in fig. 3. First, the participants
had to solve the tasks in BVMS using the HTML5 help. Afterward, the
moderator asked questions about the help, such as "Were you finding the
information you expected?" In the second part of the test, the
participants should follow the video and answer questions afterward.
Most of the tests would be conducted by the UX intern and me.
We took turns throughout the test, so I switched from being the moderator and to being
I consolidated the survey and the notes to find recurring remarks. After
that, I put together a report which I presented to my manager and the
BVMS product team.
Finding suitable test tasks
When I was planning the user test, I had been with Bosch BT for about
six weeks. Since BVMS is rather complex software, I was not an expert
myself - so I turned to the experts. I had asked our training department
if they could put together some tasks. They did not have the time but
were kind enough to provide me with their slide deck, which my
supervisor and I searched for tasks.
Finding suitable participants
Initially, I planned on testing real users on-site, face to face.
However, it was not possible to recruit real customers due to a lack of
resources to provide compensation and pay for travel expenses.
Therefore, we resorted to Bosch employees who could handle BVMS.
Fortunately, many participants were in close contact with customers, so
their feedback was bundled and passed on to us.
One of my colleagues was focusing a lot on writing down real statements
that we could include in the report later on. However, sometimes the
discussion moved on so quickly, that they could not capture everything,
resulting in incomplete sentences. This made it really hard to evaluate
the notes objectively and not interpret something that the participants
had not said. I learned that the quality and structure of the transcript
are important for a quick and clean evaluation.
The first three questions were focusing on the usage pattern of the BVMS
documentation, see fig. 4. While the survey showed, that most
participants use the documentation fairly often, they also rely on
colleagues to help them. Since we were interviewing Bosch employees,
they have access to knowledgeable colleagues. However, the average user
does not have this luxury.
Next, the participants shared their opinion on the HTML5 help.
Which help functionalities do users use to find the desired information?
Most users went for the search. They also used the new history feature
to go back to entries they had viewed before.
How easily do the users find information in the on-screen help? The
search revealed what the users were looking for most of the time. One
suggestion was to enable some kind of advanced search to narrow down
search results. Another issue was that the users did not know when a
query was unsuccessful.
How helpful is the information that they find? The general opinion was
that the documentation was focusing too much on describing
functionalities, instead of guiding through a use case.
Lastly, we collected feedback regarding the how-to video to answer the
question: "When and in which contexts are instructional videos useful?"
Most of the participants said they would expect the videos to be either
embedded in the on-screen help or be able to click a link that leads
them to YouTube or a Bosch website.
Since all the test subjects work at Bosch BT themselves, we received a
lot of feedback on how learning videos could improve their everyday
work. Apart from documentation, potential applications for how-to videos
Part of a training
Preparation for on-site visits
The user test unveiled that the HTML5 help prototype was a step in the
right direction but still had some minor UX issues to be fixed. I
provided action items for the HTML5 help and recommendations on
Furthermore, the test made clear that
video documentation is in demand, as it focuses on
actual use cases. Since BVMS has a steep learning curve, the videos can
potentially speed up the learning process and make it easier to
configure the software. The demand is so high, that a sales manager had
started publishing how-to videos himself in Spanish. I believe that the
responsibility to provide user-centric documentation lies within the
technical writing department, therefore I provided a collection of the
most wanted topics for how-to videos.
Lastly, the user test did not only show where the documentation can be
improved but also revealed pain points in the actual product. Also, I
raised the interest within the technical writing community at Bosch to
run their own user test.
I showed that is possible to gather valuable insights with a little
support. Since the technical writers provide content for multiple stages
in the customer journey, improving the content will increase the overall
Data-driven documentation. I believe collecting data
about our users and their usage patterns needs to be a cornerstone of
content creation. When technical writers and user researchers work
together, the customer experience as a whole will improve.
Nature abhors a vacuum. This principle holds true for
content as well. The test revealed that a sales manager in Spain already
provided how-to videos to satisfy the demand for simple, use
case-oriented working aids. As technical writers, we regularly need to
check if the content we create still fits the needs of our audience.
The art of user testing. Conducting a user test in
itself was a new but fun experience for me. I found it difficult to
watch users struggle and felt the urge to help them. However, their
struggle is where the insights happen, so I had to stay silent. Another
learning was that notes are the foundation for the evaluation afterward,
so making enough time for note-taking is crucial.