Title: A Manufacturing Collaboratory Case Study - Companion Document

Executive Summary:

The National Institute of Standards and Technology (NIST) and the University of Michigan (UM) embarked on a research project to examine the deployment of collaboration technology in a manufacturing setting. The primary research goals were assessment and better understanding of a) the use of collaborative tools in a manufacturing (field) setting and b) the impact of this use on team performance and effectiveness, based on a comparison of pre- and post-intervention measures. An additional goal was to provide a collection of practical recommendations to potential manufacturing users and analysts, who may be considering using these tools in todays growing global economy.

The research setting and approach

In this research, the University of Michigan (UM) worked with an industrial partner, Auto 1¹, to deploy and assess the performance and impact of network-based tools for a geographically-distributed, engineering team, CAR. UM conducted interviews and administered a voluntary, web-based, pre-intervention survey to identify collaboration needs and to determine baseline, collaboration conditions. A suite of collaboration tools was recommended to Auto 1. The CAR team management agreed to the deployment and assessment of a tool for synchronous viewing of engineering drawings and documents, Microsoft (MS) NetMeeting² (NM). In conjunction with the Auto 1 Information Technology (IT) group, UM trained key CAR team personnel upon NetMeeting deployment. After 6 months of use, a second survey was administered. UM used the two surveys to analyze the adoption and impact of network-based, collaborative tools on the teams performance. Using this analysis, previous research, and follow-up questions to Auto 1 management, UM addressed the meaning of the results in terms of the adoption, use, and impact of collaboration tools, techniques for assessment and evaluation of collaboration tool use, and practical recommendations for users.

Tool use and impact

Although the analysis of the pre- and post-intervention surveys showed that a small proportion (1/3, n=34) of the team used NetMeeting, NM users described a positive impact of this tool on their collaborative work. NetMeeting use reduced some of the difficulties indicated in the requirements analysis, such as interruptions in mid-conversation to send email attachments of drawings. With respect to the low adoption rate, results suggested a key feature of NetMeeting, specifically synchronous application sharing, was not as valuable as expected for collaborations that spanned time zones; but, it was more widely used for local collaboration. Another factor was that workarounds already in use within the team were sufficiently successful to inhibit exploration and adoption of alternative collaboration tools, even when these tools had desirable features as identified in the collaboration requirements analysis.

Assessment techniques, challenges and observations

Over the course of one year at Auto 1, UM researchers used several techniques and approaches to assess and evaluate the use and impact of collaborative tools. They made numerous visits to CAR team sites, including European locations to conduct face-to-face interviews and training sessions. In collaboration with the Auto 1 IT group, UM conducted one-on-one training of NM and tracked deployment consequences for six months. Finally, UM developed, administered, and analyzed two surveys.

These efforts were conducted within a dynamic environment that threatened support for this research at Auto 1. There was continual uncertainty with respect to the ultimate success of the CAR team's product. There was also uncertainty regarding the legal relationship between Auto 1 and the University of Michigan concerning the overall research project with respect to intellectual property rights. Additionally, halfway through the research period, the Auto 1 vice president, who had invited and nurtured the research project, left the company. Even though the deployment and evaluation phases of the project were ultimately completed, these experiences reinforced the known difficulties in conducting this type of research in a business environment [6].

Observations from these efforts included:

• Conducting this type of research in a corporate environment, within actual engineering teams, required enormous cooperation and compromise to accommodate competing priorities.
• In the Auto 1 corporate environment, where IT resources were strained, the research team expended a great deal of effort attempting to bring a critical mass of users to a minimum level of competency with the target collaborative tools. With additional support from the Auto 1's IT group, more effort could have been expended directly on observation of the tools in use or enhancing practices with the tools.
• The use of surveys had the advantage of gathering data broadly and at relatively low cost. However, low response rates reduced the capacity to draw useful conclusions.
• Overcoming inertia in organizational settings is very difficult. For instance, if a solution to a collaboration issue already existed, there was little incentive for participants to change to another solution, even with anticipated improvements.
• A high-level champion in the field is vital for success of the implementation [6] and of the research. The champion at Auto 1 played an essential role in negotiation across the two divisions from which CAR team members originated. Momentum on this project slowed considerably when that high-level champion left Auto 1.
• UM's status as outsiders allowed them to probe the CAR team members to reveal practices and procedures that the engineers themselves, and Auto 1 personnel, did not have the time or inclination to explore. However, as outsiders, researchers confronted a very steep learning curve to understand the content of the engineering work. The effort expended on mastering the elementary aspects of the engineers tasks could have been allocated to a more detailed examination of performance of the tasks and the relationship between these tasks and collaborative tools. In this way, inclusion of a veteran automotive engineer, specifically one from Auto 1, on the research team might have accelerated understanding of what the CAR team was trying to do. Yet, in the time- and resource-starved environment where CAR team members worked, it was unlikely that CAR team management or Auto 1 would have consented to an engineer/designer acting in a formal liaison.
• It is important to anticipate competing demands in this type of research. Specifically, adoption of novel tools with accompanying novel practices is not thing that unfolds on a time scale consistent with most projects. Therefore, searchers will confront the need to perform some level of "pump priming," that is, there has to be some level of collaboration-tool adoption and use to produce behavior and outcomes that can be used to evaluate the impact of collaborative tools. Realistic determination of resources required to jump-start use, particularly where none currently exists, is difficult.

Practical recommendations for managers and engineers include:

• Selected collaborative tools must meet a specific need to merit the effort of deployment, adoption, and subsequent use. For example, the requirements gathering effort conducted early in this project, successfully highlighted difficulties experienced by the CAR team members when attempting to do cross-site work. At least for some of the CAR engineers, as indicated by adoption and use rates, the set of introduced collaborative tools met the identified needs. Yet, for the collaborative tool still imposed too great a burden to learn and master, relative to the perceived benefits.
• Successful deployment of collaborative tools requires a coordinated approach that involves the IT support organization at an early stage.
• Targeted changes in work practices, such as the adoption and use of collaborative tools, need adequate support to ensure success. In this study of NM adoption and impact, engineers were asked to master both the tool itself and the choreography of working with a distant colleague via the tool. The significant investment of one-on-one training surrounding the introduction of NetMeeting was successful for individual users. However, it was not conducted on a broad enough scale to create a critical mass of use for the team in the timeframe of this study.

Concluding remarks

The primary goals of this research were an assessment and better understanding of the adoption, use, impact, and evaluation of collaborative tools in a manufacturing setting. Because the collaborative tool intervention was specifically targeted to needs identified in the requirements gathering phase, a primary expectation was to find a positive impact of tool use. While there was some indication of improved team performance with use of the tools, the overall adoption did not meet expectations. Although results were mixed, assessment of collaborative tool intervention is often difficult, even at the very coarse granularity of success and failure [1]. Therefore, one important outcome of this research is the well-documented case study [16], so that over time results can be interpreted more fully, and future research and collaborative tool deployments can draw from it.

Keywords: NetMeeting; CSCW, collaboration technologies; collaborative tool adoption

² Any commercial product identified in this document is for the purpose of describing a collaborative software environment. This identification does not imply any recommendation or endorsement by NIST.

References (a complete list of references is included in the full paper)

[1] Blythin, S., J. Hughes, S. Kristoffersen, T. Rodden, and M. Rouncefield. Recognising Success and Failure: Evaluating Groupware in a Commercial Context. Proceedings of GROUP97 (Phoenix, AZ, November 1997), ACM Press, p. 39-46.

[6] Ehrlich, S. Strategies for Encouraging Successful Adoption of Office Communication Systems. ACM Transactions on Information System, 5, 4, 1987, p. 340 357.

[16] Wierba, E. and T. Finholt. A Manufacturing Collaboratory Case Study. National Institute of Standards and Technology. GCR 01-811; 2001 May. (Available at: http://www.nist.gov/msid/collaboratory/doc/gcr-811.pdf)