Hand collecting and coding versus data-driven methods in technical and professional communication research

Background: Qualitative technical communication research often produces datasets that are too large to manage effectively with hand-coded approaches. Text-mining methods, used carefully, may uncover patterns and provide results for larger datasets that are more easily reproduced and scaled. Research questions: 1. To what degree can hand collection results be replicated by automated data collection? 2. To what degree can hand-coded results be replicated by machine coding? 3. What are the affordances and limitations of each method? Literature review: We introduce the stages of data collection and analysis that researchers typically discuss in the literature, and show how researchers in technical communication and other fields have discussed the affordances and limitations of hand collection and coding versus automated methods throughout each stage. Research methodology: We utilize an existing dataset that was hand-collected and hand-coded. We discuss the collection and coding processes, and demonstrate how they might be replicated with web scraping and machine coding. Results/discussion: We found that web scraping demonstrated an obvious advantage of automated data collection: Speed. Machine coding was able to provide comparable outputs to hand coding for certain types of data; for more nuanced and verbally complex data, machine coding was less useful and less reliable. Conclusions: Our findings highlight the importance of considering the context of a particular project when weighing the affordances and limitations of hand collecting and coding over automated approaches. Ultimately, a mixed-methods approach that relies on a combination of hand coding and automated coding should prove to be the most productive for current and future kinds of technical communication work, in which close attention to the nuances of language is critical, but in which processing large amounts of data would yield significant benefits as well.