Can we Leverage Language Models to Efficiently Code Complex Task Data for Army Job Analysis? Streamlining Army Qualitative Research with Automated Qualitative Assistants (AQA)

Abstract

Individual critical task lists (ICTLs; published in the Central Army Registry [CAR], 2023) are comprised sets of mission critical tasks which each Soldier must perform to maintain competency throughout the full range of Army operations. ICTLs are often leveraged for other manpower operational purposes, such as Soldiers' career development and pathing needs, as well as crafting job descriptions and requisitions more attractive to potential applicants (ATMTF, 2022; CASCOM, 2023). Special branch ICTLs like AMEDD (Branch #60-70s) often cover a complex, wide range of requirements, leading to challenges in obtaining complete and accurate evaluations of proficiency (Hertz et al., 2020). Therefore, identifying methods to efficiently summarize and code these task lists is an important area of research.

Traditionally, approaching this type of data would involve qualitative research, such as methods in which subject matter experts (SMEs) review, analyze, and code the information—and afterwards, collectively discuss (dis)agreements and reach synthesis. However, qualitative methods remain underused in part due to the time- and labor-intensive costs of coding and annotating these lengthy data sources.

Despite these challenges, we suggest leveraging the several language models (word vectors and semantic similarity) as an automated qualitative assistant (AQA) to facilitate the coding of lengthy, complex text data to advance traditional computer-assisted qualitative data analysis (CAQDA; Richards, 1999; Devlin et al., 2019; Joshi et al., 2019). Previously validated criterion-driven competency frameworks (The Great Eight; Bartram, 2005) can be leveraged to categorize tasks into broader competency domains. Such an approach holds promise in replacing human-facilitated coding procedures with transformer-based label prediction. As a result, the proposed approach may help ease the labor-intensive and time-consuming aspects of qualitative research and circumvent some issues that may arise with human coders (ensuring inter-rater reliability and training requirements).

The ability to summarize and label ICTL related data would have direct implications for Army special branches and functional areas (FAs), enabling commanders to quickly evaluate and compare ICTLs for manpower and personnel management. Overall, this research holds practical implications for establishing a calibrated force posture and facilitating multi-domain formations through the rapid identification of job requirements and requisite Multi-Domain Operations (MDO) capabilities needed for victory.

Keywords: Automated Qualitative Coding; Code Complex; Language Models; LLMs; Army Job Analysis; Human Resources; ATAF; KSBs