ABSTRACT

As Large Language Models (LLMs) become increasingly central to real-world applications, the demand for high-quality, instructioncompliant, and multilingual training data has surged, particularly in tier-2 languages with limited digital representation. In this work, we introduce an AI-assisted annotation framework designed to optimize authoring of training data for multilingual guardrails, specifically PII detection, in Supervised Fine-Tuning (SFT) of LLMs. Targeting 13 locales, mostly underrepresented, we operationalize a suite of AI tools to augment human annotators without replacing them. Our results demonstrate a 40+% reduction in average handling time while improving instruction compliance, semantic diversity, and data quality. The key contribution of this work is that we explore the emerging paradigm of ’LLM-as-a-Judge’, using LLM not only as generative tools but also as evaluators of human-authored training data.