AI Video Annotation Quality Lead

About the opportunity

AI Data Quality Lead – Robotics & Video Annotation | Remote About the Role We are looking for an experienced AI Data Quality Lead to own the quality standards for egocentric training data used to develop and evaluate physical-AI and robotics models. You will be responsible for building and maintaining the golden sets, annotation SOPs, rubrics, calibration systems, quality frameworks, and audit processes that ensure high-quality ground-truth data. You will also work closely with engineering and product teams to develop annotation workflows for hand tracking, object identification, and action labeling. This is a ground-floor opportunity to build a 0→1 data quality system at a fast-growing, well-funded startup. You will not simply inherit an existing process—you will help establish the quality standards, workflows, and systems from the ground up. Key Responsibilities Own the end-to-end quality framework for egocentric training data, including: Golden and benchmark datasets Acceptance criteria Frame-level tolerance thresholds Sampling strategies Audit and quality-control loops Author, maintain, and version annotation SOPs, rubrics, and edge-case guidelines as project specifications evolve. Translate ambiguous requirements into precise, actionable instructions for annotators and reviewers. Establish and manage annotator training, certification, and calibration programs. Run calibration sessions and adjudicate disagreements between annotators. Measure and report quality using metrics such as: Inter-Annotator Agreement (IAA) Kappa/alpha Tolerance-based agreement for temporal boundaries Acceptance/rejection rates Rework rates Cost per accepted hour Identify root causes of quality issues and determine whether they result from guideline ambiguity, annotation errors, or workflow issues. Design human-in-the-loop workflows where AI/model-generated labels are reviewed, corrected, and escalated by human annotators. Audit auto-generated labels and improve annotation throughput without compromising quality. Partner with engineering and product teams to develop new annotation workflows for: Hand tracking Object identification Action labeling Define label schemas, annotation-tool requirements, QC dashboards, and pilot-validation processes. Validate annotation workflows through pilot programs before scaling them into production. Work with ML teams to understand how annotation quality impacts model performance. Contribute to tooling and workflow improvements where required. Required Skills & Qualifications 5+ years of hands-on video annotation experience, including at least 1+ year working with egocentric/first-person video for robotics or embodied AI. 3+ years of experience in a lead or QA capacity, including annotator calibration, disagreement adjudication, and ownership of annotation guidelines. Demonstrated experience building 0→1 annotation programs and taking them from pilot through production. Strong command of annotation-quality methodologies, including: IAA frameworks Kappa/alpha Gold-set creation Sampling strategies Tolerance thresholds Acceptance criteria Ability to design quality systems and explain the practical difference between quality targets such as 95% and 99% accuracy. Proven experience authoring, rather than simply following, annotation SOPs and rubrics. Ability to translate ambiguous specifications into clear and actionable annotation guidelines. Hands-on experience with at least 2 professional annotation platforms, such as: CVAT Labelbox Encord V7 Label Studio Equivalent annotation tools Experience configuring label schemas and QA workflows within annotation platforms. Experience auditing model-generated or automatically generated labels. Experience designing human-in-the-loop annotation workflows. Strong written and verbal English communication skills. Strong analytical skills and proficiency with Excel and/or Google Sheets for quality reporting. Strong understanding of ML concepts and how annotation quality affects model performance. Data-driven and process-oriented mindset with strong ownership. Comfortable working in ambiguous, fast-moving, startup environments. Strong ability to work collaboratively with cross-functional and remote teams. Preferred / Bonus Qualifications Background in robotics, mechanical engineering, mechatronics, or computer vision engineering. Experience working at a leading robotics company, AI/robotics startup, or similar project. Experience managing distributed or global annotator workforces or external annotation vendors. Experience with multimodal annotation, including: 3D/depth data Joint pose Grasp outcome classification Experience training or fine-tuning autolabeling models. Experience partnering closely with ML teams responsible for automated labeling. Experience building annotation tooling or working closely with annotation-tooling engineering teams. Experience with physical AI, embodied AI, computer vision, or robotics datasets. Why This Role? The quality bar is the product. The ground-truth data produced through this role will become a benchmark against which physical-AI models are measured. You will have direct ownership of the systems that determine whether that data meets the required quality standards. This is a ground-floor opportunity at a fast-growing, well-funded startup where you will build the quality system from scratch rather than inherit an established process. Role Highlights Role: AI Data Quality Lead – Robotics & Video Annotation Work Type: Remote Engagement: [Contract / Full-Time – specify as applicable] Focus: AI Training Data, Robotics, Embodied AI & Video Annotation Experience: 5+ years relevant experience

Responsibilities

• Own the end-to-end quality framework for egocentric training data, including: Golden and benchmark datasets Acceptance criteria Frame-level tolerance thresholds Sampling strategies Audit and quality-control loops Author, maintain, and version annotation SOPs, rubrics, and edge-case guidelines as project specifications evolve.

Qualifications

• into precise, actionable instructions for annotators and reviewers.
• Establish and manage annotator training, certification, and calibration programs.
• Run calibration sessions and adjudicate disagreements between annotators.
• Measure and report quality using metrics such as: Inter-Annotator Agreement (IAA) Kappa/alpha Tolerance-based agreement for temporal boundaries Acceptance/rejection rates Rework rates Cost per accepted hour Identify root causes of quality issues and determine whether they result from guideline ambiguity, annotation errors, or workflow issues.
• Design human-in-the-loop workflows where AI/model-generated labels are reviewed, corrected, and escalated by human annotators.
• Audit auto-generated labels and improve annotation throughput without compromising quality.
• Partner with engineering and product teams to develop new annotation workflows for: Hand tracking Object identification Action labeling Define label schemas, annotation-tool
• , QC dashboards, and pilot-validation processes.