Our client is seeking a highly analytical and
technically strong Product Manager – Data & ML Evaluation to lead
the strategy and execution of product data, experimentation, and machine
learning evaluation.
In this role, you will act as the product owner for
data and evaluation frameworks, ensuring that AI-driven products are
measurable, reliable, and continuously improving. You will work
cross-functionally with Product, Engineering, Data Science, and business
stakeholders to deliver high-impact, data-driven solutions.
Key Responsibilities:
·Product Data & Analytics
ÃOwn and manage the end-to-end product
data ecosystem, including telemetry, tracking plans, event schemas, data
quality, and warehousing
ÃDefine and implement a product metrics
framework (north star metrics, leading indicators, input metrics)
ÃEnable stakeholders with accessible
dashboards and actionable insights across products and platforms
·ML Evaluation & Experimentation
ÃDesign and own machine learning
evaluation frameworks, including test set curation and KPI definition
(quality, safety, bias, latency, cost)
ÃRun and optimize offline and online
evaluations for AI models and features
ÃBuild scalable and reproducible evaluation
pipelines integrated with CI/CD workflows
ÃLead A/B testing and experimentation
strategies, including designing, analyzing, and scaling testing frameworks
·Strategy & Governance
ÃDrive data-informed pricing strategies by analyzing usage patterns, feature value, and cost structures
ÃEstablish and enforce data governance
standards, including event naming, privacy, PII handling, and data
retention policies
ÃCollaborate with Security, Legal, and
business teams to ensure compliance and best practices.
Requirements·5–8+ years of experience in Product Management with a strong background in
data, analytics, or engineering
·Proven
experience owning product analytics, instrumentation, and experimentation
frameworks
·Strong
proficiency in SQL and data analysis, with the ability to validate and
interpret experiments independently
·Hands-on
experience working with machine learning products, including model
evaluation and monitoring
·Strong
understanding of A/B testing methodologies and statistical concepts (e.g., power, MDE, guardrail metrics)
·Excellent
stakeholder management and communication skills across technical and business
teams
·Ability to operate effectively in fast-paced,
cross-functional environments.
Preferred Qualifications
·Experience with product analytics tools
(e.g., Amplitude or similar) and experiment tracking platforms (e.g., MLflow)
·Exposure
to LLM / RAG evaluation frameworks, safety metrics, and AI model testing
methodologies
·Familiarity
with data modeling tools (e.g., dbt), feature stores, and observability
platforms
·Experience
working in regulated or privacy-sensitive environments
·Comfort leveraging AI tools to prototype
analyses, dashboards, or lightweight data utilities
Work Location
·Abu Dhabi
To view other
vacancies we have, please check our website (www.blackpearlconsult.com) and follow us on our
social media accounts - LinkedIn / Instagram
Follow the Black
Pearl channel on WhatsApp.
Disclaimer: Black
Pearl will never ask for money or any form of charge our candidates just to
process or consider their application for any of our available vacancies. If
you happen to receive such a request from any members of our staff or other
individuals claiming to be part of Black Pearl, please do call our office at
+9712 622 55 03 or drop us a message on our website - www.blackpearlconsult.com.