Automated AI Tester

Job Title: Automated AI TesterLocation: DubaiIndustry: Payment Solutions / BFSI / Retail / HealthcareEmployment Type: Full TimeAbout KayanaKayana is one of the fastest-growing companies, empowering small and medium-sized businesses with cutting-edge technology to compete with larger enterprises in the ordering and payments space. As a disruptive force in the industry, we provide everything from essential point-of-sale systems to advanced solutions like self-service kiosks, enabling brands to scale and succeed.Our mission is simple: to help businesses thrive through innovative technology. With operations across the USA, UK, Europe, Australia, and the Middle East, Kayana is proud to foster an inclusive and dynamic culture. Every employee is supported with a personalized development plan, offering clear and achievable career paths for long-term growth.Important Work Arrangement NoteThis role requires full-time, on-site work from Monday to Saturday (6 days a week). Candidates should be comfortable with a fully office-based setup, as remote or hybrid work is not available for this position.Role OverviewWe are looking for an experienced Automated AI QA Tester to ensure the reliability, accuracy, safety, and performance of AI-driven ordering and payment workflows at Kayana. The ideal candidate will have 4+ years of experience in QA automation and software testing and be proficient in leveraging AI-powered testing tools and methodologies to perform end-to-end (E2E) testing across conversational AI, ordering, checkout, payment, and integration workflowsKey Responsibilities: Build, maintain, and scale automated test frameworks for AI-powered ordering and payment systems.Design conversation-based test scenarios for AI ordering assistants (menu browsing, cart building, checkout, cancellations)Develop AI-driven and automated E2E test suites that simulate real-world customer journeys.Validate payment flows (UPI, cards, wallets, failures, refunds, edge cases)Create synthetic user simulations and automated load scenarios to stress-test AI systems at scale.Evaluate AI responses for accuracy, consistency, and task completionPerform regression testing for AI model updates, prompt changes, and product releases.Identify issues in AI reasoning, API integrations, and transaction flowsIntegrate automated tests into CI/CD pipelinesCollaborate closely with engineering, product, and AI teams to improve product quality and reliability.Leverage AI-assisted testing tools and frameworks to increase test coverage, automate test generation, and improve testing efficiency.Required Skills:4+ years of experience in QA automation, software testing, or test engineering.Strong experience designing and implementing automated E2E testing frameworks.Demonstrated experience using AI tools and AI-assisted testing methodologies for test creation, execution, analysis, and validation.Strong understanding of AI/LLM testing concepts, including Prompt testing, Hallucination detection, Response evaluation, Groundedness validation, AI safety testing &Evaluation metricsExperience with automation tools such as: Playwright, Selenium, Cypress & PostmanStrong programming skills in Python and/or JavaScript.Experience with API testing and service integration validation.Familiarity with payment systems, fintech workflows, or transactional platforms.Ability to design structured test cases for conversational AI systems, chatbots, and AI agents.Experience validating multi-step user journeys across web, mobile, and backend systems.Strong debugging, root-cause analysis, and problem-solving skills.Nice to Have : Experience testing LLM-powered applications, AI agents, or autonomous workflows.Experience with Retrieval-Augmented Generation (RAG) systems and vector databases.Familiarity with AI evaluation frameworks such as: OpenAI Evals, DeepEval, LangSmith, Ragas, Similar evaluation platformsExperience in food ordering, restaurant technology, POS, e-commerce, or fintech products.Experience with CI/CD platforms such as: GitHub Actions, Jenkins, GitLab CI/CDExperience with performance, load, and reliability testing of AI-powered applications. Soft Skills : Strong analytical and investigative mindset.Exceptional attention to detail and edge-case discovery.Ability to work cross-functionally with engineering, product, and AI teams.Excellent written and verbal communication skills.Comfortable working in fast-paced startup environments with rapidly evolving products and requirements.Proactive ownership mindset with a focus on quality, reliability, and customer experience.Why Join Us?At Kayana, you’ll be part of a fast-growing global company where your contributions will make a direct impact.We offer:A clear path for career growth.The opportunity to set up and shape the compliance framework for a growing international business.A supportive and collaborative environment where your success is celebrated.Starting salary: 8K AED per month