Teach and Test

The adoption of Artificial intelligence (AI) and Machine Learning (ML) in building of application is rapidly increasing and is rendering the conventional testing methods that are used in traditional software development as obsolete. Traditional testing is usually deterministic i.e. the scenarios developed and tested are finite and can be defined in advance. Once a system has been tested for all the test cases, it is guaranteed to work as expected in production. However to test AI/ML systems a different approach is needed as the output of such systems is probabilistic in nature, and they include new paradigms in their development lifecycle -for example, processing unstructured data , managing the variety and veracity of data, choice of correct AI/ ML algorithm, evaluating the accuracy and performance of the learning models, ensuring ethical and unbiased decisioning by the new system along with regulatory and compliance adherence. Correspondingly, new testing and monitoring processes which account for the data-dependent nature of these systems come to the fore.

Teach and Test framework consists of 2 components:

Teach for Training Data Validation and Test for Model Validation. Built using technologies like Java, R and Python the framework helps select the right data, models and algorithms to produce right decisions.

  1. Teach consists of teaching / training the AI solution to detect data bias and make it ready to train machine learning models.
  2. Test involves validation of output, both in test data and production to check bias, variance, and accuracy to evaluate the AI solution.

Key Benefits

  1. Reduces the risk of bias in decisions to minimize impact of negative publicity
  2. AI validation technique reduces algorithm bugs with minimal optimal testing
  3. Paraphrase Generator dictionaries accelerates the creation of test data for Virtual agents
  4. Data scientist translation of model findings into understandable decisions that help business owners

features

Open Source AI /ML Solution

Built using R & Python - open source programming language

Metamorphic Testing Support

Framework detects bias / accuracy issues in image classification models & NLP models

Data Agnostic

Detects and removes gender / race bias from unstructured data as well as removes quality related issues and bias from structured data

Data Generation

Generates neutral data (synthetic data) as well as trains virtual agent with Semantic data generation

Model Validation

Uses Confusion Matrix to validate data model accuracy

Bias Detection

Uses Learning Curve Testing to detect bias and variance from statistical models

ARCHITECTURE

Case Studies

Our clients have been able to identify number of improvement opportunities through implementation of Teach and Test

Improves accuracy of screening resumes, attracts the right talent and improve the diversity for HR Application.

Challenges

  • Implementing AI driven solution to shortlist & recruit right resources.
  • Unintentionally bias is introduced because of some specific words / language resulting in improper / biased candidates.

Our solution

  • Bias is detected showcasing the influence on recruiting decisions.
  • Bias score is calculated for each JD / resume.
  • Debias the JD so that AI engine can be trained with unbiased data.

Contacts

Rohit Patwardhan