An AI product undergoes evaluation at several key phases of its lifecycle, including (1) during the pre-study phase, (2) during development, (3) at the point of delivery, and (4) after a period of use. Early problem detection is crucial for cost-effectiveness. The product evaluation encompasses the following components, with the results documented in an Evaluation Report:
Training Data: The evaluation assesses the availability and quality of training data, including data format correctness, the removal of outliers, handling missing data through imputation, ensuring data is in the correct physical or monetary units, and more.
Query Data: It ensures that query data is curated in a manner consistent with the training data. For online systems, it’s critical to verify the correctness of data queues and signal time stamps.
Data Storage: Typically, data is stored in a database accessible by the AI system, but in some cases, data may be read from XML, Excel, or text files. Adequate database visualization, update, and backup tools should be in place.
Training and Inference Performance: This involves evaluating both training and inference performance, with attention to variability in training time intervals, which should stay within acceptable limits.
User Interfaces: It’s essential that user interfaces are tailored to meet user expectations. For example, a medical doctor may require a different type of interface compared to a computer scientist.
Infrastructure and Communications: The evaluation verifies the system’s resilience to communication failures and power supply disruptions.
Algorithm Behavior: This involves ensuring that error messages are clear and actionable, checking for timing variations, and understanding dependencies within the system.
Infrastructure Security and Physical/Thermal Stability: The evaluation includes a comprehensive assessment of infrastructure security, including data protection, and ensuring that the AI system remains stable under varying physical and thermal conditions.
A well-structured evaluation process ensures that the AI product not only functions effectively but also aligns with user needs and remains reliable and secure throughout its lifecycle.