The way to Construct a Totally Automated Information Drift Detection Pipeline | by Khuyen Tran | Aug, 2023

The workflow includes the next duties:

1. Fetch reference information from the Postgres database.
2. Get the present manufacturing information from the net.
3. Detect information drift by evaluating the reference and present information.
4. Append the present information to the present Postgres database.
5. When there’s information drift, the next actions are taken:

• Ship a Slack message to alert the information staff.
• Retrain the mannequin to replace its efficiency.
• Push the up to date mannequin to S3 for storage.

This workflow is scheduled to run at particular occasions, similar to 11:00 AM each Monday.

General, the workflow consists of two kinds of duties: information science and information engineering duties.

Information science duties, represented by pink packing containers, are carried out by information scientists and contain information drift…