Data Processing Agent

Data Analysis Sub-Agent (MCP based) for AWS SageMaker Unified Studio

Project Overview

Developed a SubAgent Data Preprocessing that works with Amazon Q to provide the customer with a Data Science Assistant, in SageMaker Unified Studio.

Situation & Context

For a chat-based Data Science Assistant pwered by Amazon Q, I developed a sub-agent which was a MCP Server with tools. The objective of the sub-agent was data analysis and preparation, which is inherently stateful and a core technical challenge was to design the tools in away to provide the user a guided sequential and stateful experience as well as solicit inputs, in a stateless agentic system. The challenge was to design a system that could regulate the behavior of the MCP Client through a stateless server while ensuring all pre-conditions and checks were satisfied for analyzing data and preparing it for ML model training.

Task

Core Development Objectives
  • Design and implement MCP Server (Sub-agent) for data analysis with preparation capabilities.
  • Ensure pre-conditions and checks are satisfied through stateless server design
  • Regulate MCP Client behavior through clever tool return value design
Additional Responsibilities
  • Work with Product team to disambiguate scope and solve issues at different project stages
  • Create automated evaluation framework for MCP server assessment

Action & Implementation

MCP Server Development
  • Stateless Server Architecture: Designed innovative MCP server that regulates MCP Client (Q) behavior through stateless design principles
  • Scientific Novelty: Implemented pre-conditions and checks satisfaction through thorough experimentation and clever tool return value design
  • LLM Guidance System: Crafted tool return values that guide the LLM on which tool to utilize next and how, beyond just functional elements
  • ML Pipeline Integration: Added subsequent ML model training capabilities to the data analysis server
Automated Evaluation Framework
  • Proactive Framework Creation: Independently developed automated evaluation framework for MCP server with Q CLI as MCP Client
  • LLM-as-a-Judge System: Set up sophisticated evaluation system with customized user journeys for multi-dataset evaluation
  • Dynamic Correction Capability: Implemented ability to provide corrective inputs dynamically for unintended trajectories from Q CLI
  • Product Collaboration: Worked closely with Product team to disambiguate scope and solve issues throughout project lifecycle

Results & Impact

  • Innovative Server Design: Successfully implemented stateless MCP server with novel approach to client behavior regulation
  • Enhanced Data Quality Understanding: Enabled users to better understand data quality through guided data preparation processes
  • Robust Evaluation Framework: Created non-trivial automated evaluation system capable of dynamic correction for uncontrolled Q CLI outputs
  • Scalable Solution: Framework supports evaluation across multiple datasets with customized user journey scenarios
  • Enterprise Integration: Successfully integrated with SageMaker Unified Studio for enterprise-scale ML pipeline development

Keywords

  • MCP Protocol
  • Data Analysis Server
  • LLM-as-a-Judge
  • Automated Evaluation Framework

References