Models
Here is how to map your allowed AWS Bedrock models (us-east-1) to the multi-agent architecture for the **"Beyond Text Extraction"** hackathon.
Each model in your allowed list has a specific strength. Splitting them across your agents gives you optimal performance, latency, and reasoning depth.
### 1. Model Assignment Matrix
| Agent / Layer | Best Model | AWS Model ID (boto3) | Why It’s the Best Choice |
|---|---|---|---|
| **Agent 1: Classifier** | **Amazon Nova Lite** | amazon.nova-lite-v1:0 | Ultra-fast and low-cost. Perfect for quickly identifying document types (e.g., Invoice vs. Form) and checking image legibility. |
| **Agent 2: Multimodal Extractor** | **Claude 4.5 Haiku** | us.anthropic.claude-haiku-4-5-20251001-v1:0 | Outstanding visual/spatial reasoning. Best in this list for reading messy handwriting, checkboxes, complex tables, and returning strict bounding box JSON. |
| **Agent 3: Validator** | **Cohere Command R+** | cohere.command-r-plus-v1:0 | Built specifically for complex reasoning, tool usage, and structured logic. Ideal for verifying math (subtotal + tax = total), checking confidence scores, and formatting downstream payloads. |
| **Cross-Referencing / RAG** | **Cohere Embed English v3** | cohere.embed-english-v3 | High-accuracy embedding model. Use this if your Validator needs to search external rulebooks or reference databases. |
| **Basic Text Search** | **Amazon Titan Embeddings** | amazon.titan-embed-text-v1 | Lightweight text embedding fallback for simple vector matching. |
### 2. Operational Breakdown
#### **Agent 1: The Classifier**
* **Primary Choice:** amazon.nova-lite-v1:0
* **Operation:** Receives the raw uploaded document bytes.
* **Task:** Answers lightweight metadata questions: *Is this legible? Is it skewed? Is it an invoice or a tax form?*
* **Why:** You don't need heavy reasoning here. Nova Lite responds in milliseconds, keeping your pipeline snappy.
#### **Agent 2: The Multimodal Extractor (Visual Vision)**
* **Primary Choice:** us.anthropic.claude-haiku-4-5-20251001-v1:0
* **Operation:** Takes the image file and the target JSON schema.
* **Task:** Reads spatial layout, detects checked vs. unchecked boxes, extracts text values, and returns coordinates (box_2d).
* **Why:** Claude models outperform almost all other models at structured multimodal layout extraction and sticking strictly to bounding box coordinate schemas.
#### **Agent 3: The Validator & Action Generator**
* **Primary Choice:** cohere.command-r-plus-v1:0
* **Operation:** Accepts the raw JSON output from the Extractor (pure text, no image required).
* **Task:** Evaluates business logic, flags fields where confidence_score < 0.75, verifies arithmetic formulas, and formats the downstream action message (e.g., Jira ticket payload or DB record).
* **Why:** Command R+ specializes in multi-step reasoning, logical cross-referencing, and deterministic rule validation.
#### **Data Grounding & Cross-Referencing (Optional Feature)**
* **Primary Choice:** cohere.embed-english-v3
* **Operation:** Converts document text chunks or database vendor names into vector embeddings.
* **Task:** Allows the Validator to verify if an extracted vendor name matches an existing approved vendor list in your system.
### 3. How to Call Them in Python (boto3)
Here is how you configure your agent_api/agents/ calls in Python using these exact model IDs in us-east-1:
```python
import boto3
import json
# Bedrock Runtime Client in us-east-1
bedrock_runtime = boto3.client("bedrock-runtime", region_name="us-east-1")
# 1. CLASSIFIER (Amazon Nova Lite)
def run_classifier(prompt_text):
response = bedrock_runtime.invoke_model(
modelId="amazon.nova-lite-v1:0",
body=json.dumps({"inferenceConfig": {"max_new_tokens": 500}, "messages": [{"role": "user", "content": [{"text": prompt_text}]}]})
)
return json.loads(response['body'].read())
# 2. MULTIMODAL EXTRACTOR (Claude 4.5 Haiku)
def run_extractor(prompt_text, image_bytes_base64):
payload = {
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 2000,
"messages": [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": image_bytes_base64
}
},
{"type": "text", "text": prompt_text}
]
}
]
}
response = bedrock_runtime.invoke_model(
modelId="us.anthropic.claude-haiku-4-5-20251001-v1:0",
body=json.dumps(payload)
)
return json.loads(response['body'].read())
# 3. VALIDATOR (Cohere Command R+)
def run_validator(extracted_json_prompt):
payload = {
"message": extracted_json_prompt,
"max_tokens": 1000,
"temperature": 0.1
}
response = bedrock_runtime.invoke_model(
modelId="cohere.command-r-plus-v1:0",
body=json.dumps(payload)
)
return json.loads(response['body'].read())
```
Now, implement the business logic for the three AI agents in our application using the AWS Bedrock `boto3` client.
All Bedrock calls must be made in the `us-east-1` region using the modern `bedrock_client.converse()` method for payload standardization.
Target Files to Update:
1. `agent_api/agents/classifier.py`
2. `agent_api/agents/multimodal_extractor.py`
3. `agent_api/agents/validator.py`
## Instructions per File:
**1. classifier.py (Amazon Nova Lite)**
- Initialize: `boto3.client("bedrock-runtime", region_name="us-east-1")`
- Model ID: `amazon.nova-lite-v1:0`
- Logic: Create an async function `classify_document(prompt_text: str)` that uses the Bedrock Converse API to evaluate the text and determine the document type and legibility. Return the parsed JSON matching our `ClassificationResult` schema.
**2. multimodal_extractor.py (Claude 4.5 Haiku)**
- Initialize: `boto3.client("bedrock-runtime", region_name="us-east-1")`
- Model ID: `us.anthropic.claude-haiku-4-5-20251001-v1:0`
- Logic: Create an async function `extract_document_data(prompt_text: str, image_bytes: bytes)` using the Converse API.
- You MUST pass the `image_bytes` in the standard Bedrock image block format.
- Instruct the model in the system prompt to return absolute pixel bounding boxes `[x1, y1, x2, y2]`. Return the parsed JSON matching the `InvoiceData` schema.
**3. validator.py (Cohere Command R+)**
- Initialize: `boto3.client("bedrock-runtime", region_name="us-east-1")`
- Model ID: `cohere.command-r-plus-v1:0`
- Logic: Create an async function `validate_extraction(extracted_json_string: str)`.
- Pass the extracted JSON text to the Converse API. Command R+ must verify the math (subtotal + tax = total), check if any confidence scores are below 0.75, and return a parsed JSON object matching our `ValidationResult` schema.
Ensure proper try/catch blocks are implemented for `botocore.exceptions.ClientError`. Imp
lement these three files now.
aws bedrock list-foundation-models --region us-east-1 --query "modelSummaries[*].modelId"
