Skip to content

Lab 3: Research & Data Analysis Agent - Competitive Intelligence System

Lab Overview

Duration: 3-4 hours Difficulty: Intermediate-Advanced Prerequisites: Labs 1-2 completed, basic understanding of business research

In this lab, you'll build an intelligent research agent that automatically monitors your competitive landscape, analyzes market trends, and delivers actionable business insights on a regular schedule.

Learning Objectives

By the end of this lab, you will: - ✅ Design and implement automated data collection systems - ✅ Create intelligent analysis and pattern recognition workflows - ✅ Build comprehensive business intelligence reporting - ✅ Establish continuous monitoring and alert systems - ✅ Generate actionable insights for strategic decision-making

The Business Intelligence Challenge

Current Research Pain Points: - 5-10 hours weekly spent manually researching competitors - Inconsistent monitoring leads to missed opportunities - Information scattered across multiple sources - Delayed awareness of market changes and threats - Difficulty synthesizing insights from raw data

What You'll Build: A comprehensive competitive intelligence system that: - Monitors competitor websites, news, and social media 24/7 - Analyzes pricing changes and product announcements - Tracks industry trends and market movements - Delivers weekly intelligence briefings with actionable insights - Provides real-time alerts for significant developments

Part 1: Intelligence Requirements Planning (45 minutes)

Step 1: Define Your Competitive Landscape

Primary Competitors to Monitor: 

Competitor 1:
  Company Name: _________________________
  Website: ______________________________
  Key Products/Services: ________________
  Market Position: ______________________
  Why Important: ________________________

Competitor 2:
  Company Name: _________________________
  Website: ______________________________
  Key Products/Services: ________________
  Market Position: ______________________
  Why Important: ________________________

Competitor 3:
  Company Name: _________________________
  Website: ______________________________
  Key Products/Services: ________________
  Market Position: ______________________
  Why Important: ________________________

Step 2: Intelligence Categories and Sources

What You Need to Know: 

Product Intelligence:
  - New product launches and features
  - Pricing changes and promotions
  - Product discontinuations
  - Customer reviews and feedback

Market Strategy:
  - Marketing campaigns and messaging
  - Partnership announcements
  - Market expansion activities
  - Customer acquisition strategies

Business Operations:
  - Funding rounds and investments
  - Leadership changes
  - Hiring patterns and growth indicators
  - Merger and acquisition activity

Financial Performance:
  - Revenue and growth announcements
  - Public financial filings
  - Stock price movements (if public)
  - Analyst reports and ratings

Information Sources to Monitor: 

Direct Company Sources:
  - Company websites and blogs
  - Press release pages
  - Social media accounts (LinkedIn, Twitter)
  - Product documentation and pricing pages
  - Job posting sites (LinkedIn, Indeed)

Third-Party Sources:
  - Industry publications and news sites
  - Analyst reports and market research
  - Customer review platforms
  - Patent databases
  - Conference presentations and videos

Financial Sources:
  - SEC filings (for public companies)
  - Financial news outlets
  - Investor relations pages
  - Stock market data feeds

Step 3: Intelligence Delivery Requirements

Reporting Schedule: 

Real-Time Alerts (Immediate):
  - Major product announcements
  - Pricing changes >10%
  - Leadership departures
  - Acquisition announcements
  - Crisis or negative news

Daily Monitoring (Morning Briefing):
  - New job postings indicating growth areas
  - Social media activity and engagement
  - Customer review sentiment changes
  - Stock price movements (if applicable)

Weekly Intelligence Report:
  - Comprehensive competitive analysis
  - Market trend identification
  - Strategic recommendation development
  - Opportunity and threat assessment

Part 2: Research Agent Architecture (60 minutes)

Step 1: Master Intelligence Coordinator

Agent Configuration: 

Name: MarketScope Intelligence
Role: Competitive Intelligence Director
Department: Strategic Planning
Reporting To: Executive Team

Core Instructions: 

## Your Mission
Provide comprehensive, accurate, and actionable competitive
intelligence that enables superior strategic decision-making
and market positioning.

## Intelligence Collection Framework

### Competitor Monitoring Priorities
1. **Product & Pricing Intelligence (High Priority)**
   - Monitor competitor pricing pages daily
   - Track new product/feature announcements
   - Analyze customer reviews for satisfaction trends
   - Identify product gaps and opportunities

2. **Market Strategy Analysis (Medium Priority)**
   - Track marketing campaigns and messaging changes
   - Monitor partnership and integration announcements
   - Analyze social media engagement and content strategy
   - Assess market expansion activities

3. **Business Intelligence (Medium Priority)**
   - Monitor hiring patterns for strategic insight
   - Track funding and investment announcements
   - Analyze leadership changes and organizational shifts
   - Assess financial performance indicators

4. **Industry Trends (Ongoing)**
   - Monitor industry publications and reports
   - Track regulatory and policy changes
   - Analyze technology trends affecting the market
   - Identify emerging competitive threats

## Analysis Standards

### Significance Assessment
Rate each piece of intelligence on impact:
- **Critical (9-10)**: Immediate strategic response required
- **Important (7-8)**: Should influence planning within 30 days
- **Interesting (5-6)**: Good to know, monitor for trends
- **Minor (1-4)**: Archive for reference

### Source Credibility Scoring
Evaluate information sources:
- **Tier 1 (Official)**: Company press releases, SEC filings, official announcements
- **Tier 2 (Reliable)**: Established industry publications, verified journalist reports
- **Tier 3 (Useful)**: Social media, review sites, employee discussions
- **Tier 4 (Rumor)**: Unverified sources, speculation, anonymous tips

## Intelligence Synthesis

### Weekly Report Structure
1. **Executive Summary** (3 key insights maximum)
   - Most significant competitive developments
   - Market trends requiring attention
   - Immediate opportunities or threats

2. **Competitive Movements** (By company)
   - What each competitor did this week
   - Analysis of strategic implications
   - Recommended competitive responses

3. **Market Intelligence**
   - Industry trends and developments
   - Regulatory or technology changes
   - Customer behavior insights

4. **Strategic Recommendations**
   - Immediate actions (next 30 days)
   - Medium-term considerations (next quarter)
   - Long-term strategic implications

## Quality Standards
- Verify all significant claims with multiple sources
- Distinguish between facts and analysis/opinion
- Provide confidence levels for key assessments
- Include links to source materials for verification
- Flag information requiring further investigation

Step 2: Specialized Collection Agents

Web Monitoring Agent: 

Name: WebWatcher Pro
Role: Website Change Detection Specialist
Responsibilities:
  - Monitor competitor websites for changes
  - Track pricing page updates
  - Capture new content and announcements
  - Detect product page modifications

News & Social Intelligence Agent: 

Name: MediaTracker Intelligence
Role: News and Social Media Analyst
Responsibilities:
  - Monitor industry news mentions
  - Track social media activity and engagement
  - Analyze customer sentiment and feedback
  - Identify trending topics and discussions

Business Intelligence Agent: 

Name: BizIntel Analyzer
Role: Corporate Intelligence Specialist
Responsibilities:
  - Monitor job posting patterns
  - Track funding and investment news
  - Analyze financial filings and reports
  - Assess leadership and organizational changes

Part 3: Data Collection Implementation (75 minutes)

Step 1: Web Monitoring Configuration

Website Change Detection Setup: 

Monitored Pages per Competitor:
  Homepage: Daily scan for major announcements
  Pricing Page: Every 4 hours for price changes
  Product Pages: Daily for feature updates
  About/Team Pages: Weekly for leadership changes
  Press Room: Daily for new announcements
  Careers Page: Daily for hiring pattern analysis

Change Detection Settings:
  Sensitivity: Medium (ignore minor formatting)
  Archive Duration: 90 days of historical changes
  Screenshot Capture: Yes (for visual changes)
  Content Analysis: Extract key information changes

Web Monitoring Agent Instructions: 

## Website Monitoring Protocol

### Change Detection Criteria
Monitor for these types of changes:
- Pricing information (any percentage change)
- Product features and specifications
- New product or service announcements
- Team member additions or departures
- Press releases and news announcements
- Partnership or integration announcements

### Analysis Requirements
For each detected change:
1. **Capture Details**
   - What specifically changed
   - When the change was detected
   - Before and after comparison
   - Screenshots if visual changes

2. **Assess Significance**
   - Business impact rating (1-10)
   - Competitive implications
   - Customer impact potential
   - Urgency for response

3. **Generate Alert**
   - Immediate alert for significant changes (>7 rating)
   - Daily summary for minor changes
   - Archive all changes for trend analysis

### Quality Control
- Verify changes are real (not temporary site issues)
- Compare against archived versions
- Cross-reference with other sources when possible
- Flag suspicious or unclear changes for review

Step 2: News and Social Monitoring

News Tracking Configuration: 

Search Terms and Queries:
  Company Names:
    - "[Competitor Name]" (exact phrase)
    - [Competitor] AND (announcement OR launch OR funding)
    - [Competitor] AND (partnership OR acquisition OR merger)

  Industry Terms:
    - "[Your Industry]" AND (trend OR forecast OR analysis)
    - "[Key Technology]" AND (innovation OR advancement)
    - "[Your Market]" AND (growth OR opportunity OR threat)

News Sources to Monitor:
  Tier 1: TechCrunch, Forbes, Wall Street Journal, Reuters
  Tier 2: Industry trade publications
  Tier 3: Company blogs and press rooms
  Tier 4: Social media and forums (Reddit, LinkedIn)

Monitoring Frequency:
  - Breaking news: Real-time alerts
  - Industry publications: Every 4 hours
  - Company sources: Daily
  - Social media: Every 2 hours during business hours

Social Media Monitoring Setup: 

Platforms to Monitor:
  LinkedIn:
    - Company pages and updates
    - Employee posts and job changes
    - Industry group discussions

  Twitter:
    - Company accounts and mentions
    - Executive accounts
    - Industry hashtags and trends

  Reddit:
    - Industry subreddits
    - Product discussion forums
    - Customer feedback threads

Engagement Analysis:
  - Like/share/comment volumes
  - Sentiment analysis of comments
  - Viral content identification
  - Customer service interaction quality

Step 3: Financial and Business Data Collection

Business Intelligence Monitoring: 

Financial Data Sources:
  Public Companies:
    - SEC EDGAR filings (quarterly/annual reports)
    - Investor relations announcements
    - Stock price and trading volume
    - Analyst reports and ratings

  Private Companies:
    - Funding announcement tracking
    - Patent filings and IP activity
    - Key hire announcements
    - Office expansion or consolidation

Job Market Analysis:
  Platforms: LinkedIn, Indeed, Glassdoor, AngelList

  Metrics to Track:
    - Total open positions by department
    - New role types indicating strategic shifts
    - Salary ranges and compensation trends
    - Employee review sentiment

Patent and IP Monitoring:
  Sources: USPTO database, Google Patents, patent law firms

  Analysis Focus:
    - New patent filings in your technology area
    - Patent grants that could block your innovation
    - Licensing deals and IP transactions
    - Patent litigation activity

Part 4: Analysis and Intelligence Generation (60 minutes)

Step 1: Data Analysis Algorithms

Trend Analysis Framework: 

# Trend Analysis Logic (Pseudo-code)
def analyze_pricing_trends(competitor_data):
    pricing_history = get_pricing_history(competitor_data, days=90)

    trends = {
        'price_changes': calculate_price_changes(pricing_history),
        'frequency': calculate_change_frequency(pricing_history),
        'seasonal_patterns': identify_seasonal_patterns(pricing_history),
        'competitive_responses': analyze_competitive_responses(pricing_history)
    }

    insights = {
        'strategy_assessment': assess_pricing_strategy(trends),
        'predicted_changes': predict_future_changes(trends),
        'competitive_impact': assess_impact_on_us(trends),
        'recommended_response': generate_recommendations(trends)
    }

    return combine_trends_and_insights(trends, insights)

def analyze_product_announcements(announcement_data):
    analysis = {
        'feature_comparison': compare_with_our_products(announcement_data),
        'market_positioning': analyze_positioning_strategy(announcement_data),
        'customer_impact': assess_customer_response(announcement_data),
        'competitive_threat': rate_threat_level(announcement_data)
    }

    return analysis

Sentiment Analysis Configuration: 

Customer Sentiment Tracking:
  Sources: Product reviews, social media, support forums

  Metrics:
    - Overall sentiment score (-1 to +1)
    - Sentiment trend over time
    - Topic-based sentiment (pricing, features, support)
    - Comparative sentiment vs. our products

Brand Reputation Monitoring:
  Indicators:
    - Media mention sentiment
    - Crisis or controversy detection
    - Leadership reputation changes
    - Customer advocacy levels

Analysis Framework:
  - Baseline sentiment establishment
  - Change detection and alerting
  - Correlation with business events
  - Competitive comparison and benchmarking

Step 2: Insight Generation Engine

Strategic Analysis Framework: 

## Competitive Intelligence Analysis

### Opportunity Identification
Look for patterns indicating opportunities:
- Competitor price increases (pricing opportunity)
- Product feature gaps (development opportunity)
- Customer complaints (positioning opportunity)
- Market expansion announcements (partnership opportunity)
- Key employee departures (talent acquisition opportunity)

### Threat Assessment
Monitor for potential threats:
- Aggressive pricing strategies
- Superior product features
- Strong partnership announcements
- Significant funding rounds
- Key talent acquisitions

### Market Trend Analysis
Identify broader market movements:
- Technology adoption curves
- Customer behavior shifts
- Regulatory environment changes
- Economic factors affecting industry
- Competitive landscape consolidation

### Strategic Recommendations
Generate actionable recommendations:
- Immediate tactical responses (next 30 days)
- Medium-term strategic adjustments (next quarter)
- Long-term positioning considerations (next year)

Recommendation Engine Logic: 

Recommendation Types:

Defensive Moves:
  - Pricing adjustments to maintain competitiveness
  - Feature development to close gaps
  - Marketing responses to competitive campaigns
  - Customer retention strategies

Offensive Opportunities:
  - Market expansion into competitor weaknesses
  - Product differentiation strategies
  - Partnership opportunities
  - Talent acquisition from competitors

Strategic Positioning:
  - Brand messaging adjustments
  - Product roadmap prioritization
  - Market segment focus changes
  - Competitive differentiation enhancement

Part 5: Reporting and Delivery System (45 minutes)

Step 1: Automated Report Generation

Weekly Intelligence Report Template: 

# Weekly Competitive Intelligence Report
*Week of [Start Date] - [End Date]*

## Executive Summary
*Top 3 strategic insights requiring attention*

### 🚨 Critical Development
**[Most significant event]**
- **What happened:** [Description]
- **Why it matters:** [Strategic implications]
- **Recommended action:** [Specific next steps]

### 📈 Key Trend
**[Important market or competitive trend]**
- **Pattern identified:** [Trend description]
- **Business impact:** [How it affects us]
- **Timeline:** [When action needed]

### 💡 Strategic Opportunity
**[Identified opportunity]**
- **Opportunity:** [Description]
- **Why now:** [Timing rationale]
- **Potential value:** [Expected benefit]

---

## Competitive Activity Analysis

### [Competitor 1 Name]
**Activity Level:** High/Medium/Low
**Key Developments:**
- [Bullet point of significant activity]
- [Analysis of strategic implications]

**Our Assessment:**
- **Threat Level:** High/Medium/Low
- **Recommended Response:** [Specific actions]

### [Competitor 2 Name]
**Activity Level:** High/Medium/Low
**Key Developments:**
- [Bullet point of significant activity]
- [Analysis of strategic implications]

**Our Assessment:**
- **Threat Level:** High/Medium/Low
- **Recommended Response:** [Specific actions]

---

## Market Intelligence

### Industry Trends
1. **[Trend Name]**
   - **Evidence:** [Supporting data]
   - **Implications:** [What it means for us]
   - **Timeline:** [Expected development timeline]

### Customer Insights
- **Sentiment Changes:** [Analysis of customer opinion shifts]
- **Emerging Needs:** [New customer requirements identified]
- **Satisfaction Factors:** [What drives customer satisfaction]

### Technology Landscape
- **Innovation Areas:** [New technologies affecting market]
- **Adoption Patterns:** [How customers are adopting new tech]
- **Competitive Advantages:** [Technology-based differentiation opportunities]

---

## Strategic Recommendations

### Immediate Actions (Next 30 Days)
1. **[Action Item 1]**
   - **Rationale:** [Why this is important]
   - **Owner:** [Who should execute]
   - **Timeline:** [When to complete]

2. **[Action Item 2]**
   - **Rationale:** [Why this is important]
   - **Owner:** [Who should execute]
   - **Timeline:** [When to complete]

### Strategic Initiatives (Next Quarter)
1. **[Initiative 1]**
   - **Objective:** [What we want to achieve]
   - **Resources Needed:** [Investment required]
   - **Success Metrics:** [How to measure success]

### Long-term Considerations (Next Year)
- **Market Positioning:** [Strategic positioning recommendations]
- **Product Strategy:** [Product development priorities]
- **Competitive Differentiation:** [How to maintain advantage]

---

## Data Quality and Sources
- **Sources Monitored:** [Number] websites, [Number] news sources, [Number] social accounts
- **Data Points Collected:** [Number] this week
- **Confidence Level:** [High/Medium/Low] based on source verification
- **Notable Gaps:** [Areas needing additional intelligence]

*Report generated automatically by MarketScope Intelligence*
*For questions or additional analysis, contact [Intelligence Team]*

Step 2: Alert and Notification System

Real-Time Alert Configuration: 

Critical Alerts (Immediate Notification):
  Triggers:
    - Competitor price reduction >20%
    - Major product launch announcement
    - Key executive departure (C-level)
    - Acquisition or merger announcement
    - Significant negative news or crisis

  Delivery Method:
    - SMS to key stakeholders
    - Slack channel notification
    - Email to executive team
    - Dashboard alert banner

Daily Briefing Alerts:
  Triggers:
    - Multiple minor competitive changes
    - Industry trend developments
    - Customer sentiment shifts
    - Financial performance changes

  Delivery Method:
    - Email summary to strategy team
    - Dashboard update
    - Slack daily digest

Step 3: Performance Dashboard

Intelligence Dashboard Components: 

Real-Time Metrics:
  - Competitor activity level (last 24 hours)
  - Market sentiment indicators
  - News mention volume and sentiment
  - Social media engagement trends

Weekly Trends:
  - Competitive activity summary
  - Market share movement indicators
  - Customer satisfaction trends
  - Product feature gap analysis

Strategic Indicators:
  - Threat level assessment by competitor
  - Opportunity pipeline development
  - Market trend alignment with strategy
  - Competitive response effectiveness

Part 6: Testing and Validation (45 minutes)

Step 1: System Accuracy Testing

Test Scenario 1: Price Change Detection 

Test Setup:
  - Monitor known competitor pricing page
  - Create controlled price change (if possible)
  - Verify detection speed and accuracy

Expected Results:
  - Change detected within 4 hours
  - Accurate price difference calculation
  - Proper significance assessment
  - Appropriate alert generation

Test Scenario 2: News Monitoring 

Test Setup:
  - Track known upcoming product announcement
  - Monitor coverage across multiple sources
  - Verify analysis accuracy

Expected Results:
  - News captured from multiple sources
  - Proper source credibility assessment
  - Accurate competitive implications analysis
  - Timely alert delivery

Test Scenario 3: Social Media Analysis 

Test Setup:
  - Monitor social media during known campaign
  - Track engagement and sentiment
  - Verify analysis accuracy

Expected Results:
  - Engagement metrics accurately captured
  - Sentiment analysis aligns with human assessment
  - Trending topics properly identified
  - Competitive comparison provides value

Step 2: Intelligence Quality Assessment

Quality Metrics: 

Data Accuracy:
  - Source verification rate: >95%
  - False positive alerts: <5%
  - Intelligence confidence scores: >80% average

Timeliness:
  - Critical alert delivery: <30 minutes
  - Daily briefing delivery: Before 8 AM
  - Weekly report delivery: Monday 9 AM

Relevance:
  - Actionable insights percentage: >60%
  - Stakeholder satisfaction score: >8/10
  - Recommendation implementation rate: >40%

Step 3: Business Impact Validation

Value Measurement Framework: 

Research Efficiency:
  Before: _____ hours/week manual research
  After: _____ hours/week review automation results
  Time Savings: _____ hours/week

Intelligence Coverage:
  Before: _____ competitors monitored irregularly
  After: _____ competitors monitored continuously
  Coverage Improvement: _____%

Decision Speed:
  Before: _____ days from event to strategic response
  After: _____ days from event to strategic response
  Response Time Improvement: _____%

Troubleshooting and Optimization

Common Challenge 1: Information Overload

Problem: Too much data, not enough insights Solutions: 

Filter Optimization:
  - Increase significance thresholds
  - Focus on high-impact competitors
  - Reduce monitoring frequency for low-value sources
  - Improve analysis algorithms to surface key insights

Reporting Refinement:
  - Shorter executive summaries
  - More focused recommendations
  - Better data visualization
  - Stakeholder feedback integration

Common Challenge 2: False Positives

Problem: Alerts for unimportant changes Solutions: 

Detection Tuning:
  - Improve change significance algorithms
  - Add context validation (temporary vs. permanent)
  - Cross-reference multiple sources
  - Learn from false positive feedback

Alert Thresholds:
  - Raise minimum impact thresholds
  - Add confirmation requirements
  - Implement cooling-off periods
  - Use stakeholder feedback for training

Common Challenge 3: Source Access Issues

Problem: Websites blocking automated monitoring Solutions: 

Technical Workarounds:
  - Rotate IP addresses and user agents
  - Use multiple data collection methods
  - Implement respectful rate limiting
  - Focus on API-accessible sources

Alternative Sources:
  - Third-party data providers
  - Industry databases and reports
  - Social media monitoring tools
  - Public filing databases

Lab Completion Checklist

  • Competitive landscape and intelligence requirements defined
  • Master intelligence coordinator agent created
  • Specialized collection agents implemented
  • Web monitoring system functional
  • News and social media tracking operational
  • Financial and business intelligence monitoring working
  • Analysis and insight generation algorithms implemented
  • Automated reporting system delivering weekly reports
  • Real-time alert system functional
  • Performance dashboard operational
  • Testing completed with real competitive scenarios
  • Quality metrics meeting target thresholds

Final Intelligence System Metrics: - Competitors monitored: _____ - Information sources: _____ - Weekly insights generated: _____ - Alert response time: _____ minutes - Research time savings: _____ hours/week - Stakeholder satisfaction: _____/10

Congratulations! You've built a sophisticated competitive intelligence system that transforms how your organization stays informed about market dynamics. This research agent demonstrates the power of continuous automated monitoring, intelligent analysis, and actionable insight generation.

In Lab 4, you'll apply these research and analysis skills to create a customer service agent that can handle complex customer interactions while maintaining your brand voice and customer satisfaction standards.