Lesson 8.1: Course Project Overview

Project Goals

Create a robust data aggregation system that demonstrates:

• ✅ Parallel and sequential Promise execution
• ✅ Error handling and retry logic
• ✅ Rate limiting and caching
• ✅ Timeout handling
• ✅ Comprehensive testing
• ✅ Real-world best practices

What You'll Build

Core Features

1. Multi-API Data Fetcher

• Fetch data from 3+ different APIs
• Handle different response formats
• Aggregate results into unified format

2. Resilient Request Handler

• Automatic retry with exponential backoff
• Request timeout handling
• Fallback strategies

3. Performance Optimization

• Parallel requests where possible
• Request caching with TTL
• Rate limiting to prevent API throttling

4. Error Management

• Graceful degradation
• Detailed error logging
• Partial success handling

5. Testing Suite

• Unit tests for all utilities
• Integration tests for data flow
• Mock API responses

Project Structure

data-aggregator/
├── src/
│   ├── api/
│   │   ├── client.js           # Base API client
│   │   ├── weatherAPI.js       # Weather data source
│   │   ├── newsAPI.js          # News data source
│   │   └── stocksAPI.js        # Stock data source
│   │
│   ├── utils/
│   │   ├── retry.js            # Retry logic
│   │   ├── timeout.js          # Timeout wrapper
│   │   ├── cache.js            # Caching layer
│   │   ├── rateLimiter.js      # Rate limiting
│   │   └── logger.js           # Logging utility
│   │
│   ├── aggregator/
│   │   ├── index.js            # Main aggregator
│   │   ├── transformer.js      # Data transformation
│   │   └── validator.js        # Data validation
│   │
│   └── index.js                # Entry point
│
├── tests/
│   ├── unit/
│   │   ├── retry.test.js
│   │   ├── cache.test.js
│   │   └── rateLimiter.test.js
│   │
│   └── integration/
│       └── aggregator.test.js
│
├── config/
│   └── default.js              # Configuration
│
├── package.json
└── README.md

Technical Requirements

APIs to Integrate

1. Weather API (simulated)

• Endpoint: /api/weather?city=London
• Returns: Temperature, conditions, forecast

2. News API (simulated)

• Endpoint: /api/news?category=tech
• Returns: Headlines, articles, sources

3. Stocks API (simulated)

• Endpoint: /api/stocks?symbol=AAPL
• Returns: Price, change, volume

Implementation Steps

Phase 1: Setup & Basic Structure

1. Initialize project with npm
2. Set up folder structure
3. Install dependencies (Jest for testing)
4. Create configuration file

Phase 2: Core Utilities

1. Implement delay function
2. Create timeout wrapper
3. Build retry logic with exponential backoff
4. Develop caching layer
5. Implement rate limiter

Phase 3: API Clients

1. Create base API client
2. Implement weather API client
3. Implement news API client
4. Implement stocks API client
5. Add error handling to all clients

Phase 4: Data Aggregation

1. Build main aggregator
2. Implement parallel fetching
3. Add data transformation
4. Implement validation
5. Handle partial failures

Phase 5: Testing

1. Write unit tests for utilities
2. Create mock API responses
3. Write integration tests
4. Test error scenarios
5. Measure performance

Example: Basic Aggregator

// src/aggregator/index.js
class DataAggregator {
  constructor(apis, utils) {
    this.apis = apis;
    this.utils = utils;
  }
  
  async fetchAll() {
    try {
      const [weather, news, stocks] = await Promise.all([
        this.utils.retry(() => this.apis.weather.get()),
        this.utils.retry(() => this.apis.news.get()),
        this.utils.retry(() => this.apis.stocks.get())
      ]);
      
      return {
        weather,
        news,
        stocks,
        timestamp: new Date().toISOString()
      };
    } catch (error) {
      console.error('Aggregation failed:', error);
      throw error;
    }
  }
  
  async fetchWithCache() {
    const cacheKey = 'dashboard-data';
    const cached = await this.utils.cache.get(cacheKey);
    
    if (cached) {
      return cached;
    }
    
    const data = await this.fetchAll();
    await this.utils.cache.set(cacheKey, data, 300); // 5 min TTL
    
    return data;
  }
}

export default DataAggregator;

Success Criteria

• ✅ All API calls succeed with retry logic
• ✅ Requests complete in under 2 seconds (parallel)
• ✅ Caching reduces redundant API calls
• ✅ Rate limiting prevents API throttling
• ✅ Graceful error handling for partial failures
• ✅ 90%+ test coverage
• ✅ Clear logging for debugging

Next Steps

Start building your project! Use the lessons learned throughout this course to create a robust, production-ready data aggregator. Good luck! 🚀