# Plutus Payment System - Logging Best Practices

## Overview

This document outlines the enhanced logging system implemented in the Plutus Payment Processing System. The logging infrastructure provides comprehensive monitoring, security event tracking, performance analysis, and automated log management.

## Logging Architecture

### Core Components

1. **Enhanced Logging Configuration** (`logging_config.py`)
   - Structured logging with correlation IDs
   - Multiple specialized logger types
   - Automatic log formatting and rotation

2. **Middleware System** (`middleware.py`)
   - Request/response logging
   - Performance monitoring
   - Security event detection
   - Database query tracking

3. **Analytics Dashboard** (`blueprints/analytics.py`)
   - Real-time system health monitoring
   - Performance metrics visualization
   - Security event analysis
   - Log search and filtering

4. **Log Retention System** (`log_retention.py`)
   - Automated cleanup and archiving
   - Configurable retention policies
   - Disk space management

## Logger Types

### StructuredLogger
General-purpose logger with correlation ID support and structured data.

```python
from logging_config import get_logger

logger = get_logger('module_name')
logger.info("Payment processed successfully", 
            payment_id=12345, 
            amount=89.95, 
            customer_id="cus_123")
```

### SecurityLogger
Specialized logger for security events and threats.

```python
from logging_config import security_logger

security_logger.log_login_attempt("username", success=False, ip_address="192.168.1.1")
security_logger.log_payment_fraud_alert(payment_id=123, customer_id="cus_456", 
                                       reason="Unusual amount pattern", amount=5000.0)
```

### PerformanceLogger
Dedicated logger for performance monitoring and optimization.

```python
from logging_config import performance_logger

performance_logger.log_request_time("POST /payments", "POST", 1250.5, 200, user_id=1)
performance_logger.log_stripe_api_call("create_payment", 850.2, True)
```

## Log Files Structure

### File Organization
```
logs/
├── plutus_detailed.log          # Comprehensive application logs
├── performance.log              # Performance metrics and slow operations
├── security.log                 # Security events and threats
├── payment_processing.log       # Payment-specific operations
├── archive/                     # Archived logs by month
│   ├── 202409/
│   └── 202410/
└── *.log.gz                    # Compressed rotated logs
```

### Log Formats

#### Standard Format
```
2024-09-02 14:30:15,123 - [corr-abc123] - plutus.payments - INFO - Payment processed successfully {"payment_id": 12345, "amount": 89.95}
```

#### Security Format
```
2024-09-02 14:30:15,123 - SECURITY - [corr-abc123] - WARNING - LOGIN_FAILED for user: testuser {"ip_address": "192.168.1.1", "user_agent": "Mozilla/5.0..."}
```

#### Performance Format
```
2024-09-02 14:30:15,123 - PERF - [corr-abc123] - REQUEST: POST /payments - 1250.50ms - 200 {"user_id": 1, "endpoint": "/payments"}
```

## Correlation IDs

### Purpose
Correlation IDs track requests across the entire system, making it easy to trace a single operation through multiple components.

### Usage
```python
from logging_config import log_context, set_correlation_id

# Automatic correlation ID
with log_context():
    logger.info("Processing payment")  # Will include auto-generated correlation ID

# Custom correlation ID
with log_context("req-12345"):
    logger.info("Processing payment")  # Will include "req-12345"

# Manual setting
correlation_id = set_correlation_id("custom-id")
logger.info("Payment processed")
```

## Performance Monitoring

### Automatic Monitoring
The system automatically tracks:
- HTTP request response times
- Database query performance
- Stripe API call latencies
- Slow operations (>1 second requests, >100ms queries)

### Manual Performance Logging
```python
from logging_config import log_performance

@log_performance("payment_processing")
def process_payment(payment_data):
    # Function implementation
    pass

# Or manually
start_time = time.time()
result = some_operation()
duration_ms = (time.time() - start_time) * 1000
performance_logger.log_request_time("operation_name", "GET", duration_ms, 200)
```

## Security Event Monitoring

### Automatic Detection
The middleware automatically detects and logs:
- SQL injection attempts
- Cross-site scripting (XSS) attempts
- Failed authentication attempts
- Suspicious user agents
- Access to admin endpoints
- Brute force attack patterns

### Manual Security Logging
```python
from logging_config import security_logger

# Log permission violations
security_logger.log_permission_denied("username", "delete_payment", "payment/123", "192.168.1.1")

# Log fraud alerts
security_logger.log_payment_fraud_alert(payment_id=123, customer_id="cus_456", 
                                       reason="Multiple failed attempts", amount=1000.0)
```

## Log Retention and Management

### Retention Policies
Default retention periods:
- Application logs: 30 days
- Performance logs: 14 days  
- Security logs: 90 days
- Payment processing logs: 60 days

### Automated Cleanup
- Runs daily at 2:00 AM
- Compresses logs older than configured threshold
- Archives important logs before deletion
- Monitors disk space usage

### Manual Management
```python
from log_retention import retention_manager

# Get statistics
stats = retention_manager.get_log_statistics()

# Manual cleanup
cleanup_stats = retention_manager.cleanup_logs()

# Emergency cleanup (when disk space is low)
emergency_stats = retention_manager.emergency_cleanup(target_size_mb=500)
```

## Analytics Dashboard

### Access
Navigate to `/analytics/dashboard` (requires Finance+ permissions)

### Features
- **System Health**: Real-time health score and key metrics
- **Performance Monitoring**: Response times, slow requests, database performance
- **Payment Analytics**: Success rates, error analysis, trends
- **Security Events**: Failed logins, suspicious activity, fraud alerts
- **Log Search**: Full-text search with filtering and pagination

### API Endpoints
- `GET /analytics/api/system-health` - Current system health metrics
- `GET /analytics/api/performance-metrics` - Performance analysis data
- `GET /analytics/api/payment-analytics` - Payment processing statistics
- `GET /analytics/api/security-events` - Security event summary
- `GET /analytics/api/logs/search` - Search system logs

## Best Practices

### For Developers

1. **Use Structured Logging**
   ```python
   # Good
   logger.info("Payment processed", payment_id=123, amount=89.95, status="success")
   
   # Avoid
   logger.info(f"Payment {payment_id} processed for ${amount} - status: {status}")
   ```

2. **Include Context**
   ```python
   # Include relevant context in all log messages
   logger.info("Payment failed", 
               payment_id=payment.id,
               customer_id=payment.customer_id,
               error_code=error.code,
               error_message=str(error))
   ```

3. **Use Appropriate Log Levels**
   - `DEBUG`: Detailed diagnostic information
   - `INFO`: General information about system operation
   - `WARNING`: Something unexpected happened but system continues
   - `ERROR`: Serious problem that prevented function completion
   - `CRITICAL`: Very serious error that may abort the program

4. **Security-Sensitive Data**
   ```python
   # Never log sensitive data
   logger.info("Payment processed", 
               payment_id=123,
               amount=89.95,
               card_last4="1234")  # OK - only last 4 digits
   
   # Avoid logging full card numbers, CVV, passwords, etc.
   ```

### For Operations

1. **Monitor Key Metrics**
   - System health score (target: >90%)
   - Payment success rate (target: >95%)
   - Error rate (target: <5%)
   - Average response time (target: <1000ms)

2. **Set Up Alerts**
   - Health score drops below 75%
   - Payment success rate drops below 90%
   - Multiple security events in short timeframe
   - Disk space usage exceeds 80%

3. **Regular Review**
   - Weekly review of security events
   - Monthly analysis of performance trends
   - Quarterly review of retention policies
   - Annual security audit of logged events

### For Security

1. **Monitor for Patterns**
   - Multiple failed logins from same IP
   - Unusual payment amounts or frequencies
   - Access attempts to admin endpoints
   - SQL injection or XSS attempts

2. **Incident Response**
   - Use correlation IDs to trace incident across systems
   - Export relevant logs for forensic analysis
   - Coordinate with development team using structured log data

## Configuration

### Environment Variables
```bash
# Optional: Override default log retention
LOG_RETENTION_DAYS=30
LOG_CLEANUP_TIME=02:00
LOG_MAX_FILE_SIZE_MB=100
LOG_ARCHIVE_COMPRESS=true
```

### Programmatic Configuration
```python
# Custom retention configuration
custom_config = {
    'retention_policies': {
        'security.log': {'days': 180, 'compress_after_days': 7},
        'performance.log': {'days': 7, 'compress_after_days': 1},
        'default': {'days': 30, 'compress_after_days': 7}
    },
    'cleanup_schedule': '03:00',
    'max_file_size_mb': 50
}

retention_manager = LogRetentionManager(custom_config)
```

## Troubleshooting

### Common Issues

1. **Logs Not Appearing**
   - Check logs directory permissions
   - Verify logger configuration in app initialization
   - Check disk space availability

2. **High Disk Usage**
   - Run manual cleanup: `python log_retention.py`
   - Reduce retention periods for non-critical logs
   - Enable compression for all log types

3. **Performance Impact**
   - Disable DEBUG level logging in production
   - Reduce log verbosity for high-frequency operations
   - Use async logging for high-throughput scenarios

4. **Missing Correlation IDs**
   - Ensure middleware is properly initialized
   - Check that log context is being used in threaded operations
   - Verify correlation ID propagation in external API calls

### Log Analysis Commands

```bash
# Search for specific payment
grep "payment_id.*12345" logs/plutus_detailed.log

# Find all errors in last hour
grep "$(date -d '1 hour ago' '+%Y-%m-%d %H')" logs/plutus_detailed.log | grep ERROR

# Count security events by type
grep "SECURITY" logs/security.log | cut -d'-' -f5 | sort | uniq -c

# Monitor real-time logs
tail -f logs/plutus_detailed.log

# Analyze correlation ID flow
grep "corr-abc123" logs/*.log | sort
```

## Support and Maintenance

### Log File Monitoring
Set up monitoring for:
- Log file growth rates
- Error frequency patterns
- Security event trends
- System performance degradation

### Regular Maintenance
- Weekly: Review disk space and cleanup if needed
- Monthly: Analyze performance trends and optimize slow queries
- Quarterly: Review retention policies and adjust as needed
- Annually: Audit security events and update detection rules

### Contact Information
For logging system issues or questions:
- Development Team: Review code in `logging_config.py`, `middleware.py`
- Operations Team: Monitor analytics dashboard and system health
- Security Team: Review security logs and event patterns

## Version History

- **v1.0** (Phase 8): Initial enhanced logging implementation
- **v1.1** (Phase 9): Analytics dashboard and retention system
- **v1.2**: Correlation ID improvements and performance optimization

---

This logging system provides comprehensive visibility into the Plutus Payment System while maintaining security, performance, and operational efficiency. Regular review and maintenance of the logging infrastructure ensures continued reliability and usefulness for system monitoring and troubleshooting.