Rodrigo Rodriguez (Pragmatismo) d365744486 - New stuff, 6.1.

2025-11-21 23:23:53 -03:00

9.1 KiB

Raw Blame History

Meeting and Multimedia Features Implementation

Overview

This document describes the implementation of enhanced chat features, meeting services, and screen capture capabilities for the General Bots botserver application.

Features Implemented

1. Enhanced Bot Module with Multimedia Support

Location: `src/bot/multimedia.rs`

Video Messages: Support for sending and receiving video files with thumbnails
Image Messages: Image sharing with caption support
Web Search: Integrated web search capability with /search command
Document Sharing: Support for various document formats
Meeting Invites: Handling meeting invitations and redirects from Teams/WhatsApp

Key Components:

MultimediaMessage enum for different message types
MultimediaHandler trait for processing multimedia content
DefaultMultimediaHandler implementation with S3 storage support
Media upload/download functionality

2. Meeting Service Implementation

Location: `src/meet/service.rs`

Real-time Meeting Rooms: Support for creating and joining video conference rooms
Live Transcription: Real-time speech-to-text transcription during meetings
Bot Integration: AI assistant that responds to voice commands and meeting context
WebSocket Communication: Real-time messaging between participants
Recording Support: Meeting recording capabilities

Key Features:

Meeting room management with participant tracking
WebSocket message types for various meeting events
Transcription service integration
Bot command processing ("Hey bot" wake word)
Screen sharing support

3. Screen Capture with WebAPI

Implementation: Browser-native WebRTC

Screen Recording: Full screen capture using MediaStream Recording API
Window Capture: Capture specific application windows via browser selection
Region Selection: Browser-provided selection interface
Screenshot: Capture video frames from MediaStream
WebRTC Streaming: Direct streaming to meetings via RTCPeerConnection

Browser API Usage:

// Request screen capture
const stream = await navigator.mediaDevices.getDisplayMedia({
    video: {
        cursor: "always",
        displaySurface: "monitor" // or "window", "browser"
    },
    audio: true
});

// Add to meeting peer connection
stream.getTracks().forEach(track => {
    peerConnection.addTrack(track, stream);
});

Benefits:

Cross-platform: Works on web, desktop, and mobile browsers
No native dependencies: Pure JavaScript implementation
Browser security: Built-in permission management
Standard API: W3C MediaStream specification

4. Web Desktop Meet Component

Location: `web/desktop/meet/`

Full Meeting UI: Complete video conferencing interface
Video Grid: Dynamic participant video layout
Chat Panel: In-meeting text chat
Transcription Panel: Live transcription display
Bot Assistant Panel: AI assistant interface
Participant Management: View and manage meeting participants

Files:

meet.html: Meeting room interface
meet.js: WebRTC and meeting logic
meet.css: Responsive styling

Integration Points

1. WebSocket Message Types

const MessageType = {
    JOIN_MEETING: 'join_meeting',
    LEAVE_MEETING: 'leave_meeting',
    TRANSCRIPTION: 'transcription',
    CHAT_MESSAGE: 'chat_message',
    BOT_MESSAGE: 'bot_message',
    SCREEN_SHARE: 'screen_share',
    STATUS_UPDATE: 'status_update',
    PARTICIPANT_UPDATE: 'participant_update',
    RECORDING_CONTROL: 'recording_control',
    BOT_REQUEST: 'bot_request'
};

2. API Endpoints

POST /api/meet/create - Create new meeting room
POST /api/meet/token - Get WebRTC connection token
POST /api/meet/invite - Send meeting invitations
GET /ws/meet - WebSocket connection for meeting

3. Bot Commands in Meetings

Summarize: Generate meeting summary
Action Items: Extract action items from discussion
Key Points: Highlight important topics
Questions: List pending questions

Usage Examples

Creating a Meeting

const response = await fetch('/api/meet/create', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        name: 'Team Standup',
        settings: {
            enable_transcription: true,
            enable_bot: true
        }
    })
});

Sending Multimedia Message

let message = MultimediaMessage::Image {
    url: "https://example.com/image.jpg".to_string(),
    caption: Some("Check this out!".to_string()),
    mime_type: "image/jpeg".to_string(),
};

Starting Screen Capture (WebAPI)

// Request screen capture with options
const stream = await navigator.mediaDevices.getDisplayMedia({
    video: {
        cursor: "always",
        width: { ideal: 1920 },
        height: { ideal: 1080 },
        frameRate: { ideal: 30 }
    },
    audio: true
});

// Record or stream to meeting
const mediaRecorder = new MediaRecorder(stream, {
    mimeType: 'video/webm;codecs=vp9',
    videoBitsPerSecond: 2500000
});
mediaRecorder.start();

Meeting Redirect Flow

Handling Teams/WhatsApp Video Calls

External platform initiates video call
User receives redirect to botserver meeting
Redirect handler shows incoming call notification
Auto-accept or manual accept/reject
Join meeting room with guest credentials

URL Format for Redirects

/meet?meeting=<meeting_id>&from=<platform>

Examples:
/meet?meeting=abc123&from=teams
/meet?meeting=xyz789&from=whatsapp

Configuration

Environment Variables

# Search API
SEARCH_API_KEY=your_search_api_key

# WebRTC Server (LiveKit)
LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=your_api_key
LIVEKIT_SECRET=your_secret

# Storage for media
DRIVE_SERVER=http://localhost:9000
DRIVE_ACCESSKEY=your_access_key
DRIVE_SECRET=your_secret

Meeting Settings

pub struct MeetingSettings {
    pub enable_transcription: bool,  // Default: true
    pub enable_recording: bool,      // Default: false
    pub enable_chat: bool,           // Default: true
    pub enable_screen_share: bool,   // Default: true
    pub auto_admit: bool,            // Default: true
    pub waiting_room: bool,          // Default: false
    pub bot_enabled: bool,           // Default: true
    pub bot_id: Option<String>,      // Optional specific bot
}

Security Considerations

Authentication: All meeting endpoints should verify user authentication
Room Access: Implement proper room access controls
Recording Consent: Get participant consent before recording
Data Privacy: Ensure transcriptions and recordings are properly secured
WebRTC Security: Use secure signaling and TURN servers

Performance Optimization

Video Quality: Adaptive bitrate based on network conditions
Lazy Loading: Load panels and features on-demand
WebSocket Batching: Batch multiple messages when possible
Transcription Buffer: Buffer audio before sending to transcription service
Media Compression: Compress images/videos before upload

Future Enhancements

Virtual Backgrounds: Add background blur/replacement
Breakout Rooms: Support for sub-meetings
Whiteboard: Collaborative drawing during meetings
Meeting Analytics: Track speaking time, participation
Calendar Integration: Schedule meetings with calendar apps
Mobile Support: Responsive design for mobile devices
End-to-End Encryption: Secure meeting content
Custom Layouts: User-defined video grid layouts
Meeting Templates: Pre-configured meeting types
Integration APIs: Webhooks for external integrations

Testing

Unit Tests

Test multimedia message parsing
Test meeting room creation/joining
Test transcription processing
Test bot command handling

Integration Tests

Test WebSocket message flow
Test video call redirects
Test screen capture with different configurations
Test meeting recording and playback

E2E Tests

Complete meeting flow from creation to end
Multi-participant interaction
Screen sharing during meeting
Bot interaction during meeting

Deployment

Ensure LiveKit or WebRTC server is running
Configure S3 or storage for media files
Set up transcription service (if using external)
Deploy web assets to static server
Configure reverse proxy for WebSocket connections
Set up SSL certificates for production
Configure TURN/STUN servers for NAT traversal

Troubleshooting

Common Issues

No Video/Audio: Check browser permissions and device access
Connection Failed: Verify WebSocket URL and CORS settings
Transcription Not Working: Check transcription service credentials
Screen Share Black: May need elevated permissions on some OS
Bot Not Responding: Verify bot service is running and connected

Debug Mode

Enable debug logging in the browser console:

localStorage.setItem('debug', 'meet:*');

Support

For issues or questions:

Check logs in ./logs/meeting.log
Review WebSocket messages in browser DevTools
Contact support with meeting ID and timestamp

9.1 KiB Raw Blame History