Migration to Rust removal of Azure.

This commit is contained in:
Rodrigo Rodriguez (Pragmatismo) 2025-10-06 10:13:37 -03:00
parent 84d59b61da
commit b634086b37
3 changed files with 149 additions and 7459 deletions

View file

@ -1,2 +0,0 @@
[term]
quiet = true

6984
Cargo.lock generated

File diff suppressed because it is too large Load diff

612
README.md
View file

@ -1,519 +1,195 @@
# General Bots 6 (GB6) Platform ### Key Facts
- LLM Orchestrator AGPL licensed (to use as custom-label SaaS, contributing back)
- True community governance
- No single corporate control
- 5+ years of stability
- Never changed license
- Enterprise-grad
- Hosted locally or Multicloud
## Vision ## Contributors
GB6 is a billion-scale real-time communication platform integrating advanced bot capabilities, WebRTC multimedia, and enterprise-grade messaging, built with Rust for maximum performance and reliability and BASIC-WebAssembly VM.
## 🌟 Key Features <a href="https://github.com/generalbots/botserver/graphs/contributors">
<img src="https://contrib.rocks/image?repo=generalbots/botserver" />
</a>
### Scale & Performance ## Overview
- Billion+ active users support
- Sub-second message delivery
- 4K video streaming
- 99.99% uptime guarantee
- Zero message loss
- Petabyte-scale storage
### Core Services | Area | Status |
- **API Service** (gb-server) |------------------------------|----------------------------------------------------------------------------------------------------|
- Axum-based REST & WebSocket | Releases | [![General Bots](https://img.shields.io/npm/dt/botserver.svg?logo=npm&label=botserver)](https://www.npmjs.com/package/botserver/) [![.gbapp lib](https://img.shields.io/npm/dt/botlib.svg?logo=npm&label=botlib)](https://www.npmjs.com/package/botlib/) [![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)|
- Multi-tenant request routing | Community | [![StackExchange](https://img.shields.io/stackexchange/stackoverflow/t/generalbots.svg)](https://stackoverflow.com/search?q=%23generalbots&s=966e24e7-4f7a-46ee-b159-79d643d6b74a) [![Open-source](https://badges.frapsoft.com/os/v2/open-source.svg)](https://badges.frapsoft.com) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com) [![License](https://img.shields.io/badge/license-AGPL-blue.svg)](https://github.com/GeneralBots/BotServer/blob/master/LICENSE.txt)|
- Authentication & Authorization | Management | [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://gitHub.com/GeneralBots/BotServer/graphs/commit-activity) |
- File handling & streaming | Security | [![Known Vulnerabilities](https://snyk.io/test/github/GeneralBots/BotServer/badge.svg)](https://snyk.io/test/github/GeneralBots/BotServer) |
| Building & Quality | [![Coverage Status](https://coveralls.io/repos/github/GeneralBots/BotServer/badge.svg)](https://coveralls.io/github/GeneralBots/BotServer) [![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier) |
| Packaging | [![forthebadge](https://badge.fury.io/js/botserver.svg)](https://badge.fury.io) [![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/) |
| Samples | [BASIC](https://github.com/GeneralBots/BotServer/tree/master/packages/default.gbdialog) or [![TypeScript](https://badges.frapsoft.com/typescript/code/typescript.svg?v=101)](https://github.com/GeneralBots/AzureADPasswordReset.gbapp)
| [Docker Image](https://github.com/lpicanco/docker-botserver) ![Docker Pulls](https://img.shields.io/docker/pulls/lpicanco/botserver.svg) <br/> *Provided by [@lpicanco](https://github.com/lpicanco/docker-botserver)* |
- **Media Processing** (gb-media) # General Bots
- WebRTC integration
- GStreamer transcoding
- Real-time track management
- Professional recording
- **Messaging** (gb-messaging) ![General Bot Logo](https://github.com/GeneralBots/BotServer/blob/main/logo.png?raw=true))
- Kafka event processing
- RabbitMQ integration
- WebSocket communication
- Redis PubSub
- **Storage** (gb-storage) General Bot is a strongly typed LLM conversational platform package based chat bot server focused in convention over configuration and code-less approaches, which brings software packages and application server concepts to help parallel bot development.
- PostgreSQL with sharding
- Redis caching
- TiKV distributed storage
## 🏗 Architecture ## What is a Bot Server?
### Multi-Tenant Core Bot Server accelerates the process of developing a bot. It provisions all code
- Organizations base, resources and deployment to the cloud, and gives you templates you can
- Instance management choose from whenever you need a new bot. The server has a database and service
- Resource quotas backend allowing you to further modify your bot package directly by downloading
- Usage analytics a zip file, editing and uploading it back to the server (deploying process) with
no code. The Bot Server also provides a framework to develop bot packages in a more
advanced fashion writing custom code in editors like Visual Studio Code, Atom or Brackets.
### Communication Infrastructure Everyone can create bots by just copying and pasting some files and using their
- WebRTC rooms favorite tools from Office (or any text editor) or Photoshop (or any image
- Real-time messaging editor). LLM and BASIC can be mixed used to build custom dialogs so Bot can be extended just like VBA for Excel.
- Media processing
- Video conferencing
## 🛠 Installation ## Getting Started
### Prerequisites ### Prerequisites
- Rust 1.70+
- Kubernetes cluster
- PostgreSQL 13+
- Redis 6+
- Kafka 3.0+
- GStreamer
# Deploy platform Before you embark on your General Bots journey, ensure you have the following tools installed:
- **Node.js (version 20 or later)**: General Bots leverages the latest features of Node.js to provide a robust and efficient runtime environment. Download it from [nodejs.org](https://nodejs.org/en/download/).
- **Git (latest stable version)**: Essential for version control and collaborating on bot projects. Get it from [git-scm.com](https://git-scm.com/downloads).
## Linux && Mac ### Quick Start Guide
```
sudo apt update
sudo apt install brave-browser-beta Follow these steps to get your General Bots server up and running:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y 1. Clone the repository:
source "$HOME/.cargo/env" ```bash
git clone https://alm.pragmatismo.com.br/generalbots/gbserver git clone https://github.com/GeneralBots/BotServer
```
This command creates a local copy of the General Bots server repository on your machine.
apt install -y build-essential \ 2. Navigate to the project directory:
pkg-config \ ```bash
libssl-dev \ cd BotServer
gcc-multilib \ ```
g++-multilib \ This changes your current directory to the newly cloned BotServer folder.
clang \
lld \
binutils-dev \
libudev-dev \
libdbus-1-dev
```
## Build 3. Install dependencies and start the server:
```bash
npm install
npm run start
```
The `npm install` command installs all necessary dependencies for the project. `npm run start` builds your bot server locally and serves it through a development server.
### Accessing Your Bot
Once the server is running, you can access your bot at `http://localhost:4242/`. This local server allows you to interact with your bot and test its functionality in real-time. If you want to publish
without password, define [ADMIN_OPEN_PUBLISH](https://github.com/GeneralBots/BotBook/master/docs/chapter-07-gbot-reference#enviroment-variables-reference) as true in BotServer .env file.
To publish bot packages and initiate a conversation with the bot, use the command:
``` ```
/publish
```
This command prepares your bot packages for use and allows you to start interacting with your bot immediately.
### Build & Run ## Development Workflow
```bash
# Build all services
cargo build --workspace
# Run tests ### 1. Project Structure
cargo test --workspace
# Start API service The General Bots server follows a modular architecture designed for flexibility and scalability. Here's an overview of the main directories:
cargo run -p gb-server
```
BotServer/
├── packages/
│ ├── core.gbapp/ # Core bot functionality
│ ├── kb.gbapp/ # Knowledge base packages
├── src / # Main entry point
└── package.json # Project configuration
``` ```
## 📊 Monitoring & Operations This structure allows for easy navigation and management of different aspects of your bot project.
### Health Metrics ### 2. Creating Custom Packages
- System performance
- Resource utilization
- Error rates
- Latency tracking
### Scaling Operations One of the strengths of General Bots is its extensibility. You can create custom packages to enhance your bot's capabilities:
- Auto-scaling rules
- Shard management
- Load balancing
- Failover systems
## 🔒 Security - **.gbkb (Knowledge Base packages)**: Store and manage your bot's knowledge and responses.
- **.gbtheme (Theme packages)**: Customize the visual appearance of your bot interface.
- **.gbapp (Application packages)**: Add new features and functionalities to your bot.
### Authentication & Authorization Each package type has its own structure and purpose, which we'll explore in depth in their respective chapters.
- Multi-factor auth
- Role-based access
- Rate limiting
- End-to-end encryption
### Data Protection ## Samples
- Tenant isolation
- Encryption at rest
- Secure communications
- Audit logging
## 🚀 Development Several samples, including a Bot for AD Password Reset, are avaiable on the [repository list](https://github.com/GeneralBots).
### Using complete General Bots Conversational Data Analytics
![](https://user-images.githubusercontent.com/14840374/178154826-8188029e-b4f4-48aa-bc0d-126307ce5121.png)
### Project Structure
``` ```
general-bots/ TALK "General Bots Labs presents FISCAL DATA SHOW BY BASIC"
├── gb-server/ # API service
├── gb-core/ # Core functionality TALK "Gift Contributions to Reduce the Public Debt API (https://fiscaldata.treasury.gov/datasets/gift-contributions-reduce-debt-held-by-public/gift-contributions-to-reduce-the-public-debt)"
├── gb-media/ # Media processing
├── gb-messaging/ # Message brokers result = GET "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v2/accounting/od/gift_contributions?page[size]=500"
├── gb-storage/ # Data storage data = result.data
├── gb-utils/ # Utilities data = SELECT YEAR(record_date) as Yr, SUM(CAST(contribution_amt AS NUMBER)) AS Amount FROM data GROUP BY YEAR(record_date)
└── migrations/ # DB migrations
TALK "Demonstration of Gift Contributions with AS IMAGE keyword"
SET THEME dark
png = data as IMAGE
SEND FILE png
DELAY 5
TALK " Demonstration of Gift Contributions CHART keyword"
img = CHART "bar", data
SEND FILE img
``` ```
### Configuration ## Guide
```env
DATABASE_URL=postgresql://user:password@localhost:5432/gbdb
REDIS_URL=redis://localhost:6379
KAFKA_BROKERS=localhost:9092
RABBIT_URL=amqp://guest:guest@localhost:5672
```
## 🌍 Deployment [Read the General Bots BotBook Guide](https://docs.pragmatismo.com.br)
### Global Infrastructure # Videos
- Edge presence
- Regional optimization
- Content delivery
- Traffic management
### Disaster Recovery 7 AI General Bots LLM Templates for Goodness
- Automated backups [https://www.youtube.com/watch?v=KJgvUPXi3Fw](https://www.youtube.com/watch?v=KJgvUPXi3Fw)
- Multi-region failover
- Data replication
- System redundancy
## 🤝 Contributing # Contributing
1. Fork repository This project welcomes contributions and suggestions.
2. Create feature branch See our [Contribution Guidelines](https://github.com/pragmatismo-io/BotServer/blob/master/CONTRIBUTING.md) for more details.
3. Implement changes
4. Add tests
5. Submit PR
## 📝 License # Reporting Security Issues
Licensed under terms specified in workspace configuration. Security issues and bugs should be reported privately, via email, to the pragmatismo.com.br Security
team at [security@pragmatismo.com.br](mailto:security@pragmatismo.com.br). You should
receive a response within 24 hours. If for some reason you do not, please follow up via
email to ensure we received your original message.
## 🆘 Support # License & Warranty
### Issues General Bot Copyright (c) pragmatismo.com.br. All rights reserved.
- Check existing issues Licensed under the AGPL-3.0.
- Provide reproduction steps
- Include relevant logs
- Follow up on discussions
### Documentation According to our dual licensing model, this program can be used either
- API references under the terms of the GNU Affero General Public License, version 3,
- Integration guides or under a proprietary license.
- Deployment docs
- Best practices
## 🔮 Roadmap The texts of the GNU Affero General Public License with an additional
permission and of our proprietary license can be found at and
in the LICENSE file you have received along with this program.
### Short Term This program is distributed in the hope that it will be useful,
- Enhanced media processing but WITHOUT ANY WARRANTY; without even the implied warranty of
- Additional messaging protocols MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- Improved scalability GNU Affero General Public License for more details.
- Extended monitoring
### Long Term "General Bot" is a registered trademark of pragmatismo.com.br.
- Advanced analytics The licensing of the program under the AGPLv3 does not imply a
- Global expansion trademark license. Therefore any rights, title and interest in
- Enterprise features our trademarks remain entirely with us.
<a href="https://stackoverflow.com/questions/ask?tags=generalbots">:speech_balloon: Ask a question</a> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <a href="https://github.com/GeneralBots/BotBook">:book: Read the Docs</a>
| ✓ | Requirement | Component | Standard | Implementation Steps | Team pictures made with [contrib.rocks](https://contrib.rocks).
|---|-------------|-----------|-----------|---------------------| General Bots Code Name is [Guaribas](https://en.wikipedia.org/wiki/Guaribas), the name of a city in Brazil, state of Piaui.
| ✅ | TLS 1.3 Configuration | Nginx | All | Configure modern SSL parameters and ciphers in `/etc/nginx/conf.d/ssl.conf` | [Roberto Mangabeira Unger](http://www.robertounger.com/en/): "No one should have to do work that can be done by a machine".
| ✅ | Access Logging | Nginx | All | Enable detailed access logs with privacy fields in `/etc/nginx/nginx.conf` |
| ⬜ | Rate Limiting | Nginx | ISO 27001 | Implement rate limiting rules in location blocks |
| ⬜ | WAF Rules | Nginx | HIPAA | Install and configure ModSecurity with OWASP rules |
| ✅ | Reverse Proxy Security | Nginx | All | Configure security headers (X-Frame-Options, HSTS, CSP) |
| ✅ | MFA Implementation | Zitadel | All | Enable and enforce MFA for all administrative accounts |
| ✅ | RBAC Configuration | Zitadel | All | Set up role-based access control with least privilege |
| ✅ | Password Policy | Zitadel | All | Configure strong password requirements (length, complexity, history) |
| ✅ | OAuth2/OIDC Setup | Zitadel | ISO 27001 | Configure secure OAuth flows and token policies |
| ✅ | Audit Logging | Zitadel | All | Enable comprehensive audit logging for user activities |
| ✅ | Encryption at Rest | MinIO | All | Configure encrypted storage with key management |
| ✅ | Bucket Policies | MinIO | All | Implement strict bucket access policies |
| ✅ | Object Versioning | MinIO | HIPAA | Enable versioning for data recovery capability |
| ✅ | Access Logging | MinIO | All | Enable detailed access logging for object operations |
| ⬜ | Lifecycle Rules | MinIO | LGPD | Configure data retention and deletion policies |
| ✅ | DKIM/SPF/DMARC | Stalwart | All | Configure email authentication mechanisms |
| ✅ | Mail Encryption | Stalwart | All | Enable TLS for mail transport |
| ✅ | Content Filtering | Stalwart | All | Implement content scanning and filtering rules |
| ⬜ | Mail Archiving | Stalwart | HIPAA | Configure compliant email archiving |
| ✅ | Sieve Filtering | Stalwart | All | Implement security-focused mail filtering rules |
| ⬜ | System Hardening | Ubuntu | All | Apply CIS Ubuntu Linux benchmarks |
| ✅ | System Updates | Ubuntu | All | Configure unattended-upgrades for security patches |
| ⬜ | Audit Daemon | Ubuntu | All | Configure auditd for system event logging |
| ✅ | Firewall Rules | Ubuntu | All | Configure UFW with restrictive rules |
| ⬜ | Disk Encryption | Ubuntu | All | Implement LUKS encryption for system disks |
| ⬜ | SELinux/AppArmor | Ubuntu | All | Enable and configure mandatory access control |
| ✅ | Monitoring Setup | All | All | Install and configure Prometheus + Grafana |
| ✅ | Log Aggregation | All | All | Implement centralized logging (e.g., ELK Stack) |
| ⬜ | Backup System | All | All | Configure automated backup system with encryption |
| ✅ | Network Isolation | All | All | Implement proper network segmentation |
| ✅ | Data Classification | All | HIPAA/LGPD | Document data types and handling procedures |
| ✅ | Session Management | Zitadel | All | Configure secure session timeouts and invalidation |
| ✅ | Certificate Management | All | All | Implement automated certificate renewal with Let's Encrypt |
| ✅ | Vulnerability Scanning | All | ISO 27001 | Regular automated scanning with tools like OpenVAS |
| ✅ | Incident Response Plan | All | All | Document and test incident response procedures |
| ✅ | Disaster Recovery | All | HIPAA | Implement and test disaster recovery procedures |
## Documentation Requirements
1. **Security Policies**
- Information Security Policy
- Access Control Policy
- Password Policy
- Data Protection Policy
- Incident Response Plan
2. **Procedures**
- Backup and Recovery Procedures
- Change Management Procedures
- Access Review Procedures
- Security Incident Procedures
- Data Breach Response Procedures
3. **Technical Documentation**
- Network Architecture Diagrams
- System Configuration Documentation
- Security Controls Documentation
- Encryption Standards Documentation
- Logging and Monitoring Documentation
4. **Compliance Records**
- Risk Assessment Reports
- Audit Logs
- Training Records
- Incident Reports
- Access Review Records
## Regular Maintenance Tasks
- Weekly security updates
- Monthly access reviews
- Quarterly compliance audits
- Annual penetration testing
- Bi-annual disaster recovery testing
### **Key Open Source Tools in Rust/Go**:
1. **Zitadel (Go)**: Identity and access management for secure authentication.
2. **Stalwart (Rust)**: Secure email server for threat detection.
3. **MinIO (Go)**: High-performance object storage for unstructured data.
4. **Ubuntu Advantage (Go/Rust tools)**: Compliance and security tools for Ubuntu.
5. **Tantivy (Rust)**: Full-text search engine for data discovery.
6. **Drone (Go)**: CI/CD platform for DevOps automation.
7. **Temporal (Go)**: Workflow orchestration engine.
8. **Caddy (Go)**: Web server for seamless customer experiences.
9. **SeaweedFS (Go)**: Distributed file system for secure file sharing.
10. **Vector (Rust)**: Observability pipeline for monitoring.
11. **Tyk (Go)**: API gateway for secure API management.
12. **Vault (Go)**: Secrets management and encryption.
13. **Hugging Face Transformers (Rust/Go bindings)**: LLM integration and fine-tuning.
14. **Kubernetes (Go)**: Container orchestration for scalable deployments.
15. **Matrix (Rust)**: Real-time communication and collaboration.
# API:
## **File & Document Management**
/files/upload
/files/download
/files/copy
/files/move
/files/delete
/files/getContents
/files/save
/files/createFolder
/files/shareFolder
/files/dirFolder
/files/list
/files/search
/files/recent
/files/favorite
/files/versions
/files/restore
/files/permissions
/files/quota
/files/shared
/files/sync/status
/files/sync/start
/files/sync/stop
---
### **Document Processing**
/docs/merge
/docs/convert
/docs/fill
/docs/export
/docs/import
---
### **Groups & Organizations**
/groups/create
/groups/update
/groups/delete
/groups/list
/groups/search
/groups/members
/groups/members/add
/groups/members/remove
/groups/permissions
/groups/settings
/groups/analytics
/groups/join/request
/groups/join/approve
/groups/join/reject
/groups/invites/send
/groups/invites/list
---
### **Conversations & Real-time Communication**
/conversations/create
/conversations/join
/conversations/leave
/conversations/members
/conversations/messages
/conversations/messages/send
/conversations/messages/edit
/conversations/messages/delete
/conversations/messages/react
/conversations/messages/pin
/conversations/messages/search
/conversations/calls/start
/conversations/calls/join
/conversations/calls/leave
/conversations/calls/mute
/conversations/calls/unmute
/conversations/screen/share
/conversations/screen/stop
/conversations/recording/start
/conversations/recording/stop
/conversations/whiteboard/create
/conversations/whiteboard/collaborate
---
### **Communication Services**
/comm/email/send
/comm/email/template
/comm/email/schedule
/comm/email/cancel
/comm/sms/send
/comm/sms/bulk
/comm/notifications/send
/comm/notifications/preferences
/comm/broadcast/send
/comm/contacts/import
/comm/contacts/export
/comm/contacts/sync
/comm/contacts/groups
---
### **User Management & Authentication**
/users/create
/users/update
/users/delete
/users/list
/users/search
/users/profile
/users/profile/update
/users/settings
/users/permissions
/users/roles
/users/status
/users/presence
/users/activity
/users/security/2fa/enable
/users/security/2fa/disable
/users/security/devices
/users/security/sessions
/users/notifications/settings
---
### **Calendar & Task Management**
/calendar/events/create
/calendar/events/update
/calendar/events/delete
/calendar/events/list
/calendar/events/search
/calendar/availability/check
/calendar/schedule/meeting
/calendar/reminders/set
/tasks/create
/tasks/update
/tasks/delete
/tasks/list
/tasks/assign
/tasks/status/update
/tasks/priority/set
/tasks/dependencies/set
---
### **Storage & Data Management**
/storage/save
/storage/batch
/storage/json
/storage/delete
/storage/quota/check
/storage/cleanup
/storage/backup/create
/storage/backup/restore
/storage/archive
/storage/metrics
---
### **Analytics & Reporting**
/analytics/dashboard
/analytics/reports/generate
/analytics/reports/schedule
/analytics/metrics/collect
/analytics/insights/generate
/analytics/trends/analyze
/analytics/export
---
### **System & Administration**
/admin/system/status
/admin/system/metrics
/admin/logs/view
/admin/logs/export
/admin/config/update
/admin/maintenance/schedule
/admin/backup/create
/admin/backup/restore
/admin/users/manage
/admin/roles/manage
/admin/quotas/manage
/admin/licenses/manage
---
### **AI & Machine Learning**
/ai/analyze/text
/ai/analyze/image
/ai/generate/text
/ai/generate/image
/ai/translate
/ai/summarize
/ai/recommend
/ai/train/model
/ai/predict
---
### **Security & Compliance**
/security/audit/logs
/security/compliance/check
/security/threats/scan
/security/access/review
/security/encryption/manage
/security/certificates/manage
---
### **Health & Monitoring**
/health
/health/detailed
/monitoring/status
/monitoring/alerts
/monitoring/metrics
Built with ❤️ from Brazil, using Rust for maximum performance and reliability.