# cloud-disk

<p align="center">
  <a href="https://github.com/<org>/cloud-disk/blob/main/LICENSE">
    <img src="https://img.shields.io/badge/License-MIT-E11311.svg" alt="MIT License">
  </a>
  <a href="#">
    <img src="https://img.shields.io/badge/Java-21-brightgreen" alt="Java 21">
  </a>
  <a href="#">
    <img src="https://img.shields.io/badge/Spring_Boot-3.3.7-brightgreen" alt="Spring Boot 3.3.7">
  </a>
  <a href="#">
    <img src="https://img.shields.io/badge/Spring_Cloud-2023.0.3-blue" alt="Spring Cloud 2023.0.3">
  </a>
  <br/>
  <a href="./README.md"><img alt="简体中文" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
  <a href="./README_EN.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
</p>

---

**S3-compatible object storage service** built with Spring Cloud Alibaba microservices. Features pluggable multi-backend storage (MinIO / Local / Alibaba OSS), instant upload via MD5 dedup, multipart upload, recycle bin, reactive non-blocking Gateway, and 9 JMeter benchmark scenarios.

---

## ✨ Core Features

- **[S3 API Compatible](docs/deployment.md#6-完整-api-接口文档)** — PUT/GET/DELETE Bucket & Object, Multipart Upload, ListObjectsV2, CopyObject. Works with aws-cli, s3cmd, and other S3 tools.
- **[Pluggable Storage Backends](docs/architecture.md#51-为什么使用标准-s3-协议而非-s3-express)** — MinIO / Local filesystem / Alibaba OSS. Each bucket independently selects its storage type. Add new backends by implementing the `StorageBackend` interface.
- **[Instant Upload](docs/architecture.md#22-秒传instant-upload流程)** — MD5 + size dedup via Redis, O(1) lookup. 1GB re-upload completes in ~15ms, a 4,666x speedup.
- **[Multipart Upload](docs/architecture.md#21-分片上传全流程)** — UUID v7 upload IDs (time-ordered, avoids B+tree page splits). Parts state tracked in Redis. MinIO server-side composition via `composeObject`.
- **[Soft Delete & Recycle Bin](docs/architecture.md#53-为什么使用软删除)** — `NORMAL → RECYCLED (30d) → DELETED` state machine. XXL-Job scheduled cleanup at 3am daily.
- **[Reactive Non-blocking Gateway](docs/architecture.md#57-gateway-全链路非阻塞)** — Spring Cloud Gateway + `RedissonReactiveClient`. Prevents Netty event loop deadlock from blocking Redis calls. 2,000-concurrency spike test at 0% error rate.
- **[Rate Limiting & Circuit Breaking](docs/nacos-config.md)** — Sentinel Gateway flow control with Nacos dynamic data source. Rule hot-reload, no restart required.
- **[Outbox Pattern](docs/architecture.md#54-为什么使用-outbox-模式)** — At-least-once event delivery from data plane to control plane via RabbitMQ. XXL-Job retry with idempotent consumers.
- **[Redis Cluster Hot-Key Sharding](docs/architecture.md#58-hotkeyutil-热点打散)** — `HotKeyUtil` 3-shard hash-tag distribution. Reads and writes spread evenly across 3 master nodes, deviation < 20%.
- **[Database Sharding](docs/architecture.md#3-数据库设计)** — ShardingSphere 5.5 read/write split + `object_info` sharded by `bucket_id % 2`.

---

## 🏗 Architecture

```
                            ┌──────────────────────────────────────────┐
                            │       Spring Cloud Gateway (38080)       │
                            │    Nacos Discovery + Sentinel + JWT      │
                            └──────┬────────────────────┬─────────────┘
                                   │                    │
                        ┌──────────▼──────────┐ ┌───────▼──────────┐
                        │   object-storage    │ │  object-metadata │
                        │      (38081)        │ │     (38082)      │
                        │                     │ │                  │
                        │  ┌───────────────┐  │ │  ┌────────────┐  │
                        │  │StorageBackend │  │ │  │   MySQL    │  │
                        │  │ MinIO/Local   │  │ │  │ (metadata) │  │
                        │  │ /OSS (data)   │  │ │  └────────────┘  │
                        │  └───────────────┘  │ │  ┌────────────┐  │
                        │  ┌───────────────┐  │ │  │   Redis    │  │
                        │  │Redis Cluster  │  │ │  │  Cluster   │  │
                        │  │(instant/parts)│  │ │  │  (cache)   │  │
                        │  └───────────────┘  │ │  └────────────┘  │
                        │  ┌───────────────┐  │ │                  │
                        │  │  RabbitMQ     │──┼─┼──> consumer     │
                        │  │  (Outbox)     │  │ │                  │
                        │  └───────────────┘  │ │                  │
                        └─────────────────────┘ └──────────────────┘

 Infrastructure: Nacos(38848)  Sentinel(38858)  MySQL(33306/33307)
                 Redis Cluster(37000-37005)  RabbitMQ(35672)  MinIO(39000/39001)  XXL-Job(39080)
```

| Module | Role | Key Tech |
|--------|------|----------|
| `gateway` | API Gateway | Spring Cloud Gateway + JWT + Sentinel + RedissonReactiveClient |
| `object-storage` | Data Plane | StorageBackend (MinIO/Local/OSS) + Redis + RabbitMQ Outbox |
| `object-metadata` | Control Plane | MySQL + MyBatis-Plus + Redis Cache + RabbitMQ Consumer |
| `admin-ui` | Admin Dashboard | React 19 + Vite + shadcn/ui + i18n |
| `common` | Shared Library | DTOs, S3ErrorCode, StorageBackend Interface, HotKeyUtil |

## 🖥 Preview

| My Files | Bucket Browser |
|----------|---------------|
| ![My Files](img/我的文件.png) | ![test-bucket](img/test-bucket.png) |

---

## 🚀 Quick Start

### Prerequisites

- Docker & Docker Compose (2.20+)
- Java 21 & Maven 3.9+ (for building from source only; runtime is containerized)

### 1. Clone

```bash
git clone https://github.com/<org>/cloud-disk.git
cd cloud-disk
```

### 2. Build

```bash
echo "JAVA_HOME=/path/to/jdk-21" > .env
make package-fast
```

### 3. Start all services

```bash
docker compose up -d
```

Initial startup takes ~2-3 minutes. Wait for all containers to be healthy:

```bash
docker compose ps
```

### 4. Verify

```bash
curl -s -w "\nHTTP:%{http_code}" -X POST http://localhost:38080/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"123456"}'
```

### Management Consoles

| Console | URL | Credentials |
|---------|-----|-------------|
| Nacos | http://localhost:38848/nacos | `nacos` / `nacos` |
| MinIO | http://localhost:39001 | `minioadmin` / `minioadmin` |
| RabbitMQ | http://localhost:45672 | `admin` / `admin123` |
| Sentinel | http://localhost:38858 | `sentinel` / `sentinel` |
| XXL-Job | http://localhost:39080/xxl-job-admin | `admin` / `123456` |
| Admin UI | http://localhost:5173 | `admin` / `123456` |

---

## 📖 API Quick Example

All requests go through Gateway at `http://localhost:38080`. Full API docs: [docs/deployment.md](docs/deployment.md#6-完整-api-接口文档)

```bash
TOKEN="<from-login>"

# Create a bucket
curl -X PUT http://localhost:38080/my-bucket \
  -H "Authorization: Bearer $TOKEN"

# Upload an object
curl -X PUT http://localhost:38080/my-bucket/hello.txt \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/plain" \
  -d "Hello World!"

# Instant upload (MD5 dedup)
MD5=$(md5sum large-file.bin | cut -d' ' -f1)
curl -X PUT http://localhost:38080/my-bucket/large-file.bin \
  -H "Authorization: Bearer $TOKEN" \
  -H "x-cloud-disk-md5: $MD5" \
  -T large-file.bin
# Response: { "data": { "instant": true } } if file already exists

# Download (default: 302 redirect to pre-signed URL; use -L to follow)
curl -L -X GET http://localhost:38080/my-bucket/hello.txt \
  -H "Authorization: Bearer $TOKEN" -o hello.txt

# Force proxy mode (data flows through Gateway)
curl -X GET "http://localhost:38080/my-bucket/hello.txt?proxy=true" \
  -H "Authorization: Bearer $TOKEN" -o hello.txt

# List objects (S3 ListObjectsV2)
curl "http://localhost:38080/my-bucket?list-type=2&prefix=images/" \
  -H "Authorization: Bearer $TOKEN"
```

---

## 📋 S3 API Compatibility

| Feature | Status |
|---------|:------:|
| PUT / DELETE / HEAD Bucket | Y |
| GET Bucket (ListObjectsV2) | Y |
| PUT / GET / HEAD / DELETE Object | Y |
| Delete Multiple Objects | Y |
| CopyObject | Y |
| Multipart Upload (Init / Part / Complete / Abort) | Y |
| List Parts | Y |
| Pre-signed URL | Y |
| Bucket Policy / Object Lock / Versioning | N |

---

## 📚 Documentation

| Document | Content |
|----------|---------|
| [docs/architecture.md](docs/architecture.md) | Architecture design, data flows, database schema, design decisions |
| [docs/deployment.md](docs/deployment.md) | Deployment guide, configuration reference, full API docs, troubleshooting |
| [docs/development.md](docs/development.md) | Development environment, project structure, coding conventions, testing |
| [docs/nacos-config.md](docs/nacos-config.md) | Nacos dynamic config for Sentinel rules |
| [benchmark/README.md](benchmark/README.md) | JMeter benchmark suite (9 scenarios, 2,000-concurrency spike at 0% error) |

---

## 📄 License

[MIT](LICENSE)