Storage Migration

Migrate your documents between storage providers with zero downtime.

Overview

When upgrading or changing storage providers, Archivus provides a seamless migration process that moves all your existing documents to the new storage location without interrupting your workflow.

Key Features:

  • Zero-downtime migration
  • Automatic file verification
  • Progress tracking
  • Rollback capability
  • Detailed error reporting

Supported Migration Paths

From To Notes
Shared Dedicated Upgrade to isolated bucket
Shared BYOB Move to customer-owned storage
Dedicated BYOB Full data sovereignty
BYOB Dedicated Return to Archivus-managed
Dedicated Shared Downgrade (data preserved)

Migration Process

Phase 1: Preparation

  1. Configuration - New storage is configured and validated
  2. Inventory - All existing files are catalogued
  3. Capacity Check - Destination storage is verified to have sufficient space

Phase 2: Migration

  1. Dual-Write Mode - New uploads go to both old and new storage
  2. Background Copy - Existing files are copied to new storage
  3. Verification - Each file is verified after transfer (checksum)
  4. Progress Tracking - Real-time progress updates

Phase 3: Cutover

  1. Switch - All operations move to new storage
  2. Verification - Final consistency check
  3. Cleanup - Old storage files marked for deletion

Phase 4: Completion

  1. Old Storage Cleanup - Files removed from previous storage (after retention period)
  2. Audit Log - Complete migration record saved
  3. Notification - Migration completion confirmation

Starting a Migration

  1. Contact support@archivusdms.com
  2. Specify current and desired storage type
  3. Provide any configuration details for new storage
  4. Coordinate timing if needed

Via API (Admin Only) 

curl -X POST https://api.archivus.app/api/v1/admin/storage/config/{tenant_id}/migrate \
  -H "Authorization: Bearer YOUR_ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_provider": "dedicated"
  }'

Response:

{
  "id": "mig_abc123",
  "tenant_id": "550e8400-e29b-41d4-a716-446655440000",
  "source_provider": "shared",
  "target_provider": "dedicated",
  "status": "pending",
  "total_files": 1250,
  "total_bytes": 5368709120,
  "migrated_files": 0,
  "migrated_bytes": 0,
  "failed_files": 0,
  "skipped_files": 0,
  "started_at": "2024-01-15T10:30:00Z",
  "initiated_by_email": "admin@example.com"
}

Monitoring Progress

In the Dashboard

Navigate to Settings > Storage to view:

  • Overall progress percentage
  • Files migrated vs. total
  • Data transferred
  • Estimated time remaining
  • Any errors encountered

Via API 

curl -X GET https://api.archivus.app/api/v1/admin/storage/migrations/{tenant_id} \
  -H "Authorization: Bearer YOUR_ADMIN_API_KEY"

Response:

{
  "id": "mig_abc123",
  "status": "in_progress",
  "progress_percent": 45.5,
  "total_files": 1250,
  "migrated_files": 568,
  "failed_files": 2,
  "skipped_files": 5,
  "total_bytes": 5368709120,
  "migrated_bytes": 2442000000,
  "current_file": "documents/reports/2024/quarterly-report.pdf",
  "started_at": "2024-01-15T10:30:00Z",
  "estimated_completion": "2024-01-15T11:15:00Z"
}

Migration Statuses

Status Description
pending Migration created, waiting to start
in_progress Actively migrating files
completed All files successfully migrated
completed_with_errors Migration finished with some failures
failed Migration encountered critical error
cancelled Migration was cancelled

Handling Failures

Automatic Retry

Failed files are automatically retried up to 3 times with exponential backoff:

  • First retry: 1 minute
  • Second retry: 5 minutes
  • Third retry: 15 minutes

Manual Retry

After migration completes with errors, retry failed files:

curl -X POST https://api.archivus.app/api/v1/admin/storage/migrations/{migration_id}/retry \
  -H "Authorization: Bearer YOUR_ADMIN_API_KEY"

Viewing Failed Files 

curl -X GET https://api.archivus.app/api/v1/admin/storage/migrations/{migration_id}/failed \
  -H "Authorization: Bearer YOUR_ADMIN_API_KEY"

Response:

{
  "failed_files": [
    {
      "file_id": "doc_123",
      "file_path": "documents/corrupted-file.pdf",
      "error": "Source file checksum mismatch",
      "attempts": 3,
      "last_attempt": "2024-01-15T11:00:00Z"
    }
  ],
  "total_failed": 1
}

Cancelling a Migration

If needed, cancel an in-progress migration:

curl -X POST https://api.archivus.app/api/v1/admin/storage/migrations/{migration_id}/cancel \
  -H "Authorization: Bearer YOUR_ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Discovered configuration issue, need to reconfigure"
  }'

What happens when cancelled:

  • Migration stops immediately
  • Already-migrated files remain in new storage
  • New uploads continue to old storage
  • Can restart migration after fixing issues

Migration History

View past migrations:

curl -X GET https://api.archivus.app/api/v1/admin/storage/migrations/{tenant_id}/history \
  -H "Authorization: Bearer YOUR_ADMIN_API_KEY"

Response:

{
  "migrations": [
    {
      "id": "mig_abc123",
      "source_provider": "shared",
      "target_provider": "dedicated",
      "status": "completed",
      "total_files": 1250,
      "migrated_files": 1248,
      "failed_files": 2,
      "started_at": "2024-01-15T10:30:00Z",
      "completed_at": "2024-01-15T11:20:00Z"
    }
  ],
  "count": 1
}

Performance Considerations

Migration Speed

Speed depends on:

  • Number of files
  • Total data size
  • Network bandwidth between regions
  • File sizes (many small files slower than few large files)

Estimated Times

Data Size Approximate Time
1 GB 5-10 minutes
10 GB 30-60 minutes
100 GB 4-8 hours
1 TB 1-2 days

Times vary based on network conditions and file distribution.

Best Practices

  1. Schedule during off-peak hours - Reduces impact on normal operations
  2. Verify destination capacity - Ensure destination has 20% extra capacity
  3. Monitor progress - Check periodically for any issues
  4. Test with small dataset first - If possible, test migration with a subset

During Migration

While migration is in progress:

Operation Behavior
Upload new files Goes to new storage immediately
Download files Reads from wherever file currently exists
Delete files Deletes from both locations
Search/Browse Works normally
AI features Work normally

Users experience no interruption during migration.

After Migration

Verify Completion

  1. Check migration status shows “completed”
  2. Verify file counts match expectations
  3. Test accessing random files
  4. Confirm new uploads work correctly

Old Storage Cleanup

After successful migration:

  • Old storage files retained for 7 days (configurable)
  • After retention period, old files are permanently deleted
  • This provides rollback window if issues discovered

Update Any Integrations

If you have direct integrations with storage:

  • Update any custom backup scripts
  • Modify any direct S3 access configurations
  • Update monitoring dashboards

Troubleshooting

Migration Stuck at 0%

  • Check source storage is accessible
  • Verify destination configuration is valid
  • Review error logs in migration status

Slow Progress

  • Large files take longer
  • Many small files have per-file overhead
  • Cross-region migrations have network latency

Checksum Failures

Files failing checksum verification:

  • May indicate source file corruption
  • Network issues during transfer
  • Will be retried automatically

Permission Errors

For BYOB migrations:

  • Verify all required S3 permissions
  • Check bucket policy allows operations
  • Ensure credentials are not expired

FAQ

Can I use Archivus during migration?

Yes! Migration happens in the background with zero downtime. All features work normally.

What if migration fails partway through?

The system tracks progress. You can retry failed files or restart the migration without re-migrating successful files.

How long is data kept in old storage?

7 days by default after successful migration. Enterprise customers can customize this retention period.

Can I migrate back if there are issues?

Yes. Contact support to initiate a reverse migration. Your data is preserved in both locations during the retention period.

Does migration affect my storage quota?

During migration, only one copy counts toward your quota. After completion, only the new storage is counted.

API Reference Summary

Endpoint Method Description
/admin/storage/config/{tenant_id}/migrate POST Start migration
/admin/storage/migrations/{tenant_id} GET Get current migration status
/admin/storage/migrations/{tenant_id}/history GET Get migration history
/admin/storage/migrations/{migration_id}/cancel POST Cancel migration

Next Steps

Questions? Contact support@archivusdms.com