Add MongoDB→Neon migration scripts and V2 environment templates #4512

Originally written by @ccantynz-alt on GitHub.
Imported from https://github.com/ccantynz-alt/BookarideV2/pull/5

Summary

Add comprehensive migration tooling to move production data from MongoDB Atlas to Neon PostgreSQL, along with environment configuration templates for BookARide V2.

Key Changes

Migration Scripts

• migrate_real_data.py: Main migration tool that:

  • Connects to MongoDB Bookaride_db and Neon PostgreSQL
  • Cleans 1,200 seeded test bookings from Neon (reference numbers 10–1209)
  • Imports all 13 collections from MongoDB with automatic table creation
  • Supports dry-run mode (default) and live execution with --confirm flag
  • Handles JSON serialization of dates and MongoDB-specific fields
  • Provides detailed logging and final summary report

• verify_neon_data.py: Post-migration verification tool that:

  • Lists all tables with row counts
  • Flags empty tables that should have data
  • Shows sample records from key collections (bookings, users, drivers, admin_users)
  • Identifies missing expected tables
  • Generates a comprehensive verification report

• find_all_mongo_data.py: MongoDB cluster scanner that:

  • Scans multiple MongoDB Atlas clusters for data
  • Reports all databases and collections with document counts
  • Highlights key collections (bookings, drivers, users, etc.)
  • Shows sample fields from important records
  • Helps locate missing data across clusters

• clean_neon_test_data.py: Test data cleanup utility that:

  • Removes seeded test bookings by reference number range
  • Preserves real admin_users and imported data
  • Supports dry-run mode with --confirm for actual deletion
  • Reports before/after row counts

Environment Templates

• backend/.env.example: Backend configuration template with sections for:

  • Neon PostgreSQL connection
  • JWT authentication
  • Stripe payments
  • Mailgun email
  • Twilio SMS
  • Google APIs (Maps, OAuth)
  • Geoapify routing
  • Server settings

• frontend/.env.example: Frontend configuration template for:

  • Backend API URL
  • Google Maps API key

Utility Script

• sync-env-to-v2.sh: Bash script to sync shared environment variables from V1's .env to V2's GitHub repository secrets, supporting:
  • 13 shared configuration variables (Stripe, Mailgun, Twilio, Google, etc.)
  • Dry-run validation
  • GitHub CLI integration
  • Clear documentation of V2-only variables

Implementation Details

• All migration scripts use async/await for efficient database operations
• Automatic table creation with JSONB storage and GIN indexing
• Graceful error handling with detailed logging
• Support for documents without unique IDs (fallback to auto-increment)
• Comprehensive documentation in docstrings and usage comments
• Connection string examples and setup instructions included

https://claude.ai/code/session_019RWqiTR68paaqzS76ujBRG