Masked Database Dump System - Implementation Guide
The masked database dump system provides developers with a safe, anonymized copy of the production database for local development. This guide covers the complete implementation of the system from scratch.
Overview
The system consists of four main components:
- Build: Creates a masked dump of the database with sensitive data anonymized
- Upload: Stores the dump in a configured storage location
- Pull: Downloads and imports the dump to a local database
- Sanitize: Prepares the database for local development
System Flow
graph TD
A[Production Database] -->|Extract Schema| B[Schema Dump]
A -->|Extract Masked Data| C[Masked Data Dump]
B -->|Merge| D[Combined SQL Dump]
C -->|Merge| D
D -->|Compress| E[Gzipped Dump]
E -->|Upload| F[Storage S3/etc]
F -->|Download| G[Local Environment]
G -->|Extract & Import| H[Local Database]
H -->|Sanitize| I[Development Ready DB]
Dependencies
Required Packages
Add to your composer.json:
{
"require": {
"beyondcode/laravel-masked-dumper": "^1.0"
}
}
Install the package:
composer require beyondcode/laravel-masked-dumper
System Requirements
- MySQL/MariaDB database
gzipcommand available on system- Storage disk configuration (S3, local, etc.)
- Sufficient disk space for dump files
- PHP memory limit of at least 2GB for large databases
File Structure
Create the following files in your Laravel application:
app/
├── Console/Commands/
│ ├── BuildMaskedDumpCommand.php
│ ├── PullMaskedDump.php
│ ├── UploadMaskedDump.php
│ └── SanitizeLocalDatabase.php (use existing legacy version)
├── Support/
│ └── MaskedDump.php
config/
└── masked-dump.php
Implementation
1. Configuration File
Create config/masked-dump.php:
<?php
return [
'enabled' => env('MASKED_DUMP_ENABLED', true),
// Use a callable so the configuration remains serializable when caching
'default_definition' => [\App\Support\MaskedDump::class, 'define'],
// Storage disk for uploads/downloads
'storage_disk' => 'masked_dump',
// File paths
'local_path' => storage_path('masked-dump.sql'),
'compressed_path' => storage_path('masked-dump.sql.gz'),
// Performance settings
'memory_limit' => '2G',
'chunk_size' => 8192,
'data_dump_timeout' => 1800,
'gzip_timeout' => 600,
];
2. Storage Configuration
Add to config/filesystems.php:
'disks' => [
// ... existing disks
'masked_dump' => [
'driver' => 's3',
'key' => env('MASKED_DUMP_AWS_ACCESS_KEY_ID'),
'secret' => env('MASKED_DUMP_AWS_SECRET_ACCESS_KEY'),
'region' => env('MASKED_DUMP_AWS_DEFAULT_REGION', 'us-east-1'),
'bucket' => env('MASKED_DUMP_AWS_BUCKET'),
'url' => env('MASKED_DUMP_AWS_URL'),
'endpoint' => env('MASKED_DUMP_AWS_ENDPOINT'),
'use_path_style_endpoint' => env('MASKED_DUMP_AWS_USE_PATH_STYLE_ENDPOINT', false),
'throw' => false,
],
],
3. Masking Definition
Create app/Support/MaskedDump.php:
<?php
declare(strict_types=1);
namespace App\Support;
use BeyondCode\LaravelMaskedDumper\DumpSchema;
use BeyondCode\LaravelMaskedDumper\TableDefinitions\TableDefinition;
use Faker\Generator as Faker;
final class MaskedDump
{
/**
* Return the DumpSchema definition used for masked DB dumps.
*/
public static function define(): DumpSchema
{
return DumpSchema::define()
->allTables()
// Mask user sensitive data
->table('users', function (TableDefinition $table) {
// Get admin users to preserve their data
$adminUserIds = \DB::table('role_user')
->join('roles', 'roles.id', '=', 'role_user.role_id')
->whereIn('roles.name', ['admin', 'super-admin'])
->pluck('role_user.user_id')
->unique()->toArray();
$adminUserEmails = \DB::table('users')
->whereIn('id', $adminUserIds)
->pluck('email')
->unique()->toArray();
// Mask names
$table->replace('name', fn(Faker $faker) => $faker->name);
$table->replace('first_name', fn(Faker $faker) => $faker->firstName);
$table->replace('last_name', fn(Faker $faker) => $faker->lastName);
// Mask emails except admin users
$table->replace('email', function (Faker $faker, $originalValue) use ($adminUserEmails) {
return in_array($originalValue, $adminUserEmails)
? $originalValue
: $faker->safeEmail;
});
// Mask phone numbers except admin users
$adminUserPhones = \DB::table('users')
->whereIn('id', $adminUserIds)
->pluck('phone')
->unique()->toArray();
$table->replace('phone', function (Faker $faker, $originalValue) use ($adminUserPhones) {
return in_array($originalValue, $adminUserPhones)
? $originalValue
: $faker->phoneNumber;
});
// Generate new UIDs
$table->replace('uid', fn(Faker $faker) => $faker->uuid);
$table->outputInChunksOf(500);
})
// Limit historical data to reduce dump size
->table('activity_log', function (TableDefinition $table) {
$table->query(fn($query) => $query->where('created_at', '>=', now()->subMonth()));
$table->outputInChunksOf(500);
})
// Add more tables as needed...
->table('user_sessions', function (TableDefinition $table) {
$table->query(fn($query) => $query->where('created_at', '>=', now()->subWeeks(2)));
$table->outputInChunksOf(500);
})
// Mask encrypted field_progress value by nullifying it - will be re-encrypted on import
->table('field_progress', fn (TableDefinition $table) => $table->replace('value', fn () => null))
// Exclude tables with sensitive data that shouldn't be in dev
->table('admin_notifications', fn(TableDefinition $table) =>
$table->query(fn($query) => $query->whereRaw('1 = 0'))
)
->table('failed_jobs', fn(TableDefinition $table) =>
$table->query(fn($query) => $query->whereRaw('1 = 0'))
)
->table('notifications', fn(TableDefinition $table) =>
$table->query(fn($query) => $query->whereRaw('1 = 0'))
)
// Exclude monitoring/debugging tables
->exclude('telescope_entries')
->exclude('telescope_entries_tags')
->exclude('telescope_monitoring')
->exclude('pulse_entries')
->exclude('pulse_aggregates');
}
}
4. Build Command
Create app/Console/Commands/BuildMaskedDumpCommand.php:
<?php
declare(strict_types=1);
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Illuminate\Filesystem\Filesystem;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Process;
final class BuildMaskedDumpCommand extends Command
{
protected $signature = 'app:build-masked-dump';
protected $description = 'Builds a masked database dump if enabled in configuration';
private Filesystem $files;
public function __construct(Filesystem $files)
{
parent::__construct();
$this->files = $files;
}
public function handle(): int
{
// Guard: ensure masked dump is enabled
if (!config('masked-dump.enabled')) {
$this->error('Masked dump is disabled in configuration.');
return self::FAILURE;
}
$finalDumpPath = config('masked-dump.local_path');
$gzippedFinalDumpPath = config('masked-dump.compressed_path');
$tempDataDumpPath = storage_path('app/temp_masked_data.sql');
// Ensure target directory exists
$this->files->ensureDirectoryExists(dirname($finalDumpPath));
// Clean up old dumps
$this->files->delete([$finalDumpPath, $gzippedFinalDumpPath, $tempDataDumpPath]);
// Validate database driver
$connectionConfig = config('database.connections.' . config('database.default'));
if (($connectionConfig['driver'] ?? '') !== 'mysql') {
$this->error('Only MySQL connections are supported for masked dumps.');
return self::FAILURE;
}
if (!$this->dumpSchema($finalDumpPath)) {
return self::FAILURE;
}
if (!$this->runMaskedDump($tempDataDumpPath)) {
$this->files->delete([$finalDumpPath, $tempDataDumpPath]);
return self::FAILURE;
}
if (!$this->mergeDumps($finalDumpPath, $tempDataDumpPath)) {
$this->files->delete([$finalDumpPath, $tempDataDumpPath]);
return self::FAILURE;
}
if (!$this->gzipDump($finalDumpPath, $gzippedFinalDumpPath)) {
$this->files->delete([$finalDumpPath, $gzippedFinalDumpPath]);
return self::FAILURE;
}
// Upload the gzipped dump
$exitCode = $this->call('app:upload-masked-dump');
if ($exitCode !== self::SUCCESS) {
$this->error('Failed to upload masked dump.');
$this->files->delete([$gzippedFinalDumpPath]);
return self::FAILURE;
}
$this->info('Masked database dump process completed successfully.');
return self::SUCCESS;
}
private function dumpSchema(string $path): bool
{
try {
$this->info('Starting schema dump...');
$schemaSql = 'SET FOREIGN_KEY_CHECKS=0;' . PHP_EOL . PHP_EOL;
$tables = DB::connection(config('database.default'))->getSchemaBuilder()->getTableListing();
if (empty($tables)) {
$this->error('No tables found in the database.');
return false;
}
$bar = $this->output->createProgressBar(count($tables));
$bar->start();
foreach ($tables as $table) {
$bar->advance();
$result = DB::connection(config('database.default'))->select("SHOW CREATE TABLE `{$table}`");
if (!empty($result[0]->{'Create Table'})) {
$schemaSql .= $result[0]->{'Create Table'} . ';' . PHP_EOL . PHP_EOL;
} else {
$this->warn("Could not get CREATE TABLE for {$table}");
}
}
$bar->finish();
$this->info('');
$this->files->put($path, $schemaSql);
$this->info('Schema dump completed.');
return true;
} catch (\Throwable $e) {
$this->error('Schema dump failed: ' . $e->getMessage());
Log::error('Schema dump failed', ['exception' => $e]);
return false;
}
}
private function runMaskedDump(string $tempPath): bool
{
try {
$this->info('Running masked data dump...');
$cmd = [
'php',
'-d',
'memory_limit=' . config('masked-dump.memory_limit'),
'artisan',
'db:masked-dump',
$tempPath
];
if ($def = config('masked-dump.default_definition')) {
$cmd[] = '--definition=' . $def;
}
$result = Process::timeout(config('masked-dump.data_dump_timeout'))->run($cmd);
if (!$result->successful()) {
$this->error('Masked data dump failed: ' . $result->errorOutput());
Log::error('Masked data dump failed', ['output' => $result->errorOutput()]);
return false;
}
$this->info('Data dump completed.');
return true;
} catch (\Throwable $e) {
$this->error('Masked data dump exception: ' . $e->getMessage());
Log::error('Masked data dump exception', ['exception' => $e]);
return false;
}
}
private function mergeDumps(string $schemaPath, string $dataPath): bool
{
try {
$this->info('Merging dumps...');
$schemaHandle = fopen($schemaPath, 'r+');
$dataHandle = fopen($dataPath, 'r');
if (!$schemaHandle || !$dataHandle) {
throw new \RuntimeException('Failed to open dump files.');
}
fseek($schemaHandle, 0, SEEK_END);
fwrite($schemaHandle, PHP_EOL);
while (!feof($dataHandle)) {
fwrite($schemaHandle, fread($dataHandle, config('masked-dump.chunk_size')));
}
// Add foreign key checks reset at end
fwrite($schemaHandle, PHP_EOL . 'SET FOREIGN_KEY_CHECKS=1;' . PHP_EOL);
fclose($dataHandle);
fclose($schemaHandle);
$this->files->delete($dataPath);
$this->info('Merge completed.');
return true;
} catch (\Throwable $e) {
$this->error('Merge failed: ' . $e->getMessage());
Log::error('Merge failed', ['exception' => $e]);
return false;
}
}
private function gzipDump(string $inputPath, string $outputPath): bool
{
try {
$this->info('Compressing dump...');
$result = Process::timeout(config('masked-dump.gzip_timeout'))->run(['gzip', $inputPath]);
if ($result->successful() && $this->files->exists($outputPath)) {
$this->info('Compression completed.');
return true;
}
// Fallback to PHP gzencode
$content = $this->files->get($inputPath);
$gz = gzencode($content, 9);
$this->files->put($outputPath, $gz);
$this->files->delete($inputPath);
$this->info('Compression completed via fallback.');
return true;
} catch (\Throwable $e) {
$this->error('Compression failed: ' . $e->getMessage());
Log::error('Compression failed', ['exception' => $e]);
return false;
}
}
}
5. Upload Command
Create app/Console/Commands/UploadMaskedDump.php:
<?php
declare(strict_types=1);
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
final class UploadMaskedDump extends Command
{
protected $signature = 'app:upload-masked-dump';
protected $description = 'Uploads the masked database dump to the configured storage';
public function handle(): int
{
$this->info('Starting masked dump upload...');
try {
$localSqlGzPath = config('masked-dump.compressed_path');
// Check if file exists
if (!file_exists($localSqlGzPath)) {
$this->error('Masked dump file not found at: ' . $localSqlGzPath);
return self::FAILURE;
}
// Upload the file
$this->info('Uploading masked dump to storage...');
$fileContents = file_get_contents($localSqlGzPath);
$success = Storage::disk(config('masked-dump.storage_disk'))
->put('masked-dump.sql.gz', $fileContents);
if (!$success) {
$this->error('Failed to upload masked dump to storage.');
Log::error('Failed to upload masked dump to storage');
return self::FAILURE;
}
$this->info('Masked dump uploaded successfully.');
return self::SUCCESS;
} catch (\Exception $e) {
$this->error('An error occurred during upload: ' . $e->getMessage());
Log::error('Error during masked dump upload', ['exception' => $e]);
return self::FAILURE;
}
}
}
6. Pull Command
Create app/Console/Commands/PullMaskedDump.php:
<?php
declare(strict_types=1);
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Process;
use Illuminate\Support\Facades\Storage;
final class PullMaskedDump extends Command
{
protected $signature = 'app:pull-masked-dump
{--force : Force import without confirmation}
{--timeout=1800 : Timeout in seconds for import process}';
protected $description = 'Pull the masked database dump from storage and import it';
public function handle()
{
if (!$this->option('force') && !$this->confirm('This will replace your current database. Are you sure?')) {
$this->info('Operation cancelled.');
return self::SUCCESS;
}
if (app()->environment('production')) {
$this->error('This command cannot be run on the production environment');
return self::FAILURE;
}
$this->info('Starting masked dump import process...');
// Get database credentials
$connection = config('database.default');
$config = config("database.connections.$connection");
$host = $config['host'];
$port = (int) ($config['port'] ?? 3306);
$database = $config['database'];
$username = $config['username'];
$password = $config['password'];
// Validate database configuration
if (empty($host) || empty($database) || empty($username)) {
$this->error('Invalid database configuration. Please check your .env file.');
return self::FAILURE;
}
$localGzipPath = config('masked-dump.compressed_path');
$localSqlPath = config('masked-dump.local_path');
try {
// Check if dump exists in storage
if (!Storage::disk(config('masked-dump.storage_disk'))->exists('masked-dump.sql.gz')) {
$this->error('Masked dump file not found in storage.');
return self::FAILURE;
}
// Download the dump file
$this->info('Downloading masked dump...');
$contents = Storage::disk(config('masked-dump.storage_disk'))->get('masked-dump.sql.gz');
if (file_put_contents($localGzipPath, $contents) === false) {
$this->error('Failed to write dump file to storage');
return self::FAILURE;
}
// Extract the gzipped file
$this->info('Extracting dump file...');
$result = Process::run(['gzip', '-d', '-f', $localGzipPath]);
if (!$result->successful()) {
$this->error('Failed to extract the dump file: ' . $result->errorOutput());
Log::error('Failed to extract masked dump', ['error' => $result->errorOutput()]);
return self::FAILURE;
}
// Get timeout from options
$timeout = max(1, (int) $this->option('timeout'));
// Drop all tables before import
$this->info('Dropping existing tables...');
$this->dropAllTables($connection);
// Import the database
$this->info('Importing database dump (this may take a while)...');
$importCommand = $this->buildImportCommand($host, $port, $database, $username, $password, $localSqlPath);
$process = Process::timeout($timeout)->run($importCommand);
if (!$process->successful()) {
$this->error('Failed to import the database: ' . $process->errorOutput());
Log::error('Failed to import masked dump', ['error' => $process->errorOutput()]);
@unlink($localSqlPath);
return self::FAILURE;
}
// Clean up .sql file
@unlink($localSqlPath);
$this->info('Masked dump imported successfully.');
// Run the sanitize local database command
$this->call('app:sanitize-local-database');
// Seed encrypted data with local APP_KEY to make it readable
$this->info('Seeding encrypted data with local APP_KEY...');
$this->call('app:seed-encrypted-data', ['--fresh' => true, '--force' => true]);
// Run migrations
$this->info('Running migrations...');
$this->call('migrate', ['--force' => true]);
return self::SUCCESS;
} catch (\Exception $e) {
$this->error('An error occurred: ' . $e->getMessage());
Log::error('Error during masked dump import', ['exception' => $e]);
// Clean up any leftover files
if (isset($localGzipPath) && file_exists($localGzipPath)) {
@unlink($localGzipPath);
}
if (isset($localSqlPath) && file_exists($localSqlPath)) {
@unlink($localSqlPath);
}
return self::FAILURE;
}
}
protected function dropAllTables(string $connection): void
{
// Disable foreign key checks
DB::connection($connection)->statement('SET FOREIGN_KEY_CHECKS=0');
// Get all tables
$tables = DB::connection($connection)->select('SHOW TABLES');
$dbName = config("database.connections.$connection.database");
$tableKey = "Tables_in_$dbName";
// Drop each table
foreach ($tables as $table) {
if (isset($table->$tableKey)) {
$tableName = $table->$tableKey;
try {
DB::connection($connection)->statement("DROP TABLE IF EXISTS `$tableName`");
} catch (\Throwable $e) {
$this->warn("Could not drop table `$tableName`: " . $e->getMessage());
}
}
}
// Re-enable foreign key checks
DB::connection($connection)->statement('SET FOREIGN_KEY_CHECKS=1');
}
protected function buildImportCommand(string $host, int $port, string $database, string $username, ?string $password, string $sqlFile): string
{
// Create a temporary config file to avoid password in command line
$configFile = storage_path('mysql-config-' . time() . '.cnf');
$configContent = "[client]\nhost=\"$host\"\nport=\"$port\"\nuser=\"$username\"\n";
if ($password) {
$configContent .= "password=\"$password\"\n";
}
file_put_contents($configFile, $configContent);
chmod($configFile, 0600); // Secure the file
// Cleanup function
$cleanup = function () use ($configFile) {
if (file_exists($configFile)) {
@unlink($configFile);
}
};
// Register cleanup handlers
if (function_exists('pcntl_async_signals')) {
pcntl_async_signals(true);
pcntl_signal(SIGINT, $cleanup);
pcntl_signal(SIGTERM, $cleanup);
}
register_shutdown_function($cleanup);
// Build import command
return 'mysql --defaults-file=' . escapeshellarg($configFile) .
' --init-command=' . escapeshellarg('SET FOREIGN_KEY_CHECKS=0;') .
' --force ' .
escapeshellarg($database) .
' < ' . escapeshellarg($sqlFile);
}
}
Environment Variables
Add these to your .env file:
# Masked Dump Configuration
MASKED_DUMP_ENABLED=true
# AWS S3 Configuration for Masked Dumps
MASKED_DUMP_AWS_ACCESS_KEY_ID=your-access-key
MASKED_DUMP_AWS_SECRET_ACCESS_KEY=your-secret-key
MASKED_DUMP_AWS_DEFAULT_REGION=us-east-1
MASKED_DUMP_AWS_BUCKET=your-masked-dump-bucket
Usage Workflow
For Production (Building and Uploading)
# Build and upload masked dump
php artisan app:build-masked-dump
This command will:
- Create schema dump
- Create masked data dump
- Merge both dumps
- Compress the final dump
- Upload to configured storage
For Development (Pulling and Setting Up)
# Pull and import masked dump
php artisan app:pull-masked-dump
# Or with options
php artisan app:pull-masked-dump --force --timeout=1200
This command will:
- Download dump from storage
- Extract the compressed file
- Drop all existing tables
- Import the dump
- Run sanitization
- Seed and re-encrypt data with the local
APP_KEY - Run migrations
Security Considerations
1. Data Masking Strategy
- Always mask PII: Names, emails, phone numbers, addresses
- Preserve admin accounts: Keep admin user data for development access
- Limit historical data: Only include recent data to reduce dump size
- Exclude sensitive tables: Remove logs, notifications, failed jobs
2. Access Control
- Storage security: Use proper IAM policies for S3 bucket access
- Environment restrictions: Never run pull commands in production
- Credential management: Use secure credential storage
- Network security: Ensure secure transmission of dumps
3. Compliance
- GDPR compliance: Ensure masked data doesn't contain identifiable information
- Data retention: Set appropriate retention policies for stored dumps
- Audit logging: Log all dump operations
- Access reviews: Regularly review who has access to dumps
Performance Optimization
1. Memory Management
// Increase memory limit in config
'memory_limit' => '4G',
// Use chunking for large tables
$table->outputInChunksOf(1000);
2. Network Optimization
- Use compression (gzip) for all transfers
- Consider regional storage locations
- Implement retry mechanisms for failed uploads/downloads
3. Database Optimization
- Add indexes to improve query performance during masking
- Use connection pooling for large operations
- Consider read replicas for dump creation
Troubleshooting
Common Issues
-
Memory Exhaustion
php -d memory_limit=4G artisan app:build-masked-dump -
Timeout Errors
php artisan app:pull-masked-dump --timeout=3600 -
Foreign Key Constraint Errors
- Ensure foreign key checks are disabled during import
- Check table creation order in schema dump
-
Storage Permission Errors
- Verify AWS credentials and S3 bucket permissions
- Check local storage permissions
Debugging
Enable debug logging in your .env:
LOG_LEVEL=debug
Check logs at storage/logs/laravel.log for detailed error information.
Best Practices
- Regular Updates: Schedule automatic dump creation (daily/weekly)
- Size Management: Monitor dump sizes and optimize data inclusion
- Testing: Test the full cycle regularly in staging environment
- Documentation: Keep masking rules documented and reviewed
- Monitoring: Set up alerts for failed dump operations
- Validation: Verify data integrity after import
Advanced Configuration
Custom Storage Drivers
You can use any Laravel filesystem driver:
// Local storage
'masked_dump' => [
'driver' => 'local',
'root' => storage_path('app/masked-dumps'),
],
// FTP storage
'masked_dump' => [
'driver' => 'ftp',
'host' => env('MASKED_DUMP_FTP_HOST'),
'username' => env('MASKED_DUMP_FTP_USERNAME'),
'password' => env('MASKED_DUMP_FTP_PASSWORD'),
],
Scheduling
Add to app/Console/Kernel.php:
protected function schedule(Schedule $schedule)
{
// Build masked dump daily at 2 AM
$schedule->command('app:build-masked-dump')
->dailyAt('02:00')
->environments(['production'])
->onFailure(function () {
// Send notification on failure
});
}
This comprehensive guide should provide everything needed to implement the masked dump system in a new Laravel application. The system provides a secure, automated way to create development databases with production-like data while protecting sensitive information.