Imports

Introduction

The import functionality allows administrators to bulk import data into the application using CSV files. This feature supports importing various types of data including sessions, organisations, users, and session tags. The import process includes validation, data preparation, and error handling to ensure data integrity.

Import Types

The application supports the following import types, defined in the ImportTypes enum:

Sessions: Import session data including topic, type, status, compliance, dates, and relationships.
Organisations: Import organisation data including name, description, licence information, and visibility settings.
Users: Import user data including personal information, organisation relationships, and programme seats.
Session Tags: Import session tag data including labels and parent-child relationships.

Each import type has specific required and optional fields, with detailed descriptions available in the import interface.

Import Process Flow

The import process follows these steps:

Upload: User selects an import type and uploads a CSV file.
Validation: The system validates the data against predefined rules.
Review: User reviews any validation errors and can fix them in the source file.
Import: Once validation passes, the user can run the import process.
Results: The system provides feedback on the import results, including success and failure counts.

Architecture

Import Models

Import: Stores metadata about the import, including the file path, import type, and progress statistics.
ImportLog: Records detailed logs of the import process, including validation errors and success messages.

Import Classes

Each import type has a dedicated import class that extends the BaseImport class:

SessionsImport: Handles session data imports.
OrganisationsImport: Handles organisation data imports.
UsersImport: Handles user data imports.
SessionTagsImport: Handles session tag data imports.

Validation and Data Preparation

Each import class implements:

Rules Method: Defines validation rules for each field.
PrepareForValidation Method: Sanitizes and transforms data before validation.
RunImport Method: Processes each row of valid data and performs the actual import.

Example: Organisation Import

The OrganisationsImport class handles importing organisation data with the following key features:

public function rules(): array
{
    return [
        'id' => ['nullable', 'sometimes', Rule::exists('organisations', 'id')],
        'name' => ['required', 'string', 'max:255'],
        'description' => ['required', 'string', 'max:255'],
        // Additional fields...
        'licence_group_id' => ['nullable', Rule::exists('licence_groups', 'id')],
        'licence_quantity' => ['nullable', 'integer', 'min:1'],
        'licence_expiry_date' => ['nullable', 'date'],
        'optt_default_visibility' => ['nullable', Rule::in(array_map(fn ($case) => $case->value, OpttVisibility::cases()))],
    ];
}

The import process handles:

Creating or updating organisations
Assigning licence groups with quantities and expiry dates
Setting OPTT visibility preferences
Attaching account owners and users
Linking to tenants

Example: User Import

The UsersImport class handles importing user data with features such as:

Creating or updating users based on email
Attaching users to organisations
Setting organisation ownership
Linking users to sessions
Creating programme seats with contract and ignition dates
Sending welcome emails (optional)

Filament UI Integration

The import functionality is integrated into the Filament admin panel through:

ImportResource: Defines the form and table for managing imports.
Wizard Interface: Guides users through the import process with steps for setup, validation, and running the import.
Import Table Component: Displays available columns for each import type with descriptions and requirements.

Column Definitions

Column definitions for each import type are defined in the ImportTypes enum's columns method. Each column has:

Column Name: The field name in the CSV.
Required Flag: Indicates if the field is mandatory.
Description: Explains the field's purpose and format.
Route (optional): Links to related resources for reference.

Organisation Import Columns

The Organisation import type includes the following key columns:

[
    'column' => 'name',
    'required' => true,
    'description' => 'The name of the organisation (limited to 255 characters)',
],
[
    'column' => 'description',
    'required' => true,
    'description' => 'A helpful description of the organisation (limited to 255 characters)',
],
[
    'column' => 'location',
    'required' => false,
    'description' => 'The session delivery location ID for the organisation, from the delivery locations list',
],
// Licence-related fields
[
    'column' => 'licence_group_id',
    'required' => false,
    'description' => 'The licence group ID to assign to the organisation, from the licence groups list',
],
[
    'column' => 'licence_quantity',
    'required' => false,
    'description' => 'The number of licences to assign to the organisation (minimum 1)',
],
[
    'column' => 'licence_expiry_date',
    'required' => false,
    'description' => 'The expiry date for the licence group (YYYY-MM-DD format)',
],
[
    'column' => 'optt_default_visibility',
    'required' => false,
    'description' => 'The default visibility setting for OPTT items in this organisation. Options: public (Entire Organisation), private1 (Private Me), private2 (Private Me and OPAL), private3 (Private Me and department)',
],

Best Practices

When using the import functionality:

Use Templates: Download the template for the specific import type to ensure correct column headers.
Validate First: Always run validation before performing the actual import.
Check Logs: Review import logs for any errors or warnings.
Incremental Imports: For large datasets, consider breaking imports into smaller batches.
ID References: When referencing related entities (e.g., organisation_ids), ensure the IDs exist in the system.

Extending Import Functionality

To add a new import type:

Add a new case to the ImportTypes enum.
Create a new import class extending BaseImport.
Implement the required methods (rules, prepareForValidation, runImport).
Add column definitions to the columns method in ImportTypes.
Update the ImportResource to handle the new import type.

Security Considerations

The import functionality includes several security measures:

Permission Checks: Users must have appropriate permissions to access specific import types.
Validation: Strict validation rules prevent invalid or malicious data.
Sanitization: Data is sanitized before processing to prevent injection attacks.
Tenant Isolation: Imports respect tenant boundaries to maintain data isolation.