PHP Classes

How to Implement a PHP CVS Import Library that Can Import Large CSV Files without Exceeding PHP Memory Limit using the Package APL Test: Process large CSV files using Laravel queues

Recommend this page to a friend!
  Info   View files Documentation   View files View files (82)   DownloadInstall with Composer Download .zip   Reputation   Support forum   Blog    
Last Updated Ratings Unique User Downloads Download Rankings
2024-06-14 (6 days ago) RSS 2.0 feedNot yet rated by the usersTotal: Not yet counted Not yet ranked
Version License PHP version Categories
apl-test 1.0MIT/X Consortium ...7Files and Folders, Libraries, Design ..., P...
Description 

Author

This package can process large CSV files using Laravel queues.

It implements an application that provides command classes to process CSV files passed from the command line using Laravel Artisan and extract lists of users contained in CSV files.

The application uses Laravel queues to process large CVS files without using too much memory.

Picture of András Zoltán-Gyárfás
Name: András Zoltán-Gyárfás <contact>
Classes: 7 packages by
Country: Romania Romania
Innovation award
Innovation award
Nominee: 4x

Winner: 1x

Documentation

APL nextED aptitude test

The Goal

Create a small sample application using common APL tools & languages. The application will need to import a large CSV file (100k records) into a queue, where a background worker will process each individual record.

Requirements

  • Use a public Git repository
  • Set up a Homestead & VirtualBox VM environment for the application
  • Use the native Laravel framework as much as possible (Models, Routes, Config, Storage, DB)
  • Use Laravel Queue (your choice of queue)
  • Use MySQL as the destination database - Create a DB migration to setup a Users table - Use the native Laravel timestamps - Use the SoftDelete feature
  • Write a script to create a CSV file with 100k records containing a list of users - Names, emails, passwords and phone numbers (can be fully randomized) - Include a ?deleted? column, with a few records having a value set
  • Create a CSV file importer, that will accept the generated CSV file and insert the records into the Queue
  • Create a Console Command that processes the records in the queue, and update the database with the data.
  • Setup PHP Unit / Functional Tests on the importer and queue worker.

Requirements

  • _Homestead_, _Valet_ or any other local setup.
  • Laravel 8.x

Local setup

  • Clone this repository
  • Download and install `Homestead` and `VirtualBox`
  • Install the dependencies from the project folder
composer install

  • Generate the Homestead configuration file by running the
php vendor/bin/homestead make

or by copying the attached _Homestead.yaml_ file to your project directory and run

vagrant up

  • In order to connect to your virtual machine, run:
vagrant ssh
  • Go to your project folder listed in the Homestead.yaml file and create the `.env` file, than update the database connection data
copy .env.example .env

  • Run the migration
php artisan migrate

  • Add the Jobs table:
php artisan queue:table

Application Usage

Generate users to file

This application can generate users in csv, json format.

To generate a list of users, use the following command:

php artisan generate:users {format} {count=10} {filename=users}

This command will generate a users.csv file containing 100000 users to the default storage directory.

Currently there are two formats supported: json and csv. For more info, see the app/Extension/HandleItemsList and app/Extension/HandleItemsList folders for more info about further format implementations.

__Example:__

This will generate a users.csv file with 100000 user data

php artisan generate:users csv 100000

This command will generate a users.json file with 100000 users data.

php artisan generate:users json 100000

Load file into users table

Start the workers:

php artisan queue:work --queue=users

To load a file into the users table, run the below command

php artisan load:users {format=csv} {filename=users}

__Example:__

This will load the users.csv file into the users table

php artisan load:users csv users

This will load the users.json file into the users table

php artisan generate:users json 100000

Testing the app

In order to properly run tests, the .env.testing file has to be completed. After the above is completed, run:

php artisan test

Author

Andras Zoltan Gyarfas - Zoli


  Files folder image Files  
File Role Description
Files folder imageapl-csv-handler (12 files, 8 directories)
Accessible without login Plain text file Homestead.yaml Data Auxiliary data
Accessible without login Plain text file README.md Doc. Documentation

  Files folder image Files  /  apl-csv-handler  
File Role Description
Files folder imageapp (8 directories)
Files folder imagebootstrap (1 file)
Files folder imageconfig (14 files)
Files folder imagedatabase (3 directories)
Files folder imagepublic (4 files)
Files folder imageresources (3 directories)
Files folder imageroutes (4 files)
Files folder imagetests (2 files, 2 directories)
  Accessible without login Plain text file .editorconfig Data Auxiliary data
  Accessible without login Plain text file .env.example Data Auxiliary data
  Accessible without login Plain text file .env.testing Data Auxiliary data
  Accessible without login Plain text file .styleci.yml Data Auxiliary data
  Plain text file artisan Class Class source
  Accessible without login Plain text file composer.json Data Auxiliary data
  Accessible without login Plain text file composer.lock Data Auxiliary data
  Accessible without login Plain text file package.json Data Auxiliary data
  Accessible without login Plain text file phpunit.xml Data Auxiliary data
  Accessible without login Plain text file README.md Doc. Documentation
  Accessible without login Plain text file server.php Aux. Auxiliary script
  Accessible without login Plain text file webpack.mix.js Data Auxiliary data

  Files folder image Files  /  apl-csv-handler  /  app  
File Role Description
Files folder imageConsole (1 file, 1 directory)
Files folder imageExceptions (1 file)
Files folder imageExtensions (2 directories)
Files folder imageHelpers (1 file)
Files folder imageHttp (1 file, 2 directories)
Files folder imageJobs (1 file)
Files folder imageModels (1 file)
Files folder imageProviders (5 files)

  Files folder image Files  /  apl-csv-handler  /  app  /  Console  
File Role Description
Files folder imageCommands (3 files)
  Plain text file Kernel.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Console  /  Commands  
File Role Description
  Plain text file AbstractUsersListCommand.php Class Class source
  Plain text file GenerateUsersListCommand.php Class Class source
  Plain text file LoadUsersListCommand.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Exceptions  
File Role Description
  Plain text file Handler.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Extensions  
File Role Description
Files folder imageGenerateUserList (2 files)
Files folder imageHandleItemsList (2 files)

  Files folder image Files  /  apl-csv-handler  /  app  /  Extensions  /  GenerateUserList  
File Role Description
  Plain text file HandleCsvUserList.php Class Class source
  Plain text file HandleJsonUserList.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Extensions  /  HandleItemsList  
File Role Description
  Plain text file AbstractHandleItemsList.php Class Class source
  Plain text file HandleItemsListContract.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Helpers  
File Role Description
  Plain text file FilesHelper.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Http  
File Role Description
Files folder imageControllers (1 file)
Files folder imageMiddleware (8 files)
  Plain text file Kernel.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Http  /  Controllers  
File Role Description
  Plain text file Controller.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Http  /  Middleware  
File Role Description
  Plain text file Authenticate.php Class Class source
  Plain text file EncryptCookies.php Class Class source
  Plain text file PreventRequestsDuringMaintenance.php Class Class source
  Plain text file RedirectIfAuthenticated.php Class Class source
  Plain text file TrimStrings.php Class Class source
  Plain text file TrustHosts.php Class Class source
  Plain text file TrustProxies.php Class Class source
  Plain text file VerifyCsrfToken.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Jobs  
File Role Description
  Plain text file CreateUserRecord.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Models  
File Role Description
  Plain text file User.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  app  /  Providers  
File Role Description
  Plain text file AppServiceProvider.php Class Class source
  Plain text file AuthServiceProvider.php Class Class source
  Plain text file BroadcastServiceProvider.php Class Class source
  Plain text file EventServiceProvider.php Class Class source
  Plain text file RouteServiceProvider.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  bootstrap  
File Role Description
  Plain text file app.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  config  
File Role Description
  Plain text file app.php Class Class source
  Plain text file auth.php Class Class source
  Accessible without login Plain text file broadcasting.php Aux. Auxiliary script
  Accessible without login Plain text file cache.php Aux. Auxiliary script
  Accessible without login Plain text file cors.php Aux. Auxiliary script
  Accessible without login Plain text file database.php Aux. Auxiliary script
  Accessible without login Plain text file filesystems.php Aux. Auxiliary script
  Accessible without login Plain text file hashing.php Aux. Auxiliary script
  Plain text file logging.php Class Class source
  Accessible without login Plain text file mail.php Aux. Auxiliary script
  Accessible without login Plain text file queue.php Aux. Auxiliary script
  Accessible without login Plain text file services.php Aux. Auxiliary script
  Accessible without login Plain text file session.php Aux. Auxiliary script
  Accessible without login Plain text file view.php Aux. Auxiliary script

  Files folder image Files  /  apl-csv-handler  /  database  
File Role Description
Files folder imagefactories (1 file)
Files folder imagemigrations (5 files)
Files folder imageseeders (1 file)

  Files folder image Files  /  apl-csv-handler  /  database  /  factories  
File Role Description
  Plain text file UserFactory.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  database  /  migrations  
File Role Description
  Plain text file 2014_10_12_000000_create_users_table.php Class Class source
  Plain text file 2014_10_12_100000_...rd_resets_table.php Class Class source
  Plain text file 2019_08_19_000000_...iled_jobs_table.php Class Class source
  Plain text file 2021_02_08_085713_...ted_at_to_users.php Class Class source
  Plain text file 2021_02_08_090418_create_jobs_table.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  database  /  seeders  
File Role Description
  Plain text file DatabaseSeeder.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  public  
File Role Description
  Accessible without login Plain text file .htaccess Data Auxiliary data
  Plain text file index.php Class Class source
  Accessible without login Plain text file robots.txt Doc. Documentation
  Accessible without login Plain text file web.config Data Auxiliary data

  Files folder image Files  /  apl-csv-handler  /  resources  
File Role Description
Files folder imagejs (2 files)
Files folder imagelang (1 directory)
Files folder imageviews (1 file)

  Files folder image Files  /  apl-csv-handler  /  resources  /  js  
File Role Description
  Accessible without login Plain text file app.js Data Auxiliary data
  Accessible without login Plain text file bootstrap.js Data Auxiliary data

  Files folder image Files  /  apl-csv-handler  /  resources  /  lang  
File Role Description
Files folder imageen (4 files)

  Files folder image Files  /  apl-csv-handler  /  resources  /  lang  /  en  
File Role Description
  Accessible without login Plain text file auth.php Aux. Auxiliary script
  Accessible without login Plain text file pagination.php Aux. Auxiliary script
  Accessible without login Plain text file passwords.php Aux. Auxiliary script
  Accessible without login Plain text file validation.php Aux. Auxiliary script

  Files folder image Files  /  apl-csv-handler  /  resources  /  views  
File Role Description
  Accessible without login Plain text file welcome.blade.php Aux. Auxiliary script

  Files folder image Files  /  apl-csv-handler  /  routes  
File Role Description
  Accessible without login Plain text file api.php Example Example script
  Accessible without login Plain text file channels.php Example Example script
  Accessible without login Plain text file console.php Example Example script
  Accessible without login Plain text file web.php Aux. Auxiliary script

  Files folder image Files  /  apl-csv-handler  /  tests  
File Role Description
Files folder imageFeature (1 file)
Files folder imageUnit (1 file)
  Plain text file CreatesApplication.php Class Class source
  Plain text file TestCase.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  tests  /  Feature  
File Role Description
  Plain text file GenerateAndLoadUsersListTest.php Class Class source

  Files folder image Files  /  apl-csv-handler  /  tests  /  Unit  
File Role Description
  Plain text file GenerateAndLoadUsersListTest.php Class Class source

 Version Control Unique User Downloads  
 100%
Total:0
This week:0