Great for Read/Write Intensive Applications.Row Level Locking – Locks Single Row on Write.Table Level Locking – Locks Entire Table on Write.In brief the things to consider (in general) are as follows: The two main engines you are going to read about are MYISAM and InnoDB and each has their own benefits and drawbacks. Large or small database it’s always good to take a little time to consider which database engine you are going to use for your project. In that case you can upload the files manually to your server and then execute from PHPMyAdmin (see their manual for more info) or execute the command via your SSH console (assuming you have your own server) LOAD DATA INFILE '/mylargefile.csv' INTO TABLE temp_data FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' Of note, this can be executed through PHPMyAdmin, but you may still have file upload issues. The SQL function LOAD DATA INFILE was created to handle these large datasets and will significantly reduce the time it takes to handle the import process. Not only will it fail in many cases because your server won’t be able to handle a file upload as large as some of your data files due to upload size restrictions and server timeouts, but even if it does succeed, the process could take hours depending our your hardware. Importing large data files like the ones we worked with (and larger ones) can be tough to do if you go ahead and try a regular CSV insert via a tool like PHPMyAdmin.
Getting an understand of what all of the data represents, what columns related to what and what type of manipulation you need to will end up saving you time in the long run. I can’t stress this first step enough! Make sure that you take the time to analyze the data you are working with before importing it at all. Now, working with a dataset this large or larger is not the simplest of tasks and in working on it we wanted to take a moment to share some of the things you should consider and know when working with large datasets like this. Our end goal was to import and manipulate the data into a MySQL relational database to be used to power a front-end PHP script that we also developed. Our client provided us with a 50 CSV files ranging from 30 MB to 350 MB in size and all in all containing approximately 20 million rows of data and 15 columns of data. On a recent project we had the challenge of working with and manipulating a large collection of data.
0 kommentar(er)
