【问题标题】:Importing big csv file with too many columns into a table in SQL Server将包含太多列的大 csv 文件导入 SQL Server 中的表
【发布时间】:2019-09-26 14:20:17
【问题描述】:

我想将四个 .csv 文件(每个文件包含超过 300,000 行和 150 列且大小超过 0.5 GB)导入到数据库表中。在 SQL Server 中执行此操作的最简单方法是什么?

我使用的是最新的 SQL Server 2017 Express。我有 4 个大的 .csv 文件,我想将它们导入数据库。我打算将这些导入到 4 个单独的表中。但是,要创建新表,需要定义所有列名和数据类型,这很麻烦。因此,我想知道如何以更好、更轻松的方式实现这一目标。请注意,我是 SQL Server 的新手,对它不太熟悉。

【问题讨论】:

  • 导入向导可以为您创建表格,但请注意,如果您希望此信息发挥作用,则需要创建适当的设计。
  • 您有与每个 CSV 文件关联的格式文件吗?

标签: sql-server database csv import


【解决方案1】:

BULK INSERT 会为您执行此操作。

BULK INSERT dbo.YourTableName
FROM 'C:\Users\Steve\Downloads\yourfiletoIMPORT.csv'
WITH (
FirstRow = 2, --(if skipping a header row)
FIELDTERMINATOR = '\t',
ROWTERMINATOR   = '\n'
)

或者……

BULK INSERT SchoolsTemp
    FROM 'C:\Users\Steve\Downloads\yourfiletoIMPORT.csv'
    WITH
    (
    FIRSTROW = 2,
    FIELDTERMINATOR = ',',  --CSV field delimiter
    ROWTERMINATOR = '\n',   --Use to shift the control to next row
    ERRORFILE = 'C:\Users\Steve\Downloads\errors.csv',
    TABLOCK
    )

或者,只需导入文件,如下面的链接所述。

https://host4asp.net/import-csv-file-using-sql-server-management-studio/

使用 SQL Server Management Studio 导入 CSV 文件:

Step 1
At the start, please open up the SQL Server Management Studio. Log into the target database, and right click the database. Please note that you shall click on the entire database, rather than a particular table. From the Object Explorer, you shall point to the button of Tasks, and find the Import Data.

Step 2
Please note that the Wizard introduction page might be popped up. When you see such introduction page, please safely click on next. This is the screen prompting the selection of a data source. From the screen, you shall select the Flat File source from the Dropdown box, and the Browse button.

Step 3
From the Windows Explorer, you shall select the designated CSV file. In order to ensure you select the correct file type, it is the best practice to select the filetype as CSV, but not TXT. Therefore, only CSV filetype shall be displayed.

Step 4
After the selection of the CSV file, please allocate some time to configure how to import the data into the database before you click the Next > button. Note that you shall ensure the First Data Row checked, because the file shall then contain the required column names. From the following image, you shall see the Column Names from the SQL Server Management Studio shall try their best to important header row instead.

Step 5
After the review of columns, you shall examine more advanced options. The review is important before you completely import the CSV file. From the image below, by default, the SQL Server set the length of each string to be 50.

Step 6
If you have string that is larger than 50, please request the SQL Server to inspect all columns in the file. The inspection can be done by clicking on the Suggest Types button. SQL Server shall be instructed to examine only the first 100 rows, giving suggested types of each column. Error shall be pointed out during the inspection process. Depending on the file size, you can select to inspect the whole file or just selected the fields.

Step 7
You will be prompted to the Preview section from the Data Source page. That will be the last time to examine columns again.

Step 8
After your review on the import preview, you shall select your destination database.

Step 9
In this step, you shall select your destination database. The SQL Server normally selects the desired table on behalf of you. If it is not the case, please create your table. If you would like to select a different table, please click on the destination column for action.

Step 10
You are required to prompt to the option in order to save as an SSIS package. You can also leave the option unchecked as is. Please click next.

Step 11
Finally, you will be prompted to the verification screen. If you are fine with everything, please run the Import by click the Finish.

【讨论】:

    【解决方案2】:

    Microsoft BCP Utility 专门为此目的创建:

    大容量复制程序实用程序 (bcp) 大容量复制数据 Microsoft SQL Server 实例和用户指定的数据文件 格式。 bcp 实用程序可用于导入大量新的 行到 SQL Server 表中或将数据从表中导出到数据中 文件。

    【讨论】:

    • BCP在导入数据时似乎要求目标表已经存在。
    • 没错,但 BCP 是迄今为止最快的定期导入数据的方式。
    猜你喜欢
    • 2011-03-27
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2013-02-20
    • 2019-07-06
    • 1970-01-01
    • 2012-01-09
    相关资源
    最近更新 更多