存储过程和数据导出自动化答案

【问题标题】：Stored procedure and data export automation存储过程和数据导出自动化
【发布时间】：2023-04-02 15:08:02
【问题描述】：

我在 Oracle SQL Developer 中创建了一个包含一组查询的存储过程。每个查询都返回一个包含大量数据的表（每个表超过 200 万行 => 手动导出已经由于数据大小而导致问题）。

我想自动化存储过程并将每个结果表一次导出到单独的 Excel 工作表。

谢谢

【问题讨论】：

如果有一段特定的代码您需要帮助，请发布该代码，我们可以尝试帮助您解决该问题。
当然，如果您愿意，我会发布程序，但这只是一个包含大量 SELECT 查询的简单程序字段。我遇到的两个问题是如何在一个 batsh 中自动化午餐和导出，以及如何在该 batsh 中包含一个解决方案，以便能够将大量从 oracle 导出到 excel @Rene
您的操作系统和数据库都提供调度系统，您可以使用 sqlcl/sqlplpus 和 bash 脚本加上 crontab，或者您可以使用数据库中的 DBMS_SCHEDULER 作业来运行您的存储过程
这不仅仅是程序调度的问题@thatjeffsmith
@kouthe 他们是对的，所以我给了你一半的答案。如果您分享您的代码/处理特定代码问题，您也会得到更好的答案

标签： sql bash oracle shell plsql

【解决方案1】：

我创建了一个 powershell 脚本，它将指定用户的所有表导出到单独的 csv 文件中。

您可以在查询中指定所需的表列表。 select TNAME from tab where tabtype='TABLE' and tname not like 'BIN$%' 如果表有 0 行，则不会创建 csv 文件。在脚本中，您可以为字符串指定日期格式和代码页。

$NLS_NUMERIC_CHARACTERS=".,"
$NLS_DATE_FORMAT="DD.MM.YYYY HH24:MI:SS"
$NLS_LANG="AMERICAN_AMERICA.UTF8"

脚本将所有用户表导出到单独的 CSV 文件。

.\run_export_all_tables.ps1 -username SCOTT -password tiger -connect_string ORCL -csv_dir_path C:\upwork\powershell_sqlplus_export_csv\csv\  -log_file log_file.log


<#
    .SYNOPSIS
     The script exports all user tables to separate CSV files.
     Author: Dmitry Demin dmitrydemin1973@gmail.com


    .DESCRIPTION
     In the script, the format for displaying the date and decimal separator is configured.

    .PARAMETER username
    Specify the username  for example SCOTT

    .PARAMETER password
    Specify the password  for example TIGER

    .PARAMETER connect_string
    Specify the connect_string(TNS alias)  for connect to database from $ORACLE_HOME/network/admin/tnsnames.ora.  

    .PARAMETER csv_dir_path
    Specify the csv directory for csv files

    .PARAMETER  log_file
    Specify the  log file for this script.  


    .EXAMPLE
      The script exports all user tables to separate CSV files.
    .\run_export_all_tables.ps1 -username SCOTT -password tiger -connect_string ORCL -csv_dir_path C:\upwork\powershell_sqlplus_export_csv\csv\  -log_file log_file.log
#>


param(
[string]$username = "scott", 
[string]$password = "tiger",
[string]$connect_string = "esmd",
[string]$csv_dir_path = "C:\upwork\powershell_sqlplus_export_csv\csv\",
[string]$log_file = "C:\upwork\powershell_sqlplus_export_csv\log_file.log"
)
# Column separator for csv file 
$COLSEP=";"
# NLS_NUMERIC_CHARACTERS
$NLS_NUMERIC_CHARACTERS=".,"
$NLS_DATE_FORMAT="DD.MM.YYYY HH24:MI:SS"
# Log file 
$full_log_path=$log_file
# CSV directory
$full_csv_path=$csv_dir_path
#csv file extension
$csv_ext=".csv"
#Set NLS_LANG for session sqlplus 
#"RUSSIAN_CIS.UTF8"
#"RUSSIAN_CIS.CL8MSWIN1251"
#"AMERICAN_AMERICA.UTF8"
#$NLS_LANG="RUSSIAN_CIS.CL8MSWIN1251"
$NLS_LANG="AMERICAN_AMERICA.UTF8"

#Set NLS_LANG for session sqlplus 
[Environment]::SetEnvironmentVariable("NLS_LANG",$NLS_LANG , [System.EnvironmentVariableTarget]::PROCESS)
$env_path_NLS=[Environment]::GetEnvironmentVariable("NLS_LANG", [EnvironmentVariableTarget]::PROCESS)

echo "SET session NLS_LANG: $env_path_NLS" | tee-object -Append  -filepath $full_log_path


$sqlQuery_show_user_tables = 
@"
set heading off
set termout OFF
SET FEEDBACK OFF
SET TAB OFF
set pause off
set verify off
SET UNDERLINE OFF
set trimspool on
set timing off
set echo off
set linesize 1000
set pagesize 100
select  TNAME  from tab where tabtype='TABLE' and tname not like 'BIN$%'
;
exit
"@

$SqlQueryExportTable1 = 
@"
set heading off
set termout OFF
SET FEEDBACK OFF
SET TAB OFF
set pause off
set verify off
SET UNDERLINE OFF
set trimspool on
set timing off
set echo off
set linesize 10000
set pagesize 0
SET COLSEP '$COLSEP'
ALTER SESSION SET NLS_NUMERIC_CHARACTERS='$NLS_NUMERIC_CHARACTERS';
ALTER SESSION SET NLS_DATE_FORMAT='$NLS_DATE_FORMAT'; 
select * from   
"@

$SqlQueryExportTable2 =
@"

exit
"@


function Check_File
{
     param (
          [string]$pathfile 
     )


try {
$A=Get-Content -Path $pathfile  -ErrorAction Stop
}
catch [System.UnauthorizedAccessException]
{
#Write-Host "File $pathfile  is not accessible."
echo "File $pathfile  is not accessible." | tee-object -Append  -filepath $full_log_path


exit
}

catch [System.Management.Automation.ItemNotFoundException]
{
#Write-Host "File $pathfile  is not found."
echo "File $pathfile  is not found." | tee-object -Append  -filepath $full_log_path
exit
}
catch {
Write-Host "File $pathfile.  Other type of error was found:"
#Write-Host "Exception type is $($_.Exception.GetType().Name)"
   echo "Exception type is $($_.Exception.GetType().Name)" | tee-object -Append  -filepath $full_log_path
exit
}

}


echo "===========================================================================================" | tee-object -Append  -filepath $full_log_path



$date_time_start = Get-Date -Format "yyyy-MM-dd HH:mm:ss"            
$date_time_log = Get-Date -Format "yyyyMMddHHmmss"            

Write-host "Script start time : $date_time_start "
try
{
echo "Script start time :  $date_time_start ">>$full_log_path
}
catch {
Write-Host "Log File $full_log_path.  Other type of error was found:"
Write-Host "Exception type is $($_.Exception.GetType().Name)"
exit
}


chcp 1251

$sqlQuery =  $sqlQuery_show_user_tables

$sqlOutput = $sqlQuery | sqlplus -s  $username/$password@$connect_string

$UserList =$sqlOutput| where {$_-notlike "" }
$i=1

   echo  "Found tables for export : " | tee-object -Append  -filepath $full_log_path

foreach ($user_tab in $UserList)
{
   echo " $i $user_tab"  | tee-object -Append  -filepath $full_log_path
   $i=$i+1
}


foreach ($user_tab in $UserList)
{
$sqlQuery_show_table_all=""

$sqlQuery_show_table_all=$SqlQueryExportTable1+ $user_tab+ $SqlQueryExportTable2

$full_csv_path_file=$full_csv_path +  $user_tab + "_" + $date_time_log + $csv_ext

echo "-------------------------------------------------------------------------------------------"  | tee-object -Append  -filepath $full_log_path
echo "For table : $user_tab will be created new csv file: $full_csv_path_file" | tee-object -Append  -filepath $full_log_path


echo  "Script will run for table: $user_tab "  | tee-object -Append  -filepath $full_log_path

$sqlOutput_tab = $sqlQuery_show_table_all | sqlplus -s $username/$password@$connect_string
$sqlOutput_count = $sqlOutput_tab.count
 if ($sqlOutput_tab.count -gt 0) 
{
Out-File -filepath $full_csv_path_file -append -inputobject $sqlOutput_tab -encoding default
echo  "Exported rows:  $sqlOutput_count "  | tee-object -Append  -filepath $full_log_path
}
else
{
echo  "No exported rows: 0 row"  | tee-object -Append  -filepath $full_log_path
echo  "$full_csv_path_file file not created "  | tee-object -Append  -filepath $full_log_path
}

echo "-------------------------------------------------------------------------------------------"  | tee-object -Append  -filepath $full_log_path
}

例如输出脚本

Script start time :  2020-01-16 14:25:28 
Found tables for export : 
 1 TEST_TABLE5
 2 TEST_TABLE4
 3 TEST_TABLE3
 4 TEST_TABLE2
 5 TEST_TABLE1
 6 TEST2
 7 TEST1
 8 SALGRADE
 9 EMP
 10 DEPT
 11 CL_TAB1
 12 BONUS
-------------------------------------------------------------------------------------------
For table : TEST_TABLE5 will be created new csv file: .\csv\TEST_TABLE5_20200116142528.csv
Script will run for table: TEST_TABLE5 
No exported rows: 0 row
.\csv\TEST_TABLE5_20200116142528.csv file not created 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : TEST_TABLE4 will be created new csv file: .\csv\TEST_TABLE4_20200116142528.csv
Script will run for table: TEST_TABLE4 
No exported rows: 0 row
.\csv\TEST_TABLE4_20200116142528.csv file not created 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : TEST_TABLE3 will be created new csv file: .\csv\TEST_TABLE3_20200116142528.csv
Script will run for table: TEST_TABLE3 
No exported rows: 0 row
.\csv\TEST_TABLE3_20200116142528.csv file not created 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : TEST_TABLE2 will be created new csv file: .\csv\TEST_TABLE2_20200116142528.csv
Script will run for table: TEST_TABLE2 
No exported rows: 0 row
.\csv\TEST_TABLE2_20200116142528.csv file not created 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : TEST_TABLE1 will be created new csv file: .\csv\TEST_TABLE1_20200116142528.csv
Script will run for table: TEST_TABLE1 
No exported rows: 0 row
.\csv\TEST_TABLE1_20200116142528.csv file not created 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : TEST2 will be created new csv file: .\csv\TEST2_20200116142528.csv
Script will run for table: TEST2 
Exported rows:  213 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : TEST1 will be created new csv file: .\csv\TEST1_20200116142528.csv
Script will run for table: TEST1 
Exported rows:  2 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : SALGRADE will be created new csv file: .\csv\SALGRADE_20200116142528.csv
Script will run for table: SALGRADE 
Exported rows:  5 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : EMP will be created new csv file: .\csv\EMP_20200116142528.csv
Script will run for table: EMP 
Exported rows:  14 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : DEPT will be created new csv file: .\csv\DEPT_20200116142528.csv
Script will run for table: DEPT 
Exported rows:  7 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : CL_TAB1 will be created new csv file: .\csv\CL_TAB1_20200116142528.csv
Script will run for table: CL_TAB1 
Exported rows:  4 
-------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------
For table : BONUS will be created new csv file: .\csv\BONUS_20200116142528.csv
Script will run for table: BONUS 
No exported rows: 0 row
.\csv\BONUS_20200116142528.csv file not created 
-------------------------------------------------------------------------------------------

【讨论】：

【解决方案2】：

要将大量数据导出到 XLS 文件，您可以使用：

https://www.oracle.com/webfolder/community/oracle_database/3784064.html

在

中也有解释

Create an Excel File (.xlsx) using PL/SQL

【讨论】：