如何从 URL 下载所有文件？

Question

我是 PowerShell 的新手，需要一个脚本来从 URL 下载所有文件： https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/daily/kl/recent/

当我将单个文件的 URL 存储在列表中时，我已经设法下载了这些文件（参见代码）。我尝试了不同的方法来自动生成列表，但我没有成功。

$list = get-content "D:\ListURL.txt"

foreach($url in $list)
{
    $filename =[System.IO.Path]::GetFileName($url) 
    $file =[System.IO.Path]::Combine($outputdir, $filename) 
    Invoke-WebRequest -Uri $url -OutFile $file

}

谁能帮我编写一些代码来从 URL 创建列表？非常感谢。

Answer 1

查看该网址上的文件列表，这对我有用：

$outputdir = 'D:\Downloads'
$url       = 'https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/daily/kl/recent/'

# enable TLS 1.2 and TLS 1.1 protocols
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12, [Net.SecurityProtocolType]::Tls11

$WebResponse = Invoke-WebRequest -Uri $url
# get the list of links, skip the first one ("../") and download the files
$WebResponse.Links | Select-Object -ExpandProperty href -Skip 1 | ForEach-Object {
    Write-Host "Downloading file '$_'"
    $filePath = Join-Path -Path $outputdir -ChildPath $_
    $fileUrl  = '{0}/{1}' -f $url.TrimEnd('/'), $_
    Invoke-WebRequest -Uri $fileUrl -OutFile $filePath
}

Answer 2

这应该可以解决问题：

Add-Type -AssemblyName System.Web
Add-Type -AssemblyName System.Web.Extensions

$url       = 'https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/daily/kl/recent/'
$destPath  = 'C:\temp\'

$response  = Invoke-WebRequest -Uri $url -Method GET

$files = @( $response.Content -split [environment]::NewLine | ? { $_ -like '*tageswerte*.zip*' } | % { $_ -replace '^(.*>)(tages.*\.zip)<.*', '$2' } )

foreach( $file in $files ) {

    Invoke-WebRequest -Uri ($url + $file) -Method GET -OutFile ($destPath + $file) | Out-Null
}