使用 wget 克隆网站的 Bash 脚本答案

【问题标题】：Bash script using wget to clone a website使用 wget 克隆网站的 Bash 脚本
【发布时间】：2021-01-26 10:25:24
【问题描述】：

我正在尝试通过修改here 上的 wget 命令来创建一个 bash 脚本来克隆网站。

这是我目前所拥有的，但我在正确解析 URL 时遇到问题：

URL=$1
DOMAIN=`echo $URL | sed -e 's/[^/]*\/\/\([^@]*@\)\?\([^:/]*\).*/\2/'`
echo $DOMAIN

wget \
     --recursive \ # Download the whole site.
     --page-requisites \ # Get all assets/elements (CSS/JS/images).
     --adjust-extension \ # Save files with .html on the end.
     --span-hosts \ # Include necessary assets from offsite as well.
     --convert-links \ # Update links to still work in the static version.
     --restrict-file-names=windows \ # Modify filenames to work in Windows as well.
     --domains $DOMAIN \ # Do not follow links outside this domain.
     --no-parent \ # Don't follow links outside the directory you pass in.
         $URL # The URL to download

问题似乎是从 URL 确定域。

这是我尝试克隆网页时的结果：

./clone_website.sh https://www.elliman.com/newyork/sales/detail/612-l-566-14_h344874/5-paumanok-road-water-mill-ny-11976
www.elliman.com
--2021-01-26 11:06:11--  http://%20/
Resolving   ( )... failed: Name or service not known.
wget: unable to resolve host address ‘ ’
--2021-01-26 11:06:11--  http://download/
Resolving download (download)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘download’
--2021-01-26 11:06:11--  http://the/
Resolving the (the)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘the’
--2021-01-26 11:06:11--  http://whole/
Resolving whole (whole)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘whole’
--2021-01-26 11:06:11--  http://site./
Resolving site. (site.)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘site.’
./clone_website.sh: line 7: syntax error near unexpected token `('
./clone_website.sh: line 7: `     --page-requisites \ # Get all assets/elements (CSS/JS/images).'

我该如何解决这个问题？

【问题讨论】：

您能告诉我们修改之前和之后的 URL 是什么吗？
问题是什么？您想如何解决什么？
@RamanSailopal 问题已更新以反映这一点
@umläute 问题已更新以显示错误消息。那应该澄清我想要解决的问题
\ # Download the whole site. cmets 是脚本的一部分吗？

标签： bash shell ubuntu

【解决方案1】：

\ 必须是一行中的最后一个字符。后面不允许评论。你想要：

wget \
     --recursive \
     --page-requisites \
     --adjust-extension \
     --span-hosts \
     --convert-links \
     --restrict-file-names=windows \
     --domains "$DOMAIN" \
     --no-parent \
         "$URL" # The URL to download This comment is fine

您的脚本正在执行：

wget --recursive ' #' Download the whole site.

--page-requisites ' #' Get all assets/elements (CSS/JS/images).
                                                             ^^ - syntax error

etc.

而wget 尝试下载名为http:// #/ 和http://Download/ 等的网站...

请记住引用变量扩展以禁用文件名扩展和分词（如果是 url，请引用参数以禁用在后台创建 & 子shell）。在脚本中使用 bash 数组更容易管理：

runme=(
     wget # yay bash array - you can comment and no need for '\'
     --recursive # another comment
     --page-requisites # to this and that
     --adjust-extension
     --span-hosts
     --convert-links
     --restrict-file-names=windows
     --domains "$DOMAIN"
     --no-parent 
     "$URL" # The URL to download This comment is fine
)
"${runme[@]}"

【讨论】：

【解决方案2】：

# replace the double-slash between protocol and domain with `%`:
DOMAIN_WITH_PROTOCOL=${URL/\/\//%}
# strip away all the path components
DOMAIN_WITH_PROTOCOL=${DOMAIN_WITH_PROTOCOL%%/*}
# replace the `%` (that we used to delimit protocol from domain) back to `//`
DOMAIN_WITH_PROTOCOL=${DOMAIN_WITH_PROTOCOL/\%///}

或者只是分步选择域和协议，这需要多一步，但我发现更容易遵循：

# overything before the first `://` is the protocol
proto=${URL%%://*}
# strip away the protocol, which gives us <domain>+<path>
domain=${URL#*://}
# strip the path from the domain:
domain=${domain%%/*}
# and reassemble protocol+domain:
DOMAIN_WITH_PROTOCOL="${proto://${domain}"

【讨论】：