[Bash]从 txt 文件中读取 URL答案

【问题标题】：[Bash]Read URL from txt file[Bash]从 txt 文件中读取 URL
【发布时间】：2020-07-23 10:51:54
【问题描述】：

我有一个文本文件，我想从该文本文件中逐一读取 URL，并检查它们是否在 30 天或更长时间内过期。

考试网址：

example01.example.com:8080
example02.example.com:8002
example03.example.com:8003
https://example04.example.com:8111
...

我有一个检查 SSL 证书到期日期的 Bash 脚本：

 #!/bin/bash
TARGET="example01.example.com:8080";
DAYS=30;
echo "checking if $TARGET expires in less than $DAYS days";
expirationdate=$(date -d "$(: | openssl s_client -connect $TARGET -servername $TARGET 2>/dev/null \
                              | openssl x509 -text \
                              | grep 'Not After' \
                              |awk '{print $4,$5,$7}')" '+%s'); 
time=$(($(date +%s) + (86400*$DAYS)));
if [ $time -gt $expirationdate ]; then
    echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d @$expirationdate '+%Y-%m-%d')" ;
    echo $TARGET on $(date -d @$expirationdate '+%Y-%m-%d') > expireEndpoint.txt;
else
    echo "OK - Certificate expires on $(date -d @$expirationdate '+%Y-%m-%d')";
fi;

有些网址也是重复的。

【问题讨论】：

标签： linux bash for-loop awk sed

【解决方案1】：

修改了上面的脚本来修复openssl错误

> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
    do
        if [ -z "$line" ]; then
            continue   # ignore if line is empty
        fi

        protocol=`echo $line | awk -F: '{print $1}'` 
        temp=(${line//:/ })
        PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
        temp=`echo $line | awk -F/ '{print $3}'`
        TARGET=`echo $temp | awk -F: '{print $1}'`
        DAYS=30;

        # 1. Check if it is uses secure HTTP
        if [ $protocol == "https" ]; then
            echo "$TARGET is secure: uses HTTPS"
        else
            echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
            echo "$TARGET uses HTTP: insecure" 
        fi

        # 2. check if server is reachable
        if curl --output /dev/null --silent --head --fail "$TARGET"; then
            echo "$TARGET is reachable "
        else
            echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
            echo "============" >> unreachableOrInsecure.txt
            echo "$TARGET is unreachable " 
            echo "============"
            continue   # do not continue anymore, check the next target
        fi

        # 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
        echo "checking if $TARGET:$PORT expires in less than $DAYS days";
        expirationdate=$(timeout 3  openssl s_client -servername $TARGET -connect $TARGET:$PORT </dev/null 2>/dev/null | \
                         openssl x509 -noout -dates 2>/dev/null | \
                         awk -F= '/^notAfter/ { print $2; exit }')
        expire_epoch=$(date +%s -d "$expirationdate")
        epoch_warning=$(($DAYS*86400)) #look for 30 days
        today_epoch="$(date +%s)"
        timeleft=`expr $expire_epoch - $today_epoch`

        if [[ $timeleft -le $epoch_warning ]]; then 
            echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d @$expire_epoch)" ;
            echo "Cert expires for $TARGET on $(date -d @$expire_epoch '+%Y-%m-%d')" >> certLessThan30Days.txt;
            echo "============" >> certLessThan30Days.txt
        else
            echo "OK - Certificate expires on $(date -d @$expire_epoch)";
            echo "Cert expires for $TARGET on $(date -d @$expire_epoch '+%Y-%m-%d')" >> certMoreThan30Days.txt;
            echo "============" >> certMoreThan30Days.txt
        fi;

        echo "============"
    done < "$input"

现在的输出如下所示

msn.com is secure: uses HTTPS
msn.com is reachable 
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on Thursday 07 April 2022 02:18:44 AM IST
============
google.com is secure: uses HTTPS
google.com is reachable 
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on Wednesday 23 September 2020 02:13:12 AM IST
============
example.com uses HTTP: insecure
example.com is reachable 
checking if example.com:80 expires in less than 30 days
KO - Certificate for example.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable 
checking if www.google.ie: expires in less than 30 days
KO - Certificate for www.google.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable 
checking if www.facebook.com: expires in less than 30 days
KO - Certificate for www.facebook.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable 
checking if www.kooba.ie: expires in less than 30 days
KO - Certificate for www.kooba.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============

【讨论】：

【解决方案2】：

有点老套，但我想出了这个快速而肮脏的解决方案......这是我的看法

编辑：修复了一个小错误

> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
    do
        if [ -z "$line" ]; then
            continue   # ignore if line is empty
        fi

        protocol=`echo $line | awk -F: '{print $1}'` 
        temp=(${line//:/ })
        PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
        temp=`echo $line | awk -F/ '{print $3}'`
        TARGET=`echo $temp | awk -F: '{print $1}'`
        DAYS=30;

        # 1. Check if it is uses secure HTTP
        if [ $protocol == "https" ]; then
            echo "$TARGET is secure: uses HTTPS"
        else
            echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
            echo "$TARGET uses HTTP: insecure" 
        fi

        # 2. check if server is reachable
        if curl --output /dev/null --silent --head --fail "$TARGET"; then
            echo "$TARGET is reachable "
        else
            echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
            echo "============" >> unreachableOrInsecure.txt
            echo "$TARGET is unreachable " 
            echo "============"
            continue   # do not continue anymore, check the next target
        fi

        # 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
        echo "checking if $TARGET:$PORT expires in less than $DAYS days";
        expirationdate=$(date -d "$(: | timeout 3 openssl s_client -connect $TARGET:$PORT -servername $TARGET 2>/dev/null \
                              | openssl x509 -text \
                              | grep 'Not After' \
                              |awk '{print $4,$5,$7}')" '+%s'); 
        time=$(($(date +%s) + (86400*$DAYS)));

        if [ $time -gt $expirationdate ]; then
            echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d @$expirationdate '+%Y-%m-%d')" ;
            echo $TARGET on $(date -d @$expirationdate '+%Y-%m-%d') >> certLessThan30Days.txt;
            echo "============" >> certLessThan30Days.txt
        else
            echo "OK - Certificate expires on $(date -d @$expirationdate '+%Y-%m-%d')";
            echo $TARGET on $(date -d @$expirationdate '+%Y-%m-%d') >> certMoreThan30Days.txt;
            echo "============" >> certMoreThan30Days.txt
        fi;

        echo "============"
    done < "$input"

编辑：添加 url.txt 和输出这是我的 url.txt


    https://msn.com:443/services/f?a
    https://google.com:443
    http://example.com:80/c/ca/s
    https://www.google.ie/ 
    https://www.facebook.com/ 
    https://www.kooba.ie/

脚本的输出

msn.com is secure: uses HTTPS
msn.com is reachable 
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on 2022-04-06
============
google.com is secure: uses HTTPS
google.com is reachable 
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on 2020-09-22
============
example.com uses HTTP: insecure
example.com is reachable 
checking if example.com:80 expires in less than 30 days
unable to load certificate
140024794944832:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for example.com expires in less than 30 days, on 2020-07-23
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable 
checking if www.google.ie: expires in less than 30 days
unable to load certificate
139738534053184:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.google.ie expires in less than 30 days, on 2020-07-23
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable 
checking if www.facebook.com: expires in less than 30 days
unable to load certificate
139623076582720:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.facebook.com expires in less than 30 days, on 2020-07-23
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable 
checking if www.kooba.ie: expires in less than 30 days
unable to load certificate
139831347127616:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.kooba.ie expires in less than 30 days, on 2020-07-23
============

【讨论】：

注意，input="url.txt" 是包含 URL 列表的文件的路径。您可以将其替换为绝对路径。
感谢您的回复。代码效果很好，但有一个小问题。某些原因它不读取 txt 文件的最后一行（跳过最后一行/URL）。你知道为什么吗？还有有没有办法分开 HTTP 和 https ？就像其中一个 URL 证书在本月到期（> 30 天），所以它在 echo "KO - certs..." 和 HTTP（不安全或不可访问的 URL 也属于它）下。所以通过查看输出的txt文件真的很难分辨。有没有办法将它们分开。感谢您的帮助
HTTP 和 HTTPS 在我的脚本中已经是分开的，它是在 $protocol 中提取的。看下面部分代码if [ $protocol == "https" ]; then ...
在我的测试中找不到那个错误。你能提供我你的文本文件来测试吗？
对不起。它的所有内部服务器，所以我无法给你文件。关于https。我的意思是，如果证书在 30 天内过期，它将输出 echo KO - Certificate for URL01 expires in less than 30 days；如果服务器不可访问或不安全，它也会输出相同的语句；好简单的方法：我试图将输出分成三个 txt 文件。 First txt：仅显示证书在 30 天内过期的端点；第二个 txt ：仅显示证书超过 30 天的端点；第三个txt：显示所有其他。即无法到达的端点/HTTP/不安全；