As a demonstration using only standard utilities, the three scripts below 1.sh, 2.sh and 3.sh will
1. accept a list of domains and output the required Base64URL-encoded DNS request
2. make a single HTTP connection to retrieve all the responses in binary format and write them to a file
3. convert the binary file to text, suitable for manipulation with text-formatting utilities so the DNS data can be added to HOSTS file or a zone file
bindparser is from curveprotect project; it converts BIND-style output from drill to tinydns zone file format.
Most HTTP servers on the internet do have pipelining enabled, sometimes with a max-limit of 100. The three examples listed below were pipelining enabled with max-limit greater than 100 last time I checked.
Example usage:
1.sh < list-of-domains > 1.txt
2.sh 001 < 1.txt > 1.bin
3.sh < 1.bin > 1.txt
bindparser 1.txt > 1.zone
x=tinydns/root/data;cat 1.zone >> $x/data;cd $x;
awk '!x[$0]++' data > data.tmp;mv data.tmp data;tinydns-data;cd -
# 1.sh
#!/bin/sh
while IFS= read -r x;do printf $x' ';
drill -q /dev/stdout $x @0.0.0.0 a|sed 's/;.*//'|xxd -p -r \
|openssl base64|sed 's/+/-/g;s|/|_|g;s/=.*$//'|tr -d '\n';echo;
done
# 2.sh
#!/bin/sh
case $1 in
"")sed '/;;[0-9]*/!d' $0;
echo usage: $0 provider-number \< output-from-script1.txt;exit
;;001)x=doh.powerdns.org;y=1
;;002)x=ibuki.cgnat.net;y=1
;;003)x=dns.aa.net.uk;y=1
esac;
case $y in
1) sed 's/\(.* \)\(.*\)/GET \/dns-query?dns=\2 HTTP\/1.1\r\nHost: \1\r\n/;
$!s/$/Connection: keep-alive\r\n/;$s/$/Connection: close\r\n\r\n/;' \
|socat -,ignoreeof ssl:$x:443,verify=0 > 1.bin
esac;
3.sh < 1.bin
# 3.sh
#!/bin/sh
while IFS= read -r x;do sed -n /${1-\.}/p\;/${1-\.}/q|xxd -p|drill -i /dev/stdin 2>/dev/null;done
