wenku8_fetch_se

From NAT, 6 Years ago, written in Bash, viewed 823 times. This paste is a reply to wenku8_fetch from NAT - view diff

URL https://code.nat.moe/view/cefa1fdc Embed

Download Paste or View Raw

#!/bin/bash
# A tool to download all novels on wenku8.com, the second edition. Minor BUGs fixed.
#
# - Use fallback URL to fetch the real address to ensure that all the novels are downloaded.
# - Removed '?' after the file name.
 
STORE_PATH=./save/
FETCH_URL='http://www.wenku8.com/modules/article/articlelist.php?page='
DOWNLOAD_TYPE="utf8"
DOWNLOAD_URL="http://dl.wenku8.com/txt$DOWNLOAD_TYPE/__K/__ID.txt"
DOWNLOAD_FALLBACK="http://dl.wenku8.com/down.php?type=$DOWNLOAD_TYPE&id="
TEMP=temp.tmp
FROM=1
TO=93
TIMESTEMP_FORMAT="%H:%M:%S"
 
for page in $(seq $FORM $TO)
do
        echo "[$(date +$TIMESTEMP_FORMAT)] Starting page $page..."
        curl $FETCH_URL$page 2> /dev/null > $TEMP
        cat $TEMP | iconv -f gbk -t utf-8 | grep 'font-size:13px;' | sed -e 's/.*book\///g; s/.htm">/ /g; s/<\/a><\/b>//g;' > title$TEMP
        _ids=$(cat $TEMP | iconv -f gbk -t utf-8 | grep 'font-size:13px;' | sed -e 's/.*book\///g; s/\.htm.*//g')
        for novel in $_ids
        do
                echo "[$(date +$TIMESTEMP_FORMAT)] Downloading $(cat title$TEMP|grep $novel)"
                _this_url="$(echo $DOWNLOAD_URL|sed -e "s/__K/1/; s/__ID/$novel/;")"
                _this_save="$STORE_PATH/$(cat title$TEMP|grep $novel|tr ' ' '_'|dos2unix 2>/dev/null)"
                curl $_this_url > $_this_save 2> /dev/null
                [[ ! -z $(cat $_this_save | grep '404 Not Found') ]] && {
                        _this_url="http://dl.wenku8.com$(curl -I "$DOWNLOAD_FALLBACK$novel" 2> /dev/null | grep Location | awk -F:\  '{print $2}')"
                        curl $_this_url > $_this_save 2> /dev/null
                }
        done
done

Replies to wenku8_fetch_se

Title	Name	Language	When
wenku8_fetch_te	NAT	bash	6 Years ago.

Reply to "wenku8_fetch_se"

Here you can reply to the paste above

Author What's your name?

Title Give your paste a title.

Language What language is your paste written in?

Your paste Paste your paste here

#!/bin/bash
# A tool to download all novels on wenku8.com, the second edition. Minor BUGs fixed.
#
# - Use fallback URL to fetch the real address to ensure that all the novels are downloaded.
# - Removed '?' after the file name.

STORE_PATH=./save/
FETCH_URL='http://www.wenku8.com/modules/article/articlelist.php?page='
DOWNLOAD_TYPE="utf8"
DOWNLOAD_URL="http://dl.wenku8.com/txt$DOWNLOAD_TYPE/__K/__ID.txt"
DOWNLOAD_FALLBACK="http://dl.wenku8.com/down.php?type=$DOWNLOAD_TYPE&id="
TEMP=temp.tmp
FROM=1
TO=93
TIMESTEMP_FORMAT="%H:%M:%S"

for page in $(seq $FORM $TO)
do
	echo "[$(date +$TIMESTEMP_FORMAT)] Starting page $page..."
	curl $FETCH_URL$page 2> /dev/null > $TEMP
	cat $TEMP | iconv -f gbk -t utf-8 | grep 'font-size:13px;' | sed -e 's/.*book\///g; s/.htm">/ /g; s/<\/a><\/b>//g;' > title$TEMP
	_ids=$(cat $TEMP | iconv -f gbk -t utf-8 | grep 'font-size:13px;' | sed -e 's/.*book\///g; s/\.htm.*//g')
	for novel in $_ids
	do
		echo "[$(date +$TIMESTEMP_FORMAT)] Downloading $(cat title$TEMP|grep $novel)"
		_this_url="$(echo $DOWNLOAD_URL|sed -e "s/__K/1/; s/__ID/$novel/;")"
		_this_save="$STORE_PATH/$(cat title$TEMP|grep $novel|tr ' ' '_'|dos2unix 2>/dev/null)"
		curl $_this_url > $_this_save 2> /dev/null
		[[ ! -z $(cat $_this_save | grep '404 Not Found') ]] && {
			_this_url="http://dl.wenku8.com$(curl -I "$DOWNLOAD_FALLBACK$novel" 2> /dev/null | grep Location | awk -F:\  '{print $2}')"
			curl $_this_url > $_this_save 2> /dev/null
		}
	done
done

Create Shorturl Create a shorter url that redirects to your paste?

Private Private paste aren't shown in recent listings.

Delete After When should we delete your paste?

Spam protection Type in the letters

nat-code

wenku8_fetch_se

Replies to wenku8_fetch_se

Reply to "wenku8_fetch_se"