Difuse/packages
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

adblock: release 3.0

Browse source 
* add kresd & turris omnia support
* add dnscrypt-proxy support
* change start priority to 30, to fix possible trigger issues on slow
booting hardware
* simplify suspend/resume handling (no longer use a hideout directory)
* enhanced LuCI frontend
* many small changes & improvements
* default config change (please update your config!), adblock is now
disabled by default
* documentation update

Signed-off-by: Dirk Brenken <dev@brenken.org>
This commit is contained in:
Dirk Brenken 2017-09-09 17:35:42 +02:00
parent 7cdb6c7031
commit 206c847bb5
5 changed files with 238 additions and 188 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
net/adblock/Makefile
View file

@ -6,7 +6,7 @@
include $(TOPDIR)/rules.mk include $(TOPDIR)/rules.mk
PKG_NAME:=adblock PKG_NAME:=adblock
PKG_VERSION:=2.8.5 PKG_VERSION:=3.0.0
PKG_RELEASE:=1 PKG_RELEASE:=1
PKG_LICENSE:=GPL-3.0+ PKG_LICENSE:=GPL-3.0+
PKG_MAINTAINER:=Dirk Brenken <dev@brenken.org> PKG_MAINTAINER:=Dirk Brenken <dev@brenken.org>

104
net/adblock/files/README.md
View file

@ -29,6 +29,8 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* => daily updates, approx. 150 entries * => daily updates, approx. 150 entries
* [reg_cn](https://easylist-downloads.adblockplus.org/easylistchina+easylist.txt) * [reg_cn](https://easylist-downloads.adblockplus.org/easylistchina+easylist.txt)
* => regional blocklist for China, daily updates, approx. 1.600 entries * => regional blocklist for China, daily updates, approx. 1.600 entries
* [reg_id](https://easylist-downloads.adblockplus.org/abpindo+easylist.txt)
* => regional blocklist for Indonesia, daily updates, approx. 800 entries
* [reg_nl](https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt) * [reg_nl](https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt)
* => regional blocklist for the Netherlands, weekly updates, approx. 1300 entries * => regional blocklist for the Netherlands, weekly updates, approx. 1300 entries
* [reg_pl](http://adblocklist.org) * [reg_pl](http://adblocklist.org)
@ -57,12 +59,12 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* => daily updates, approx. 440 entries * => daily updates, approx. 440 entries
* zero-conf like automatic installation & setup, usually no manual changes needed * zero-conf like automatic installation & setup, usually no manual changes needed
* simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that * simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
* automatically selects dnsmasq, unbound, bind or kresd (experimental!) as dns backend. * supports five different dns backends / block list formats: dnsmasq, unbound, named (bind), kresd and dnscrypt-proxy
* automatically selects uclient-fetch or wget as download utility (other tools like curl or aria2c are supported as well) * automatically selects uclient-fetch or wget as download utility (other tools like curl or aria2c are supported as well)
* support http only mode (without installed ssl library) for all non-SSL blocklist sources * provides 'http only' mode without installed ssl library for all non-SSL block list sources
* automatically supports a wide range of router modes, even AP modes are supported * supports a wide range of router modes, even AP modes are supported
* full IPv4 and IPv6 support * full IPv4 and IPv6 support
* supports tld compression (top level domain compression), this feature removes thousands of needless host entries from the block list and lowers the memory footprint for the dns backends * provides top level domain compression ('tld compression'), this feature removes thousands of needless host entries from the block list and lowers the memory footprint for the dns backends
* each block list source will be updated and processed separately * each block list source will be updated and processed separately
* block list source parsing by fast & flexible regex rulesets * block list source parsing by fast & flexible regex rulesets
* overall duplicate removal in central block list (adb_list.overall) * overall duplicate removal in central block list (adb_list.overall)
@ -70,10 +72,11 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* quality checks during block list update to ensure a reliable dns backend service * quality checks during block list update to ensure a reliable dns backend service
* minimal status & error logging to syslog, enable debug logging to receive more output * minimal status & error logging to syslog, enable debug logging to receive more output
* procd based init system support (start/stop/restart/reload/suspend/resume/query/status) * procd based init system support (start/stop/restart/reload/suspend/resume/query/status)
* procd based hotplug support, the adblock start will be solely triggered by network interface triggers * procd network interface trigger support or classic time based startup
* suspend & resume adblock actions temporarily without block list reloading * suspend & resume adblock actions temporarily without block list reloading
* runtime information available via LuCI & via 'status' init command * output comprehensive runtime information via LuCI or via 'status' init command
* query function to quickly identify blocked (sub-)domains, e.g. for whitelisting * query function to quickly identify blocked (sub-)domains, e.g. for whitelisting
* strong LuCI support
* optional: force dns requests to local resolver * optional: force dns requests to local resolver
* optional: force overall sort / duplicate removal for low memory devices (handle with care!) * optional: force overall sort / duplicate removal for low memory devices (handle with care!)
* optional: 'manual mode' to re-use blocklist backups during startup, get fresh lists only via manual reload or restart action * optional: 'manual mode' to re-use blocklist backups during startup, get fresh lists only via manual reload or restart action
@ -85,16 +88,16 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported * a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
* a download utility: * a download utility:
* to support all blocklist sources a full version (with ssl support) of 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required * to support all blocklist sources a full version (with ssl support) of 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required
* for limited devices with real memory constraints, adblock provides also a plain http option and supports wget-nossl and uclient-fetch (without libustream-ssl), too * for limited devices with real memory constraints, adblock provides also a 'http only' option and supports wget-nossl and uclient-fetch (without libustream-ssl) as well
* for more configuration options see examples below * for more configuration options see examples below
## LEDE trunk Installation & Usage ## Installation & Usage
* install 'adblock' (_opkg install adblock_) and that's it - the adblock start will be automatically triggered by procd interface trigger * install 'adblock' (_opkg install adblock_)
* at minimum configure the appropriate dns backend ('dnsmasq' by default) and enable the adblock service in _/etc/config/adblock_
* control the adblock service manually with _/etc/init.d/adblock_ start/stop/restart/reload/suspend/resume/status or use the LuCI frontend * control the adblock service manually with _/etc/init.d/adblock_ start/stop/restart/reload/suspend/resume/status or use the LuCI frontend
* enable/disable your favored block list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
## LuCI adblock companion package ## LuCI adblock companion package
* for easy management of the various block list sources and all other adblock options you can also use a nice & efficient LuCI frontend * for easy management of the various block list sources and all other adblock options you should use the provided LuCI frontend
* install 'luci-app-adblock' (_opkg install luci-app-adblock_) * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
* the application is located in LuCI under 'Services' menu * the application is located in LuCI under 'Services' menu
@ -105,7 +108,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* **add white- / blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below) * **add white- / blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
* **backup & restore block lists:** enable this feature, to restore automatically the latest compressed backup of your block lists in case of any processing error (e.g. a single block list source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this * **backup & restore block lists:** enable this feature, to restore automatically the latest compressed backup of your block lists in case of any processing error (e.g. a single block list source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
* **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below) * **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
* **restrict procd interface trigger:** restrict the procd interface trigger to a (list of) certain interface(s) (default: wan). To disable it at all, remove all entries * **change startup behaviour:** by default the startup will be triggered by the 'wan' procd interface trigger. Choose 'none' to disable automatic startups, 'timed' to use a classic timeout (default 30 sec.) or select another trigger interface.
* **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_ * **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
* **domain query:** to query the active block list for a specific domain, please run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below) * **domain query:** to query the active block list for a specific domain, please run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
* **add new list sources:** you could add new block list sources on your own via uci config, all you need is a source url and an awk one-liner (see example below) * **add new list sources:** you could add new block list sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
@ -113,12 +116,14 @@ A lot of people already use adblocker plugins within their desktop browsers, but
## Further adblock config options ## Further adblock config options
* usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section: * usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section:
* adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled) * adb\_enabled => main switch to enable/disable adblock service (default: '0', disabled)
* adb\_debug => enable/disable adblock debug output (default: '0', disabled) * adb\_debug => enable/disable adblock debug output (default: '0', disabled)
* adb\_iface => set the procd interface trigger to a (list of) lan / wan interface(s) (default: 'wan') * adb\_dns => select the dns backend for your environment: 'dnsmasq', 'unbound', 'named', 'kresd' or 'dnscrypt-proxy' (default: 'dnsmasq')
* adb\_fetch => full path to a different download utility, see example below (default: not set, use wget) * adb\_dnsdir => target directory for the generated blocklist 'adb_list.overall' (default: not set, use dns backend default)
* adb\_fetchparm => options for the download utility, see example below (default: not set, use wget options) * adb\_trigger => set the startup trigger to a certain interface, to 'timed' or to 'none' (default: 'wan')
* adb\_triggerdelay => additional trigger delay in seconds before adblock processing starts (default: '2') * adb\_triggerdelay => additional trigger delay in seconds before adblock processing begins (default: '1')
* adb\_fetch => full path to a dedicated download utility, see example below (default: not set, use wget default)
* adb\_fetchparm => options for the download utility, see example below (default: not set, use wget default options)
* adb\_forcedns => force dns requests to local resolver (default: '0', disabled) * adb\_forcedns => force dns requests to local resolver (default: '0', disabled)
* adb\_forcesrt => force overall sort on low memory devices with less than 64 MB RAM (default: '0', disabled) * adb\_forcesrt => force overall sort on low memory devices with less than 64 MB RAM (default: '0', disabled)
* adb\_manmode => do not automatically update block lists during startup, use backups instead (default: '0', disabled) * adb\_manmode => do not automatically update block lists during startup, use backups instead (default: '0', disabled)
@ -132,7 +137,7 @@ If you use manual configuration for unbound, then just include the following lin
include: "/var/lib/unbound/adb_list.overall" include: "/var/lib/unbound/adb_list.overall"
</code></pre> </code></pre>
**change default dns backend to 'bind':** **change default dns backend to 'named' (bind):**
<pre><code> <pre><code>
Adblock deposits the sorted and filtered block list (adb_list.overall) in '/var/lib/bind' where bind can find them. Adblock deposits the sorted and filtered block list (adb_list.overall) in '/var/lib/bind' where bind can find them.
To use the block list please modify the following bind configuration files: To use the block list please modify the following bind configuration files:
@ -159,9 +164,9 @@ create the new file '/etc/bind/db.rpz' and add:
**change default dns backend to 'kresd':** **change default dns backend to 'kresd':**
<pre><code> <pre><code>
The knot-resolver (kresd) support is only available to turris omnia users. At this stage there's no package for kresd in the official LEDE / OpenWrt package repository. The knot-resolver (kresd) is only available on turris omnia devices. Currently there's no package for kresd in the official LEDE / OpenWrt package repository.
Adblock deposits the sorted and filtered block list (adb_list.overall) in '/tmp/kresd' where kresd can find them. Adblock deposits the sorted and filtered block list (adb_list.overall) in '/etc/kresd' where kresd can find them.
To use the block list please modify the following kresd configuration files (experimental / untested!): To use the block list please create/modify the following kresd configuration files:
edit '/etc/config/resolver' and uncomment the following option: edit '/etc/config/resolver' and uncomment the following option:
option include_config '/etc/kresd/custom.conf' option include_config '/etc/kresd/custom.conf'
@ -169,25 +174,27 @@ edit '/etc/config/resolver' and uncomment the following option:
in the same file change the 'forward_upstream' option like that: in the same file change the 'forward_upstream' option like that:
forward_upstream '0' forward_upstream '0'
edit '/etc/kresd/custom.conf' and add: create '/etc/kresd/custom.conf' and add:
policy.add(policy.rpz(policy.DENY, '/etc/kresd/db.rpz')) policy.add(policy.rpz(policy.DENY, '/etc/kresd/adb_list.overall'))
policy.add(policy.all(policy.FORWARD('8.8.8.8'))) policy.add(policy.all(policy.FORWARD('8.8.8.8')))
policy.add(policy.all(policy.FORWARD('8.8.4.4'))) policy.add(policy.all(policy.FORWARD('8.8.4.4')))
</code></pre>
**change default dns backend to 'dnscrypt-proxy':**
<pre><code>
Adblock deposits the sorted and filtered block list (adb_list.overall) by default in '/tmp' where DNSCrypt-Proxy can find them.
The blacklist option is not supported by default, because DNSCrypt-Proxy is compiled without plugins support.
Take a custom LEDE build with plugins support to use this feature:
create the new file '/etc/kresd/db.rpz' and add: edit '/etc/config/dnscrypt-proxy' and add the following option per dnscrypt-proxy instance:
$TTL 2h list blacklist 'domains:/tmp/adb_list.overall'
$ORIGIN rpz.
@ SOA localhost. root.localhost. (1 6h 1h 1w 2h)
NS localhost.
$INCLUDE /tmp/kresd/adb_list.overall
</code></pre> </code></pre>
**configuration for different download utilities:** **configuration for different download utilities:**
<pre><code> <pre><code>
wget (default): wget (default):
option adb_fetch="/usr/bin/wget" option adb_fetch '/usr/bin/wget'
option adb_fetchparm="--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 --no-check-certificate -O" option adb_fetchparm '--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 --no-check-certificate -O'
aria2c: aria2c:
option adb_fetch '/usr/bin/aria2c' option adb_fetch '/usr/bin/aria2c'
@ -206,13 +213,13 @@ curl:
<pre><code> <pre><code>
/etc/init.d/adblock status /etc/init.d/adblock status
::: adblock runtime information ::: adblock runtime information
status : active + adblock_status : enabled
adblock_version : 2.8.0 + adblock_version : 3.0.0
blocked_domains : 122827 + blocked_domains : 37406
fetch_info : wget (built-in) + fetch_utility : wget (built-in)
dns_backend : dnsmasq + dns_backend : kresd (/etc/kresd)
last_rundate : 26.06.2017 17:00:27 + last_rundate : 08.09.2017 21:21:21
system : LEDE Reboot SNAPSHOT r4434-b91a38d647 + system_release : OpenWrt omnia 15.05
</code></pre> </code></pre>
**cronjob for a regular block list update (/etc/crontabs/root):** **cronjob for a regular block list update (/etc/crontabs/root):**
@ -250,17 +257,16 @@ This entry does not remove:
**query active block list for a certain (sub-)domain, e.g. for whitelisting:** **query active block list for a certain (sub-)domain, e.g. for whitelisting:**
<pre><code> <pre><code>
/etc/init.d/adblock query example.www.doubleclick.net /etc/init.d/adblock query example.www.doubleclick.net
::: results for (sub-)domain 'example.www.doubleclick.net' (max. 5) root@turris:~# /etc/init.d/adblock query example.www.doubleclick.net
- no match ::: max. ten results for domain 'example.www.doubleclick.net'
::: results for (sub-)domain 'www.doubleclick.net' (max. 5) - no match
- no match ::: max. ten results for domain 'www.doubleclick.net'
::: results for (sub-)domain 'doubleclick.net' (max. 5) - no match
+ doubleclick.net ::: max. ten results for domain 'doubleclick.net'
+ feedads.g.doubleclick.net + doubleclick.net
+ survey.g.doubleclick.net
The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain(s). The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain.
For every (sub-)domain it returns the first five relevant results. For every (sub-)domain it returns the first ten relevant results.
In the example above whitelist "doubleclick.net" to free the submitted domain. In the example above whitelist "doubleclick.net" to free the submitted domain.
</code></pre> </code></pre>

12
net/adblock/files/adblock.conf
View file

@ -2,12 +2,12 @@
# see 'https://github.com/openwrt/packages/blob/master/net/adblock/files/README.md' # see 'https://github.com/openwrt/packages/blob/master/net/adblock/files/README.md'
config adblock 'global' config adblock 'global'
option adb_enabled '1' option adb_enabled '0'
option adb_debug '0' option adb_debug '0'
option adb_forcesrt '0' option adb_forcesrt '0'
option adb_forcedns '0' option adb_forcedns '0'
option adb_iface 'wan' option adb_dns 'dnsmasq'
option adb_triggerdelay '2' option adb_trigger 'wan'
option adb_whitelist '/etc/adblock/adblock.whitelist' option adb_whitelist '/etc/adblock/adblock.whitelist'
option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}' option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}'
option adb_backup '0' option adb_backup '0'
@ -86,6 +86,12 @@ config source 'reg_cn'
option adb_src_rset '{FS=\"[|^]\"} \$0 ~/^\|\|([A-Za-z0-9_-]+\.){1,}[A-Za-z]+\^$/{print tolower(\$3)}' option adb_src_rset '{FS=\"[|^]\"} \$0 ~/^\|\|([A-Za-z0-9_-]+\.){1,}[A-Za-z]+\^$/{print tolower(\$3)}'
option adb_src_desc 'focus on chinese ads, daily updates, approx. 1.600 entries' option adb_src_desc 'focus on chinese ads, daily updates, approx. 1.600 entries'
config source 'reg_id'
option enabled '0'
option adb_src 'https://easylist-downloads.adblockplus.org/abpindo+easylist.txt'
option adb_src_rset '{FS=\"[|^]\"} \$0 ~/^\|\|([A-Za-z0-9_-]+\.){1,}[A-Za-z]+\^$/{print tolower(\$3)}'
option adb_src_desc 'focus on indonesian ads plus generic easylist additions, weekly updates, approx. 800 entries'
config source 'reg_nl' config source 'reg_nl'
option enabled '0' option enabled '0'
option adb_src 'https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt' option adb_src 'https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt'

26
net/adblock/files/adblock.init
View file

@ -1,7 +1,7 @@
#!/bin/sh /etc/rc.common #!/bin/sh /etc/rc.common
# #
START=50 START=30
USE_PROCD=1 USE_PROCD=1
EXTRA_COMMANDS="suspend resume query status" EXTRA_COMMANDS="suspend resume query status"
@ -16,7 +16,6 @@ adb_script="/usr/bin/adblock.sh"
boot() boot()
{ {
adb_boot=1 adb_boot=1
ubus -t 30 wait_for network.interface 2>/dev/null
rc_procd start_service rc_procd start_service
} }
@ -26,7 +25,11 @@ start_service()
then then
if [ -n "${adb_boot}" ] if [ -n "${adb_boot}" ]
then then
return 0 local trigger="$(uci_get adblock.global.adb_trigger)"
if [ "${trigger}" != "timed" ]
then
return 0
fi
fi fi
procd_open_instance "adblock" procd_open_instance "adblock"
procd_set_param command "${adb_script}" "${@}" procd_set_param command "${adb_script}" "${@}"
@ -74,14 +77,13 @@ status()
service_triggers() service_triggers()
{ {
local iface="$(uci -q get adblock.global.adb_iface)" local trigger="$(uci_get adblock.global.adb_trigger)"
local delay="$(uci -q get adblock.global.adb_triggerdelay)" local delay="$(uci_get adblock.global.adb_triggerdelay)"
PROCD_RELOAD_DELAY=$((${delay:=2} * 1000)) if [ "${trigger}" != "none" ] && [ "${trigger}" != "timed" ]
for name in ${iface} then
do PROCD_RELOAD_DELAY=$((${delay:=1} * 1000))
procd_add_interface_trigger "interface.*.up" "${name}" "${adb_init}" start procd_add_interface_trigger "interface.*.up" "${trigger}" "${adb_init}" start
done fi
PROCD_RELOAD_DELAY=1000 procd_add_reload_trigger "adblock"
procd_add_config_trigger "config.change" "adblock" "${adb_init}" start
} }

282
net/adblock/files/adblock.sh
View file

@ -10,33 +10,31 @@
# #
LC_ALL=C LC_ALL=C
PATH="/usr/sbin:/usr/bin:/sbin:/bin" PATH="/usr/sbin:/usr/bin:/sbin:/bin"
adb_ver="2.8.5" adb_ver="3.0.0"
adb_sysver="$(ubus -S call system board | jsonfilter -e '@.release.description')" adb_sysver="$(ubus -S call system board | jsonfilter -e '@.release.description')"
adb_enabled=1 adb_enabled=0
adb_debug=0 adb_debug=0
adb_minfree=2
adb_manmode=0 adb_manmode=0
adb_forcesrt=0 adb_forcesrt=0
adb_forcedns=0 adb_forcedns=0
adb_triggerdelay=0
adb_backup=0 adb_backup=0
adb_backupdir="/mnt" adb_backupdir="/mnt"
adb_whitelist="/etc/adblock/adblock.whitelist"
adb_whitelist_rset="\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}"
adb_fetch="/usr/bin/wget" adb_fetch="/usr/bin/wget"
adb_fetchparm="--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 --no-check-certificate -O" adb_fetchparm="--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 --no-check-certificate -O"
adb_dnslist="dnsmasq unbound named kresd" adb_dns="dnsmasq"
adb_dnsprefix="adb_list" adb_dnsprefix="adb_list"
adb_dnsfile="${adb_dnsprefix}.overall" adb_dnsfile="${adb_dnsprefix}.overall"
adb_rtfile="/tmp/adb_runtime.json" adb_rtfile="/tmp/adb_runtime.json"
adb_sources="" adb_action="${1:-"start"}"
adb_src_cat_shalla="" adb_cnt=0
adb_action="${1}" adb_rc=0
# f_envload: load adblock environment # f_envload: load adblock environment
# #
f_envload() f_envload()
{ {
local services dns_up cnt=0 local dns_up cnt=0
# source in system libraries # source in system libraries
# #
@ -89,55 +87,60 @@ f_envload()
# set dns backend environment # set dns backend environment
# #
while [ ${cnt} -le 20 ] case "${adb_action}" in
start|restart|reload)
> "${adb_rtfile}"
if [ "${adb_action}" = "start" ] && [ "${adb_trigger}" = "timed" ]
then
sleep ${adb_triggerdelay}
fi
;;
esac
while [ ${cnt} -le 30 ]
do do
services="$(ubus -S call service list 2>/dev/null)" dns_up="$(ubus -S call service list "{\"name\":\"${adb_dns}\"}" 2>/dev/null | jsonfilter -l1 -e "@[\"${adb_dns}\"].instances.*.running" 2>/dev/null)"
if [ -n "${services}" ] if [ "${dns_up}" = "true" ]
then then
for dns in ${adb_dnslist} case "${adb_dns}" in
do dnsmasq)
dns_up="$(printf "%s" "${services}" | jsonfilter -l1 -e "@.${dns}.instances.*.running")" adb_dnsuser="${adb_dns}"
if [ "${dns_up}" = "true" ] adb_dnsdir="${adb_dnsdir:-"/tmp/dnsmasq.d"}"
then adb_dnsformat="awk '{print \"local=/\"\$0\"/\"}'"
case "${dns}" in break
dnsmasq) ;;
adb_dns="${dns}" unbound)
adb_dnsdir="${adb_dnsdir:="/tmp/dnsmasq.d"}" adb_dnsuser="${adb_dns}"
adb_dnshidedir="${adb_dnsdir}/.adb_hidden" adb_dnsdir="${adb_dnsdir:-"/var/lib/unbound"}"
adb_dnsformat="awk '{print \"local=/\"\$0\"/\"}'" adb_dnsformat="awk '{print \"local-zone: \042\"\$0\"\042 static\"}'"
break 2 break
;; ;;
unbound) named)
adb_dns="${dns}" adb_dnsuser="bind"
adb_dnsdir="${adb_dnsdir:="/var/lib/unbound"}" adb_dnsdir="${adb_dnsdir:-"/var/lib/bind"}"
adb_dnshidedir="${adb_dnsdir}/.adb_hidden" adb_dnsformat="awk '{print \"\"\$0\" IN CNAME .\n*.\"\$0\" IN CNAME .\"}'"
adb_dnsformat="awk '{print \"local-zone: \042\"\$0\"\042 static\"}'" break
break 2 ;;
;; kresd)
named) adb_dnsuser="root"
adb_dns="${dns}" adb_dnsdir="${adb_dnsdir:-"/etc/kresd"}"
adb_dnsdir="${adb_dnsdir:="/var/lib/bind"}" adb_dnsformat="awk '{print \"\"\$0\" CNAME .\n*.\"\$0\" CNAME .\"}'"
adb_dnshidedir="${adb_dnsdir}/.adb_hidden" adb_dnsheader="\$TTL 2h"$'\n'"@ IN SOA localhost. root.localhost. (2 6h 1h 1w 2h)"$'\n'" IN NS localhost."
adb_dnsformat="awk '{print \"\"\$0\" IN CNAME .\n*.\"\$0\" IN CNAME .\"}'" break
break 2 ;;
;; dnscrypt-proxy)
kresd) adb_dnsuser="nobody"
adb_dns="${dns}" adb_dnsdir="${adb_dnsdir:-"/tmp"}"
adb_dnsdir="${adb_dnsdir:="/tmp/kresd"}" adb_dnsformat="awk '{print \$0}'"
adb_dnshidedir="${adb_dnsdir}/.adb_hidden" break
adb_dnsformat="awk '{print \"\"\$0\" CNAME .\n*.\"\$0\" CNAME .\"}'" ;;
break 2 esac
;;
esac
fi
done
fi fi
sleep 1 sleep 1
cnt=$((cnt+1)) cnt=$((cnt+1))
done done
if [ -z "${adb_dns}" ] || [ -z "${adb_dnsformat}" ] || [ ! -x "$(command -v ${adb_dns})" ] || [ ! -d "${adb_dnsdir}" ] if [ -z "${adb_dns}" ] || [ -z "${adb_dnsformat}" ] || [ ! -x "$(command -v ${adb_dns})" ] || [ ! -d "${adb_dnsdir}" ]
then then
f_log "error" "no active/supported DNS backend found" f_log "error" "'${adb_dns}' not running, DNS backend not found"
fi fi
# force dns to local resolver # force dns to local resolver
@ -180,6 +183,7 @@ f_envcheck()
f_rmdns f_rmdns
f_dnsrestart f_dnsrestart
fi fi
f_jsnupdate
f_log "info " "adblock is currently disabled, please set adb_enabled to '1' to use this service" f_log "info " "adblock is currently disabled, please set adb_enabled to '1' to use this service"
exit 0 exit 0
fi fi
@ -192,7 +196,8 @@ f_envcheck()
if [ "$(readlink -fn "${adb_fetch}")" = "/usr/bin/wget-nossl" ] if [ "$(readlink -fn "${adb_fetch}")" = "/usr/bin/wget-nossl" ]
then then
adb_fetchparm="--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 -O" adb_fetchparm="--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 -O"
elif [ "$(readlink -fn "/bin/wget")" = "/bin/busybox" ] || [ "$(readlink -fn "${adb_fetch}")" = "/bin/busybox" ] elif [ "$(readlink -fn "${adb_fetch}")" = "/bin/busybox" ] ||
([ "$(readlink -fn "/bin/wget")" = "/bin/busybox" ] && [ "$(readlink -fn "${adb_fetch}")" != "/usr/bin/wget" ])
then then
adb_fetch="/bin/busybox" adb_fetch="/bin/busybox"
adb_fetchparm="-q -O" adb_fetchparm="-q -O"
@ -217,16 +222,6 @@ f_envcheck()
fi fi
adb_fetchinfo="${adb_fetch##*/} (${ssl_lib})" adb_fetchinfo="${adb_fetch##*/} (${ssl_lib})"
# create dns hideout directory
#
if [ ! -d "${adb_dnshidedir}" ]
then
mkdir -p -m 660 "${adb_dnshidedir}"
chown -R "${adb_dns}":"${adb_dns}" "${adb_dnshidedir}" 2>/dev/null
else
rm -f "${adb_dnshidedir}/${adb_dnsprefix}"*
fi
# create adblock temp file/directory # create adblock temp file/directory
# #
adb_tmpload="$(mktemp -tu)" adb_tmpload="$(mktemp -tu)"
@ -235,7 +230,7 @@ f_envcheck()
# prepare whitelist entries # prepare whitelist entries
# #
if [ -s "${adb_whitelist}" ] if [ -s "${adb_whitelist}" ] && [ -n "${adb_whitelist_rset}" ]
then then
awk "${adb_whitelist_rset}" "${adb_whitelist}" > "${adb_tmpdir}/tmp.whitelist" awk "${adb_whitelist_rset}" "${adb_whitelist}" > "${adb_tmpdir}/tmp.whitelist"
fi fi
@ -259,10 +254,10 @@ f_rmdns()
{ {
if [ -n "${adb_dns}" ] if [ -n "${adb_dns}" ]
then then
rm -f "${adb_dnsdir}/${adb_dnsprefix}"* > "${adb_dnsdir}/${adb_dnsfile}"
rm -f "${adb_backupdir}/${adb_dnsprefix}"*.gz
rm -rf "${adb_dnshidedir}"
> "${adb_rtfile}" > "${adb_rtfile}"
rm -f "${adb_dnsdir}/.${adb_dnsfile}"
rm -f "${adb_backupdir}/${adb_dnsprefix}"*.gz
fi fi
} }
@ -270,19 +265,15 @@ f_rmdns()
# #
f_dnsrestart() f_dnsrestart()
{ {
local dns_up mem_free cnt=0 local dns_up cnt=0
"/etc/init.d/${adb_dns}" restart >/dev/null 2>&1 "/etc/init.d/${adb_dns}" restart >/dev/null 2>&1
while [ ${cnt} -le 10 ] while [ ${cnt} -le 10 ]
do do
dns_up="$(ubus -S call service list "{\"name\":\"${adb_dns}\"}" | jsonfilter -l1 -e "@.${adb_dns}.instances.*.running")" dns_up="$(ubus -S call service list "{\"name\":\"${adb_dns}\"}" | jsonfilter -l1 -e "@[\"${adb_dns}\"].instances.*.running")"
if [ "${dns_up}" = "true" ] if [ "${dns_up}" = "true" ]
then then
mem_free="$(awk '/^MemFree/ {print int($2/1000)}' "/proc/meminfo")" return 0
if [ ${mem_free} -ge ${adb_minfree} ]
then
return 0
fi
fi fi
cnt=$((cnt+1)) cnt=$((cnt+1))
sleep 1 sleep 1
@ -334,6 +325,11 @@ f_list()
else else
eval "${adb_dnsformat}" "${adb_tmpdir}/${adb_dnsfile}" > "${adb_dnsdir}/${adb_dnsfile}" eval "${adb_dnsformat}" "${adb_tmpdir}/${adb_dnsfile}" > "${adb_dnsdir}/${adb_dnsfile}"
fi fi
if [ -n "${adb_dnsheader}" ]
then
printf '%s\n' "${adb_dnsheader}" | cat - "${adb_dnsdir}/${adb_dnsfile}" > "${adb_tmpdir}/${adb_dnsfile}"
cat "${adb_tmpdir}/${adb_dnsfile}" > "${adb_dnsdir}/${adb_dnsfile}"
fi
adb_rc=${?} adb_rc=${?}
;; ;;
esac esac
@ -358,25 +354,24 @@ f_switch()
{ {
local source target status mode="${1}" local source target status mode="${1}"
if [ -d "${adb_dnshidedir}" ] if [ -s "${adb_dnsdir}/${adb_dnsfile}" ] && [ "${mode}" = "suspend" ]
then then
if [ -s "${adb_dnsdir}/${adb_dnsfile}" ] && [ "${mode}" = "suspend" ] source="${adb_dnsdir}/${adb_dnsfile}"
then target="${adb_dnsdir}/.${adb_dnsfile}"
source="${adb_dnsdir}/${adb_dnsfile}" status="suspended"
target="${adb_dnshidedir}" elif [ -s "${adb_dnsdir}/.${adb_dnsfile}" ] && [ "${mode}" = "resume" ]
status="suspended" then
elif [ -s "${adb_dnshidedir}/${adb_dnsfile}" ] && [ "${mode}" = "resume" ] source="${adb_dnsdir}/.${adb_dnsfile}"
then target="${adb_dnsdir}/${adb_dnsfile}"
source="${adb_dnshidedir}/${adb_dnsfile}" status="resumed"
target="${adb_dnsdir}" fi
status="resumed" if [ -n "${status}" ]
fi then
if [ -n "${status}" ] cat "${source}" > "${target}"
then > "${source}"
mv -f "${source}"* "${target}" f_dnsrestart
f_dnsrestart f_jsnupdate
f_log "info " "adblock processing ${status}" f_log "info " "adblock processing ${status}"
fi
fi fi
} }
@ -393,21 +388,74 @@ f_query()
printf "%s\n" "::: no active block list found, please start / resume adblock first" printf "%s\n" "::: no active block list found, please start / resume adblock first"
elif [ -z "${domain}" ] || [ "${domain}" = "${tld}" ] elif [ -z "${domain}" ] || [ "${domain}" = "${tld}" ]
then then
printf "%s\n" "::: invalid domain input, please submit a specific (sub-)domain, e.g. 'www.abc.xyz'" printf "%s\n" "::: invalid domain input, please submit a single domain, e.g. 'doubleclick.net'"
else else
cd "${adb_dnsdir}" cd "${adb_dnsdir}"
while [ "${domain}" != "${tld}" ] while [ "${domain}" != "${tld}" ]
do do
search="${domain//./\.}" search="${domain//./\.}"
result="$(grep -Hm5 "[/\"\.]${search}[/\"]" "${adb_dnsfile}" | awk -F ':|=|/|\"' '{printf(" + %s\n",$4)}')" if [ "${adb_dns}" = "dnsmasq" ] || [ "${adb_dns}" = "unbound" ]
printf "%s\n" "::: results for (sub-)domain '${domain}' (max. 5)" then
printf "%s\n" "${result:=" - no match"}" result="$(awk -F '/|\"' "/[\/\"\.]${search}/{i++;{printf(\" + %s\n\",\$2)};if(i>9){exit}}" "${adb_dnsfile}")"
else
result="$(awk "/(^[^\*][a-z]*[\.]+${search}|^${search})/{i++;{printf(\" + %s\n\",\$1)};if(i>9){exit}}" "${adb_dnsfile}")"
fi
printf "%s\n" "::: max. ten results for domain '${domain}'"
printf "%s\n" "${result:-" - no match"}"
domain="${tld}" domain="${tld}"
tld="${domain#*.}" tld="${domain#*.}"
done done
fi fi
} }
# f_jsnupdate: update runtime information
#
f_jsnupdate()
{
local status rundate="$(/bin/date "+%d.%m.%Y %H:%M:%S")"
if [ ${adb_rc} -gt 0 ]
then
status="error"
elif [ ${adb_enabled} -ne 1 ]
then
status="disabled"
elif [ -s "${adb_dnsdir}/.${adb_dnsfile}" ]
then
status="paused"
else
status="enabled"
if [ -s "${adb_dnsdir}/${adb_dnsfile}" ]
then
if [ "${adb_dns}" = "named" ] || [ "${adb_dns}" = "kresd" ]
then
adb_cnt="$(( ( $(wc -l < "${adb_dnsdir}/${adb_dnsfile}") - $(printf "%s" "${adb_dnsheader}" | grep -c "^") ) / 2 ))"
else
adb_cnt="$(( $(wc -l < "${adb_dnsdir}/${adb_dnsfile}") - $(printf "%s" "${adb_dnsheader}" | grep -c "^") ))"
fi
fi
fi
if [ -z "${adb_fetchinfo}" ] && [ -s "${adb_rtfile}" ]
then
json_load "$(cat "${adb_rtfile}" 2>/dev/null)"
json_select data
json_get_var adb_fetchinfo "fetch_utility"
fi
json_init
json_add_object "data"
json_add_string "adblock_status" "${status}"
json_add_string "adblock_version" "${adb_ver}"
json_add_string "blocked_domains" "${adb_cnt}"
json_add_string "fetch_utility" "${adb_fetchinfo}"
json_add_string "dns_backend" "${adb_dns} (${adb_dnsdir})"
json_add_string "last_rundate" "${rundate}"
json_add_string "system_release" "${adb_sysver}"
json_close_object
json_dump > "${adb_rtfile}"
}
# f_status: output runtime information # f_status: output runtime information
# #
f_status() f_status()
@ -416,21 +464,14 @@ f_status()
if [ -s "${adb_rtfile}" ] if [ -s "${adb_rtfile}" ]
then then
if [ -s "${adb_dnsdir}/${adb_dnsfile}" ]
then
value="active"
else
value="no domains blocked"
fi
printf "%s\n" "::: adblock runtime information" printf "%s\n" "::: adblock runtime information"
printf " %-15s : %s\n" "status" "${value}"
json_load "$(cat "${adb_rtfile}" 2>/dev/null)" json_load "$(cat "${adb_rtfile}" 2>/dev/null)"
json_select data json_select data
json_get_keys keylist json_get_keys keylist
for key in ${keylist} for key in ${keylist}
do do
json_get_var value "${key}" json_get_var value "${key}"
printf " %-15s : %s\n" "${key}" "${value}" printf " + %-15s : %s\n" "${key}" "${value}"
done done
fi fi
} }
@ -453,6 +494,8 @@ f_log()
f_rmdns f_rmdns
f_dnsrestart f_dnsrestart
fi fi
adb_rc=1
f_jsnupdate
exit 1 exit 1
fi fi
fi fi
@ -462,7 +505,7 @@ f_log()
# #
f_main() f_main()
{ {
local src_name src_rset shalla_archive enabled url hash_old hash_new cnt=0 local src_name src_rset shalla_archive enabled url hash_old hash_new
local mem_total="$(awk '/^MemTotal/ {print int($2/1000)}' "/proc/meminfo")" local mem_total="$(awk '/^MemTotal/ {print int($2/1000)}' "/proc/meminfo")"
f_log "info " "start adblock processing ..." f_log "info " "start adblock processing ..."
@ -488,7 +531,7 @@ f_main()
# manual / backup mode # manual / backup mode
# #
if [ ${adb_manmode} -eq 1 ] && [ -z "${adb_action}" ] && [ "${src_name}" != "blacklist" ] if [ ${adb_manmode} -eq 1 ] && [ "${adb_action}" = "start" ] && [ "${src_name}" != "blacklist" ]
then then
f_list restore f_list restore
if [ ${adb_rc} -eq 0 ] && [ -s "${adb_tmpfile}" ] if [ ${adb_rc} -eq 0 ] && [ -s "${adb_tmpfile}" ]
@ -536,7 +579,10 @@ f_main()
if [ -s "${adb_tmpfile}" ] if [ -s "${adb_tmpfile}" ]
then then
f_tldcompression "${adb_tmpfile}" f_tldcompression "${adb_tmpfile}"
f_list backup if [ "${src_name}" != "blacklist" ]
then
f_list backup
fi
else else
f_list restore f_list restore
fi fi
@ -562,7 +608,7 @@ f_main()
# #
if [ -f "${adb_dnsdir}/${adb_dnsfile}" ] if [ -f "${adb_dnsdir}/${adb_dnsfile}" ]
then then
hash_old="$(sha256sum "${adb_dnsdir}/${adb_dnsfile}" | awk '{print $1}')" hash_old="$(sha256sum "${adb_dnsdir}/${adb_dnsfile}" 2>/dev/null | awk '{print $1}')"
fi fi
if [ -s "${adb_tmpdir}/${adb_dnsfile}" ] if [ -s "${adb_tmpdir}/${adb_dnsfile}" ]
then then
@ -574,30 +620,20 @@ f_main()
else else
> "${adb_dnsdir}/${adb_dnsfile}" > "${adb_dnsdir}/${adb_dnsfile}"
fi fi
hash_new="$(sha256sum "${adb_dnsdir}/${adb_dnsfile}" | awk '{print $1}')" chown "${adb_dnsuser}":"${adb_dnsuser}" "${adb_dnsdir}/${adb_dnsfile}" 2>/dev/null
cnt="$(wc -l < "${adb_dnsdir}/${adb_dnsfile}")" f_rmtemp
# conditional restart of the dns backend and runtime information export # conditional restart of the dns backend and runtime information export
# #
chown "${adb_dns}":"${adb_dns}" "${adb_dnsdir}/${adb_dnsfile}" 2>/dev/null hash_new="$(sha256sum "${adb_dnsdir}/${adb_dnsfile}" 2>/dev/null | awk '{print $1}')"
f_rmtemp if [ -z "${hash_old}" ] || [ -z "${hash_new}" ] || [ "${hash_old}" != "${hash_new}" ]
if [ "${hash_old}" != "${hash_new}" ]
then then
f_dnsrestart f_dnsrestart
fi fi
if [ ${?} -eq 0 ] if [ ${?} -eq 0 ]
then then
json_init f_jsnupdate "${adb_cnt}"
json_add_object "data" f_log "info " "block list with overall ${adb_cnt} domains loaded successfully (${adb_sysver})"
json_add_string "adblock_version" "${adb_ver}"
json_add_string "blocked_domains" "${cnt}"
json_add_string "fetch_info" "${adb_fetchinfo}"
json_add_string "dns_backend" "${adb_dns}"
json_add_string "last_rundate" "$(/bin/date "+%d.%m.%Y %H:%M:%S")"
json_add_string "system" "${adb_sysver}"
json_close_object
json_dump > "${adb_rtfile}"
f_log "info " "block list with overall ${cnt} domains loaded successfully (${adb_sysver})"
else else
f_log "error" "dns backend restart with active block list failed" f_log "error" "dns backend restart with active block list failed"
fi fi