Unfortunately it seems there's now a desire for browsers (and soon, maybe other applications/systems will follow) to make DNS requests inside HTTPS tunnels (DoH), but maybe that'll just encourage more the use of MITM proxies which have almost become a taboo amongst the force of "HTTPS everything" security-authoritarianism. Web security these days may be just as much focused on securing the profits of the advertising and tracking companies as it is against malware and actual user hostilities...
I also think it’ll encourage consolidation onto the existing major providers because everyone is going to want to be on an unblock-able IP block.
One day soon someone will figure out that the right way to deliver any and all content is through an end-to-end, cryptographically secure server -> HTTPS -> NIC -> HDCP -> screen pipe. And then we'll all have TVs instead of PCs, but at least they'll be secure™.
* Look at the actual content if it's HTTP. HTTPS prevents this.
* Look at the DNS query to determine the domain. DoH prevents this.
* Look at the SNI header to determine the domain. ESNI prevents this.
* Look at the IP address to see where the traffic is going. However, this isn't fine grained enough, especially if the traffic is going to a CDN.
* MitM with a self-signed CA. This is the hardest option. It's impossible on many devices (because you can't install a root CA) and a huge PITA to install a custom root CA on devices that support it. It's also much more invasive than the other options and makes it possible to accidentally log sensitive information.
Right now a lot of devices ignore your network config and use 18.104.22.168 for DNS. At least with DNS as a separate protocol, you can block external DNS servers and most devices will reluctantly use your local DNS (which can be used to block ads and tracking).
However, once DoH starts getting adopted, those devices (and many apps) will start to use it because you can't block HTTP(S). Literally all a network admin is going to see is a bunch of encrypted connections to CDNs (Cloudflare, Cloudfront, etc.).
IMHO, even though it's sold as privacy tech, it's a huge loss for the average person that can spin up a DNS based ad-blocker right now. It's going to get to the point where you no longer have control over your own network. The huge tech companies and ISPs will control everything.
But the mitm has its place. Hostile iot devices should be sanboxed.
What's worrying is that even if browser extensions retained their power, DoH being used on Electron apps and native apps will nerf any ability to control network requests on those.
And for iOS: https://chromium.googlesource.com/chromium/src/+/master/docs...
This is also why there are Chromium-derived browsers on iOS and Android (eg Brave).
 - https://developer.mozilla.org/en-US/docs/Web/HTTP/Public_Key...
Wouldn't one be able to install their own certificate (EG: Let's Encrypt) on their devices, and use their own local DNS over HTTPS that way?
I honestly don't remember the last time I watched actual live TV. I just recorded it and watch it later, then fast forward through all the ads.
Even if I'm watching it the same night, I'll just sit down to watch it slightly late, so I've got enough of a buffer to skip the ads, and catch back up to live by the end of the show.
Then again, what about clips from the show that you're watching, it might start blocking part of the show as it thinks it's an advert.
I modeled my change on similar changes made by Streamlink's Twitch plugin recently: https://github.com/streamlink/streamlink/pull/2372
This feature was in MythTV literally 10+ years ago. Commercials are very easy to detect using simple methods .
ETA: Looking it up it looks like America has laws about this now. I'm in Canada I'm not sure if it's the same here.
And on further reading, it seems like advertisers may be using excessive compression to artificially boost loudness and skirt the law.
Even if peak volume is under some prescribed maximum, compressed audio can feel very loud. Commercials often leave very little dynamic space, which makes them jarring in that they feel "loud."
"The process is CPU-intensive and can take several minutes to complete, depending on the recording duration. On a reasonably fast CPU, we typically see a 30-minute recording take 2-4 minutes to process."
Assuming they know what they are doing, real time blocking seems not possible with a reasonably modern approach.
"The recording is analyzed on various characteristics such as black frames, silences and changes in aspect ratio.
Based on this information Comskip segments the recording in blocks and using heuristics, together with additional information such as the presence of logo, the scene change rate, Close Captioning information and other information sources Comskip tries to determine what blocks of the recording are to be characterized as commercials."
It doesn't say it explicitly, but based on this and the note on high CPU utilization, I get the impression it has to look back and forward to determine where the commercial is.
If you want to watch a prerecorded show then skipping is feasible but for watching a live event at a distance (proper tele-vision) it’s obviously not.
I don’t think many fans of live sports TV would consider watching the game starting 20 minutes after to be able to skip commercial breaks.
I guess it depends on the type of programmes you watch.
If you live in the US, and broadcasts are delivered to you with IPTV, then reading about SCTE-35 might be of interest.
Though I don’t know if these control messages are actually included in the video stream that is broadcast to the viewers, so it might be a dead end, but I think I read that at least some channels do include them in the broadcast stream. Might be worth looking into.
You and your neighbor may not be seeing the same ads. The cable companies and advertising companies can show each subscriber something different on the same channel. But they only do that during ad time- when more customized content is a way to make money. During the show, they're currently incentivized to show you both the same content. Even if this type of thing isn't going on, it's been obvious for 20 years that some ad time is given to local ads vs national ads.
All we need then is a system where each member shares some small hash of what they're seeing on this channel every second or so. When the group sees their hashes diverge, it's ad-time. Turn on your relaxing music.
So long as this doesn't become too common, the cable networks would have a hard time changing their behavior to try to fight this. They make too much additional money with the customized advertising.
When I was a Google intern almost a decade ago, another intern explained to me that her division of Google had bought out 100% of the ad time on a cable network and resold the ad time with targeted customer demographics, making a profit doing so. I can only imagine how crazy the targeting systems are now.
Pre-CALM, you might have been able to use the relative volume change to detect ads. Not sure if that will work anymore. Some kind of visual/audio fingerprinting might work best these days, collecting information about new ads in the process.
It would sit between your set-top box and your TV, analysing the incoming HDMI signal, changing things in the video stream, and then outputting the new signal to the TV.
I haven’t configured this setup, but when I watch the shows delayed on my pc (on delay, not live) the ads are blocked due to my browser ad blocking - which shortens the overall length of the show as they seem to be injected rather than part of the stream
I haven’t used MythTV in years but I remember it being quite decent at commercial detection. Sometimes it missed the boundary between show and commercial by 1-2s, but otherwise was great.
(I bet they check for volume changes as one of the algorithms)
I suspect that other commercial skipping systems would have the same problem.
By far the best way to watch TV these days IMO.
They did get sued into bankrupcy for it
To me the real issue with commercials isn't the fact that they're selling me something, it's the fact that they're interrupting my entertainment. That will happen regardless of whatever pops up on my screen.
In saying all that, I do like the concept. Perhaps video taping a show and removing the ads afterwards would have the optimal effect? Just googling around I seem to find some video taping software already has this feature, not sure how well it all works though.
All you need is a USB formatted to EXT4 and you're good to go.
But a question for anyone who's done this Presumably most of the computing devices in your home make their way onto other networks from time to time. So you need a per-device solution to the ad infestation in any case. What's the point of adding a house-wide one?
This is the consideration that's stopped me doing this so far - the adblocker I currently use is going to have to remain in place regardless, so what would a Pi Hole add (other than a pleasurable hour or so toying around)?
[Edit: I don't have a smart tv, google home or similar net-connected but unhackable device]
It doesn't stop you from running adblockers on devices in addition. As it takes about 10 mins to setup it's totally worth trying out.
2) You get nice graphs. This is surprisingly addictive. I’m just trying to feed it into Grafana now, to make one dashboard to rule them all.
3) You can block other stuff easily, which is handy for sites you don’t want visited accidentally (which is also possible in my adblocker).
4) Removes DHCP and DNS from the crappy ISP router.
5) It’s on a Pi and as mentioned elsewhere here, that somehow makes a chore into something fun.
That doesn't solve the problem when you're out of the house although you can set up your devices to use your pi as a dns server when you're away from home.
I have a pretty hefty pfSense box but running pfBlockerNG caused all sorts of weird slow downs of DNS resolution through the pfSense DNS Resolver for all clients.
The parent company (Netgate) has price their hardware higher than I believe the market supports.
In addition to acting like pi hole and blocking certain hostnames, it allows you to encrypt your DNS lookups for anything forwarded to DoH or DNS Crypt supported services.
I have to enable JS or third-party content much more often than that, but the trade off to protect my privacy is worth it.
Sometimes, I really do want to click that ad on Google search, or I'm looking at some "deal site" which routes URLs through an advertiser, and there's no way for me to open that link. It gets really annoying. Not only that, blocking the request is nothing like a real ad-blocker removing the element, most of the time it still leaves boxes with errors in the middle of the content you're browsing.
I've only had a couple of things that didn't work because of the pihole, and honestly I found it faster to just make a phone be a 4g hotspot, connect my computer to that hotspot, get past the "hump", then switch back to my home network. It's not ideal, but it doesn't happen often.
Beyond the obvious, one use I’ve had for it is when traveling in an RV and using a bandwidth-limited connection. Not eating up quota with all of the ad assets is very nice.
With browser-based tools it is super easy to turn these things off when it breaks things (e.g. flight booking sites often fail miserably with an ad/tracker blocker I've found). If you come across a website that breaks with PiHole do you have to change your DNS settings to get around it?
But you can't disable it on a per website base as it's working on DNS level.
Not perfect, but easier than explaining the pi-hole admin interface and consequences of their actions.
So all your bulk data transfer (HTTP, etc.) would go at the normal speeds.
DNS itself doesn't involve a lot of work for the server to do. It's a pretty simple protocol / system that doesn't require powerful hardware. So assuming their blacklisting implementation doesn't bog things down, it should be able to serve DNS requests at the normal speeds. In some cases, it could actually speed up DNS because you'd have a local caching server which you might not otherwise have.
I run mine on old, low-wattage PC hardware that cost me about $30 all-in and boots in 20 seconds, so at least I'm not at the mercy of unpredictable mSD cards or an ESXi server that takes 10 minutes to start launching VMs after a power outage.
Edit: if that's not the case I still think applications round robin also defeating the purpose.
I if I recall correctly Windows, Linux, and MacOS all have the resolving behavior of only querying alternative servers if absolutely no response at all is received within some default time frame with the order of attempted servers being the order listed in the DNS configuration. I believe that this is also the behavior recommended by RFC. These methods should also result in some level of local caching as well by default if you're on Windows or MacOS, but Linux will vary by distro.
Because alternatives will only be tried if the preceding server fails it should be safe to manually add something like Cloudflare as a back up DNS server or distribute the alternative via DHCP (along with pihole or via pihole if it's also acting as DHCP).
On Unix-y things, resolv.conf is specified to be used in the listed order. Local caching resolvers will default to remembering the last successful server and treating the list as circular, so after the first DNS server has a failure it won't be tried again until each subsequent server has failed. Every time a distro has switched to having a local caching resolver by default you'll find big threads of people confused over the altered behavior.
And default behaviors are meant to be over-ridden, so in places where I have dnsmasq providing DNS for a network (PiHole, EdgeRouters) I set all-servers so that every upstream is queried simultaneously and the fastest response wins -- an exceptionally bad configuration option when having upstreams with differing views of DNS, but for the general use case it makes the Which public DNS is fastest debate moot.
Is this definitely correct? I’ve read that primary and secondary are more or less equals and it’s just 2 servers and either may be picked. If you have a secondary that isn’t a Pihole, some percentage will bypass the Pihole.
Disclaimer: I have no expertise and just read what Pihole and a few guides had to say. It was recommended to point primary and secondary at the same address or 2 separate Piholes to avoid issues.
Pihole is basically a front end for dnsmasq which may indeed forward to servers in a round-robin fashion. Servers capable of forwarding usually have other options for determining who to forward to than just an ordered list. For example, if I recall correctly, Unbound by default distributes at random and then favors what it determines to be the fastest.
What I'd like is for multiple DNS servers to receive their configuration from, and forward their metrics to, a single management instance.
I'm curious if any websites use JS to check if the ad was successfully loaded.
I found that with the default lists it runs very smoothly but if you start adding a lot of domains it might break some sites. If you break a site you can always whitelist it or disable blocking for a while through the web interface.
Fortunately working around is as easy as going to http://pi.hole/admin, hitting "disable for 30 seconds" and reloading.
My experience since installing has been overwhelmingly positive.
[Edit: pi.hole is http not https.]
Solve all my problems with ads and also effective against social, porn and gaming addiction.
This approach has way less overhead than pihole... But now someone else knows what you're browsing.
Edit: the top comment specifies the hostname incorrectly. It should be: dns.adguard.com (not .org)
(it's why my raspi's are all on 5V/3.5A supplies instead)
I'd also recommend a USB tester like the RuiDeng UM24, especially if you use a lot of SBCs or USB powered devices.
- better supply
- temperature (heatsink / fan ?)
- sd card fault
Does anyone know how they're able to do this and if there is a solution?
Then, app developers could https tunnel out to a well-known ip running a dns resolver, I suppose.
At that point, end users could configure routers to block all outgoing ip’s that weren’t recently returned by the local dns resolver.
And so on. It is an arms race.
“Like any other project I run everything in a Docker container, and this project should be no different“
Why? I assume they don’t maintain their own image for home use
It helps isolation, and doesn’t pollute the LXC-host itself.
When I want to remove/replace something deleting the container is guaranteed to do a 100% cleanup.
It also helps migrating apps/services across hosts/servers. Now I don’t do that very often, but the few times I do, it’s a godsend.
Small network of about 10 clients.
I turned off my pi hole.
Personal anecdote only
Speaking of that, does anyone know of a router that lets you run any operating system (I was thinking OpenBSD) but with better energy usage than a full computer?
I run pfSense on an Intel box I got from AliExpress (i5-6200U, 8GB ram) as my firewall and use its 6x Ethernet ports for basic routing. Despite having a decently powered spec, in operation it takes 8-9W power.
Searching AliExpress for pfSense returns lots of options, many will have lower consumption.
For WiFi I have a tp-link AP (EAP-225) that takes about 3W.
This is a bit more than an all-in-one consumer unit (the one I replaced was a couple of watts), but I'm very happy with its power consumption.
You can install several packages including adblock and cryptdns-proxy.
Here's a list of supported routers: https://openwrt.org/toh/start
It doesn't run OpenBSD, but it runs OpenWRT-based TurrisOS, and also supports LXC containers (generic ARM-based Linux distros like Debian, Ubuntu etc). By the way, it's possible to install PiHole inside such a container. Besides that, it has builtin WiFi (2.4 and 5 GHz radios) and a plenty of hardware resources (2 Gb RAM, powerful CPU).