The danger here, of course, is that while these compromised devices are currently being monetized in a relatively benign way (other than bandwidth costs to the user, which to be sure can be nontrivial)… it would be naive to assume that there isn't a Command and Control capability that can assign less benign tasking - or can be compromised to do so.
This also puts some of the troubling reports that politicians were bringing phones into SCIFs into context…
On a server that used to get below 1k HTTP requests a day, I've had several cases of mass scraping from these botnets exceeding 100req/s. The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
A compromised device inside the White House network could presumably be doing something _much_ worse than participating in a DDOS against random websites.
And it's right there in the second line of the article:
> One of the proxy services is a Russian company whose founder’s web hosting business was ground zero for cyberattacks against election systems in Arizona and Illinois in 2016.
Surely a Russian company known for cyberattacks against the US having direct access to the White House network is far more serious than just DDOSing random websites. This is espionage at the highest level. (Again.)
>> A compromised device inside the White House network
A whitehouse network. It isn't like everything done in that building run through a single wifi router. It is likely an unprotected network for personal use with zero connecting to anything important.
I'm a screen reader user (which is as "weird and unusual" as it gets when fingerprinting mechanisms are concerned), and I am not really noticing much difference versus 2022 or so. Sure, there's little bits of anubis here or there, and a few sites (which I know for a fact are experiencing targeted DDoS attacks unrelated to AI scraping) having annoying Cloudflare verification screens, but that's about it.
My setup (other than the screen reader) is as vanilla as it gets. Safari, AdGuard for Ad blocking, no VPN except work VPN when necessary, no PiHole, no settings changed. If you're using something more esoteric, maybe try going mainstream and see if that improves your experience.
I am not sure what you are doing to get "harassed" by anubis but I use privacy respecting browsers with ad blockers and vpns myself, and anubis is by far the least annoying, almost invisible even.
In comparison I get cloudflare recaptcha on every site that uses it, sometimes multiple times per session and sometimes not even letting me pass at all due to "error".
And if the website uses Google recaptcha then I close the tab because I know it will just end up in endless recaptcha hell.
Google too. 'We are detecting unusual traffic from your address'. I don't know how often I've seen this and there isn't a byte that goes in or out on this connection without monitoring so if it really would be 'unusual' I would hope I would be on it before Google told me, especially not from a session that's been active for days or weeks.
Please substantiate this claim. What do you think makes it DRM? What kind of browser are you using that results in "harassment"? I use tor browser and other oddball firefox forks all the time, Anubis never gave me any trouble. I think it just requires you to compute some hashes.
It’s a software tool that specifically exists to allow content creators to enforce rules about how their content is used. It meets the dictionary definition of DRM.
It’s DRM that I think of as silly larping that doesn’t really bother me, but it’s clearly drm.
1) The proxy isn't the scraper. There are plenty of scrapers not using residential proxies.
2) They're clearly talking from the perspective of the user and network admin. A residential proxy virus is in fact quite benign for both.
3) No matter what, DoS is quite a low priority in terms of security. And it's not even really DDoS, you'll know a real DDoS when you're on the receiving end.
Let's not get ahead of ourselves... If someone plants a bomb under your car, you drive to work, then it blows up while you're out for lunch, should you be prosecuted?
I'm all for creating consequences for people who negligently don't secure their systems and people get hurt because of it, but people who installed a mobile game that has a proxy library in it? Please, chill.
> The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
Indeed. I recently set up a bot on my local machine that I can interact with via Telegram. I gave it the ability to retrieve the contents of a URL via Playwright (with modifications). Basic things like getting the Readability view and getting the site as Markdown with links.
I was essentially mimicking the feature OpenAI etc offer where it can do a web search for you and get you the answer you seek. I just didn't want to pay a company for that feature.
It's not autonomous. It only acts when I tell it to. The actual search is via the Kagi API, but the URL contents is via Playwright. I also ensured it can't do lots of requests at a time. There's a mandatory (random) sleep of between 1 and 5 seconds between two requests.
In less than a week, I've been flagged. When I do a Google search (as a human), I have to prove I'm human. Many sites have simply blocked me - I can't buy stuff from various vendors[1], I cannot access several news sites, etc.
The thing is, I didn't really access that many sites with the bot! At most it's the equivalent of someone mildly addicted to the web. Yet they caught me quickly.
Obviously, I disabled the bot's capability to get site content. I wonder how long my IP will be in the blacklist.
I know some people will say "Good riddance" to the outcome and to what I did. I'd love to have a real, serious discussion on whether what I did was immoral. But whatever your thoughts on the matter, do ponder over the fact that the system has made it hard for average Joes like you and me to have this "feature", but if you pay a giant tech company a monthly fee, you can have it.
[1] Thankfully, Amazon didn't block me. In the past I'd avoid them, but now I'm in the dilemma where I don't want to buy stuff from sites that make it difficult for me to buy stuff from them!
I think that major gateways like Google and Cloudflare and Akamai are doing things like TLS-fingerprinting and smearing.
I normally use a VPN to access the Internet. It's an ordinary consumer-grade VPN, and I use it for pretty innocuous things like watching the international versions of Netflix or avoiding being geographically-profiled and charged more when I book travel. The VPN itself, I'm sure, is being used for all sorts of naughty stuff: when I look it up in whatismyip, it comes up as VPN, "abuse detected", and "residential proxy detected". I expect to get blocked occasionally on it and just pause the VPN then.
What I found surprising is that pausing the VPN doesn't always clear the CAPTCHAs. The next request will also fail, sometimes. For a while I was wondering if I have a residential proxy on my network, but I went hunting with lsof and bandwidth monitoring and a list of all the open connections through my router, and it seems clear. Yet even with VPN disabled, I'll get CAPTCHAs from Google and Cloudflare.
I suspect they're using browser/TLS fingerprinting and then smearing the results of that across bad IPs. If you connect to them through a known-bad IP, they not only will CAPTCHA you, but they will record a fingerprint of any data they can. Then if you connect again with that fingerprint, regardless of the IP, they block that too.
Ironically, if you are actually malicious, you can get around this pretty easily. Because you'll know not to use a fingerprintable browser, or to rotate fingerprints. So you have one completely separate identity on a given IP (that is not yours), and if it gets blocked, you roll up another completely separate identity on a different IP. But this is too much work for an ordinary consumer who just wants to keep their IP private. So you end up with this excluded middle where the only rational strategies are to either do exactly what the website expects you to do (with no funny business, even if it's innocuous), or to go whole-hog as the aggressor where you're determined to skirt their policies no matter what, and if you're doing that you might as well amortize the effort by extracting everything of value you can from them.
I just did this as well. I also added paywall bypass by default along with inline article chat. The internet is such trash these days.
I made my modifications at the network level and auto redirect a configurable set of sites to go to reader mode/paywall bypass by default with a MITM proxy and a custom CA cert I trust
Expect to remain blocked, that's how defense works, some services plenty use will have is bot or known proxy exit node set to true, and some websites use the same defense services, request velocity is just one factor, if you don't have random human like scrolling and jitter with a bot, it's a dead profile walking, metaphorically speaking
It depends on the DHCP part and your provider lease time. When you access the Internet through a modem and landline, it's an highly dynamical system, where it's difficult to guarantee every caller/user gets its own IP. Especially when IP sharing is used among the clients. Even more difficult it is with the mobile Internet. It's a constant connection loosing/reconnecting. Actually resetting and freeing up IPs for new devices, the whole time. With cable Internet (the TV cable) one is having a kind of stationary access point. The network provider can see it's topology and it never changes once a leafe has joined the network. I typically have the same access creds over months. Even when I restart my cable modem, it's still the same IP. When I was on modem, I could get a new IP every second of I wanted. Now I have to think twice, because I just can't get a new IP, even if is forcedly, not.
That is entirely ISP dependent and many don't actually work like that. Your cable modem gets a specific IP for as long as Comcast or whoever sees fit, restarting does nothing on the WAN side.
Doesn't work like that, only on mobile phones you can toggle airplane mode to get a new ip assigned and cgnat makes sure mobile ips are "trusted" , but they fingerprint you beyond ip address
While your server handles that, what does your CPU do with the other 99% of available cycles?
I run https://bunnystomp.com on an Always Free VM from a major cloud provider. It handles 1,930 req/s at 33 ms latency with no tuning/optimization. It's literally just python behind cloudflare.
You seem to know a lot about my server. From what I could see, all cores were pinned to 100%, but maybe my budget AMD CPU is hiding fantastical amounts of compute from the kernel. I'd love to know how to make use of it.
I have no idea what the site you linked does, but it doesn't look like a cgit instance to me; that's what got hammered on my end, and when hosting it, one very quickly finds out that many git operations are CPU-bound.
Funny. My site is a search engine and cgit is one of many tarpits that I block. Otherwise the index gets filled with garbage and the crawler hits TCP conn limits because the bytes trickle in too slowly.
If you want credibility from the get-go, you need to be up front about your application. I certainly took it to be a generic host for an unpopular site in your first comment.
I'll just note that "compromised" is perhaps too strong. These services often come from VPN plugins (eg Hola VPN) that lets users watch sports from different restricted geographic regions. There is a trade here that's in the TOs: you get someone else's IP address, they get yours.
If I'm using one of these services while connected to a guest wifi network, with the usual configuration for guest access (like no intranet connections), is that guest wifi network compromised? What's the threat model?
I'd say yes since you've allowed random people unauthorized use of the connection. It makes no difference if it's your connection or not. For example, if that proxy gets used as part of a DDoS attack, your device compromised that network.
how is this different than Tor? You seem a bit on the surveillance side. The computer/network is compromised if someone gains unauthorized access to it not if you share it.
If that guest WiFi network is running inside the White House? Yes, I'd say that is compromised. The threat model being that guests on that network may be diplomatic officials and staff.
That's not how Hola
works. You're not sharing your IP with some other Hola user. You're sharing it with bots used for all sorts of good and bad things.
What part of that isn’t technically “compromised”? You can argue there’s no evidence of it currently being weaponized but I’d say some shitbird running a free vpn inside the White House is not a good look regardless.
Not sure which direction you're intending this. The NSA is making special phones to protect politicians, or to make it easier to ensure they can read everything?
They’re too polite to say it explicitly, but when they say:
> Most of the activity happens in the evening, Eastern time: about half of it falls between 5 p.m. and midnight, and almost none happens in the early morning. That pattern suggests a person rather than a public guest Wi-Fi network: someone regularly on the White House network late into the evening, whose device is infected with a proxy SDK. Our data shows the exits, not the device or the person behind them.
There really aren’t very many candidates for who that person might be and it really seems like it would be someone who, I don’t know, maybe lives where the traffic is originating.
I'm no expert but even I've read that "Works at the White House" is bigger than the executive residence and oval office, including the West Wing and the big office building that's directly connected to it by basement walkways. The National Security Council maintains 24-hour watch operations in the Situation Room (which is more a 'complex' than 'room'), while regional desk officers routinely work late evenings to overlap with morning business hours in Europe and Asia.
I asked Gemini if my recollection was correct and got: "The EOP network (managed by the EOP Office of Administration in coordination with Presidential Information Technology Community entities like the White House Communications Agency) operates as a single campus enterprise network. Its high-speed backbone links the West Wing, the EEOB, the East Wing, and nearby administrative facilities such as the New Executive Office Building (NEOB) on 17th Street."
So, definitely an issue that there's a device with some rogue malware installed but the total workforce across all shifts at the White House Complex is roughly between 2,500 and 4,000.
Because it's not the treasury building lol. There are multiple buildings near the White House that are under this umbrella, and I wouldn't think Treasury is one of them since they need their own office space.
the show that came out recently, The Residence, while obviously fiction, made me think about the white house as a semi-public event venue in near constant use by dozens if not hundreds of people. It made for an interesting setting for a whodunit.
Of course it is. It's "The People's House." Look up the Great Cheese Levy of 1837, Lincoln's Office Hours, Coolidge's reception on New Years 1927 or Jackie Kennedy literally giving an "MTV Cribs" on TV in 1962. Or consider, if you can remember, the outcry when they permanently closed Pennslvania Avenue in 1995. Or when they closed Lafayette Square in 2020.
It's amazing thinking about Teddy Roosevelt following the New Year's Day tradition of shaking every single person's hand who showed up at the White House.
Which in the 1907 case meant 8,513 people's hands.
I can't imagine you'd need very many days of traffic to narrow it down to a very small group of people though. If a device is coming into and out of the campus, that should be fairly easy to reconcile with visitor records. Given that this is one of the most secure locations in the world, I'm reasonably sure a comprehensive list of "who is and isn't on campus" at any given time is available.
My prosumer ubiquiti gear will give me the MAC address and AP that the device is connected to, along with signal strength. WiFiman and a 5 minute walk will give me the mapping of where had that signal strength from that AP and at that point I could find the device with a physical search. There is no way that this device can’t be found in a few hours by anyone who is even vaguely paying attention.
Even if many sometimes work at night, you would expect them to also regularly work in the morning, which isn't the case here? So the fact that almost none of the traffic is in the early morning is a further clue pointing to a certain resident whose office hours start around 11 AM...
Cleaner that comes in after normal office hours, with a phone which connects to guest wifi, which is segregated on the LAN side, but has public IPs from the Whitehouse AS
Do you believe the availability of wifi for staff/guests is somehow unreasonable? Or, is there a belief that no internet (wifi or cellular) connected devices are allowed on premises? Is them using cellular data, rather than a guest network, somehow more secure?
Guest networks are configured to disallow intranet connectivity, so the "security" risk would be from the device having access to the internet, on a guest network IP. But a guest network exists for the purpose of giving untrusted devices access to the internet, without compromising the security of the intranet. My local library, the local airport, Home Depot, the movie theater, etc all have public wifi, yet nobody would claim they were directly compromised. I understand the connection could be used for receiving and sending packets over the internet, like a DDoS (which would trigger rate limits, even at Home Depot), but framing a guest network, giving an untrusted device access to the internet, as some security issue seems very odd. The isolation is the mitigation.
The difference is the home depot does not have spies applying for custodial roles. I’d expect nonessential staff to put their phone in a box at the security check. We also live in a world where israel can remotely explode telecom devices they rig in supply chain intercepts.
As someone elsewhere in the thread noted, this could well be the press. Could also be that this isn’t the White House network at all despite that attribution.
Right, we know the White House owns that public IP, but this could be as bad as an unsecured, unmanaged device that is joined to a private network that is supposed to be secured, OR, on the “all the way good” side, that IP could be the exit for a completely throwaway, untrusted, isolated LAN segment meant only for some iOT device and which has no route to any other peer on the LAN side, and some civilian who has access to that for work brought in his laptop with a “free VPN!” Plugin and joined it to that network because it’s all he had credentials for.
If it keeps happening I think that at least suggests the source is someone or some people who are “above the rules” most people would be bound by. You can tell the cleaning crew to get off the damned WiFi, others not so much.
Even at midnight I can't imagine there being less than 100 people working at the WH in various capacities. While it's probably not the duty/desk officers in the Situation Room/Security Council, there are plenty of maintenance staff that work at night. 5-midnight seems like it overlaps perfectly with a shift schedule.
But how many of them would risk their job by smuggling in a device that violates government policies? Every device in the White House should be accounted for and meeting a number of security criteria. There are really only two or three people there who would know that those rules don't apply to them.
They are not saying that is _on purpose_, in fact they are saying the opposite: a compromised device, could be even an Android phone. It could also be as device planted on purpose, of course.
Given that, why are there so many loose phones at all? Here is one compromised that we know about, in the biggest target for espionage on planet earth.
You’re misreading what this says. It’s someone bringing in a device they managed to get infected.
Oh and also the part where no one has caught this internally which is possibly the bigger takeaway. It’s a zero-trust network made out of incompetence.
There's nothing in the article that suggests it's not just the guest wifi. Which wouldn't be monitored. Why would you? The whole point is that it's segmented and can't actually interact with anything important.
Eh, I would not be surprised if they have a separate wifi that guests can use and you can hook up your private devices to. Basically every large company i have worked for, for the last two decades had that. I suspect deep in the bowels of the White House you don’t get good reception.
Why are you assuming the person knows and is actively smuggling in the device? And why would you then assume that person is more significant than a shift worker there?
Probably because hegseth decided to actively use signal to discuss military planning instead of secure channels that don’t invite such screwups like adding random citizens to these discussions.
The White House doesn't shut down at 5pm. There's whole books, TV series, and movies about the people that are working late every night at the White House. Likely dozens of people are on staff and work shifts that last until midnight every night of the week.
Service staff, including the storied hundred or so White House butlers, work whatever hours are needed. Then you have Secret Service, Kitchen staff, Natalie, house admin staff like the Chief Usher, maybe even NPS staff as the WH grounds are a national park as well, any of whom I would assume might be given access to the secure but unclassified WH wifi and likely a government-issued device. One device with a proxy among all of those people? Entirely unsurprising.
Maybe it's just the printer that Natalie carries around in her purse.
That timing could also align with a device that is turned on and off at specific times, for example an insecure smart TV that is generally turned on to watch TV in the evening, and turned off around midnight.
Though the fact that it lines up with an 8 hour shift also makes it seem human-like, so I think I'd rank a human carrying an infected device as more likely than an infected periodically-active device.
If you regard 1,000 or less people as few, then this is true. If you regard 1,000 people as "too many" then you rate the national security of the USA at the cost of 1 in 1000 means it's not worth bothing to find out.
There’s a ton of people it could be. Any staffer that works 5-12, which wouldn’t be a particularly weird shift for cooks, cleaners, etc that bring their infected phone with them.
If it was someone who lived there it wouldn’t be exclusively that time period. Why wouldn’t it happen overnight or in the morning or any other time of day?
Unless it is running on a government issued iPhone, it is not the Individual #1 you are alluding to.
He very famously does not know how to use computers and can only basically manage tweeting from his phone, which he is mostly not, actually, doing himself.
Now, there are two or three terminally online people around him. I would be more quick to point the finger at his foul-mouthed spokesman but it’s admittedly because I hate that guy and he seems rather stupid. But perhaps it could be the devoted portable printer lady, or the overpromoted unblinking former golf caddy guy.
All of them are around to help with the, er, modern day presidential communications.
> He very famously does not know how to use computers and can only basically manage tweeting from his phone, which he is mostly not, actually, doing himself.
This article seems to cast residential proxies in a wholly negative light. In fact the company is openly antagonistic to anonymity. I find that perspective diminishes my trust in the rest of their claims.
Residential proxy is the industry euphemism. In most cases they aren’t set up willingly. Access is sold to various bad actors, for data collection, botnets, or for a clean residential exit IP
> There is no evidence that physically ties this to the WH.
They don't need to be physically inside a specific building to be a networking security problem.
The core issue is whether an outsider has bypassed your firewall and can send+receive traffic from a node which is getting more trust than it deserves. If they've compromised the computer's microphone/camera/wifi-control that's an additional problem.
You can only _add_ jitter. So presumably you make people think you're farther away than you really are. I guess if you're especially clever, you can manufacture responses to packets you haven't sent yet to throw people off.
> I get a hit in my vulnerability log, like in the threat log on my Palo Alto firewall, and it’s for a SCADA vulnerability. ... A SCADA vulnerability. This makes no sense. SCADA is the control systems used in large-scale industrial settings. Think pumps, valves, conveyor belts, compressors, elevators, railway switches. This is where SCADA systems live. Why in the world is this box that’s here to deliver TV and movies attempting to trigger a SCADA exploit on D3ada55’ network? This is very concerning.
Such things are threats to humanity imho. They can get us back to the times we couldn't exploit the internet to run a society.
Interesting! According to this, a lot more going on beyond the residential proxy element, which was the only bit I had considered (or heard about). I wish there was an accompanying article with some more specifics.
That is the least bad thing they could do btw. Unless your internet is data restricted, go for it if you want free cable. If questioned by your ISP (extremely unlikely but theoretically possible) pretend you have no idea what they're talking about.
Firewall it from your local network if you want to - it'll still work - they're not after that.
Absolutely, and this is why I refuse to run connected devices on my network that do run software I can reasonably trust. This is also why every computer, phone or appliance I own runs Linux. This includes my TV box, which I recently migrated to an x86 port of Android with Claude's help.
There are plenty of people selling IPTV access on g2g, few bucks for a month, all TV channels from entire world. I am not sure if it is profitable to let such piracy boxes work forever, even with them being used as residential proxy, but I also don't know how low you can go if you skip middlemen.
I have T-mobile 5g. My IP shows up as a residential proxy. I can't control the T-Mobile 5g device. It's a black-box. And, I only have one Linux desktop on the home network (so no Windows/Mac malware).
Not sure why my IP is listed as a residential proxy.
There’s only that many ipv4 addresses and I believe all 5G are shared ones. That’s why they’re among the most valuable ip ranges for scraping. You can’t just naively ban that IP or you’ll ban a large group of users
It should be a balancing act though, somewhere between what you said and my first sentence.
Rate limiting, or additional layers of input sanitisation for IP addresses known to have been malicious in the past.
It's unfortunate that good users will have to endure some difficulties because of bad users on the same network, but if security is the priority, then measures must be taken, you shouldn't just leave to door open to potentially known bad users / traffic.
IPv4 is a shared resource, but some ranges have worse reputations than others, and all are owned by organisations that have the ability to monitor and filter for malicious traffic. It gets into a fraught discussion about censorship, but it is possible for network owners to monitor for, and block / restrict traffic detected as 'bad'. It's just a matter of cost and scale, and those that put the effort in should have their users rewarded with fewer 'trust-issues' roadblocks.
> It's unfortunate that good users will have to endure some difficulties because of bad users on the same network, but if security is the priority, then measures must be taken, you shouldn't just leave to door open to potentially known bad users / traffic
Is this like a corporate support response? I legit can't tell. Try explaining to a retail consumer why they can't buy from your site because they have TMobile service.
They’re saying T-Mobile should be monitoring traffic on their network and block traffic from bad actors that ruin it for everyone for everyone else? Or at least they do in the next paragraph.
When the bad actor is grandma who has an infected computer, T-Mobile will be dealing with a lot of support calls from objectively non-malicious actors.
>you shouldn't just leave to door open to potentially known bad users / traffic
Why not? If security is the priority, the worst that could happen from that is clutter in logs and higher resource use. Nobody wastes 0days spraying them on random hosts, and a real motivated attacker targeting your server will not be stopped by an ip blacklist.
The blocking of residential proxies might be the thing that moves the world to ipv6. These days what we need is a dynamic map of all the IPs running proxies so servers can block them. If every device gets it's own v6 address, you could block just the TV running the proxy. Modern crowd sourced security products could identify the proxies and share their IP live so you can ban and unban addresses as the device moves.
Not sure about US but where I live I can manually set the access point of my phones mobile network and it gives me a unique ipv4 (new one each time in fact)
You resell a game, bundling a botnet proxy inside, and make it freemium: free if you accept T&C with a service allowing using your device connection for research purposes (or whatever wording could be even more confusing), or pay $20.
It's no longer a botnet, it's a "residential proxy" and you are doing clean business.
"User accepted T&C" is enough to get away with almost anything. It works very well for top IT companies. Upholding new business to higher standards would be unfair.
Freaky. They have a page explaining the concept (https://ipinfo.io/data/residential-proxy); when I checked it, there was a FOUC showing a full-screen, stretched image of what I think was Dario Amodei's face.
Neat. I almost said "surely this will only detect if your internet connection is itself being routed through a residential proxy", and then I realised that's effectively the same thing!
It detects if your IP is a residential proxy, not if you are using a residential proxy. From the perspective of the outside Internet, they are the same thing, the only one who cares about the difference is you.
My most worrisome IoT device I have on my network is a digital picture frame my wife connected, but I was surprised the company routinely updates it even though it's on Android 6 (!).
It's been some time since I ran this myself but I used to have an openwrt router sit behind my ISP modem/router (it was one unit).
Multiple times I caught unexpected traffic and was able to block it piecewise including a lightbulb(!) that was sending mb packets of god knows what home.
If they were ethernet based I reckon some smart VLAN capable switches would be nice even at home to keep them out of each others business.
I moved home a few years ago and now I'm back with the ISP router and do often feel a little in the dark about what my network is doing!
I fear this is just the tip of the iceberg. It's the WH's own TOR node for any paying customer. It's manchurian candidate. It makes Clinton's emails look like ice cream.
Brightdata still sucks at disclosing to users that they're opted in to this and that their connection is being used for this. It's "on by default", not opt-in and often buried in an unskippable "Accept the terms" or dig-to-opt-out prompt.
It's described at a high level in media often. Here's more detail though:
Person buys a product, or downloads some software - that unknowing to them, was reaching out to a central server elsewhere to establish a network tunnel. And over that tunnel, the other server routes traffic onto that person's home network. Which can then of course go onto the internet, and it appears like the traffic originated from their home internet connection.
Other people rent access to this network. The goal being, you have surreptitious access to appear as any one of tens of thousands of home network IP addresses on the public internet. Bad guys can use this to rotate through many different IP addresses when their access gets blocked because of malicious (or benign but unwanted) behavior on the internet. Residential IP addresses are more 'valuable' because they don't have the negative connotation that datacentre and commercial IPs can have, and they can be spread geographically or across many ISPs.
A very common way for people being a part of these residential proxy networks is by buying dodgy streaming boxes, and also installing shady apps on TV app marketplaces.
This isn't a theoretical problem either. My employer's website was being hammered by residential proxies a month ago. It wasn't until we setup WAF captcha rules that they mostly subsided. It wasn't even the typical exploit scanner. I don't know what the purpose was but it was effectively a DDoS.
It's definitely not all VPNs (typically the ones you pay for will buy their own hardware and internet connections in the places you can route your traffic through), but it is a common situation for the shadier (especially 'free') ones, and in that case it's usually not just a peer-to-peer thing, they make money by selling your connection to people who pay because they want a large pool of relatively reputable and disconnected IPs to send traffic through.
You're right, a lot of 'free VPN' software also uses your own internet as a residential proxy. It's often mentioned in the ToS but the users don't read that.
Honestly, I expect any of those free VPN services that require an app install to be doing precisely this. At this point, I just assume anything with "free" in it will cost me more than if I just paid for something instead.
It means some networked devices in there are being remotely piloted to tunnel network traffic to somewhere else. This scenario usually happens with more "unprofessional" ISPs, which sell bandwidth from their networked clients for people in need of such services.
Such as ignoring web scraping countermeasures, and DDoS.
Some good answers in this thread. I had the same question but I'm still a tad confused:
* I initially thought this meant someone in the WH network was using commercial proxies to appear as if they were exiting from a different network (which would align with "some user is firing up a proxy on their phone to exit in a different location")
* But then the description of residential proxies seemed to align more with traffic hitting an endpoint in the White House, and _exiting_ from the network there. But this didn't explain the timing (why isn't it 24/7).
* Finally it sounds like some person, legitimately on the WH network, has a device (cell phone, TV, whatever) that, when activated, is proxying outside traffic into the WH network.
There are companies that provide proxy services for scraping services, and residential proxies route through consumer networks (like Comcast) which is flagged less than a datacenter IP.
Many questionable TV streaming boxes pretend to deliver you video, but in reality they are also piggy backing on your personal ISP to power those companies.
> Residential proxy services sell access to everyday devices, letting customers send traffic through them so it appears to come from that device's network.
So what? Packets flowing through machines is just how the internet works. Stop believing all this propaganda that other peoples papers are EVIL and if you route them you are getting in BIG BIG trouble. This is harmful messaging that does not reflect reality.
There are services you can pay for such that you can route internet traffic thru a specific IP in a specific part of the world. People (usually unwillingly and unknowingly) run software that effectively makes them part of a giant botnet. A computer running in Romania might have some kind of weather application or gambling/casino app that I could use to make an HTTP request as if I were on their network. So if you need access to 1,000 or 10,000 IP's to circumvent rate limits or perform crawling tasks, this is how you could do it.
I was under the impression that actual ISP providers embed into their modems/routers ability to resell YOUR connection to proxy services and charge for that without customer's consent.
Same happens within cellphones - via native mobile apps pimped by ISP/Mobile providers.
Can anyone recommend home router that makes handling that threat easy for average Joe? As in a simple list, for TVs detected on my network, with streaming services they should be able to connect to, blocking all other traffic from it?
If residential proxies detected on White House network, why have network admistrators blocked the device?
And
why have they not physically located the device OR at least vastly reduced its possible location , by determining the flow of the traffic from network elements ?
Answer my own question - It was actually an outside entity that detected the problem. However , it gives the network admins a way to determine if this activity is happening on networks they are responsible for .....
Google just banned a chinese Android tv manufacturer for backdooring and installing malware on their devices. This I think is such a device someone pirate sports or porn streaming device.
I didn't know that residential proxies were a thing. That sounds pretty handy because going online via a VPN gives constant nag popups from the likes of cloudflare :( I hate them.
It would be nice if the VPNs would integrate this option.
Step one, find your public IP, I like ipinfo.io but there are many other services that will show it. Then pop it into a tool like the one that's hosting TFA[1]
Unexpected traffic would be the biggest gross red flag. If it constrains itself to moving traffic when it's otherwise perform its desired function (if it's one of those "free TV" boxes and it only proxies when you're playing video) it might be hard to tease-out the unwanted traffic purely looking at bytes in/out.
Beyond that gross metric you're getting into capturing traffic and decoding it.
Check network traffic from smart devices would be a good start. If your TV does gigabytes of traffic in the night when you don’t use it, it most likely is acting as a residential proxy.
I've never heard the term 'residential proxy' before, but it seems like a Tailscale exit node would qualify for example? This doesn't seem necessarily malicious or even inadvertently dangerous to me?
Residential IP addresses tend to have better reputations than datacenter networks, so there are providers who sell access to home internet connections. Oftentimes the people running the proxies have no idea they're running on their network, they're quasi malicious.
Generally not really. Residential proxies are services that can provide customers with the ability to send and receive traffic from a wide range of apparently legitimate networks (i.e. people's home internet connections). Almost always this is done without the informed consent of the person who owns the device that is used as a proxy, whether it be buried in some agreement somewhere or just outright malware.
The biggest danger is that said proxies can also allow access to internal networks, and as you might guess from how they are installed, are not usually operated by the most scrupulous of organizations.
Yes, some of these companies offer libraries to turn your mobile app into an exit node and get paid for the traffic. You can find news about Google Play playing whac-a-mole against apps doing so silently, often involving law enforcement as well.
Also some free VPNs use their users as exit nodes for their actual paying customers, good luck finding a user who understands that. It's the obvious way to monetize such a service other than ads.
it’s a well known term in shady markets of the internet, they allow users and abusers to bypass bans and create multiple accounts that cannot be linked with each other by networking intel
Not all residential proxy providers are seedy - but I only know of one that isn't out of probably 20 or so major players so I guess fair call out on "infected".
It is tricky. Monitoring network traffic should be enough, the infected device has to connect to Command and Control, to receive orders and set up tunnels.
It got harder after malware creators started using Telegram, Discord or even good old IRC for those purposes - it will be asy to notice only if they use a service no one else uses in your network.
Why not? Cass Sunstein even suggested such an approach in his book in the 1990s, back when he was the Presidential Science Advisor. It's no different than the King of England going out among his people in disguise to get the lay of the land.
OTOH it could be a honeypot, OR possibly somebody was careless. But there is no reason it has to be suspicious.
Pure speculation, I don't have any concrete answers. I'm fairly scorched-earth with my blocking, but I'm only protecting personal environments, so no one other than scatter-gun-everything AI-scraper-bots would even bother.
I'd be very interested in AI capabilities in detecting anomalous activity given that it should be very effective at pattern matching across large data sets. What would be required, however, is significant training with industry/application specific 'normal' browsing traffic patterns to determine acceptable tolerances, and then design escalating deterrence if and when tolerances are breached.
How is the scraper process stepping through the site? It must be programmatic, even if it's programmed to be random. What does that look like in traffic data in terms of timing and content requests?
What hazards can be put in place that a normal user would never go near, but a scraper is likely to trip over?
If scraping is being performed from multiple locations in parallel, what might that look like? ie. joining multiple sessions together that may show a more 'scraper' kind of behaviour.
Under IPv6 wouldn't detection start burning a lot of addresses? Yes, the pool is large, but that just means it's a scaling issue from IPv4 to IPv6, rather than a new problem requiring new thinking to solve? (assuming ways of detection were successful).
I'm being somewhat devil's advocate in my blocking rhetoric. The internet is in a weird confluence of wild-west and surveillance state. I like that there are ways and means of being anon-/pseudon-ymous; I like a bit of anarchaos, that's where the spirit of hacking lives. But I also don't want it sticking to my boots when I walk in the house. I'm also not fundamentally against scraping; there's definitely good scraping. There is an irony, though, in protecting scraped data or services that are built on scraped data (and I'm not accusing you, personally, just my general logical ideology of scraping). On residential proxies, I think they're 99% run by erstwhile criminals leveraging what were, until recently, referred to as botnets into a service they can launder their botnets through as 'I am legitimate business man'. I think most, basically all, residential proxies are run involuntarily or unwittingly and that blocking them would provide a greater society benefit than it would a drawback. Anything that promotes the sanitisation of end user devices can only be a good thing. The whole ecosystem is in a terrible state and regulation doesn't even seem to be on the distant horizon.
Maybe I'm hypocritical, or I'm trying to maintain mutually exclusive positions. This is where I'm at though.
PS: I own proxybase.xyz, and I think we’re the most transparent residential proxy network in the industry.
I don’t think it’s fair to call all residential proxy networks shady. Not all of us run our businesses that way. Our code is open, we have nothing to hide, and we use US-based stablecoins for transactions. User consent is our number one priority. I haven’t seen anyone else in the industry match that level of transparency.
Burning a compromised host just to run a public proxy client for pennies would be the worst trade-off imaginable for an attacker. It is loud, easily caught.
I would actually feel better if this were the case, but given that we have "the very best people" currently running our cybersecurity infrastructure I doubt it.
Most likely scenario is a 20-something holdover from DOGE who accidentally had his phone p0wned while downloading groyper memes.
Most likely scenario is a janitor who installed an app (I'm guessing it was called LegitWorldCupWatch420696969) on his phone (which might be connecting to the wifi network from whatever locker he leaves it in while he's on the clock, the WH is not a giant SCIF). Or any number of other mundane, easy to overlook employees who work outside of the actual secured areas of the building.
Let's assume the worst, that it's a VIP. Ok, now let's do some basic frequency statistics:
If you click the AS2650 link (1) in the article, you will learn that AS2650 is announcing 65,792 addresses in 5 prefixes. Roughly 40-70% of the IP addresses in a typical corporate ASN are associated with actual computers humans are using. Let's go with 40%. You're looking at 26,317 people. Assume they're all adults (please, dear lord, let them all be adults). 3-8% of the workforce is in IT. Let's go with 3%, that's 789 people. I'm guessing one of those 789 folks has thought this through and has a few VLANs set up to keep VIP and family-of-VIP, pattern of life stuff in a box.
As for the thesis that these are all window-licking low-IQ goons, that's just not how the US government cultivates its workforce. It's no accident that Harvard Business School characterizes themselves as "the West Point of capitalism." West Point isn't someone else's anything. The US Government is in it for the long haul and captures folks young with impossible-to-resist opportunities like the military academies, ROTC, SMART scholars, etc. Let's just stick with the academies: 12,000 people a year who were likely competitive for the Ivies, Division 1 athletics, or both, but chose their parents poorly. Free, top tier education and a job is very attractive for kids from fly-over states. Hell, in big states like California, New York, and Texas, it can actually be even more competitive. Something north of 60% of those graduates stick around for at least 20 years of service (because the high three pension + VA benefits is a pretty nice floor for retirement when your options back home are line supervisor at the cotton mill, the sugar mill, the coal mine, or ramp manager for United), usually addicted to the crazy level of mission. Sort of the opposite of finance where people really struggle mentally if they're not making pallets of money at an accelerating rate. Government service pays dogshit, but the work can super interesting, especially as you gain seniority.
Those folks then often loop back into other agencies and do another 20 years. Are there goombas in the Government? Oh, for sure. Absolutely. But it's roughly the same fraction as any other workforce in the rest of the economy.
But by the time you get to top tier stuff like the White House staff, the sorting function has been strong. To make the point: you also see top tier folks in other organizations with government stink on them. I can think of at least two directors at Google who are prior military. A fat pile of McKinsey folks. Something like 8% of Fortune 500 CEOs have served in the military, above the national average (6-7% of Americans have served in the military at some point).
And it's hilarious to watch people simultaneously assert the government is a bunch of idiots, and then get all excited about fighter pilots, the CIA, the CDC, ambassadors, the NIH. As though those people just appeared from the head of Zeus fully formed. Where do you think they worked when they were young? They were stamping visas for the Department of State in some embassy in central Asia. They were rolling blankets and digging wells in Gambia for the Peace Corps. Checking corporate filings for mostly unknown companies at SEC. Studying aerospace engineering at Embry Riddle and then flying cleared checks between clearing houses across the state overnight in a Cessna.
... and then Donald Trump as President, Stephen Miller as, whatever he is, and Peter Hegseth as Defense Secretary (he did serve in the military, if that's worth anything).
I think it's pretty wild to make all these bold assertions about the absolute top tier of elite talent that the US federal gov recruits and retains, and then compare/contrast that with the current staffers of the present incarnation of the executive branch. It just doesn't jive. Seriously, do a deep dive into Hegseth's resume and background of the nonprofit he was the figurehead for. He's more qualified than some known 3/4-star generals? Seriously? Tulsi Gabbard was the most qualified person for DNI? Seriously?
I mean supply chain-attacks are everywhere. Smart-TV's with certain marketplace apps, IoT devices. Also, I'm surprised the white house's network is not monitored more.
The joke that S in "IoT" stands for Security is as old as the term itself, but there were no such clear incentives for exploiting them before, beyond basic DDoS and more recently crypto mining.
Seems like the president found a business opportunity and gave internal orders to rent out the network infrastructure to 'ethically sourced' providers :^) Jokes aside, might be some infected IoT crap.
There exist services that will pay you a fraction of money to host their residential proxy service on a computer you own. But I do think the majority of residential proxies are hijacked.
There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.
Ethically, if someone just wants to play a free game and doesn't care about it, is it okay? How knowledgable does a user have to be in order to have informed consent?
> There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.
Betcha that means there was "consent" language on page 55 of some click-through EULA.
And making DDoS-attacks nigh impossible to IP block.
Residential proxies are in many cases at least shady, both on the side of who uses them and for what and on providers that are often not above using dark pattern to hide them in some innocuous app for a smart-whatever that silently runs a proxy on a unaware user's connection. And in many places in the world if illegal stuff in done using your connection you are liable, regardless if you knew about it.
I wouldn't call turning on CloudFlare's bot blocker an "absurd amount of effort" from the perspective of site owners.
But getting around it sure is annoying, sometimes I get flagged IP's from Comcast and can't visit sites like costco.com despite not doing any questionable stuff on my network.
Residential proxies are a very important part of the internet
So important that we have to trick people into using them. Remember when folks were running SETI@Home? Of course you do, because it was widely thought to be a good thing, and folks willingly signed up.
How many would buy the "box" if the advertising were honest about what's actually in the box?
Most tools can be dual-use. Residential proxies are also used by malevolent actors to hack infrastructure or financial institutions and let some other poor soul get investigated for it. This is why we can't have nice things.
True, though unfortunately people can not be trusted to declare their intentions so the residential proxies will ultimately have to be sacrificed and given they are fairly easy to identify by behavior I could foresee the feds now having a reason to round 'em up faster. They already took out NetNut, IPIDEA, SocksEscort, Doppellganger and the "Kim Wolf" Offshoots.
Some of the more legit providers have KYC in place, and do literally require you to declare you intentions to open an account, and block certain traffic proactively unless you can prove a legitimate use.
They list the IP addresses of all the residential proxies? Is there a link to the netsets of all the IPv4 addresses they are using I can download once a day? (preferably over rsync but git over https is fine too)
I don't think even the most "ethical" providers are going to volunteer that information, since that defeats the entire purpose of having a residential proxy in the first place.
No...? That's a completely different question than what we were talking about above? The customers of a residential proxy who would be subject to KYC are the people using the addresses -- this has nothing to do with the exit addresses of those who have joined the network.
That's a completely different question than what we were talking about above?
In fairness to me, I did say "Most tools can be dual-use. Residential proxies are also used by malevolent actors to hack infrastructure or financial institutions and let some other poor soul get investigated for it. This is why we can't have nice things."
So it could be Alice. I know Alice has good intentions. It could be Bob, we know what Bob is up to but I will give him a pass regardless because Bob did nice things in the past. But then there is Mallory ... Mallory Malfoid is up to no good. I need to know which IP's are used by Alice and Bob so I can exclude them from my automated protections. Mallory will automatically get rate limited to 16kb/s, 1 HTTP request every 5 seconds and if they bounce around different residential proxies they get redirected to a bathtub regardless of which residential proxy they use.
Virtually all of it is the product of malware. This is like you saying that robbery is very important because you like buying cheap tools at pawnshops.
The danger here, of course, is that while these compromised devices are currently being monetized in a relatively benign way (other than bandwidth costs to the user, which to be sure can be nontrivial)… it would be naive to assume that there isn't a Command and Control capability that can assign less benign tasking - or can be compromised to do so.
This also puts some of the troubling reports that politicians were bringing phones into SCIFs into context…
> benign
On a server that used to get below 1k HTTP requests a day, I've had several cases of mass scraping from these botnets exceeding 100req/s. The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
How is DDOSing every site on the web benign?
Relatively benign.
A compromised device inside the White House network could presumably be doing something _much_ worse than participating in a DDOS against random websites.
And it's right there in the second line of the article:
> One of the proxy services is a Russian company whose founder’s web hosting business was ground zero for cyberattacks against election systems in Arizona and Illinois in 2016.
Surely a Russian company known for cyberattacks against the US having direct access to the White House network is far more serious than just DDOSing random websites. This is espionage at the highest level. (Again.)
The owner of the proxy is not great, but the fact that its an open door to snoop on everything else on the entire network is the realy fucker there.
>> A compromised device inside the White House network
A whitehouse network. It isn't like everything done in that building run through a single wifi router. It is likely an unprotected network for personal use with zero connecting to anything important.
> The web is getting harder and harder to use
What are you doing to make this true?
I'm a screen reader user (which is as "weird and unusual" as it gets when fingerprinting mechanisms are concerned), and I am not really noticing much difference versus 2022 or so. Sure, there's little bits of anubis here or there, and a few sites (which I know for a fact are experiencing targeted DDoS attacks unrelated to AI scraping) having annoying Cloudflare verification screens, but that's about it.
My setup (other than the screen reader) is as vanilla as it gets. Safari, AdGuard for Ad blocking, no VPN except work VPN when necessary, no PiHole, no settings changed. If you're using something more esoteric, maybe try going mainstream and see if that improves your experience.
What do you mean by “anubis” here?
It's a DRM middleware used to harass users of customized and privacy respecting browsers which is popular with many smaller websites.
I am not sure what you are doing to get "harassed" by anubis but I use privacy respecting browsers with ad blockers and vpns myself, and anubis is by far the least annoying, almost invisible even.
In comparison I get cloudflare recaptcha on every site that uses it, sometimes multiple times per session and sometimes not even letting me pass at all due to "error".
And if the website uses Google recaptcha then I close the tab because I know it will just end up in endless recaptcha hell.
If you need to compare the project to Google and Cloudflare in order to look less bad that's not exactly a good sign.
Google too. 'We are detecting unusual traffic from your address'. I don't know how often I've seen this and there isn't a byte that goes in or out on this connection without monitoring so if it really would be 'unusual' I would hope I would be on it before Google told me, especially not from a session that's been active for days or weeks.
Is that the one that shows an anime figure holding a magnifyig glass while it is checking your browser?
Yes, and at least in the past it did proof of work to verify that it wasn't a bot. Though apparently that is no longer effective against all bots.
Please substantiate this claim. What do you think makes it DRM? What kind of browser are you using that results in "harassment"? I use tor browser and other oddball firefox forks all the time, Anubis never gave me any trouble. I think it just requires you to compute some hashes.
It’s a software tool that specifically exists to allow content creators to enforce rules about how their content is used. It meets the dictionary definition of DRM.
It’s DRM that I think of as silly larping that doesn’t really bother me, but it’s clearly drm.
This is such a wild re-imagining of 'preventing uncountable AI crawlers perform DoS attacks on a whim to make line go up'
Not OP but most likely https://anubis.techaro.lol/
https://anubis.techaro.lol/
1) The proxy isn't the scraper. There are plenty of scrapers not using residential proxies.
2) They're clearly talking from the perspective of the user and network admin. A residential proxy virus is in fact quite benign for both.
3) No matter what, DoS is quite a low priority in terms of security. And it's not even really DDoS, you'll know a real DDoS when you're on the receiving end.
> A residential proxy virus is in fact quite benign for both.
That's only because it isn't being prosecuted appropriately.
Let's not get ahead of ourselves... If someone plants a bomb under your car, you drive to work, then it blows up while you're out for lunch, should you be prosecuted?
I'm all for creating consequences for people who negligently don't secure their systems and people get hurt because of it, but people who installed a mobile game that has a proxy library in it? Please, chill.
> The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
Indeed. I recently set up a bot on my local machine that I can interact with via Telegram. I gave it the ability to retrieve the contents of a URL via Playwright (with modifications). Basic things like getting the Readability view and getting the site as Markdown with links.
I was essentially mimicking the feature OpenAI etc offer where it can do a web search for you and get you the answer you seek. I just didn't want to pay a company for that feature.
It's not autonomous. It only acts when I tell it to. The actual search is via the Kagi API, but the URL contents is via Playwright. I also ensured it can't do lots of requests at a time. There's a mandatory (random) sleep of between 1 and 5 seconds between two requests.
In less than a week, I've been flagged. When I do a Google search (as a human), I have to prove I'm human. Many sites have simply blocked me - I can't buy stuff from various vendors[1], I cannot access several news sites, etc.
The thing is, I didn't really access that many sites with the bot! At most it's the equivalent of someone mildly addicted to the web. Yet they caught me quickly.
Obviously, I disabled the bot's capability to get site content. I wonder how long my IP will be in the blacklist.
I know some people will say "Good riddance" to the outcome and to what I did. I'd love to have a real, serious discussion on whether what I did was immoral. But whatever your thoughts on the matter, do ponder over the fact that the system has made it hard for average Joes like you and me to have this "feature", but if you pay a giant tech company a monthly fee, you can have it.
[1] Thankfully, Amazon didn't block me. In the past I'd avoid them, but now I'm in the dilemma where I don't want to buy stuff from sites that make it difficult for me to buy stuff from them!
I think that major gateways like Google and Cloudflare and Akamai are doing things like TLS-fingerprinting and smearing.
I normally use a VPN to access the Internet. It's an ordinary consumer-grade VPN, and I use it for pretty innocuous things like watching the international versions of Netflix or avoiding being geographically-profiled and charged more when I book travel. The VPN itself, I'm sure, is being used for all sorts of naughty stuff: when I look it up in whatismyip, it comes up as VPN, "abuse detected", and "residential proxy detected". I expect to get blocked occasionally on it and just pause the VPN then.
What I found surprising is that pausing the VPN doesn't always clear the CAPTCHAs. The next request will also fail, sometimes. For a while I was wondering if I have a residential proxy on my network, but I went hunting with lsof and bandwidth monitoring and a list of all the open connections through my router, and it seems clear. Yet even with VPN disabled, I'll get CAPTCHAs from Google and Cloudflare.
I suspect they're using browser/TLS fingerprinting and then smearing the results of that across bad IPs. If you connect to them through a known-bad IP, they not only will CAPTCHA you, but they will record a fingerprint of any data they can. Then if you connect again with that fingerprint, regardless of the IP, they block that too.
Ironically, if you are actually malicious, you can get around this pretty easily. Because you'll know not to use a fingerprintable browser, or to rotate fingerprints. So you have one completely separate identity on a given IP (that is not yours), and if it gets blocked, you roll up another completely separate identity on a different IP. But this is too much work for an ordinary consumer who just wants to keep their IP private. So you end up with this excluded middle where the only rational strategies are to either do exactly what the website expects you to do (with no funny business, even if it's innocuous), or to go whole-hog as the aggressor where you're determined to skirt their policies no matter what, and if you're doing that you might as well amortize the effort by extracting everything of value you can from them.
I just did this as well. I also added paywall bypass by default along with inline article chat. The internet is such trash these days.
I made my modifications at the network level and auto redirect a configurable set of sites to go to reader mode/paywall bypass by default with a MITM proxy and a custom CA cert I trust
Expect to remain blocked, that's how defense works, some services plenty use will have is bot or known proxy exit node set to true, and some websites use the same defense services, request velocity is just one factor, if you don't have random human like scrolling and jitter with a bot, it's a dead profile walking, metaphorically speaking
I don't think this is an immoral application at all from your description, but would that be clear on the receiving end?
Just restart your home router to rotate your IP address.
It depends on the DHCP part and your provider lease time. When you access the Internet through a modem and landline, it's an highly dynamical system, where it's difficult to guarantee every caller/user gets its own IP. Especially when IP sharing is used among the clients. Even more difficult it is with the mobile Internet. It's a constant connection loosing/reconnecting. Actually resetting and freeing up IPs for new devices, the whole time. With cable Internet (the TV cable) one is having a kind of stationary access point. The network provider can see it's topology and it never changes once a leafe has joined the network. I typically have the same access creds over months. Even when I restart my cable modem, it's still the same IP. When I was on modem, I could get a new IP every second of I wanted. Now I have to think twice, because I just can't get a new IP, even if is forcedly, not.
That is entirely ISP dependent and many don't actually work like that. Your cable modem gets a specific IP for as long as Comcast or whoever sees fit, restarting does nothing on the WAN side.
In my case a downtime of over 4 hours will switch it. Might just be lease time doing its job.
This is simply called the dhcp lease time. Restarting has never forced a change of ip on a system using a dhcp client.
True, I've had the same IPv4 address since 2018 on Spectrum despite rebooting at least 10 times.
Doesn't work like that, only on mobile phones you can toggle airplane mode to get a new ip assigned and cgnat makes sure mobile ips are "trusted" , but they fingerprint you beyond ip address
> I know some people will say "Good riddance" to the outcome and to what I did.
I will instead say "skill issue".
Using Playwright results in some very obvious and easy to detect fingerprints. I'd expect better of someone on hacker news.
Mine hasn't been detected as far as I can tell, so I know it's possible to do better.
elitist nonesense. "Look I am soooo cool, and you are not as cool as me."
Weird to hide basic stuff like random mouse-paths, mouse wiggle.
Compared to, russian intel scraping beyond demented word of mouth.
> 100req/s
While your server handles that, what does your CPU do with the other 99% of available cycles?
I run https://bunnystomp.com on an Always Free VM from a major cloud provider. It handles 1,930 req/s at 33 ms latency with no tuning/optimization. It's literally just python behind cloudflare.
You seem to know a lot about my server. From what I could see, all cores were pinned to 100%, but maybe my budget AMD CPU is hiding fantastical amounts of compute from the kernel. I'd love to know how to make use of it.
I have no idea what the site you linked does, but it doesn't look like a cgit instance to me; that's what got hammered on my end, and when hosting it, one very quickly finds out that many git operations are CPU-bound.
Funny. My site is a search engine and cgit is one of many tarpits that I block. Otherwise the index gets filled with garbage and the crawler hits TCP conn limits because the bytes trickle in too slowly.
If you want credibility from the get-go, you need to be up front about your application. I certainly took it to be a generic host for an unpopular site in your first comment.
Nah this is hn, just trust people upfront
I'll just note that "compromised" is perhaps too strong. These services often come from VPN plugins (eg Hola VPN) that lets users watch sports from different restricted geographic regions. There is a trade here that's in the TOs: you get someone else's IP address, they get yours.
Compromised describes exactly what you are talking about. Any free rider on a network that the operator is unaware of means the device is compromised.
If I'm using one of these services while connected to a guest wifi network, with the usual configuration for guest access (like no intranet connections), is that guest wifi network compromised? What's the threat model?
> is that guest wifi network compromised?
I'd say yes since you've allowed random people unauthorized use of the connection. It makes no difference if it's your connection or not. For example, if that proxy gets used as part of a DDoS attack, your device compromised that network.
how is this different than Tor? You seem a bit on the surveillance side. The computer/network is compromised if someone gains unauthorized access to it not if you share it.
TOR doesn't use unsuspecting participants as exit nodes, that's something you have to specificially enable.
If that guest WiFi network is running inside the White House? Yes, I'd say that is compromised. The threat model being that guests on that network may be diplomatic officials and staff.
That's not how Hola works. You're not sharing your IP with some other Hola user. You're sharing it with bots used for all sorts of good and bad things.
What part of that isn’t technically “compromised”? You can argue there’s no evidence of it currently being weaponized but I’d say some shitbird running a free vpn inside the White House is not a good look regardless.
We'd need a pedantic definition of "compromised" to be sure
Why wouldn't an adversary be using it to conceal command and control traffic?
> This also puts some of the troubling reports that politicians were bringing phones into SCIFs into context…
I mean politicians are so addicted to twitter the nsa makes special phones…
Not sure which direction you're intending this. The NSA is making special phones to protect politicians, or to make it easier to ensure they can read everything?
Por que no los dos?
It's always better to protect politicians from someone else
Politicians bringing phones isn’t a scif isn’t necessarily a bad thing.
They’re too polite to say it explicitly, but when they say:
> Most of the activity happens in the evening, Eastern time: about half of it falls between 5 p.m. and midnight, and almost none happens in the early morning. That pattern suggests a person rather than a public guest Wi-Fi network: someone regularly on the White House network late into the evening, whose device is infected with a proxy SDK. Our data shows the exits, not the device or the person behind them.
There really aren’t very many candidates for who that person might be and it really seems like it would be someone who, I don’t know, maybe lives where the traffic is originating.
A lot of people work at the White House. Many of them doubtless work a night shift. The set of candidates here is actually quite large.
> A lot of people work at the White House.
I'm no expert but even I've read that "Works at the White House" is bigger than the executive residence and oval office, including the West Wing and the big office building that's directly connected to it by basement walkways. The National Security Council maintains 24-hour watch operations in the Situation Room (which is more a 'complex' than 'room'), while regional desk officers routinely work late evenings to overlap with morning business hours in Europe and Asia.
I asked Gemini if my recollection was correct and got: "The EOP network (managed by the EOP Office of Administration in coordination with Presidential Information Technology Community entities like the White House Communications Agency) operates as a single campus enterprise network. Its high-speed backbone links the West Wing, the EEOB, the East Wing, and nearby administrative facilities such as the New Executive Office Building (NEOB) on 17th Street."
So, definitely an issue that there's a device with some rogue malware installed but the total workforce across all shifts at the White House Complex is roughly between 2,500 and 4,000.
"and the big office building that's directly connected to it by basement walkways"
I ask in good faith, but why didn't you just say the Treasury Building?
Most people don't know that the Treasury Building is a big office building directly connected to the White House by basement walkways.
I didn’t.
Fair point!
Because it's not the treasury building lol. There are multiple buildings near the White House that are under this umbrella, and I wouldn't think Treasury is one of them since they need their own office space.
Because it's about Eisenhower Executive Office Building
Why is this omission of any note? I know the building and I can tell you that it is not particularly remarkable.
the show that came out recently, The Residence, while obviously fiction, made me think about the white house as a semi-public event venue in near constant use by dozens if not hundreds of people. It made for an interesting setting for a whodunit.
Of course it is. It's "The People's House." Look up the Great Cheese Levy of 1837, Lincoln's Office Hours, Coolidge's reception on New Years 1927 or Jackie Kennedy literally giving an "MTV Cribs" on TV in 1962. Or consider, if you can remember, the outcry when they permanently closed Pennslvania Avenue in 1995. Or when they closed Lafayette Square in 2020.
It's amazing thinking about Teddy Roosevelt following the New Year's Day tradition of shaking every single person's hand who showed up at the White House.
Which in the 1907 case meant 8,513 people's hands.
There used to be around the clock anti-nuclear vigils in Peace Park, with soup kitchens feeding folks who lived at their signs.
I can't imagine you'd need very many days of traffic to narrow it down to a very small group of people though. If a device is coming into and out of the campus, that should be fairly easy to reconcile with visitor records. Given that this is one of the most secure locations in the world, I'm reasonably sure a comprehensive list of "who is and isn't on campus" at any given time is available.
My prosumer ubiquiti gear will give me the MAC address and AP that the device is connected to, along with signal strength. WiFiman and a 5 minute walk will give me the mapping of where had that signal strength from that AP and at that point I could find the device with a physical search. There is no way that this device can’t be found in a few hours by anyone who is even vaguely paying attention.
Even if many sometimes work at night, you would expect them to also regularly work in the morning, which isn't the case here? So the fact that almost none of the traffic is in the early morning is a further clue pointing to a certain resident whose office hours start around 11 AM...
Cleaner that comes in after normal office hours, with a phone which connects to guest wifi, which is segregated on the LAN side, but has public IPs from the Whitehouse AS
But if you’re the cleaning staff at the White House do you just get on the WiFi?
Do you believe the availability of wifi for staff/guests is somehow unreasonable? Or, is there a belief that no internet (wifi or cellular) connected devices are allowed on premises? Is them using cellular data, rather than a guest network, somehow more secure?
I'd hope the guest network is completly separate.
I don’t understand. Is it not?
Maybe I'm confused.
Guest networks are configured to disallow intranet connectivity, so the "security" risk would be from the device having access to the internet, on a guest network IP. But a guest network exists for the purpose of giving untrusted devices access to the internet, without compromising the security of the intranet. My local library, the local airport, Home Depot, the movie theater, etc all have public wifi, yet nobody would claim they were directly compromised. I understand the connection could be used for receiving and sending packets over the internet, like a DDoS (which would trigger rate limits, even at Home Depot), but framing a guest network, giving an untrusted device access to the internet, as some security issue seems very odd. The isolation is the mitigation.
The difference is the home depot does not have spies applying for custodial roles. I’d expect nonessential staff to put their phone in a box at the security check. We also live in a world where israel can remotely explode telecom devices they rig in supply chain intercepts.
it is easy to correlate with known public travel...
But countable.
Not if you exclude the ones who behave as if the rules apply to them.
Most people working shifts will work varied shifts, sometimes nights, sometimes days, sometimes they'll be missing for days or weeks on end
Could even be cleaning or security.
As someone elsewhere in the thread noted, this could well be the press. Could also be that this isn’t the White House network at all despite that attribution.
Right, we know the White House owns that public IP, but this could be as bad as an unsecured, unmanaged device that is joined to a private network that is supposed to be secured, OR, on the “all the way good” side, that IP could be the exit for a completely throwaway, untrusted, isolated LAN segment meant only for some iOT device and which has no route to any other peer on the LAN side, and some civilian who has access to that for work brought in his laptop with a “free VPN!” Plugin and joined it to that network because it’s all he had credentials for.
One would expect network activity related to the press to be concentrated during daytime hours, not between 5 PM and midnight.
If it keeps happening I think that at least suggests the source is someone or some people who are “above the rules” most people would be bound by. You can tell the cleaning crew to get off the damned WiFi, others not so much.
Even at midnight I can't imagine there being less than 100 people working at the WH in various capacities. While it's probably not the duty/desk officers in the Situation Room/Security Council, there are plenty of maintenance staff that work at night. 5-midnight seems like it overlaps perfectly with a shift schedule.
But how many of them would risk their job by smuggling in a device that violates government policies? Every device in the White House should be accounted for and meeting a number of security criteria. There are really only two or three people there who would know that those rules don't apply to them.
They are not saying that is _on purpose_, in fact they are saying the opposite: a compromised device, could be even an Android phone. It could also be as device planted on purpose, of course.
Given that, why are there so many loose phones at all? Here is one compromised that we know about, in the biggest target for espionage on planet earth.
You’re misreading what this says. It’s someone bringing in a device they managed to get infected.
Oh and also the part where no one has caught this internally which is possibly the bigger takeaway. It’s a zero-trust network made out of incompetence.
There's nothing in the article that suggests it's not just the guest wifi. Which wouldn't be monitored. Why would you? The whole point is that it's segmented and can't actually interact with anything important.
Eh, I would not be surprised if they have a separate wifi that guests can use and you can hook up your private devices to. Basically every large company i have worked for, for the last two decades had that. I suspect deep in the bowels of the White House you don’t get good reception.
Why are you assuming the person knows and is actively smuggling in the device? And why would you then assume that person is more significant than a shift worker there?
Probably because hegseth decided to actively use signal to discuss military planning instead of secure channels that don’t invite such screwups like adding random citizens to these discussions.
The White House doesn't shut down at 5pm. There's whole books, TV series, and movies about the people that are working late every night at the White House. Likely dozens of people are on staff and work shifts that last until midnight every night of the week.
Service staff, including the storied hundred or so White House butlers, work whatever hours are needed. Then you have Secret Service, Kitchen staff, Natalie, house admin staff like the Chief Usher, maybe even NPS staff as the WH grounds are a national park as well, any of whom I would assume might be given access to the secure but unclassified WH wifi and likely a government-issued device. One device with a proxy among all of those people? Entirely unsurprising.
Maybe it's just the printer that Natalie carries around in her purse.
Actually sound like the opposite of someone who lives there...Sounds like a shift worker.
That timing could also align with a device that is turned on and off at specific times, for example an insecure smart TV that is generally turned on to watch TV in the evening, and turned off around midnight.
Though the fact that it lines up with an 8 hour shift also makes it seem human-like, so I think I'd rank a human carrying an infected device as more likely than an infected periodically-active device.
Or the cleaning or security staff working the night shift.
If you regard 1,000 or less people as few, then this is true. If you regard 1,000 people as "too many" then you rate the national security of the USA at the cost of 1 in 1000 means it's not worth bothing to find out.
Is that really how national security works?
What makes you believe this is related to national security?
There’s a ton of people it could be. Any staffer that works 5-12, which wouldn’t be a particularly weird shift for cooks, cleaners, etc that bring their infected phone with them.
If it was someone who lived there it wouldn’t be exclusively that time period. Why wouldn’t it happen overnight or in the morning or any other time of day?
Unless it is running on a government issued iPhone, it is not the Individual #1 you are alluding to.
He very famously does not know how to use computers and can only basically manage tweeting from his phone, which he is mostly not, actually, doing himself.
Now, there are two or three terminally online people around him. I would be more quick to point the finger at his foul-mouthed spokesman but it’s admittedly because I hate that guy and he seems rather stupid. But perhaps it could be the devoted portable printer lady, or the overpromoted unblinking former golf caddy guy.
All of them are around to help with the, er, modern day presidential communications.
> He very famously does not know how to use computers and can only basically manage tweeting from his phone, which he is mostly not, actually, doing himself.
Despite the constant negative press covfefe
Most of it is Harp or Scavino, as far as I have read.
Does it continue when the prez is in Florida? Seems a good test on the hypo
There’s at least 500 people on just the staff of the White House, not including many others who work nearby.
With the authority to walk in with a streaming stick, plug it into a TV and connect to WiFi!
Do you imply this residential proxy is also a Presidential proxy?
It’s probably Natalie.
Yeah dude you’re right Trump bought an android tv from banggood and got pwned. You’re a genius.
He never buys what he can grift for free. Probably a gift from some autocrat in exchange for politcal favors. Much like the Great Seal bug.
He got gifted an entire jet!
This article seems to cast residential proxies in a wholly negative light. In fact the company is openly antagonistic to anonymity. I find that perspective diminishes my trust in the rest of their claims.
No one would agree to host a residential proxy if they knew what they were signing up for.
The whole industry is dirty.
Residential proxy is the industry euphemism. In most cases they aren’t set up willingly. Access is sold to various bad actors, for data collection, botnets, or for a clean residential exit IP
So they don't actually know what this network is, just who owns the IP space... time for a blog post to advertise our week-old service!
There is no evidence that physically ties this to the WH. It could be in a security lab for all they know. Or a guest network for the news media.
> There is no evidence that physically ties this to the WH.
They don't need to be physically inside a specific building to be a networking security problem.
The core issue is whether an outsider has bypassed your firewall and can send+receive traffic from a node which is getting more trust than it deserves. If they've compromised the computer's microphone/camera/wifi-control that's an additional problem.
"We don't know where the devices that appear to have breached the WH network are located. For all we know, they could be in Beijing or Moscow."
That seems significantly worse.
Is it some like like NordVPN? It has an option to act as a residential proxy (useful to evade VPN IP blocks)
Perhaps ping time measurements could help identify where the devices are physically located.
If you were really sneaky, could you add some small jitter to the network responses to make it more difficult to locate?
You can only _add_ jitter. So presumably you make people think you're farther away than you really are. I guess if you're especially clever, you can manufacture responses to packets you haven't sent yet to throw people off.
Jitter is not a defense against statistical analysis. Additional lag might be what you want.
Additional constant delay? Isn't that also easily defeated when you're measuring from multiple nodes?
Or just block inbound ICMP.
No. You would need to add hundreds of milliseconds depending on the sender's location.
How did nobody make a (p)residential proxy joke yet?
Gotta have your SuperBox set up to watch 24 hr cable news working late into the night.
For those who don't know what this is, here's a decent podcast about it:
https://darknetdiaries.com/episode/172/
There's a transcript if you want to read, not listen. At the start, you'll be "hey! I gotta get me one!" and later you'll be like "burn it with fire!"
> I get a hit in my vulnerability log, like in the threat log on my Palo Alto firewall, and it’s for a SCADA vulnerability. ... A SCADA vulnerability. This makes no sense. SCADA is the control systems used in large-scale industrial settings. Think pumps, valves, conveyor belts, compressors, elevators, railway switches. This is where SCADA systems live. Why in the world is this box that’s here to deliver TV and movies attempting to trigger a SCADA exploit on D3ada55’ network? This is very concerning.
Such things are threats to humanity imho. They can get us back to the times we couldn't exploit the internet to run a society.
Interesting! According to this, a lot more going on beyond the residential proxy element, which was the only bit I had considered (or heard about). I wish there was an accompanying article with some more specifics.
I aso would like a technical deepdive, the transcript comes out as not very well informed and borderline conspiratorial.
“ Most of the infected devices were manufactured in China. ” uh… most devices are manufactured in China period. Very strange article/podcast.
I have seen these at mall kiosks advertising "free cable", and today I learned they are proxying from your home network.
That is the least bad thing they could do btw. Unless your internet is data restricted, go for it if you want free cable. If questioned by your ISP (extremely unlikely but theoretically possible) pretend you have no idea what they're talking about.
Firewall it from your local network if you want to - it'll still work - they're not after that.
Absolutely, and this is why I refuse to run connected devices on my network that do run software I can reasonably trust. This is also why every computer, phone or appliance I own runs Linux. This includes my TV box, which I recently migrated to an x86 port of Android with Claude's help.
That was the first thing I thought of!
A friend in Texas claims they recently bought one at Wal-Mart. I don't know anything about how these work but that can't be true ... right?
There are plenty of people selling IPTV access on g2g, few bucks for a month, all TV channels from entire world. I am not sure if it is profitable to let such piracy boxes work forever, even with them being used as residential proxy, but I also don't know how low you can go if you skip middlemen.
It's absolutely true, these are sold all over the place
Those are better than residential proxies. Those are White House proxies. I would pay $10/GB for those babies.
Those proxies are paid for in rubles, not dollars.
They are (p)residential proxies really
Executive residential proxies
I have T-mobile 5g. My IP shows up as a residential proxy. I can't control the T-Mobile 5g device. It's a black-box. And, I only have one Linux desktop on the home network (so no Windows/Mac malware).
Not sure why my IP is listed as a residential proxy.
There’s only that many ipv4 addresses and I believe all 5G are shared ones. That’s why they’re among the most valuable ip ranges for scraping. You can’t just naively ban that IP or you’ll ban a large group of users
You certainly can ban that IP.
It should be a balancing act though, somewhere between what you said and my first sentence.
Rate limiting, or additional layers of input sanitisation for IP addresses known to have been malicious in the past.
It's unfortunate that good users will have to endure some difficulties because of bad users on the same network, but if security is the priority, then measures must be taken, you shouldn't just leave to door open to potentially known bad users / traffic.
IPv4 is a shared resource, but some ranges have worse reputations than others, and all are owned by organisations that have the ability to monitor and filter for malicious traffic. It gets into a fraught discussion about censorship, but it is possible for network owners to monitor for, and block / restrict traffic detected as 'bad'. It's just a matter of cost and scale, and those that put the effort in should have their users rewarded with fewer 'trust-issues' roadblocks.
> It's unfortunate that good users will have to endure some difficulties because of bad users on the same network, but if security is the priority, then measures must be taken, you shouldn't just leave to door open to potentially known bad users / traffic
Is this like a corporate support response? I legit can't tell. Try explaining to a retail consumer why they can't buy from your site because they have TMobile service.
They’re saying T-Mobile should be monitoring traffic on their network and block traffic from bad actors that ruin it for everyone for everyone else? Or at least they do in the next paragraph.
When the bad actor is grandma who has an infected computer, T-Mobile will be dealing with a lot of support calls from objectively non-malicious actors.
>you shouldn't just leave to door open to potentially known bad users / traffic
Why not? If security is the priority, the worst that could happen from that is clutter in logs and higher resource use. Nobody wastes 0days spraying them on random hosts, and a real motivated attacker targeting your server will not be stopped by an ip blacklist.
The blocking of residential proxies might be the thing that moves the world to ipv6. These days what we need is a dynamic map of all the IPs running proxies so servers can block them. If every device gets it's own v6 address, you could block just the TV running the proxy. Modern crowd sourced security products could identify the proxies and share their IP live so you can ban and unban addresses as the device moves.
A list of proxy lists: https://proxy-scraper.de/sources/
The lists generated by this tool can be useful as well: https://github.com/maximilianfeix/proxy-scraper (which point to here: https://github.com/maximilianfeix/free-proxy-list)
There is also this bot proxy IP list: https://github.com/antoinevastel/avastel-bot-ips-lists
The TV can just pick a different address when it notices that it is being blocked.
In that case you’d just block the whole subnet that the TV is using.
Not sure about US but where I live I can manually set the access point of my phones mobile network and it gives me a unique ipv4 (new one each time in fact)
Care to elaborate? Your mobile network operator gives you a long list of APNs that you can use?
You might just be experiencing T-Mobile's CGNAT on mobile broadband?
You're almost certainly behind a cgnat shared with a bunch of compromised/residential proxy running phones and other traffic.
Or ones who are running it knowingly to make money. Can that even be called a compromise?
T-Mobile esp. their 5G network is behind CGNAT
So somebody likes to appear as if he/she works from the white house, but actually isn‘t there? Why not use a VPN for that?
There's no commercial server with an IP address at the White House.
https://layer3intel.com/what-is-a-residential-proxy
"Residential proxies" are just botnets: it's mass-spread software under direct central control without the knowledge of the owners of the machines.
Thanks for clarifying the language. I don’t understand why we softened ‘botnet’, it’s much clearer and powerful.
You resell a game, bundling a botnet proxy inside, and make it freemium: free if you accept T&C with a service allowing using your device connection for research purposes (or whatever wording could be even more confusing), or pay $20.
It's no longer a botnet, it's a "residential proxy" and you are doing clean business.
In my head this is Joe Pesci saying "the money is now laundered" in Lethal Weapon 2.
I didn’t know this was a thing. Yikes. I’m gonna freak out now and find a way to prevent that in our home network.
And I’d argue that’s not clean business, the user doesn’t know what their connection is being used for :)
"User accepted T&C" is enough to get away with almost anything. It works very well for top IT companies. Upholding new business to higher standards would be unfair.
Trillion dollar AI companies use/used these to scrape the Internet. You don’t want to smear their good names do you?
Out of curiosity, how would I check my own device for a residential proxy?
Scroll down to "anonymization": https://ipinfo.io/what-is-my-ip
If there's anything there, monitor your network to figure out exactly what
It lists VPN, Proxy, Tor, etc. Presumably a minus sign in a circle means you don't have the thing?
I had same confusion for a second but the html does clear it up: yes, it means you don't have the thing.
Freaky. They have a page explaining the concept (https://ipinfo.io/data/residential-proxy); when I checked it, there was a FOUC showing a full-screen, stretched image of what I think was Dario Amodei's face.
Neat. I almost said "surely this will only detect if your internet connection is itself being routed through a residential proxy", and then I realised that's effectively the same thing!
AFAIU that tells you if your access to ipinfo.io is thru a residential proxy, not if somebody else is using a residential proxy on your network.
It's effectively the same thing. If a residential proxy is running on your network, your IP will soon become known as a residential proxy exit.
It detects if your IP is a residential proxy, not if you are using a residential proxy. From the perspective of the outside Internet, they are the same thing, the only one who cares about the difference is you.
Fantastic question. There is so much IOT garbage out there that someone might slip into my network, what can I do to easily detect it?
I was aghast when my partner connected a pet watering bowl to the WiFi.
My most worrisome IoT device I have on my network is a digital picture frame my wife connected, but I was surprised the company routinely updates it even though it's on Android 6 (!).
https://support.frameo.com/hc/en-us/articles/23433637149842-...
… TIL such things exist. I assume it's networking so that it can grab photos to display from "the cloud" somewhere, and periodically change?
Yes. Smart frames often provide an endpoint where friends or family can send photos that will be automatically added to the rotation.
Great gifts! (At least for those unconcerned with privacy or security, which is most people.)
It's been some time since I ran this myself but I used to have an openwrt router sit behind my ISP modem/router (it was one unit).
Multiple times I caught unexpected traffic and was able to block it piecewise including a lightbulb(!) that was sending mb packets of god knows what home.
If they were ethernet based I reckon some smart VLAN capable switches would be nice even at home to keep them out of each others business.
I moved home a few years ago and now I'm back with the ISP router and do often feel a little in the dark about what my network is doing!
https://openwrt.org/
I suggest you just firewall it away from your important machines and then don't worry about it.
This one actually works: https://news.ycombinator.com/item?id=49713037
Watch egress at the router for anything unaccounted for, tbh.
I fear this is just the tip of the iceberg. It's the WH's own TOR node for any paying customer. It's manchurian candidate. It makes Clinton's emails look like ice cream.
It's incorporated into a lot of consumer devices legally. https://brightdata.com/proxy-types/residential-proxies
"Legal". The law is truly an ass.
Thankfully security researchers exist to fill the gaps that lawmakers haven't gotten to / won't get to: https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...
These are the specific URLs mentioned:
I use these regular expressions to block any connections to the domains or subdomains: No idea if this actually prevents proxy functionality altogether, but prevents some phoning home if you're infected.See below for his this "consent" actually works. (The screenshot of the game.)
https://spur.us/blog/smart-tv-apps-residential-proxy-sdks
Brightdata still sucks at disclosing to users that they're opted in to this and that their connection is being used for this. It's "on by default", not opt-in and often buried in an unskippable "Accept the terms" or dig-to-opt-out prompt.
What is a residential proxy?
It's described at a high level in media often. Here's more detail though:
Person buys a product, or downloads some software - that unknowing to them, was reaching out to a central server elsewhere to establish a network tunnel. And over that tunnel, the other server routes traffic onto that person's home network. Which can then of course go onto the internet, and it appears like the traffic originated from their home internet connection.
Other people rent access to this network. The goal being, you have surreptitious access to appear as any one of tens of thousands of home network IP addresses on the public internet. Bad guys can use this to rotate through many different IP addresses when their access gets blocked because of malicious (or benign but unwanted) behavior on the internet. Residential IP addresses are more 'valuable' because they don't have the negative connotation that datacentre and commercial IPs can have, and they can be spread geographically or across many ISPs.
A very common way for people being a part of these residential proxy networks is by buying dodgy streaming boxes, and also installing shady apps on TV app marketplaces.
This isn't a theoretical problem either. My employer's website was being hammered by residential proxies a month ago. It wasn't until we setup WAF captcha rules that they mostly subsided. It wasn't even the typical exploit scanner. I don't know what the purpose was but it was effectively a DDoS.
Everyone's is. Far as anyone can tell, it's probably AI companies.
It wasn't any sensible AI scraper. It kept requesting the same pages a thousand times over.
Yep that's what AI scrapers do
I was expecting that VPN’s (maybe not all) also use their own customers as relays, but you don’t mention them. Am I wrong?
It's definitely not all VPNs (typically the ones you pay for will buy their own hardware and internet connections in the places you can route your traffic through), but it is a common situation for the shadier (especially 'free') ones, and in that case it's usually not just a peer-to-peer thing, they make money by selling your connection to people who pay because they want a large pool of relatively reputable and disconnected IPs to send traffic through.
You're right, a lot of 'free VPN' software also uses your own internet as a residential proxy. It's often mentioned in the ToS but the users don't read that.
Honestly, I expect any of those free VPN services that require an app install to be doing precisely this. At this point, I just assume anything with "free" in it will cost me more than if I just paid for something instead.
I'm surprised it doesn't cause more chaos, even as simple as people getting copyright infringement notices from their ISP?
That kind of VPN is considered a residential proxy.
It means some networked devices in there are being remotely piloted to tunnel network traffic to somewhere else. This scenario usually happens with more "unprofessional" ISPs, which sell bandwidth from their networked clients for people in need of such services.
Such as ignoring web scraping countermeasures, and DDoS.
Some good answers in this thread. I had the same question but I'm still a tad confused:
* I initially thought this meant someone in the WH network was using commercial proxies to appear as if they were exiting from a different network (which would align with "some user is firing up a proxy on their phone to exit in a different location")
* But then the description of residential proxies seemed to align more with traffic hitting an endpoint in the White House, and _exiting_ from the network there. But this didn't explain the timing (why isn't it 24/7).
* Finally it sounds like some person, legitimately on the WH network, has a device (cell phone, TV, whatever) that, when activated, is proxying outside traffic into the WH network.
There are companies that provide proxy services for scraping services, and residential proxies route through consumer networks (like Comcast) which is flagged less than a datacenter IP.
Many questionable TV streaming boxes pretend to deliver you video, but in reality they are also piggy backing on your personal ISP to power those companies.
Even not so questionable TV's do this too. https://spur.us/blog/smart-tv-apps-residential-proxy-sdks
>Many questionable TV streaming boxes
+ Sketchy Apps
+ Malware
+ Weird Android Variants
The 4th sentence in the article has a link to What is a Residential Proxy?[0]
[0]: https://layer3intel.com/what-is-a-residential-proxy
From the article:
> Residential proxy services sell access to everyday devices, letting customers send traffic through them so it appears to come from that device's network.
They really should have said "botnet malware detected", it turns the White House into an exit IP for someone's else's botnet.
Brightdata offers it as a paid service https://brightdata.com/proxy-types/residential-proxies
Essentially lets bad actors use your home network.
Bad actors as defined by who. Why am I bad for not living in the right country to access a website.
Because the person whose IP you’re using as an exit node probably hasn’t fully consented to it being used as such.
So what? Packets flowing through machines is just how the internet works. Stop believing all this propaganda that other peoples papers are EVIL and if you route them you are getting in BIG BIG trouble. This is harmful messaging that does not reflect reality.
There are services you can pay for such that you can route internet traffic thru a specific IP in a specific part of the world. People (usually unwillingly and unknowingly) run software that effectively makes them part of a giant botnet. A computer running in Romania might have some kind of weather application or gambling/casino app that I could use to make an HTTP request as if I were on their network. So if you need access to 1,000 or 10,000 IP's to circumvent rate limits or perform crawling tasks, this is how you could do it.
Second paragraph
I was under the impression that actual ISP providers embed into their modems/routers ability to resell YOUR connection to proxy services and charge for that without customer's consent.
Same happens within cellphones - via native mobile apps pimped by ISP/Mobile providers.
Could it be the case?
Can anyone recommend home router that makes handling that threat easy for average Joe? As in a simple list, for TVs detected on my network, with streaming services they should be able to connect to, blocking all other traffic from it?
Any router capable of being flashed with OpenWRT or their own router is a good start.
This site is barely a year old. Where is the data and proof? What is the methodology?
vibe coded "black box" ie: probably shit.
If residential proxies detected on White House network, why have network admistrators blocked the device?
And why have they not physically located the device OR at least vastly reduced its possible location , by determining the flow of the traffic from network elements ?
Answer my own question - It was actually an outside entity that detected the problem. However , it gives the network admins a way to determine if this activity is happening on networks they are responsible for .....
Good luck turning this presidents kit off
A person would think that ... security software running on the users PC would / could detect this problem .......
Google just banned a chinese Android tv manufacturer for backdooring and installing malware on their devices. This I think is such a device someone pirate sports or porn streaming device.
I didn't know that residential proxies were a thing. That sounds pretty handy because going online via a VPN gives constant nag popups from the likes of cloudflare :( I hate them.
It would be nice if the VPNs would integrate this option.
I don't need one from the white house though :)
How would one detect this on their home network and protect themselves from?
Step one, find your public IP, I like ipinfo.io but there are many other services that will show it. Then pop it into a tool like the one that's hosting TFA[1]
1: https://layer3intel.com/context
There is also iphub.info and iphunter.info.
In my experience, out of the two, iphunter.info has the more comprehensive database, I think the owner/operator frequents HN.
Unexpected traffic would be the biggest gross red flag. If it constrains itself to moving traffic when it's otherwise perform its desired function (if it's one of those "free TV" boxes and it only proxies when you're playing video) it might be hard to tease-out the unwanted traffic purely looking at bytes in/out.
Beyond that gross metric you're getting into capturing traffic and decoding it.
Check network traffic from smart devices would be a good start. If your TV does gigabytes of traffic in the night when you don’t use it, it most likely is acting as a residential proxy.
I've never heard the term 'residential proxy' before, but it seems like a Tailscale exit node would qualify for example? This doesn't seem necessarily malicious or even inadvertently dangerous to me?
Residential IP addresses tend to have better reputations than datacenter networks, so there are providers who sell access to home internet connections. Oftentimes the people running the proxies have no idea they're running on their network, they're quasi malicious.
https://cloud.google.com/blog/topics/threat-intelligence/goo...
"Quasi" malicious? What's quasi about it? It sounds completely malicious and fraudulent to do this without the knowledge of the network owner.
It's pretty much exactly a Tailscale exit node, just one that somebody else installed without your knowledge or consent.
Generally not really. Residential proxies are services that can provide customers with the ability to send and receive traffic from a wide range of apparently legitimate networks (i.e. people's home internet connections). Almost always this is done without the informed consent of the person who owns the device that is used as a proxy, whether it be buried in some agreement somewhere or just outright malware.
The biggest danger is that said proxies can also allow access to internal networks, and as you might guess from how they are installed, are not usually operated by the most scrupulous of organizations.
> whether it be buried in some agreement somewhere
is that actually a thing that was caught before?
Yes, some of these companies offer libraries to turn your mobile app into an exit node and get paid for the traffic. You can find news about Google Play playing whac-a-mole against apps doing so silently, often involving law enforcement as well.
Also some free VPNs use their users as exit nodes for their actual paying customers, good luck finding a user who understands that. It's the obvious way to monetize such a service other than ads.
it’s a well known term in shady markets of the internet, they allow users and abusers to bypass bans and create multiple accounts that cannot be linked with each other by networking intel
And bypass restrictions that are keeping data paywalled as well.
And use stolen credit card numbers while appearing to originate from the real cardholder's area to avoid triggering fraud detection
I was just thinking, what company's reviews could you trust the least than a residential proxy provider.
https://www.trustpilot.com/review/myxproxy.com?languages=all
The next administration needs to redo the whole infrastructure from scratch. They have to assume everything is compromised
Not all residential proxy providers are seedy - but I only know of one that isn't out of probably 20 or so major players so I guess fair call out on "infected".
https://archive.ph/ado1H
Not sure why the text is all black, but it's there
It would be really great to have some info about how to detect such things locally.
It is tricky. Monitoring network traffic should be enough, the infected device has to connect to Command and Control, to receive orders and set up tunnels. It got harder after malware creators started using Telegram, Discord or even good old IRC for those purposes - it will be asy to notice only if they use a service no one else uses in your network.
Step 1: have you installed anything from an app store such as Google Play or Apple's app store?
Step 2: have you bought any devices made by large companies like LG or Samsung?
Why not? Cass Sunstein even suggested such an approach in his book in the 1990s, back when he was the Presidential Science Advisor. It's no different than the King of England going out among his people in disguise to get the lay of the land.
OTOH it could be a honeypot, OR possibly somebody was careless. But there is no reason it has to be suspicious.
Based. Can we turn every IP address that exists into a residential proxy, and then finally end this IP reputation stuff once and for all?
I'd still find a way to block your scrapers.
Such as what? I've tried to block scrapers on my website, and only the low effort ones are easy.
Pure speculation, I don't have any concrete answers. I'm fairly scorched-earth with my blocking, but I'm only protecting personal environments, so no one other than scatter-gun-everything AI-scraper-bots would even bother.
I'd be very interested in AI capabilities in detecting anomalous activity given that it should be very effective at pattern matching across large data sets. What would be required, however, is significant training with industry/application specific 'normal' browsing traffic patterns to determine acceptable tolerances, and then design escalating deterrence if and when tolerances are breached.
How is the scraper process stepping through the site? It must be programmatic, even if it's programmed to be random. What does that look like in traffic data in terms of timing and content requests?
What hazards can be put in place that a normal user would never go near, but a scraper is likely to trip over?
If scraping is being performed from multiple locations in parallel, what might that look like? ie. joining multiple sessions together that may show a more 'scraper' kind of behaviour.
Under IPv6 wouldn't detection start burning a lot of addresses? Yes, the pool is large, but that just means it's a scaling issue from IPv4 to IPv6, rather than a new problem requiring new thinking to solve? (assuming ways of detection were successful).
I'm being somewhat devil's advocate in my blocking rhetoric. The internet is in a weird confluence of wild-west and surveillance state. I like that there are ways and means of being anon-/pseudon-ymous; I like a bit of anarchaos, that's where the spirit of hacking lives. But I also don't want it sticking to my boots when I walk in the house. I'm also not fundamentally against scraping; there's definitely good scraping. There is an irony, though, in protecting scraped data or services that are built on scraped data (and I'm not accusing you, personally, just my general logical ideology of scraping). On residential proxies, I think they're 99% run by erstwhile criminals leveraging what were, until recently, referred to as botnets into a service they can launder their botnets through as 'I am legitimate business man'. I think most, basically all, residential proxies are run involuntarily or unwittingly and that blocking them would provide a greater society benefit than it would a drawback. Anything that promotes the sanitisation of end user devices can only be a good thing. The whole ecosystem is in a terrible state and regulation doesn't even seem to be on the distant horizon.
Maybe I'm hypocritical, or I'm trying to maintain mutually exclusive positions. This is where I'm at though.
"Infected with a proxy SDK" wouldn't be my first guess. Am I jaded? A realist?
PS: I own proxybase.xyz, and I think we’re the most transparent residential proxy network in the industry.
I don’t think it’s fair to call all residential proxy networks shady. Not all of us run our businesses that way. Our code is open, we have nothing to hide, and we use US-based stablecoins for transactions. User consent is our number one priority. I haven’t seen anyone else in the industry match that level of transparency.
Why do you use stablecoins and not credit cards like every other non-shady business on the internet?
We have a no-KYC policy to protect our users’ privacy. If we accepted credit cards, we could link users to their real identities. We don't want that.
Sounds pretty shady
Uhh, and what exactly is the benefit for end users to have their devices send out ddos and scraping traffic?
It should be made illegal.
Hmm, all our proxy traffic goes through a centralized gateway, so we have security policies in place to help protect against DDoS attacks.
Do you really think your users aren’t installing onto every device they can break into to make a buck? Ignorance is bliss I guess.
Burning a compromised host just to run a public proxy client for pennies would be the worst trade-off imaginable for an attacker. It is loud, easily caught.
But her e-mail server!
Love me some buttery males
The biggest operator of residential proxy networks (BrightData) is based in Israel, which I'm sure has no connection to this at all...
Domain is feeling the HN hug
Is the first word in that headline missing a P at the beginning?
5:00 PM Eastern Standard Time (EST) is 6:00 AM the next day in Beijing.
But … HER EMAILS!!
NSA honeypots
I would actually feel better if this were the case, but given that we have "the very best people" currently running our cybersecurity infrastructure I doubt it.
Most likely scenario is a 20-something holdover from DOGE who accidentally had his phone p0wned while downloading groyper memes.
Most likely scenario is a janitor who installed an app (I'm guessing it was called LegitWorldCupWatch420696969) on his phone (which might be connecting to the wifi network from whatever locker he leaves it in while he's on the clock, the WH is not a giant SCIF). Or any number of other mundane, easy to overlook employees who work outside of the actual secured areas of the building.
Let's assume the worst, that it's a VIP. Ok, now let's do some basic frequency statistics:
If you click the AS2650 link (1) in the article, you will learn that AS2650 is announcing 65,792 addresses in 5 prefixes. Roughly 40-70% of the IP addresses in a typical corporate ASN are associated with actual computers humans are using. Let's go with 40%. You're looking at 26,317 people. Assume they're all adults (please, dear lord, let them all be adults). 3-8% of the workforce is in IT. Let's go with 3%, that's 789 people. I'm guessing one of those 789 folks has thought this through and has a few VLANs set up to keep VIP and family-of-VIP, pattern of life stuff in a box.
As for the thesis that these are all window-licking low-IQ goons, that's just not how the US government cultivates its workforce. It's no accident that Harvard Business School characterizes themselves as "the West Point of capitalism." West Point isn't someone else's anything. The US Government is in it for the long haul and captures folks young with impossible-to-resist opportunities like the military academies, ROTC, SMART scholars, etc. Let's just stick with the academies: 12,000 people a year who were likely competitive for the Ivies, Division 1 athletics, or both, but chose their parents poorly. Free, top tier education and a job is very attractive for kids from fly-over states. Hell, in big states like California, New York, and Texas, it can actually be even more competitive. Something north of 60% of those graduates stick around for at least 20 years of service (because the high three pension + VA benefits is a pretty nice floor for retirement when your options back home are line supervisor at the cotton mill, the sugar mill, the coal mine, or ramp manager for United), usually addicted to the crazy level of mission. Sort of the opposite of finance where people really struggle mentally if they're not making pallets of money at an accelerating rate. Government service pays dogshit, but the work can super interesting, especially as you gain seniority.
Those folks then often loop back into other agencies and do another 20 years. Are there goombas in the Government? Oh, for sure. Absolutely. But it's roughly the same fraction as any other workforce in the rest of the economy.
But by the time you get to top tier stuff like the White House staff, the sorting function has been strong. To make the point: you also see top tier folks in other organizations with government stink on them. I can think of at least two directors at Google who are prior military. A fat pile of McKinsey folks. Something like 8% of Fortune 500 CEOs have served in the military, above the national average (6-7% of Americans have served in the military at some point).
And it's hilarious to watch people simultaneously assert the government is a bunch of idiots, and then get all excited about fighter pilots, the CIA, the CDC, ambassadors, the NIH. As though those people just appeared from the head of Zeus fully formed. Where do you think they worked when they were young? They were stamping visas for the Department of State in some embassy in central Asia. They were rolling blankets and digging wells in Gambia for the Peace Corps. Checking corporate filings for mostly unknown companies at SEC. Studying aerospace engineering at Embry Riddle and then flying cleared checks between clearing houses across the state overnight in a Cessna.
(1) https://layer3intel.com/context/AS2650
... and then Donald Trump as President, Stephen Miller as, whatever he is, and Peter Hegseth as Defense Secretary (he did serve in the military, if that's worth anything).
I think it's pretty wild to make all these bold assertions about the absolute top tier of elite talent that the US federal gov recruits and retains, and then compare/contrast that with the current staffers of the present incarnation of the executive branch. It just doesn't jive. Seriously, do a deep dive into Hegseth's resume and background of the nonprofit he was the figurehead for. He's more qualified than some known 3/4-star generals? Seriously? Tulsi Gabbard was the most qualified person for DNI? Seriously?
I mean supply chain-attacks are everywhere. Smart-TV's with certain marketplace apps, IoT devices. Also, I'm surprised the white house's network is not monitored more.
Monitoring is the easy part, remedying the problems has proved harder. From early in the first Trump administration:
https://lieu.house.gov/media-center/in-the-news/what-can-we-...
That is not an environment for serious people anymore so that's expected.
probably something IOT with a vulnerability, happens all the time.
The joke that S in "IoT" stands for Security is as old as the term itself, but there were no such clear incentives for exploiting them before, beyond basic DDoS and more recently crypto mining.
Typical modern government bullshit.
Root it out or let it rot us
>that poorly secured proxy SDKs can allow internal network access and lateral movement.
Let me guess. The SDK being used does properly handle this and this whole article is anti residential proxy FUD.
Slop
Seems like the president found a business opportunity and gave internal orders to rent out the network infrastructure to 'ethically sourced' providers :^) Jokes aside, might be some infected IoT crap.
I buy and use residential proxies, though.
Vibe coders really having fun on their side quests these days
Residential proxies are a very important part of the internet, they allow scraping for archiving, automation and can be used for extra anonymity.
While they have grown a bit too much, they are a natural response to the absurd amount of effort every website is pouring into anti-scraping tech.
> Residential proxies are a very important part of the internet, they allow scraping for archiving, automation and can be used for extra anonymity.
Aren't they almost always run unknowingly, by people tricked into running them?
There exist services that will pay you a fraction of money to host their residential proxy service on a computer you own. But I do think the majority of residential proxies are hijacked.
There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.
Ethically, if someone just wants to play a free game and doesn't care about it, is it okay? How knowledgable does a user have to be in order to have informed consent?
> There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.
Betcha that means there was "consent" language on page 55 of some click-through EULA.
The landscape spans the gamut of everything from malware to explicit consent: https://packetstream.io/share-bandwidth/
Leveraging compromised devices is a very important part of the Internet??? This is an incredibly twisted perspective.
Help me understand. As far as I'm aware, these residential proxies are almost always malware.
"Malware is an important part of the Internet"
> and can be used for extra anonymity
Great, so you can conduct illegal transactions and have some unsuspecting grandma take the blame and have her front door kicked in?
Just like Tor, encryption, VPNs, etc, there are tons of legal uses for these proxies as well.
People who run Tor exit nodes generally do so by choice, and know the risks implications involved.
They didn't just connect a coffee pot to their WiFi one day.
Not all of these networks are built via malware.
e.g. https://packetstream.io/share-bandwidth/
"Malware is an important part of the Internet
Technically correct statement though
And making DDoS-attacks nigh impossible to IP block.
Residential proxies are in many cases at least shady, both on the side of who uses them and for what and on providers that are often not above using dark pattern to hide them in some innocuous app for a smart-whatever that silently runs a proxy on a unaware user's connection. And in many places in the world if illegal stuff in done using your connection you are liable, regardless if you knew about it.
Did a residential proxy provider write this
Huh, and I wonder why I've been getting tons of "you might be a bot" CAPTCHA challenges from Google lately.
Frontier lab training data “collector”
I wouldn't call turning on CloudFlare's bot blocker an "absurd amount of effort" from the perspective of site owners.
But getting around it sure is annoying, sometimes I get flagged IP's from Comcast and can't visit sites like costco.com despite not doing any questionable stuff on my network.
Hard disagree. They are malware 99.9% of the time. The utility does not erase the fact they are abusive.
Residential proxies are a very important part of the internet
So important that we have to trick people into using them. Remember when folks were running SETI@Home? Of course you do, because it was widely thought to be a good thing, and folks willingly signed up.
How many would buy the "box" if the advertising were honest about what's actually in the box?
Most tools can be dual-use. Residential proxies are also used by malevolent actors to hack infrastructure or financial institutions and let some other poor soul get investigated for it. This is why we can't have nice things.
They're also used by the good guys who are going after those exact same bad guys.
True, though unfortunately people can not be trusted to declare their intentions so the residential proxies will ultimately have to be sacrificed and given they are fairly easy to identify by behavior I could foresee the feds now having a reason to round 'em up faster. They already took out NetNut, IPIDEA, SocksEscort, Doppellganger and the "Kim Wolf" Offshoots.
Some of the more legit providers have KYC in place, and do literally require you to declare you intentions to open an account, and block certain traffic proactively unless you can prove a legitimate use.
How do I identify which residential proxies have gone through KYC and which are wanting to pour hot sauce in my under garments?
The ones that do KYC list it pretty prominently on their website.
They list the IP addresses of all the residential proxies? Is there a link to the netsets of all the IPv4 addresses they are using I can download once a day? (preferably over rsync but git over https is fine too)
I don't think even the most "ethical" providers are going to volunteer that information, since that defeats the entire purpose of having a residential proxy in the first place.
No...? That's a completely different question than what we were talking about above? The customers of a residential proxy who would be subject to KYC are the people using the addresses -- this has nothing to do with the exit addresses of those who have joined the network.
That's a completely different question than what we were talking about above?
In fairness to me, I did say "Most tools can be dual-use. Residential proxies are also used by malevolent actors to hack infrastructure or financial institutions and let some other poor soul get investigated for it. This is why we can't have nice things."
So it could be Alice. I know Alice has good intentions. It could be Bob, we know what Bob is up to but I will give him a pass regardless because Bob did nice things in the past. But then there is Mallory ... Mallory Malfoid is up to no good. I need to know which IP's are used by Alice and Bob so I can exclude them from my automated protections. Mallory will automatically get rate limited to 16kb/s, 1 HTTP request every 5 seconds and if they bounce around different residential proxies they get redirected to a bathtub regardless of which residential proxy they use.
Yes, but KYC is a process where the service provider is making that determination. Either to a legal standard and/or their own standards.
In no world does "KYC" mean "expose your customer list to the whole world", let alone undermine the entire purpose of the product you're selling.
"I want to dox all of the customers of a proxy provider" is just an unreasonable ask. We wouldn't expect or want ISPs or VPNs or telecoms to do this.
In no world does "KYC" mean "expose your customer list to the whole world"
Agreed. Alice Rhodes, Bob Bojangle and Mallory Malfoid will all be treated equally.
This was a fun thread, I have always wanted to give them last names.
s/Residential proxies/DDoS networks/, then re-read your comment.
>absurd amount of effort every website is pouring into anti-scraping tech.
Maybe some people do not want to provide their entire website as free training material to [insert LLM lab here]?
You... Do know that people scrape websites for non-AI reasons, right?
Why should the site owners be interested in allowing them to do that?
Because it's legal to scrape websites? And, oh, I don't know, maybe people like me might want to archive those websites for historical reasons?
Virtually all of it is the product of malware. This is like you saying that robbery is very important because you like buying cheap tools at pawnshops.