I wouldn’t normally comment on things geeky or for that matter on foreign political affairs, some might say I don’t normally comment on anything much, however I was recently linked to an article which was so badly written it hurt. The technical “experts” in the article are also either being grossly misrepresented or are privy to further information that hasn’t been made available to the rest of us – that or they’ve checked the brains in before commenting, I really rather hope it’s the first.
So the thrust of the article at The slate is that Mr Trump, or at least his organization, has been running a secret server to communicate with a Russian bank. This story seem to have originated on the dark net and thence to various bulletin boards and thence to the media. A mirror of the original data can be found at: http://gdd.i2p.xyz. I assume this is an accurate mirror I of course have no way of knowing, nor anyway of verifying much of the data. I’ve also created a local copy here, so you can look at the same stuff I looked at just in case it gets taken down. There also a slightly skeptical article about the matter here.
I must warn you that I may get a little technical, this touches directly on my day job and I’m potentially going to call in to question what very well-respected experts in my field have been quoted as saying and suggest that they may be very well talking out of their white hats. I’m going to have to make all sorts of assumptions whilst discussing this but I’ll try to be transparent about them. The reason why I’m giving this so much attention is because there are implications for all of us far beyond how this impacts the US presidential elections. So please try to stick with me.
So the first bit that really worries me and this is the bit that has implications for all of us, which the slate article has blithely glossed over is this:
“Computer scientists have built a set of massive DNS databases, which provide fragmentary histories of communications flows, in part to create an archive of malware: a kind of catalog of the tricks bad actors have tried to pull, which often involve masquerading as legitimate actors. These databases can give a useful, though far from comprehensive, snapshot of traffic across the internet. Some of the most trusted DNS specialists—an elite group of malware hunters, who work for private contractors—have access to nearly comprehensive logs of communication between servers.“
Emphasis mine. A database of DNS lookups doesn’t indicate communication as such, any more than your calling directory inquiries shows that you’ll then subsequently call the number you’ve been given. Yes that may often be the case but not always and it gives no indication as to what the nature of the communication will be. However we’ve just been told that an unknown and private group of people have access to a large database of which computers are looking up the number for which other computers. Also if they’re logging unfiltered DNS queries which they kind of have to be it’s not just communication between servers. Worse the people with access to this database seem to be assuming that it’s indicative of traffic, it is remarkably simple to get a computer to perform a look up of any address you like especially if it’s a server. Servers regularly look up the address of anything that connects to them to check if it’s legitimate traffic. So with a little bit of work a server could easily be caused to look up “illegal-kiddy-porn.com”, without the server owner having any intention of doing so or even knowing, and then our ever so smart white hats here would see that in their super secret logs and go “ah ha! That server is communicating to “illegal-kiddy-porn.com” they must be a wrong un”, in simplified form here’s how that works:
Bad guys -> innocent Server : Hello server I’m illegal-kiddy-porn.com
innocent Server -> DNS : Please tell me the address of illegal-kiddy-porn.com so I can tell if it’s really them
That’s it. Then that look up is in the super secret database and “innocent Server” is in the frame for talking to illegal-kiddy-porn.com. DNS lookups are really indicative of very very little, but the entire case being presented is based on only two things DNS lookups and public, trivially forged, whois data. Now I’m not saying our secret-nerds are wrong in their conclusion, but I am saying that if they presented me with that evidence in my day job I’d ask then to go back and do some rudimentary trouble shooting and maybe recommend they go on a training course. They may of course have done this trouble shooting and aren’t sharing it with us, but I’m working on the assumption that they are actually sharing almost all of the evidence they have. I say only almost because they’re obviously not sharing how they construct their super secret data base and thus how many of us they’re spying on.
Oh and by the way if you’re using a modern browser and have read this far chances are your computer has now looked up “illegal-kiddy-porn.com” and may be in their super secret data base, as I put a link to it in the article and many browsers pre-fetch links to make your browsing experience faster.
Now we’re told that our “good guys” are working with our ISP’s and:
“have cameras posted on the internet’s stoplights and overpasses. They are entrusted with something close to a complete record of all the servers of the world connecting with one another.”
So our ISPs are logging every domain name we look up and handing that data over to this third party for unknown purposes and with unknown safe guards, not anonymized and kept for at least 5 months. I would hope that the EFF and such are investigating this, but I suspect they’re not as this is being done by the “good guys” not the government – so it’s all ok that secret organisations are collecting this data on us. We’re told that the data from the alleged Trump server was found by accident and they have 5 months worth of data, this means it was collected as part of a general data gathering exercise that stores all the data it sucks in for at least 5 months. Which one would hope would have raised alarms with our normal defenders of privacy.
Now this is where things start to really really hurt, and the super-nerd needs to go back to computer forensics 101 (I’m assuming that they are being reported accurately).:
“In late July, one of these scientists—who asked to be referred to as Tea Leaves, a pseudonym that would protect his relationship with the networks and banks that employ him to sift their data—found what looked like malware emanating from Russia. The destination domain had Trump in its name, which of course attracted Tea Leaves’ attention. But his discovery of the data was pure happenstance—a surprising needle in a large haystack of DNS lookups on his screen. “I have an outlier here that connects to Russia in a strange way,” he wrote in his notes. He couldn’t quite figure it out at first. But what he saw was a bank in Moscow that kept irregularly pinging a server registered to the Trump Organization on Fifth Avenue.”
So they saw that a server he associated with a bank in Moscow was looking up a server associated with Mr Trumps server and from there they’re making the following assumptions:
1) That the Trump server is connecting to Russia, “that connects to Russia in a strange way”, this based on the Russian server looking up the Trump server. This is the equivalent of assuming that you’ve called me because I looked up your phone number. If they mean “connects” just in terms of a link a Russian IP probing a server associated with Trump isn’t that strange.
2) He’s assuming that the Russian server was communicating to the Trump server, all he actually knows is that the Russian server was looking up the address of the Trump server – as observed earlier this doesn’t prove communication anymore than looking up a phone number proves that a phone call was made.
I kind of hope that the article is massively mis-reporting and that there’s a whole load of other evidence not been made available to us as this sentence really makes me worry about the abilities of the people who are considered experts
” I also spoke with academics who vouched for Tea Leaves’ integrity and his unusual access to information. “This is someone I know well and is very well-known in the networking community,” said Camp. “When they say something about DNS, you believe them. This person has technical authority and access to data.”
They may know about DNS, but there’s no evidence of it in the data they provide quite the reverse, and they are leaping to some very hasty conclusions – again based on the data they’ve chosen to make available to us. Part of the reason why I say this is because they state:
“The irregular pattern of server lookups actually resembled the pattern of human conversation—conversations that began during office hours in New York and continued during office hours in Moscow.”
Now all DNS records have a thing called a TTL, this tells other DNS servers how long they can trust a DNS record for before they have to look it up again, it’s typically set to a day sometimes a lot less sometimes more but it always exists. It’s basically how long a computer can remember the phone number for before it has to call directory inquiries again. This means that unless the record was set to be very forgettable the frequency of lookups for the Trump address can’t be correlated with actual communication (ignoring the fact that DNS lookups do not directly correlate to communication anyway). Now it may be that the address had a very short time to live, we don’t know as our uber-geeks don’t provide us with a look up taken at the time to show what that TTL was, this is one of those bits which come under basic trouble shooting. So either the record had a very short life, or the behaviour isn’t typical of normal DNS traffic, we don’t know which as our hero’s haven’t given us basic data.
Now I’ve no reason to assume that the server being looked up by the Russians wasn’t registered by Mr Trumps organisation, but the fact that the whois data is consistent proves less than nothing. It’s the equivalent of saying that a van with “property of the Queen” painted on it, must belong to the Queen. Whois data is public data, and by and large isn’t validated so there is nothing at all to stop anyone looking up other domains registered by the Trump organization and copying those details into a domain they register. Yes the domain probably belongs to part of one of his organisatons but the whois data doesn’t prove this, and our experts should know that and be clear about it. They go on to say that it’s a capacious server:
“this capacious server handled a strangely small load of traffic, such a small load that it would be hard for a company to justify the expense and trouble it would take to maintain it.”
Now unless they’ve done a lot more hacking of the server in question than they’re telling us, they have no way to know how capacious it is (it could be a Rasberry-pi), nor if it’s being used for other things, and there are many good reasons to have a server idling as you may use it only now and then for large loads but it makes sense to keep it ready. Likewise unless they’re snooping the traffic or have access to the server they have no way to tell how much mail it is or isn’t handling. It could have other names after all, and in fact it does! After this evening that servers IP address is also known as “email.trump1.anonymong.org” because I decided to set up that record. By the logic of our investigators that means there’s a link between the Trump organisation and us here at Anonymong. I could even cause things to look up that address and make it look oddly busy. So again either our investigators have done a lot more than they’re telling us, or they’re jumping to some very odd conclusions.
Apparently they got error messages when they tried to connect to the server, they don’t tell us what the message was or how they tried to connect, but reading between the lines I’m assuming they tried to send e-mail to it and weren’t allowed. They consider this odd, even though it’s perfectly normal for an outbound mail-relay. From the evidence they’ve presented we’re meant to assume that the Russian server is managing to send e-mail to it because it looks up the address. They have presented zero evidence that the Russian server was any more successful than them at sending it e-mail. Whilst were talking about e-mail the normal set of look ups to send e-mail to a server are first try to look up the mail exchange (MX) record, that tell you where to send the e-mail then you look up the A record of the name given to you in the MX record. So for normal mail exchange you always have two looks ups first MX then A. In the entire 5 months of data provided to us, the only thing we see are A record look ups:
10-May-2016 16:12:48 client 167.73.110.8 query: mail1.trump-email.com IN A + (66.216.133.29)
So either this isn’t sending e-mail or the super secret database is only recording A record lookups, and our bold investigators are making wild assumptions about what they think is going on. I really can’t tell you which it is. Though either the Trump organisation needs to have words with the people who host trump-email.com or the people that host the Trump servers DNS need to have words with their up-stream provider as apparently someone is handing over query level DNS logging to secret third parties. Query level logging is expensive and generally avoided, so this is deliberate and apparently quite wide-spread surveillance. Most organisations don’t log individual DNS queries outside of targeted investigations, because it has legal ramifications for discovery requests and it’s a lot of data (a not terribly busy DNS server will handle 1000’s of queries per second!). This is why I question either our investigators knowledge, or what they’re telling us they’re making far too many unfounded assumptions based on just DNS look ups.
Talking of jumping to conclusions :
“It’s pretty clear that it’s not an open mail server,” Camp told me. “These organizations are communicating in a way designed to block other people out.”
So they have no evidence of actual communication, just DNS lookups. DNS is protocol unaware so a DNS look up to send e-mail is indistinguishable from a DNS lookup to hack a server. In exactly the same way as the operator has no idea what you’ll talk about when they give you someones phone number. So on the back of unusual DNS lookups which aren’t typical of normal e-mail exchange they’re assuming that some sort of locked down communication is happening. Now I know people are generally idiots, but if you wanted to conduct covert communication via e-mail, why not do it by:
1) an encrypted secret network
2) using a domain name which wasn’t clearly associated with you
3) not using a domain name at all when you could set up IP based e-mail routing
4) using a busier server so that the traffic gets lost in the noise
Also our investigators are going from:
A server hosted by an external company on behalf of a trump organization is being looked up by a server in Russia
to:
Donald Trump is communicating secretly with the Russians!!!
This is roughly the equivalent of going from:
Someone in the White House looked up the phone number for Hooters
to:
President Obama is murdering sex workers
And these people we’re told are experts and we’re lead to believe they’ve made all the relevant evidence available – one of these has to be false (I hope). Oh and Mr Vixie I’m talking to you here as well, how do you get from a server looking up the A record of another server to evidence of any sort of two-way communication? No communication trail has been observed, a lot of calls to directory enquiries have been observed, which are odd but it’d be a stretch to even call that circumstantial evidence.
We do finally get to something approaching a factual statement:
“In the parlance that has become familiar since the Edward Snowden revelations, the DNS logs reside in the realm of metadata. We can see a trail of transmissions, but we can’t see the actual substance of the communications. And we can’t even say with complete certitude that the servers exchanged email. “
We don’t see a trail of transmissions, we see lookups coming from one side only in an odd pattern from a server which seems to belong to a Russian bank doing DNS lookups for an address associated with a Trump organisation. We don’t know what the Russian servers are so they could be DNS proxies just passing on requests from other things behind them. We have no evidence at all that the two servers were communicating in any way what-so-ever. We’re so far from “complete certitude” it’s a joke, and this passes for expertise!
““I’m seeing a preponderance of the evidence, but not a smoking gun,” he said. Richard Clayton, a cybersecurity researcher at Cambridge University”
If an odd sequence of A record look ups amounts to seeing a preponderance of evidence then some cybersecurity researchers should have gone to specsavers. The Trump server is running some sort of mail server is true here’s what you see when you connect to it:
Connected to 66.216.133.29.
Escape character is '^]'.
521 lvpmta14.lstrk.net does not accept mail from you (#.#.#.#)
Connection closed by foreign host.
Which is exactly what you’d expect from an outbound mail server configured for sending out bulk mail, and as observed by another cynic is in the middle of a load of other bulk mail servers, and is responding with the name of the bulk mail sending company. Perhaps when our various experts checked it responded with:
521 Mrtrumps-private-email-server does not accept mail from you
But as they haven’t seen fit to share that bit of their basic trouble shooting with us who knows?
I’m going to stop now as this is driving me to despair. There’s no evidence of anything much here. Except that security researchers with apparent access to a huge amount of DNS lookup data have jumped to all sorts of wild conclusions based on no evidence (assuming they’ve actually shared their evidence), and in publishing it have abused their ability to spy on all of us and quite frankly have discredited the cybersecurity industry in the process by failing to do even the most basic of investigation. Or at least failing to present the same trouble shooting any of us could have done, we’re meant to just trust that they did it. The “suspicious” change of server name isn’t that odd for a hosting company as IP addresses get reused and reconfigured we have no idea what’s physically behind it – and people are really bad at cleaning up old data until they need to do something else with it and even then they often do a half arsed job of it.
Really as ErrataSec says the story here isn’t that a Russian server was making DNS lookups of a server name belonging to the Trump organization, it’s that:
1) Large amounts of DNS lookups are being recorded and preserved by an unknown group of private individuals allowing them to draw fallacious conclusions about large numbers of people
2) These secret researchers seem to be incapable of fairly rudimentary data gathering and analysis
3) They’re quite happy to abuse their access and publish the data when it suits them
4) ISPs are looking this data and passing it over to this private organisation as well as who knows who else.
But hey who actually cares about mass data surveillance when it can be used to put Donald Trump in a bad light.
Update: the slate article is so bad even the Gruniad has debunked it and the intercept also takes it to pieces. Sadly both articles ignore the abuse of access of these alleged white-hats and the implications it has about mass data gathering and internet monitoring.
