Mastodon

On the Slate story about Trump's private email server

Posted on Nov 1, 2016

Note in 2019: So this post was originally on Medium, but eventually ended up in the following places:

Of those, I gave interviews to Washington Post and Bloomberg, and had a email exchange with Mic.

I still stand by everything here, despite believing Trump is a crook, almost certainly compromised by his business dealings with authoritarians and is ultimately a fascist…

Re: this article…

The Evidence

For more detail, please check out Krypt3ia’s look into things.

The evidence gathered by the anonymous group are BIND logs for ns1.cdcservices.com, ns2.cdcservices.com, ns3.cdcservices.com, owned by a hotel marketing agency (Cendyn.com), regarding a domain called trump-email.com which was also registered by them. It’s unclear whether or not these are complete or even how they were obtained. There is a higher proportion of lookups received from Alfa Bank’s servers than anything else, and occasionally a healthcare provider, Spectrum Health shows up as well.

These lookups are in the small 10s to small 100s per day.

When the NYT approached the Trump campaign, the domain was migrated to trump1.contact-client.com. Servers from Alfa Bank immediately started pinging it soon afterwards.

The anonymous group consider these suspicious because of their relative frequency, and the fact that the Alfa Bank server immediately looked up the new server.

The rebuttals

Alfa Bank has responded after an investigation carried out by Mandiant, who did the big expose on China’s APT1 who were targeting the New York Times. They say that Mandiant believe that the requests were triggered by a spam attack from the marketing servers which triggered security software.

The FBI has also said that they find the evidence weak.

Dyn said they’ve seen more requests come in from other IP addresses.

Errata Security also has a debunk.

And Krypt3ia did not weaken his scepticism as the Slate suggested he had.

The DNS angle

DNS is in a category of tools called service discovery. It’s most frequent use is the conversion of names like ‘medium.com’ to IP addresses, like 104.16.124.127. Though that’s probably common knowledge after the Dyn attack the other week.

Two different DNS resolution modes

A DNS client, on your phone or computer will connect to a group of nameservers that provide results on lookups. Normally, they’ll be provided by an ISP or you might be using Google DNS - 8.8.8.8 or OpenDNS or whatever.

If you’re in a corporate setting, you’ve probably got Microsoft Active Directory configured to do your DNS name resolution, or BIND.

The wrong corporate DNS setup - recursive lookup

These can be configured in a number of ways. By default, they will operate recursively.

That is, the DNS server has knowledge of a number of root servers, and it will query these for the nameservers which manage .com - all of it. It does this by issuing an NS lookup, which lists the authoritative nameservers for that entry. This will be by name, so it will have to issue an A lookup request to get the IP address for the nameserver.

Having got that information, it will then connect to .com, and ask for the nameservers that manage medium.com, followed by the A lookup against those servers (these might be on another domain causing another recursive set of lookups). It will then connect to the nameservers for medium.com and ask for the A record for www.medium.com.

In addition, all the results have a TTL - a time to live, which says for how long in seconds those results are valid for. A smart server can keep that information in memory so it doesn’t have to keep looking it up every time.

The correct corporate DNS setup - forwarding nameservers

If the company in question has set up their internal DNS correctly, then instead of recursively looking up DNS records, they will just forward their requests directly to another nameserver, and let them do the work. People should definitely do this as you leave yourself open to man in the middle attacks by using recursive DNS. It’s so important, that Microsoft actually warn you about this when setting up modern Windows infrastructure.

At home, your router will be in forwarding mode by default as their DNS servers are usually not complex enough to do recursion.

Another benefit of forwarding nameservers is the servers you forward to can do a lot of smart things - particularly if they’re fielding requests from millions of customers, like ISPs or Google. Most likely, they’ll have very large caches, which mean they don’t have to do lookups very often. They’re less likely to be configured badly and showing incorrect behaviour.

Why this matters

Alfa Bank’s DNS infrastructure, at least those bits related to email were setup in recursive mode, which is why it shows up in the logs. This is pretty bad for a bank. If they were using a DNS forwarder, they wouldn’t show up at all in the logs Can DNS be used for secret transmission?

Yes, the most common method being tunnelling using TXT records (which allow bigger chunks of data) with short TTLs. But that’s not what we’re seeing here. The lookups are happening in the order of <40 per day and they’re A record lookups, nothing more fancy.

Transactional Email Services & Email Validation

The servers fingered in the reports are owned by Cendyn, a hotel marketing platform. These email servers fall into the category of “transactional email services” - they’re designed for high volume sending, don’t receive email back (hence all those do-not-reply@blahblah.com spams you get). and do one thing and do it well.

They have to be very well secured, because if an attacker gets access, you’ll potentially be blacklisted by everyone, and then your customers wont be able to spam their customers with the latest pumpkin spiced latte mailshot.

However, they do need to do one thing - they need to verify their existence if asked because of a number of (actual, not just marketing) anti-spam countermeasures employed by incoming mail services. A lot of people mentioned SPF records. These are TXT records in DNS that say which servers are authoritative for sending emails. The Cendyn servers are configured correctly with regards to this.

Another simple method employed by email systems, is to do a reverse DNS lookup. When one mail server attempts to deliver email to another server, it will send a “EHLO” with its server identifier. The receiver may do a reverse DNS lookup to make sure that the sender is who they say they are.

Here’s me pretending to be trump-email.com to one of Alfa Bank’s mail servers:

alt text

Access Denied

Oddly, their other email server lets me continue:

alt text

Sorry Alfa. It’s not really hacking, is it though?

Sorry Alfa. It’s not really hacking, is it though?I’m going to guess they’ve got two different systems as a form of redundancy. That can be a valid decision security wise. It’s odd that they’re configured differently though.

Standardising infrastructure changes

Allegedly, when the NYT contacted the Trump campaign, trump-email.com was shut off and trump1.contact-client.com was set up in it’s place. If I were running the infrastructure at Cendyn, I might want to move off trump-email.com because it’s yet another domain name I have to care about. It’s much easier if I can get a customer on to a subdomain of something I always have to look after. It could be Cendyn offered to move the domain onto more standardised infrastructure when they were queried by the Trump campaign. That could be as much as simplifying the DNS setup. Ask anyone who’s had to manage a load of domain names if they haven’t forgot to renew one.

Email systems are terrible

One of the main reasons I discounted malfeasance is that email systems are terrible.

As an example, in a previous job, we kept seeing a dodgy request to download a font from our website every 2 seconds. We know that we used the font in our own email marketing (using Mailkimp, for what it’s worth). We also knew from the logs that it came from a Windows XP machine, that it was the Trident engine trying to do this (i.e. Internet Explorer engine), so that meant it was probably Outlook. Even more weirdly, it seemed to come from a call centre 2 blocks away when we did a reverse lookup. We worked around the problem by getting our web server to just say “yeah, fine, whatever” to these requests. They did eventually stop - presumably when the machine got totally infected. We never found the root cause, but the suspects included:

The fact is email is terrible, and the fact that it still works is because of decades of workarounds and hacks to make it marginally secure. And they go wrong, all the time. Add that to the fact that Alfa’s DNS systems were almost certainly not configured according to best practice, then it seems to indicate system malfunction as the most plausible explanation.

Alfa’s infrastructure

I managed to find out from sales brochures that Alfa, at least in 2009 were using the following email scanners:

Their product manual allows you configure the reverse DNS lookups as follows:

alt text

Who knows what anti-zombie protection is, but it sounds cool

From this sales presentation, it also appears they use Kaspersky Antivirus for endpoint and server protection, which is perfectly decent if they kept it up to date.

The DNC and Cendyn!!!!111oneoneone

When I was searching for Cendyn’s SMTP servers, I came across this from Wikileaks. If you look at the source, then you can see a marketing email coming from a hotel into the inbox of Scott Cormer:

alt text

I wish I could use GIMP

They’re using Appriver SecureTide as their email scanner. It clearly looks up the IP address of the sender, and does an additional reverse lookup and adds it as headers to the email. This should clarify that this is standard practice, and that the activity from Alfa shouldn’t immediately be understood as malfeasance.

Conclusions from the garden of forking paths

I think one of the big issues here is the plethora of data that can be sifted through to find a correlation, and it’s here that we find ourselves in what Andrew Gelman calls the garden of forking paths. It’s something that’s common when you start off without a clear hypothetical model. What can happen when presented with a large data set is that you’re free to cut up and compare the data in so many different ways that a spurious correlation is almost certain to appear. In statistics, it’s known as p-hacking.

I don’t per se have a problem against starting off without a hypothesis - exploration is fine - in many cases you’re not going to have a choice - but it needs to be done with caution.

Share Tweet