Biden administration doubles down on its previous warnings of possible Russian cyberattacks in the wake of hefty economic sanctions imposed on Russia by the US.
For the Beijing 2022 Winter Olympics, the Chinese government requires everyone to download an app onto their phone. It has many security/privacy concerns, as CitizenLab documents. However, another researcher goes further, claiming his analysis proves the app is recording all audio all the time. His analysis is fraudulent. He shows a lot of technical content that looks plausible, but nowhere does he show anything that substantiates his claims.
Average techies may not be able to see this. It all looks technical. Therefore, I thought I'd describe one example of the problems with this data -- something the average techie can recognize.
His "evidence" consists screenshots from reverse-engineering tools, with red arrows pointing to the suspicious bits. An example of one of these screenshots is this on:
This screenshot is that of a reverse-engineering tool (Hopper, I think) that takes code and "disassembles" it. When you dump something into a reverse-engineering tool, it'll make a few assumptions about what it sees. These assumptions are usually wrong. There's a process where the human user looks at the analyzed output, does a "sniff-test" on whether it looks reasonable, and works with the tool until it gets the assumptions correct.
That's the red flag above: the researcher has dumped the results of a reverse-engineering tool without recognizing that something is wrong in the analysis.
It fails the sniff test. Different researchers will notice different things first. Famed google researcher Tavis Ormandy points out one flaw. In this post, I describe what jumps out first to me. That would be the 'imul' (multiplication) instruction shown in the blowup below:
It's obviously ASCII. In other words, it's a series of bytes. The tool has tried to interpret these bytes as Intel x86 instructions (like 'and', 'insd', 'das', 'imul', etc.). But it's obviously not Intel x86, because those instructions make no sense.
That 'imul' instruction is multiplying something by the (hex) number 0x6b657479. That doesn't look like a number -- it looks like four lower-case ASCII letters. ASCII lower-case letters are in the range 0x61 through 0x7A, so it's not the single 4-byte number 0x6b657479 but the 4 individual bytes 6b 65 74 79, which map to the ASCII letters 'k', 'e', 't', 'y' (actually, because "little-endian", reverse order, so "ytek").
No, no. Techies aren't expected to be able to read hex this way. Instead, we are expected to recognize what's going on. I just used a random website to interpret hex bytes as ASCII.
There are 26 lower case letters, roughly 10% of the 256 possible values for a byte. Thus, the chance that a random 4 byte number will consist of all lower-case letters is 1-in-10,000. Moreover, multiplication by strange constants happens even more rarely. You'll commonly see multiplications by small numbers like 48, or large well-formed numbers like 0x1000000. You pretty much never see multiplication by a number like 0x6b657479, baring something rare like an LCG.
QED: this isn't actually an Intel x86 imul instruction, it's ASCII text that the tool has tried to interpret as x86 instructions.
Conclusion
At first glance, all those screenshots by the researcher look very technical, which many will assume supports his claims. But when we actually look at them, none of them support his claims. Instead, it's all just handwaving nonsense. It's clear the researchers doesn't understand them, either.
The reason you don't really understand NFTs is because the journalists describing them to you don't understand them, either. We can see that when they attempt to sell an NFT as part of their stories (e.g. AP and NYTimes). They get important details wrong.
The latest is Reason.com magazine selling an NFT. As libertarians, you'd think at least they'd get the technical details right. But they didn't. Instead of selling an NFT of the artwork, it's just an NFT of a URL. The URL points to OpenSea, which is known to remove artwork from its site (such as in response to DMCA takedown requests).
If you buy that Reason.com NFT, what you'll actually get is a token pointing to:
This is just the metadata, which in turn contains a link to the claimed artwork:
If either OpenSea or Google removes the linked content, then any connection between the NFT and the artwork disappears.
It doesn't have to be this way. The correct way to do NFT artwork is to point to a "hash" instead which uniquely identifies the work regardless of where it's located. That $69 million Beeple piece was done this correct way. It's completely decentralized. If the entire Internet disappeared except for the Ethereum blockchain, that Beeple NFT would still work.
This is an analogy for the entire blockchain, cryptocurrency, and Dapp ecosystem: the hype you hear ignores technical details. They promise an entirely decentralized economy controlled by math and code, rather than any human entities. In practice, almost everything cheats, being tied to humans controlling things. In this case, the "Reason.com NFT artwork" is under control of OpenSea and not the "owner" of the token.
Journalists have a problem. NFTs selling for millions of dollars are newsworthy, and it's the journalists place to report news rather than making judgements, like whether or not it's a scam. But at the same time, journalists are trying to explain things they don't understand. Instead of standing outside the story, simply quoting sources, they insert themselves into the story, becoming advocates rather than reporters. They can no longer be trusted as an objective observers.
From a fraud perspective, it may not matter that the Reason.com NFT points to a URL instead of the promised artwork. The entire point of the blockchain is caveat emptor in action. Rules are supposed to be governed by code rather than companies, government, or the courts. There is no undoing of a transaction even if courts were to order it, because it's math.
But from a journalistic point of view, this is important. They failed at an honest description of what actually the NFT contains. They've involved themselves in the story, creating a conflict of interest. It's now hard for them to point out NFT scams when they themselves have participated in something that, from a certain point of view, could be viewed as a scam.
Tina Peters, the election clerk in Mesa County (Colorado) went rogue and dumped disk images of an election computer on the Internet. They are available on the Internet via BitTorrent [Mesa1][Mesa2], The Colorado Secretary of State is now suing her over the incident.
The lawsuit describes the facts of the case, how she entered the building with an accomplice on Sunday, May 23, 2021. I thought I'd do some forensics on the image to get more details.
Specifically, I see from the Mesa1 image that she logged on at 4:24pm and was done acquiring the image by 4:30pm, in and (presumably) out in under 7 minutes.
In this blogpost, I go into more detail about how to get that information.
The image
To download the Mesa1 image, you need a program that can access BitTorrent, such as the Brave web browser or a BitTorrent client like qBittorrent. Either click on the "magnet" link or copy/paste into the program you'll use to download. It takes a minute to gather all the "metadata" associated with the link, but it'll soon start the download:
What you get is file named EMSSERVER.E01. This is a container file that contains both the raw disk image as well as some forensics metadata, like the date it was collected, the forensics investigator, and so on. This container is in the well-known "EnCase Expert Witness" format. EnCase is a commercial product, but its container format is a quasi-standard in the industry.
Some freeware utilities you can use to open this container and view the disk include "FTK Imager", "Autopsy", and on the Linux command line, "ewf-tools".
However you access the E01 file, what you most want to look at is the Windows operating-system logs. These are located in the directory C:\Windows\system32\winevtx. The standard Windows "Event Viewer" application can load these log files to help you view them.
When inserting a USB drive to create the disk image, these event files will be updated and written to that disk before the image was taken. Thus, we can see in the event files all the events that happen right before the disk image happens.
Disk image acquisition
Here's what the event logs on the Mesa1 image tells us about the acquisition of the disk image itself.
The person taking the disk image logged in at 4:24:16pm, directly to the console (not remotely), on their second attempt after first typing an incorrect password. The account used was "emsadmin". Their NTLM password hash is 9e4ec70af42436e5f0abf0a99e908b7a. This is a "role-based" account rather than an individual's account, but I think Tina Peters is the person responsible for the "emsadmin" roll.
The program "Access Data FTK Imager 4.2.0.13" was run from the USB drive (F:\FTK Imager\FTK Imager.exe) in order to image the system. The image was taken around 4:30pm, local Mountain Time (10:30pm GMT).
It's impossible to say from this image what happened after it was taken. Presumably, they immediately hit "eject" on the drive, logged off, and disconnected the hard drive. Thus from beginning to end, it took about 7 minutes to take the image once they sat down at the computer.
Dominion Voting Systems
The disk image is that of a an "EMS Server" part of the Dominion Voting Suite. This is a server on an air-gapped network (not connected to any other network) within the count offices.
Most manuals for Colorado are online, though some bits and pieces are missing, and can be found in documents posted to other state's websites (though each state does things a little different, so such cross referencing can't be completely trusted).
The locked room with an air-gapped network you see in the Mesa County office appears to look like the following, an "EMS Standard" configuration (EMS stands for Election Management System).
This small network is "air gapped", meaning there is no connection from this network to any other network in the building, nor out to the Internet. By looking at the logs from the Mesa1 image, we can see what this network looks like:
- The EMS Server is named "EMSERVER" with IP address 192.168.100.10 and MAC address 44-A8-42-30-01-5D. The hard drive matches Dominion's specs: a 1-terabyte boot drive (C:) and a 2-terabyte data drive (D:) that is shared with the rest of the network as \\EMSERVER\NAS. This also acts as the network's DHCP and DNS server.
- At least one network printer, model Dell E310dw.
- Two EMS Workstations (EMSCLIENT01 and EMSCLIENT02). This is where users spend most of their time, before an election to create the ballots, and after all the ballots have been counted to construct the final tally.
- Four ImageCast Central (ICC) (ICC01 - ICC04) scanners, for automatically scanning and tabulating ballots.
- Two Adjudication Workstations (ADJCLIENT01 and ADJCLIENT03). These are used when the scanners reject ballots, such as when somebody does a write-in candidate, or marks two candidates. Humans need to get involved to make the final judgement on what the ballot actually says.
Note this isn't the machines you'd expect to see in a precinct when you vote (which would be "ballot marking devices" predominantly). These are the machines in the back office that count the votes and store the official results.
Conclusion
What we see here is that "system logs" can tell us a lot of interesting things about the system. There's good reason to retain them in the future.
On the other hand, they generally can't answer the most important question: whether the system was hacked and votes flipped.
Mike Lindell claims to have "Absolute Proof" that Chinese hackers flipped votes throughout the country, including Maricopa County. If so, this would've been the system that the Chinese hackers would've hacked. Yet, in the system image, there is no evidence of this. By this, I mean the Mesa1 image, the one from before the system logs were deleted (obviously, there would be nothing in the Mesa2 image).
This lack of hacking evidence in the logs isn't proof that it didn't happen, though. The fact is, the logs aren't comprehensive enough to record most hacks, and the hackers could've deleted the logs anyway. That's why system logs aren't considered "election records" and that laws don't mandate keeping them: they could have some utility, as I've shown above, but they really wouldn't show the things that we most want to know.
The Alfa-Trump conspiracy-theory has gotten a new life. Among the new things is a report done by Democrat operative Daniel Jones [*]. In this blogpost, I debunk that report.
If you'll recall, the conspiracy-theory comes from anomalous DNS traffic captured by cybersecurity researchers. In the summer of 2016, while Trump was denying involvement with Russian banks, the Alfa Bank in Russia was doing lookups on the name "mail1.trump-email.com". During this time, additional lookups were also coming from two other organizations with suspicious ties to Trump, Spectrum Health and Heartland Payments.
This is certainly suspicious, but people have taken it further. They have crafted a conspiracy-theory to explain the anomaly, namely that these organizations were secretly connecting to a Trump server.
We know this explanation to be false. There is no Trump server, no real server at all, and no connections. Instead, the name was created and controlled by Cendyn. The server the name points to for transmitting bulk email and isn't really configured to accept connections. It's built for outgoing spam, not incoming connections. The Trump Org had no control over the name or the server. As Cendyn explains, the contract with the Trump Org ended in March 2016, after which they re-used the IP address for other marketing programs, but since they hadn't changed the DNS settings, this caused lookups of the DNS name.
This still doesn't answer why Alfa, Spectrum, Heartland, and nobody else were doing the lookups. That's still a question. But the answer isn't secret connections to a Trump server. The evidence is pretty solid on that point.
Daniel Jones and Democracy Integrity Project
The report is from Daniel Jones and his Democracy Integrity Project.
It's at this point that things get squirrely. All sorts of right-wing sites claim he's a front for George Soros, funds Fusion GPS, and involved in the Steele Dossier. That's right-wing conspiracy theory nonsense.
But at the same time, he's clearly not an independent and objective analyst. He was hired to further the interests of Democrats.
If the data and analysis held up, then partisan ties wouldn't matter. But they don't hold up. Jones is clearly trying to be deceptive.
The deception starts by repeatedly referring to the "Trump server". There is no Trump server. There is a Listrak server operated on behalf of Cendyn. Whether the Trump Org had any control over the name or the server is a key question the report should be trying to prove, not a premise. The report clearly understands this fact, so it can't be considered a mere mistake, but a deliberate deception.
People make assumptions that a domain name like "trump-email.com" would be controlled by the Trump organization. It's wasn't. When Trump Hotels hired Cendyn to do marketing for them, Cendyn did what they normally do in such cases, register a domain with their client's name for the sending of bulk emails. They did the same thing with hyatt-email.com, denihan-email.com, mjh-email.com, and so on. What clear is that the Trump organization had no control, no direct ties to this domain until after the conspiracy-theory hit the press.
Finding #1 - Alfa Bank, Spectrum Health, and Heartland account for nearly all of the DNS lookups for mail1.trump-email.com in the May-September timeframe.
Yup, that's weird and unexplained.
But it concludes from this that there were connections, saying the following:
In the DNS environment, if "computer X" does a DNS look-up of "Computer Y," it means that "Computer X" is trying to connect to "Computer Y".
This is false. That's certainly the assumption we usually make, that it's probably true in most cases. But it's not something we insist upon if there's reason to doubt it. And since there's reason to doubt it here, we would need more evidence to make that conclusion.
For example, before the contract was canceled in March 2016, there were DNS lookups for the "mail1.trump-email.com" name from all over the place. That's because the Listrak server was pumping out bulk emails ("spam") promoting Trump Hotels. Servers receiving the emails would often check the identity of the server through DNS lookups, but without any attempt to connect. This fact is footnoted in the Jones report even as it claims otherwise in the main text.
Obviously, that's no longer the case after March 2016, when the contract was canceled. But if Cendyn repurposes the server for something else, such lookups can still happen without connections. The DNS records hadn't changed. So if the server sends out new things from that IP address, unrelated to Trump Org, it'd still cause DNS lookups for the "trump-email.com" domain to happen. It wouldn't mean anybody was trying to connect to the server.
This is indeed what Cendyn claims, that they repurposed the resources for their hotel meetings app (whereby hotels can schedule conferences and things on their premises).
It's still suspicious that only those three organizations were involved, but at the same time, it's clearly false to assume this is evidence of connections.
Finding #2 - Comparison with denihan-email.com.
The Jones report compared the DNS logs of trump-email.com with the domain of another of Cendyn's client, Denihan. Cendyn registered the domain denihan-email.com. This is another hotel company.
This comparison was obviously bogus. The contract with Cendyn ended in March 2016, after which Cendyn claims it repurposed the server. Jones uses the timeframe August 2016 through September 2016 to compare traffic for those two domains. Of course they'd be different. A valid comparison would be a t timeframe before March 2016, when both were clients of Cendyn.
Since Jones documents the fact the contract between Cendyn and Trump Org was ended, they are knowingly comparing an apple to an orange. Thus, it's not a mistake but a deception.
This also points to the fundamental problem with the data-set. We don't really have a full picture of what happened, such as data going back to 2015. We have a carefully curated subset of the data designed to show just what they want us to see.
Everything points to trump-email.com domain and Listrak servers being just normal Cendyn stuff used for Cendyn's purposes. As far as we can tell, that domain worked the same as other Cendyn clients, such as denihan-email.com, hyatt-email.com, mjh-email.com, and so on. These domains are controlled by Cendyn, not their client's. Cendyn in turn points those names at Listrak servers for sending bulk email.
Finding #3 - Missing SPF record
The Jone's report points to missing SPF records, showing that the server is not configured correctly for sending mass emails. It includes this exhibit.
The SPF and DMARC standards were not as widely used in 2016, so misconfigurations were common. Moreover, the domains also lacked a DMARC record. Without DMARC, despite SPF being bad, many receivers won't reject the emails.
Listrak/Cendyn still fail to have proper DMARC records for their clients, which means that some of their bulk email is getting rejected. They should probably fix that. This doesn't mean Listrak/Cendyn aren't in the bulk email business, only that they could be better at it.
Thus, we've shown that trump-email.com had the perfectly normal Cendyn SPF records. Far from proving this isn't a bulk email server, the consistency with Cendyn's normal configuration proves unequivocally that it is.
Finding #4 - Accepts emails only from specific senders
The Jones report shows that the server in question (66.216.133.29) accepts incoming email, but rejects email from the public, accepting email only from specific senders. They assume the specific senders would be those from Alfa Bank, Spectrum, and Heartland.
Again, they don't compare properly to other Cendyn/Listrak systems. If they had, they'd have found that they all are configured the same way. There's an entire subnet of servers you can test this way:
All these servers show the same messages, allowing incoming email connections but not incoming email messages.
This is a vestigial configuration common to bulk email senders. Spammers only send email. One way to test if somebody is spammer is to connect back. This configuration makes it appear they'll accept email even if they won't, passing the test.
In no way is this evidence of secret communications. It's not evidence of their claim that somehow Alfa Bank, Spectrum Health, and Heartland would be on the list of allowed senders. We would need additional evidence to make that claim, not an assumption.
Finding #5 - Evidence of human interaction and coordination
The report claims a direct link between Alfa and Trump with the following:
On September 23, 2016, two days after The New York Times approached Alfa Bank, the Trump Organization deleted the email server "mail1.trump-email.com" ... it would have been a deliberate human action taken by a someone working on behalf of the Trump Organization and not by Alfa Bank. An analyst, quoted in the Slate article by Franklin Foer, observed that "the knee was struck in Moscow, and the leg kicked in New York."
This 'finding' is an excellent demonstration of how to identify conspiracy-theories: anomalies that cannot otherwise be explained become proof of the conspiracy. After all, the conspiracy-theory can explain everything.
When I debunked the Alfa-Trump thing back in 2017, reporters grilled me on this specific point. They demanded I come up with an explanation for this coincidence. I told them I had none, but just because I didn't have one, it didn't mean it was proof of the conspiracy theory. There could be lots of explanations, just because we don't know them doesn't mean they don't exist. Just because the conspiracy-theory explains it doesn't mean this is evidence for the conspiracy.
But now we do have another explanation: the FBI called Cendyn on the morning of September 23 and asked them about the domain. As the agent reported back:
“Followed up this morning with Central Dynamics [Cendyn] who confirmed that the mail1.trump-email.com domain is an old domain that was set up in approximately 2009 when they were doing business with the Trump Organization that was never used." -- *
Thus, it's not NYT contacting Alfa Bank that caused the deletion, it's the FBI calling Cendyn. Thus, there's no evidence Alfa Bank or Trump Org were even involved. The evidence is quite clear that only Cendyn was involved.
After Cendyn deletes the domain "mail1.trump-email.com", lookups of that name started to fail. The Jones report notes that Alfa Bank then switched to "trump1.contact-client.com". It weaves this in to the conspiracy thusly:
The fact that Alfa Bank was the first entity (IP address) to conduct a DNS look-up for "trump1.contact-client.com" in the data-set could indicate that someone at Alfa Bank was in some manner made aware of the new Trump Organization server name.
The name "contact-client.com" is part of Cendyn's infrastructure. For their "mail1.customer-email.com" domains, there's a matching "customer1.contact-client.com" domain. We can see test that live right now:
This is totally consistent with Cendyn's re-use of the infrastructure for a new purpose, as it would treat both domain names the same. Rather than evidence suggesting human interaction, it's evidence suggesting the opposite, that there was no human interaction.
6. The Mandiant report doesn't refuted these findings
After this thing hit the news, Alfa Bank hired Mandiant to come to their offices and investigate. Their report was inconclusive. They didn't find anything.
Note the difference in language. Things Mandiant can't explain demonstrates Mandiant's incompetence, while things Jones can't explain prove the conspiracy-theory. If Mandiant's report should be treated as inclusive and proof of nothing, then so too should the Jones report. The Jones report has even less evidence than the Mandiant report.
7. The public statements by Trump et al. are contradictory and incomplete
Duh.
The Trump Org, Alfa, and Spectrum Health have no idea what happened. Their statements are consistent with knowing they don't have secret communications, but not knowing where this DNS data came from. They are unable to refute the allegations, but at the same time, are concerned for their reputations, and behave accordingly. Which, of course, means the guess at what's going on with more confidence than is warranted.
If there were secret communications among them, you'd expect they'd do a better job at coordinating their stories.
Conclusion
In this blogpost, I've refuted all the findings of the Jones report. There is still the question where this DNS anomaly came from, but the allegation that this proves a secret connect between Alfa Bank and a Trump server is clearly false.
Moreover, I've shown that the Jones report is not merely wrong, but deliberately deceptive. They repeatedly reference a "Trump Organization Server" even though it's quite clear from the text they know that no such server exists.
For example, when Cendyn removed the "mail1.trump-email.com" DNS record, it was described as the "Trump Organization deleted the email server". It's clear they know that Cendyn simply removed the mail1.trump-email.com record, and that the Listrak server wasn't touched. Yet, they deliberate phrase things this way in order to deceive.
What we have is Alfa Bank doing DNS queries. What we don't have is any connection to the Trump Org. Since Jones couldn't create the conclusion based on evidence that Trump Org was involve, he instead made it the premise.
This in turn makes it easy to disprove the entire Jones report: since there's not only no evidence of Trump Org involvement, and quite a lot of evidence Trump Org had no control over the domain or servers, it disprove the entire theory that there was secret connections with Alfa Bank.
One of the most important classic sci-fi stories is the book "Dune" from Frank Herbert. It was recently made into a movie. I thought I'd write a quick review.
The summary is this: just read the book. It's a classic for a good reason, and you'll be missing a lot by not reading it.
But the movie Dune (2021) movie is very good. The most important thing to know is see it in IMAX. IMAX is this huge screen technology that partly wraps around the viewer, and accompanied by huge speakers that overwhelm you with sound. If you watch it in some other format, what was visually stunning becomes merely very pretty.
This is Villeneuve's trademark, which you can see in his other works, like his sequel to Bladerunner. The purpose is to marvel at the visuals in every scene. The story telling is just enough to hold the visuals together. I mean, he also seems to do a good job with the story telling, but it's just not the reason to go see the movie. (I can't tell -- I've read the book, so see the story differently than those of you who haven't).
Beyond the story and visuals, many of the actor's performances were phenomenal. Javier Bardem's "Stilgar" character steals his scenes. Stellan Skarsgård exudes evil. The two character actors playing the mentats were each perfect. I found the lead character (Timothée Chalamet) a bit annoying, but simply because he is at this point in the story.
Villeneuve's splits the book into two parts. This movie is only the first part. This presents a problem, because up until this point, the main character is just responding to events, not the hero who yet drives the events. It doesn't fit into the traditional Hollywood accounting model. I really want to see the second film even if the first part, released in the post-pandemic turmoil of the movie industry, doesn't perform well at the box office.
In short, if you haven't read the books, I'm not sure how well you'll follow the storytelling. But the visuals (seen at IMAX scale) and the characters are so great that I'm pretty sure most people will enjoy the movie. And go see it on IMAX in order to get the second movie made!!
Tina Peters, the elections clerk from Mesa County (Colorado) went rogue, creating a "disk-image" of the election server, and posting that image to the public Internet. Conspiracy theorists have been analyzing the disk-image trying to find anomalies supporting their conspiracy-theories. A recent example is this "forensics" report. In this blogpost, I debunk that report.
I suppose calling somebody a "conspiracy theorist" is insulting, but there's three objective ways we can identify them as such.
The first is when they use the logic "everything we can't explain is proof of the conspiracy". In other words, since there's no other rational explanation, the only remaining explanation is the conspiracy-theory. But there can be other possible explanations -- just ones unknown to the person because they aren't smart enough to understand them. We see that here: the person writing this report doesn't understand some basic concepts, like "airgapped" networks.
This leads to the second way to recognize a conspiracy-theory, when it demands this one thing that'll clear things up. Here, it's demanding that a manual audit/recount of Mesa County be performed. But it won't satisfy them. The Maricopa audit in neighboring Colorado, whose recount found no fraud, didn't clear anything up -- it just found more anomalies demanding more explanation. It's like Obama's birth certificate. The reason he ignored demands to show it was that first, there was no serious question (even if born in Kenya, he'd still be a natural born citizen -- just like how Cruz was born in Canada and McCain in Panama), and second, showing the birth certificate wouldn't change anything at all, as they'd just claim it was fake. There is no possibility of showing a birth certificate that can be proven isn't fake.
The third way to objectively identify a conspiracy theory is when they repeat objectively crazy things. In this case, they keep demanding that the 2020 election be "decertified". That's not a thing. There is no regulation or law where that can happen. The most you can hope for is to use this information to prosecute the fraudster, prosecute the elections clerk who didn't follow procedure, or convince legislators to change the rules for the next election. But there's just no way to change the results of the last election even if wide spread fraud is now proven.
The document makes 6 individual claims. Let's debunk them one-by-one.
#1 Data Integrity Violation
The report tracks some logs on how some votes were counted. It concludes:
If the reasons behind these findings cannot be adequately explained, then the county's election results are indeterminate and must be decertified.
This neatly demonstrates two conditions I cited above. The analyst can't explain the anomaly not because something bad happened, but because they don't understand how Dominion's voting software works. This demand for an explanation is a common attribute of conspiracy theories -- the ignorant keep finding things they don't understand and demand somebody else explain them.
Secondly, there's the claim that the election results must be "decertified". It's something that Trump and his supporters believe is a thing, that somehow the courts will overturn the past election and reinstate Trump. This isn't a rational claim. It's not how the courts or the law works or the Constitution works.
#2 Intentional purging of Log Files
This is the issue that convinced Tina Peters to go rogue, that the normal Dominion software update gets rid of all the old system-log files. She leaked two disk-images, before and after the update, to show the disappearance of system-logs. She believes this violates the law demanding the "election records" be preserved. She claims because of this, the election can't be audited.
Again, we are in crazy territory where they claim things that aren't true. System-logs aren't considered election records by any law or regulation. Moreover, they can't be used to "audit" an election.
Currently, no state/county anywhere treats system-logs as election records (since they can't be used for "audits"). Maybe this should be different. Maybe you can create a lawsuit where a judge rules that in future elections they must be treated as election records. Maybe you can convince legislatures to pass laws saying system-logs must be preserved. It's not crazy to say this should be different in the future, it's just crazy to say that past system-logs were covered under the rules.
And if you did change the rules, the way to preserve them wouldn't be to let them sit on the C: boot-drive until they eventually rot and disappear (which will eventually happen no matter what). Instead, the process to preserve them would be to copy them elsewhere. The way Dominion works is that all election records that need to be preserved are copied over to the D: data drive.
Which means, by the way, that this entire forensics report is bogus. The Mesa disk image was only of the C: boot-drive, not of the D: data drive. Thus, it's unable to say which records/logs were preserved or not. Everyone knows that system-logs probably weren't, because they aren't auditable election records, so you can still make the claim "system-logs weren't preserved". It's just that you couldn't make that claim based on a forensics of the C: boot-drive. Again, we are in crazy statements territory that identify something as a conspiracy-theory, weird claims about how reality works.
System-logs cannot be used to audit the vote. That's confusing the word "audit" with "forensics". The word "audit" implies you are looking for a definitive result, like whether the vote count was correct, or whether all procedures were followed. Forensics of system-logs can't tell you that. Instead, they can only lead to indeterminate results.
That's what you see here. This "forensics" report cannot make any definitive statement based upon the logs. It can find plenty of anomalies, meaning things the forensics investigator can't understand. But none of that is positive proof of anything. If a hacker had flipped votes on this system, it's unlikely we would have seen evidence in the log.
#3 Evidence of network connection
The report claims the computer was connected to a network. Of course this is true -- it's not a problem. The network was the one shown in the diagram below:
Specifically, this Mesa image was of the machine labeled "EMS Server" in the above diagram. From my forensics of the network logs, I can see that there are other computers on this network:
- Four ICC workstations (named ICC01 through ICC04)
- Two Adjudication Workstations (named ADJCLIENT01 and ADJCLINET03, I don't know what happened to number 2).
- Two EMS Workstations (named EMSCLIENT01 and EMSCLIENT02).
- A printer, model Dell E310dw.
The word "airgapped" doesn't mean the EMS Server is airgapped from any network, but that this entire little network is airgapped from anything else. The security of this network is physical security, the fact that nobody can enter the room who isn't authorized.
I did my own forensics on the Mesa image and could find none of the normal signs that the server accessed the Internet, and pretty good evidence that most of the time, it was unconnected (it gets mad when it can't find the Internet and produces logs stating this). This doesn't mean I proved conclusively no Internet connection was ever made. It's possible that somebody will find some new thing in that image that shows an Internet connection. It's just that currently, there's no reason to believe the "airgap" guarantee of security was violated.
The claimed evidence about the "Microsoft Report Server" is wrong.
#4 Lack of Software Updates
This is just stupid. The cybersecurity community does have this weird fetish demanding that every software update be applied immediately, but there's good reasons why they aren't, and ways of mitigating the security risk when they can't be applied.
Software updates sometimes break things. In sensitive environments where computers must be absolutely predictable, they aren't applied. This includes live financial systems, medical equipment, and industrial control systems.
This also includes elections. It's simply not acceptable canceling or delaying an election because a software update broke the computer.
This is why Dominion does what they call a "Trusted Build" process that wipes out the boot-drive (deleting system-logs). To update software, they build an entire new boot image with all the software in a tested, known state. They then apply that boot disk image to all the county machines, which replaces everything on the C: boot-drive with a new version of Windows and all the software. This leaves the D: data drive untouched, where the records are preserved.
If you didn't do things this way, then sometimes elections will fail.
This is also why having an "airgapped" network is important. The voting machines aren't going to have software updates regularly applied, so they need to be protected. Firewalls would also be another mitigation strategy.
#5 Existence of SQL Server Management Studio.
This is just a normal part of having an SQL server installed.
Yes, in theory it would make it easy for somebody to change records in the database. But at the same time, such a thing is pretty easy even without SSMS installed. One way is command-line scripts.
#6 Referential Integrity
This "referential integrity" is a reliability concern, not an anti-hacking measure. It just means hackers would need only an extra step if they wanted to delete or change records.
Conclusion
Evidence is something that the expert understands. It's something they can show, explain, and defend against challengers.
This report contained none of that. It contained instead anomalies the writer couldn't explain.
Note that this doesn't mean they weren't an expert. Obviously, they needed enough expertise to get as far as they did. It's just a consequence of conspiracy-theories. When searching for proof of your conspiracy-theory when there is none, it means going off into the weeds past your are of expertise.
Give that forensics image to any expert, and they'll find anomalies they can't explain. That includes me, I've posted some of them to Twitter and had other experts explain them to me. The difference is that I attributed the lack of an explanation to my own ignorance, not a conspiracy.
At some point, we have to call out conspiracy-theories for what they are. This isn't defending the integrity of elections. If it were, it'd be proposing solutions for future elections. Instead, it's an attack on the integrity of elections, fighting the peaceful transfer of power by unfounded conspiracy-theory claims.
And we can say this objectively. As I stated above, there's three objective tests. These are:
- Anomalies that can't be explained are claimed to be evidence -- when in fact they come from simple ignorance.
- Demands that something needs explaining, when it really doesn't, and which won't satisfy them anyway.
- Statements of a world view (like that the election can be "decertified" or that system-logs are "election records") that nobody agrees with.
So, as a nerd, let's say you need 100 terabytes of home storage. What do you do?
My solution would be a commercial NAS RAID, like from Synology, QNAP, or Asustor. I'm a nerd, and I have setup my own Linux systems with RAID, but I'd rather get a commercial product. When a disk fails, and a disk will always eventually fail, then I want something that will loudly beep at me and make it easy to replace the drive and repair the RAID.
Some choices you have are:
- vendor (Synology, QNAP, and Asustor are the vendors I know and trust the most)
- number of bays (you want 8 to 12)
- redundancy (you want at least 2 if not 3 disks)
- filesystem (btrfs or ZFS) [not btrfs-raid builtin, but btrfs on top of RAID]
- drives (NAS optimized between $20/tb and $30/tb)
- networking (at least 2-gbps bonded, but box probably can't use all of 10gbps)
- backup (big external USB drives)
The products I link above all have at least 8 drive bays. When you google "NAS", you'll get a list of smaller products. You don't want them. You want somewhere between 8 and 12 drives.
The reason is that you want two-drive redundancy like RAID6 or RAIDZ2, meaning two additional drives. Everyone tells you one-disk redundancy (like RAID5) is enough, they are wrong. It's just legacy thinking, because it was sufficient in the past when drives were small. Disks are so big nowadays that you really need two-drive redundancy. If you have a 4-bay unit, then half the drives are used for redundancy. If you have a 12-bay unit, then only 2 out of the 12 drives are being used for redundancy.
The next decision is the filesystem. There's only two choices, btrfs and ZFS. The reason is that they both healing and snapshots. Note btrfs means btrfs-on-RAID6, not btrfs-RAID, which is broken. In other words, btrfs contains its own RAID feature that you don't want to use.
Over long periods of time, errors creep into the file system. You want to scrub the data occasionally. This means reading the entire filesystem, checksuming the files, and repairing them if there's a problem. That requires a filesystem that checksums each block of data.
Another thing you want snapshots to guard against things like ransomware. This means you mark the files you want to keep, and even if a workstation attempts to change or delete the file, it'll still be held on the disk.
QNAP uses ZFS while others like Synology and Asustor use btrfs. I really don't know which is better.
It's cheaper to buy the NAS diskless then add your own disk drives. If you can't do this, then you'll be helpless when a drive fails and needs to be replaced.
Drives cost between $20/tb and $30/tb right now. This recent article has a good buying guide. You probably want to get a NAS optimized hard drive. You probably want to double-check that it's CMR instead of SMR -- SMR is "shingled" vs. "conventional" magnetic recording. SMR is bad. There's only three hard drive makers (Seagate, Western Digital, and Toshiba), so there's not a big selection.
Working with such large data sets over 1-gbps is painful. These units allow 802.3ad link aggregation as well as faster Ethernet. Some have 10gbe built-in, others allow a PCIe adapter to be plugged in.
However, due to the overhead of spinning disks, you are unlikely to get 10gbps speeds. I mention this because 10gbps copper Ethernet sucks, so is not necessarily a buying criteria. You may prefer multigig/NBASE-T that only does 5gbps with relaxed cabling requirements and lower power consumption.
This means that your NAS decision is going to be made with your home networking decision. I use a couple of these multigig switches as something that doesn't cost too much for home networking.
Even though RAID is pretty darn reliable, you still need a backup solution. The way I do this is wither external USB hard drives. I schedule the NAS to backup to those drives automatically. As a home user, tapes aren't an effective solution, so you are stuck with USB drives.
In the end, this means that your total storage costs, with the NAS server, the drives, and the backup drives, is going to cost you 3x the price of the raw storage. Spinning drives fail often. If you plan on keeping your data around for the next decade, there's no way to do this without 3x the cost for storage.
I choose Synology because I have the most familiarity with the software, and its software gets the best reviews. But QNAP and Asustor also have great reputations.
Note that I've made the assumption here that you'll want "desktop NAS" solutions. There are also rackmount solutions available.
Author: Robert Graham (@erratarob)
Later today (Friday, September 24, 2021), Republican auditors release their final report on what they found with elections in Maricopa county. Draft copies of the report have already circulated online. In this blogpost, I write up my comments on the cybersecurity portions of their draft.
https://arizonaagenda.substack.com/p/we-got-the-senate-audit-report
The three main problems are:
- They misapply cybersecurity principles that are meaningful for normal networks, but which don’t really apply to the "air gapped" networks we see here.
- They make some errors about technology, especially networking.
- They are overstretching themselves to find dirt, claiming the things they don't understand are evidence of something bad.
In the parts below, I pick apart individual pieces from that document to demonstrate these criticisms. I focus on section 7, the cybersecurity section, and ignore the other parts of the document, where others are more qualified than I to opine.
In short, when corrected, section 7 is nearly empty of any content.
7.5.2.1.1 Software and Patch Management, part 1
They claim Dominion is defective at one of the best-known cyber-security issues: applying patches.
It’s not true. The systems are “air gapped”, disconnected from the typical sort of threat that exploits unpatched systems. The primary security of the system is physical. Frequent patching isn't expected.
This is a standard in other industries with hard reliability constraints, like industrial or medical. Patches in those systems can destabilize computers and kill people, so these industries are risk averse and resist applying them. They prefer to mitigate the threat in other ways, such as with firewalls and air gaps.
Yes, this approach is controversial. There are some in the cybersecurity community who use lack of patches as a bludgeon with which to bully any who don’t apply every patch immediately. But this is because patching is more a political issue than a technical one. In the real, non-political world we live in, most things don’t get immediately patched all the time.
7.5.2.1.1 Software and Patch Management, part 2
The auditors claim new software executables were applied to the system, despite the rules against new software being applied. This isn’t necessarily true.
There are many reasons why Windows may create new software executables even when no new software is added. One reason is “Features on Demand” or FOD. You’ll see new executables appear in C:\Windows\WinSxS for these. Another reason is their .NET language, which causes binary x86 executables to be created from bytecode. You’ll see this in the C:\Windows\assembly directory.
The auditors simply counted the number of new executables, with no indication which category they fell into. Maybe they are right, maybe new software was installed or old software updated. It’s just that their mere counting of executable files doesn’t show understanding of these differences.
7.5.2.1.2 Log Management
The auditors claim that a central log management system should be used.
This obviously wouldn’t apply to “air gapped” systems, because it would need a connection to an external network.
Dominion already designates their EMSERVER as the central log repository for their little air gapped network. Important files from C: are copied to D:, a RAID10 drive. This is a perfectly adequate solution, adding yet another computer to their little network would be overkill, and add as many security problems as it solved.
One could argue more Windows logs need to be preserved, but that would simply mean archiving the from the C: drive onto the D: drive, not that you need to connect to the Internet to centrally log files.
7.5.2.1.3 Credential Management
Like the other sections, this claim is out of place given the airgapped nature of the network.
Dominion simply uses “role based security” instead of normal user accounts. It’s a well known technique, and considered very appropriate for this sort of environment.
The auditors claim account passwords must “be changed every 90 days”. This is a well-know fallacy in cybersecurity. It took years to get NIST to remove it from their recommendations. If CISA still has it in their recommendations for election systems, then CISA is wrong.
Ideally, accounts wouldn’t be created until they were needed. In practice, system administrators aren’t available (again, it’s an airgapped system, so no remote administration). Dominions alternative is to create the accounts ahead of time, suc has “adjuser09”, waiting for the 9th person you hire that might use that account.
They are all given the same default password to start, like “Arizona2019!!!”. Some customers choose to change the default password, but obviously Maricopa did not. This is weak – but not a big deal, since the primary security is from controlling physical access.
7.5.2.1.4 Lack of Baseline for Host and Network Activity
They claim sort of baselining should be done. This is absurd. Baselines are always problematic, but would be especially so in this case.
The theory of baselines is that a networks traffic is somewhat predictable on a day-to-day basis. This obviously doesn’t apply to elections systems, which are highly variable day-to-day, especially on election day.
Baselining is the sort of thing you do with a dedicated threat hunting team. It’s incredibly inappropriate for a small installation like this.
7.5.3.1.1 Network Related Data
The auditors asked for an unreasonable access to network data, in the worst way possible, triggering the refusal to hand it over. They didn’t ask for reasonable data. They blame Maricopa Count for the conflict, but it’s really themselves who are to blame.
A reasonable request would take the MAC addresses from the election machines and ask for any matching records the Maricopa might have in their Splunk, DHCP, or ARP logs. Matches shouldn’t be found, but if they were, the auditors should then ask for flow data for the associated IP addresses.
They are correct in identifying this as a very important issue. Dominion security depends upon an airgap. If auditors find a netowrk connection, it’s bad. It’s not catastrophic, and sometimes machines are disconnected from one network and attached to a network during other times than the election. But this would very much be a useful part of a report – if only they had made a reasonable request that didn’t demand Maricopa spend their entire yearly budget to comply.
7.5.3.1.? Other Devices Connected to the Election Network
The auditors complain they weren’t given access to the router identified by 192.168.100.1.
It probably doesn’t exist.
Routers aren’t needed by devices that are on the same local Ethernet. They wouldn’t exist on a single-segment air gapped network. But typical operating-system configuration demands one be configured anyway, so it’s common to put in a dummy router address even if it’s unused.
If you see messages like this one in the logs, it means the router wasn’t there:
The auditors are right in identifying this as an important issue. If there were such a router, then this would cast doubt whether the network was “airgapped”.
Note that if such a router did exist, it would almost certainly be a NAT. This would still offer some firewall protection, just not as strong as an air gap.
7.5.4 Anonymous Logins
They see something in the security logs they don’t understand, and blame Maricopa’s lack of network data ("the routers") for their inability to explain it.
This is an extraordinarily inappropriate claim, based not on expert understanding of what they see in the logs, but complete ignorance. There’s no reason to believe that getting access to Maricopa Count network logs would explain what’s going on here.
This demonstrates they are on a phishing expedition, and that everything they see that they can’t explain is used as evidence of a conspiracy, either of Maricopa to withhold data, or of election fraud.
The Dominion suite of applications and services is oddly constructed and will produce anomalies. Comparing against a general Windows server not running Dominion’s suite is meaningless.
7.5.5 Dual Boot System Discovered
The auditors claim something about “dual-homed hosts” or “jump-boxes”. That’s not how these terms are normally used. These terms normally refer to a box with access to two separate networks, not a box with two separate operating systems.
This requires no nefarious explanation. This is commonly seen in corporate networks, either because somebody simply added a new drive to re-install the operating-system, or repurposed an old drive from another system as a data drive, and simply forgot to wipe it. The BIOS points to one they intend to boot from and ignore the fact that the other can also boot.
There are endless non-nefarious explanations for what is seen here that doesn’t require a nefarious one. It’s not even clear its a failure of their build process, which focuses on what’s on the boot drive and not what’s on other drives in the system.
7.5.6 EMS Operating System Logs Not Preserved
It is true the EMS operating-system logs are not preserved (well, generally not preserved). By this I refer to the generic Windows logs, the same logs that your own Windows desktop keeps.
The auditors falsely claim that this violates the law. This is false. The “electron records” laws don’t cover the operating-system. The laws instead are intended to preserve the records of the election software running on top of the operating-system, not those of the operating-system itself.
This issue has long been known. You don’t need an auditor’s report to tell you that these logs aren’t generally preserved – everyone has known this for a long time, including those who certified Dominion.
The subtext of this claim is the continued argument by Republicans that the fact they can’t find evidence for 2020 election fraud is because key data is missing. That’s the argument of Tina Peters, the former clerk of a county in the neighboring state of Colorado, who claims their elections cannot be audited because they don’t have the Windows operating-system logs.
It’s not true. System logs are as likely to cause confusion, as they do above with the “anonymous logins” issue. They are unlikely to provide proof of votes being flipped in a hack. If there was massive fraud, as detected by recounts of paper ballots, I’d certainly want such system logs to search for how it happened. But I wouldn’t use such logs in order to audit the vote.
Note that the description of “deleting” log entries by overfilling the logs is wrong. If it were important to preserve such logs, then they would be copied right after the election. They wouldn’t be left to rot on the boot drive for months afterwards.
As a forensics guy, I would certainly support the idea that Dominion should both enable more logs and preserve them after each election. They don’t require excessive storage and can be saved automatically in the last phase of an election. But their lack really isn’t all that important, they are mostly just full of junk.
Conclusion
We live in a pluralistic democracy, meaning there are many centers of power, each competing with each other. It's inherently valid for one side to question and challenge the other side. But this can go too far, to the point where you are challenging the stability of our republic.
The Republican party is split. Some are upholding that principle of pluralism, wanting to make sure future elections are secure and fair. Others are attacking that principle, challenging the peaceful transfer of power in the last election with baseless accusations of fraud.
This split is seen in Arizona, where Republicans have demanded an audit by highly partisan auditors. An early draft of their report straddles that split, containing some reasonable attempt to create recommendations for future elections, while simultaneous providing fodder for the other side to believe the last election was stolen.
A common problem with auditors is that when they can’t find the clear evidence they were looking for, the fill their reports with things they don’t understand. I think I see that here. The auditors make technical errors in ways that question their competence, but that’s likely not true. Instead, they kept searching past where they were strong into areas where they were weak, looking for as much dirt as possible. Thus, in this report, we see where they are technically weak.
Trumpists, meaning those attacking the peaceful transfer of power with baseless accusations of fraud, will certainly use this report to champion their cause, despite the headline portion that confirms the vote count. But for the rest of us, we should welcome this report. Elections do need to be fixed, and while it’s unlikely we’ll fix them in the ways suggested in this report, it will add visibility into the process which we can use to debate improvements.
This blogpost is only a first draft. While the technical bits in section 7 look fairly straightforward to me, I'm guessing that people who don't understand them will come up with weird conspiracy-theories about them. Thus, I'm guessing I'll have to write another blogpost in a week debunking some of the crazier ideas.