Chasing APTs: How a Hunt Evolves
Last week Infocyte was doing a product demo for a partner who wants to do compromise assessments (like these guys). They chose an existing client to do a limited scan using our product, selecting a handful of systems... and we found something interesting.
Now that's not the interesting part; we find things all the time. In fact, more than half of organizations that our product is run on (i.e. compromise assessments) have some unauthorized or malicious code in their network. The interesting part here was the evolution of this particular assessment and follow on incident response engagement + the fact that they didn't ask for privacy or an NDA (which we usually do), so I get to talk about it (which I usually don't).
As with many incident response engagements, this one was full of audible facepalms, raised eyebrows, excitement, and then deflation... all in about a 4 hour period.
TL;DR: Attribution is hard. The engagement went from skepticism from the client, to apathetic acceptance, to "oh shiza, it's APT/China!", to disappointment as we realized the IOCs and YARA signatures matched were too generic to confirm and the evidence was circumstantial. Regardless, we found a significant problem and helped fix it.
This compromise assessment test run used Infocyte HUNT which is an agentless endpoint hunt platform. It collects forensic information on the state of each system including process, volatile memory analysis, enumeration of persistence mechanisms, and identification of manipulation (rootkit indicators).
After running the scan, the client reported that they found two items referenced in registry autorun keys that were flagged as malware. They were both categorized as generic trojans with limited functionality (i.e. check the system, kill the AV, and download the next payloads), Multiple AV matches to boot. Pretty clear cut.
First, the Skepticism
"We believe these are false positives. Our admin found no VirusTotal match (hash lookup) and he looked up the filename (maintenanceservice.exe) in google and it looks like a mozilla file."
"Your product also falsely flagged VNC Server, an authorized tool, which also leads us to believe this is wrong. Please provide additional proof at how you came to your conclusions."
After some back and forth, we were able to explain how malware often masquerades as legitimate files to evade detection or throw off investigations. They finally submitted it to VirusTotal and got back 51 matches.
From the initial findings we didn’t have any active connections, the file wasn't active, and they didn’t give us logs or any other info besides the samples and our scan results (HUNT pops the sample in OPSWAT, analyzes it with a proprietary static analysis algorithm, and detonates it in a sandbox and presents abridged results). From this, we described it as a generic trojan with primary functionality of scanning the system and bringing down more tools to provide remote access.
We made some attempts to get access to the system and figure out any more information from it but we were denied.
"... lets just remove it."
Down the Attribution Rabbit Hole
Microsoft categorized the files as Win32/Viking (Trojan) variants. This is a broad categorization so not much to it. But 20 minutes after the VT submission, we were notified that one of the samples matched a new YARA rule which included IOCs from APT3's latest campaign reported by Symantec on 6 Sept 2016. Attribution is an imprecise art but this raised an eyebrow for our team. APT3, also known as BuckEye or UPS, is considered a very sophisticated threat actor out of China - i've ran in to them before in a previous life and I had a bone to pick.
From the sandbox results, the samples looked pretty nasty. As an example, maintenanceservice.exe scanned the sandbox file system, tried to killed antiviruses, dropped other files and ran them, and called out to another domain to pull down an additional payload. Similar activity for the other samples.
The file calls out to a domain to pull down another file:
down.97725[.]com which is associated with a LOT of reported malware and malicious activity. I used Threatcrowd to get a Maltego Transform of all the associations available online:
My first question after this was: why would a chinese group known to target US and UK targets get their malware in South America (where the client is Headquartered)?
Well, Symantec researches give us a good reason for that:
“The group casts a wide net while trawling for targets but only remains active on the networks of organizations it is interested in. Symantec determined a more accurate picture of Buckeye’s targets by looking at where Buckeye remained active on the network longer than a day, deployed additional tools, and spread onto multiple computers." - Symantec Buckeye Report, 6 Sept 2016
Aww... so perhaps they were just collateral damage? ¯\_(ツ)_/¯
Although the callback domain
down.97725[.]com was down (did not resolve) and we could not pull the secondary payload, the domain was maliciously active at least between April 2013 and August 2016. Reports for the parent domain
97725[.]com go back as far back as 2007 - yes, 9 years ago (I can't even count how long that is in internet years) with multiple subdomains hosting malware for various purposes.
So obviously we found a persistent threat group.
As we went down the attribution rabbit hole though, we couldn't produce any other evidence to point it to APT3. We could spend hours going through other sites and samples connected to this group till we find a reputable correlation or report but we were only doing the work to justify the additional work - have to cut it off some time.
The key here is to run YARA with -s option as there may be multiple conditions in the signature (and there were). It was a high quality signature built off the symantec-provided APT3 malware samples including some unique string combinations. But it also had some generic opcode matching which tends to flag on a lot of simple trojans like this. Ours matched on the latter.
Results using the following signature:
---DiparaQtdPalete.exe 223C7480F3DD9212A16E23F4FEE8ADB0DB06E25ACBA9F78E7206B14E450B30DD Pirpi_1609_A .\DiparaQtdPalete.exe 0x1d4f:$op1: 74 08 C1 CB 0D 03 DA 40 EB 0x1d39:$op2: 03 F5 56 8B 76 20 03 F5 33 C9 49 0x1d61:$op3: 03 DD 66 8B 0C 4B 8B 5E 1C 03 DD 8B 04 8B 03 C5 ---maintenanceservice.exe 7126A1A681F3FF9CB5B703615A9456CC5CECBF2BAC25DB8798EA51988529CA9E ---rundl132.exe D99B555F85E148945A4A5BDEE6B1069980ACDBB1FC039D463525DE531C975B05 Pirpi_1609_A .\rundl132.exe 0x1d4f:$op1: 74 08 C1 CB 0D 03 DA 40 EB 0x1d39:$op2: 03 F5 56 8B 76 20 03 F5 33 C9 49 0x1d61:$op3: 03 DD 66 8B 0C 4B 8B 5E 1C 03 DD 8B 04 8B 03 C5
Damn... not APT3?
In the end, we ended up reporting to the client that we have strong confidence that they were hit by an APT-type attack and should conduct a more broad scan of their network and possibly an investigation. Circumstantial evidence points it to a group operating in or through China (like many threat groups in and out of China). Although it flagged on some Symantec-provided Indicators of Compromise (IoCs) for APT3 (aka Buckeye, UPS, etc), we could find no other direct evidence from the initial investigation. Even if it was APT3, this client isn't in the sets of targets that would be interesting to them. Ultimately, we proved we had some chops in this area and trust was built between a new international partner and a growing security company in Texas.
This was definitely a fun exercise. I live for this - it's why I asked Russ Morris to start Infocyte with me: so we could help people find hackers and get them out of their networks. I wrote this case to illustrate the process a hunt or investigation can go through - including the ups and downs, human factors, and technical sources of information that can help come to a conclusion. Below are a few takeaways I got from it, I hope this post helps someone out there going through similar experiences for the first time:
You should probably start hunting
A lot of organizations are currently hacked and don't even know it. Nor do they know the severity when they do find something. A hunt capability can be a relatively inexpensive addition to your defense in depth (which should start with basic hygiene policies, antivirus, and a security-aware firewall/proxy setup).
If you're not ready for a hunt capability, consider bringing in the professionals to have a compromise assessment done. Our partners that use Infocyte HUNT can do them for a few thousand bucks depending on the size of the organization.
Most People Want Binary Answers
- In general - most non-security people want no more complexity than a binary answer. Yes or no. Is it good or is it bad. The answer of: "It's a generic 1st stage trojan reaching out to currently unresolved domain" is garbage to most listeners. But, "you're compromised. It's China." Those I can accept if the source is credible enough.
Credibility is earned
- Our company is still small and new. We're unknown. As can be expected, our findings were immediately questioned. We threw the file in OPSWAT and showed them 20 AV results, 12 of which flagged it as a generic trojan. 12 Reputable vendors like Symantec and Kasperky (just not the AV they currently had installed). VirusTotal results gave us 51/56 - even better - can't be fired for listening to google.
There is a lot at stake when you find something.
- Even when presented evidence and scan results from 20 Antivirus Engines, we still got push back to prove and explain it. We really struggled to figure out what else we could say to get them to investigate further. But this is the reality, customers have a lot at stake when you find something like this and psychology and human factors come into play. Most security professionals have been burned by false positives and have cried wolf. Sure, fast action is needed but when you are employed you have to be sure and be able to answer objections when the company board is thinking about ramifications, loses, and impact. That's why we built Infocyte HUNT, so that answers like this could be obtained in seconds/minutes and at most, hours, and not at the conclusion of a months long investigation.
Don't trust IOCs and threat intel blindly.
The YARA signature that popped said "APT3" dataset coming from a recent Symantec report. It might be enough to convince an inexperienced analyst, but experience has taught me to merely raises my eyebrow and dig deeper.
A review of the signature showed it was actually well made, it's not just some simple string match - it's got some meat and multiple factors. But after looking at which part flagged the signature, it turned out to be just the OPCODES matching - these can be pretty generic for this type of trojan. DEFLATED! Our strongest evidence reduced to a generic match.
Attribution is hard.
- It is. But we still need to do it -- it's the only way these guys will stop. We aren't going to patch every vulnerability or make a silver bullet security product in our lifetime - the only way we can reduce hacking on the macro level is consequences and deterrence; and both of those require reliable attribution.