The security breach in Log4J / Log4Shell is one of the largest IT vulnerabilities in years. It was discovered on Dec. 9, 2021.. Photo: Adobe Stock

Criminal actors used a software vulnerability to gain access to the Suffolk County clerk’s computer network in December 2021, months earlier than previously disclosed by county officials, an ongoing independent digital forensics investigation has found.

The attackers used the incursion into the clerk’s system to steal files and system credentials, install cryptocurrency mining software, malware and remote tools, and create new users with administrative access. By August, they had gained “lateral access” to other county networks from the clerk’s system, including the county’s main computer environment, as well as the health department and sheriff’s department. They then used that access to install malware and other malicious tools and steal sensitive data stored on county servers.

This activity continued through Sept. 8, when the attackers encrypted files — rendering them inaccessible to county staff — and demanded ransom of $2.5 million. The county refused to pay the ransom, officials said.

The attackers, a professional hacking group that calls itself BlackCat and ALPHV, posted some stolen personal information on the dark web. They claimed to have encrypted more than 4 terabytes of data from county computer systems and those of government contractors.

The Sept. 8 ransom demand set the county scrambling to shut down servers and systems in a too-late attempt to stop an attack that had already been going on for months. The county’s response disabled county email systems and internet access, forcing government operations to fall back on methods and procedures used before digital era.

Though some systems and operations have been restored in the nearly four months that followed the discovery of the attack, others remain offline, including the county’s main public-facing websites.

On Wednesday, Suffolk County Executive Steve Bellone released a memorandum by the incident response contractor investigating the attack, Unit 42 by Palo Alto Networks. The document provides a timeline and summary of “key events” in the incident as determined by the investigation.

Hackers exploited a vulnerability in a popular software product

Investigators found that on Dec. 19 and Dec. 20, 2021, the attackers exploited a known vulnerability in Log4j2, a popular software library used for logging error messages. The vulnerability enables a remote attacker to gain control over a string of code and trick an application into executing malicious code under the attacker’s control. This allows attackers to remotely take over any internet-connected service that uses certain versions of the library.

The vulnerability is a very serious threat because it affects Java, a ubiquitous software used in most digital environments — everything from video games and smart TVs to Google, Amazon and Microsoft cloud storage. The vulnerability was discovered Dec. 9, 2021 and software vendors very quickly began publishing security fixes. But the sheer volume of vulnerable systems meant tens of thousands of publicly accessible systems remained at risk even months after the discovery. And malicious actors took advantage of it, launching millions of attempted exploits, according to security researchers.

The cybercriminals that exploited the vulnerability to attack the county’s systems installed cryptocurrency mining software on multiple county clerk servers in January, investigators learned. They continued to use cryptocurrency miners in January and February, the report states.

There is no known connection between this incident and another cryptocurrency mining operation allegedly run on county servers from February to August 2021 by an assistant manager of information technology operations in the county clerk’s office, according to Chief Deputy County Executive Lisa Black. The employee, Christopher Naples of Mattituck, was charged in September 2021 with grand larceny, computer trespass and official misconduct. The case is still pending in Southampton Town Justice Court.

Digital forensic investigators find serious security lapses in clerk’s network

In March the hackers installed remote monitoring and management tools and began harvesting user credentials on the county clerk’s systems. In April, they created a user named “John” and added it to the administrators group on the county clerk’s servers. This behavior continued, with new users given administrator access, allowing the hackers to poke around the clerk’s network undetected, looking for sensitive data to steal and other systems that could be accessed, according to investigators.

Among the stolen files were credential files — text files with names like “SvcAccounts.txt” and “TMpasswords.txt” — that were stored in a folder named “IronKey.” Iron Key is the brand name of an encrypted USB portable storage device used by governments to securely store password, credentials and sensitive information. The encrypted portable storage device is supposed to be taken off premises for security reasons, Black said in an interview yesterday.

“Why do you keep a folder on your server called ‘IronKey?’ And it gets better — when you open up that folder there’s a file that says ‘passwords,’” Black said. “I don’t know why you’d put that in a text file that’s not encrypted or secured in any way. Storing it on your network is wild to me,” she said.

“In three hours they had control of the clerk’s network,” said Deputy County Executive Peter Scully, who also sat in on yesterday’s interview with RiverheadLOCAL.

The digital forensic investigation determined that the “SvcAccounts.txt” file was deleted from the clerk’s server on Sept. 29, Black said.

The investigation has been complicated by a lack of cooperation from the internal IT department at the county clerk’s office, according to Black. The clerk’s office has its own internal IT department, separate from the county’s main IT department, and the clerk’s office has not been allowing investigators from the county’s main IT department to access the clerk’s systems to investigate, Black said.

Investigators have therefore been unable to determine if files containing sensitive information remain in existing backups on the clerk’s servers, because the clerk’s office has prevented the county’s main IT staff and even the digital forensic incident response team from accessing the clerk’s systems, Black said.

“We do know that every other domain in the county has been deemed validated and clean,” Black said.

Without access to the clerk’s systems, the digital forensic incident response team said there was nothing else it could do.

“They said, ‘Look, if we don’t get in there, we have to we’re gonna have to close out.’ We said, ‘You can’t do that. If there’s something still out there. We’re responsible for it and we have a moral obligation to our residents to to provide the information.’ The investigators need to get in there and do the work they’ve been able to do countywide thus far,” Black said.

That prompted the county executive to reassign the clerk’s IT staff to the main county department of information technology, Black said. In addition, the clerk’s information technology director has been put on paid administrative leave, a move that was announced by the county executive Wednesday.

County Clerk Judith Pascale, who is leaving office Dec. 31, did not return a phone call yesterday seeking comment for this story.

County lacks central control over IT security systems

To this point, there has not been one central protocol or central control over information technology and security in departments run by independently elected officials, such as the county clerk and comptroller.

The county legislature has supported the administration’s request for a Central Information Security Officer, Black said, so there will be one central team monitoring security for all county domains.

“Anything that’s a dot-gov domain should be governed by CISA standards,” Black said, referring to the federal Cybersecurity and Infrastructure Security Agency. “For example, if there is a patch that comes out, if there’s a vulnerability on any one software, the federal government requires you to patch it in two weeks. If you do not patch it within two weeks, you lose your domain. That’s it,” Black said. “We just don’t have that level of governance” at the county level.

Tip about potential malware from court officer to DA’s office in June mishandled by clerk’s IT director, officials say

On June 21, the DA’s office received a tip about “potential malware” and referred it to the county’s IT security officer, according to Scully.

“The DA’s office had gotten a call from a court officer who got a call from a retired detective who got a call from an FBI agent,” Scully said. “The county’s IT security officer immediately contacted the FBI agent and the court officer and was able to determine, based on that conversation, that the activity that was being referenced was not in the county system, but was in fact in the clerk’s office — between the clerk’s office and the court system.” The clerk’s office files judgments and other documents filed by county and state courts in Suffolk.

“And because the clerk’s office has insisted that our central IT not have the ability to monitor their system, the only recourse they had was to reach out to the clerk’s office and ask if there was anything to it,” Scully said.

The clerk’s IT director responded on June 22, saying there was “nothing going on on our side,” Scully said.

Six hours later, the same official sent out an email to his clerk’s office staff saying effective July 1, virtual clients would not be able to use the system remotely because of a significant security flaw, Scully said.

No county central IT security staff was copied on that email, Scully said.

“When the county’s central IT security learned of the email, they reached out to the county clerk’s office and said, ‘What can you tell us about the significant security flaw,’ they got no response,” Scully said.

Since that was a full two months before the hackers moved into the county’s central systems, the county had an opportunity to prevent the expanded hack, officials believe.

“It’s entirely possible we could have arrested the problem before they got into the other systems, but we’ll never know because the information wasn’t shared on June 21 and 22,” Scully said.

Bellone on Wednesday also released the emails Scully referred to in the interview.

Digital forensic investigation to continue, now with access to clerk’s network

The digital forensic investigation needs to continue because “there are still some outstanding items that they know to have been taken, that we know exist in the clerk’s office that we just haven’t been able to find,” Black said. “And it’s incumbent upon us to be able to investigate what that is and determine whether or not it had PII (personally identifiable information) on it,” Black said. To do that requires access, she said.

“It’s not only read-only access. We need to go in there and do some assessments ourselves, so that we can bring the digital forensics team back in and then they can get started. Not to mention the restoration. There’s quite a bit of restoration that still exists in the clerk’s network that we need to pursue,” she said.

“The next step is to really dig down and see what of their systems was accessed and what did they take from it, what was exfiltrated, what was put on the dark web. And if we can’t find something we know or believe was there, was it deleted,” Black said.

The county executive announced on Nov. 24 that the personal data of individuals who were issued nearly half a million moving violations in the Suffolk County Police District during the past decade may have been accessed by hackers. Earlier this month, Bellone announced the Social Security numbers of 26,000 county employees and retirees were exposed in the attack.

County officials do not believe town systems that interact with the clerk and other county systems were affected by the hack.

“I don’t think you can ever say no, unequivocally,” Black said. “It depends on what kind of systems that they’re operating too… We have to figure out how the municipal systems are speaking to the clerk’s office,” she said. “We don’t have enough of an understanding of the operation of the clerk’s office at this stage to make a comment on that right now.”

Black said the administration has been making a push to centralize IT security. But only 80 out of the 180 IT titles in county government today are in the county’s central IT office.

Time frame for full restoration still uncertain

County officials still aren’t sure when its websites will be up and running again.

“We want to want to make sure we do the website when we have every domain clean,” Black said. “So it really depends on what they would they find over the next few days and give us an estimate on either restoration rebuild,” she said. It’s going to take some time to assess before they can get starring on a restoration plan, according to Black.

“Optimistically, by the end of January,” Black said, but she acknowledged she still can’t say for sure.

The FBI is separately investigating the attack, county officials said. They have briefed county officials but really haven’t shared a lot of detail.

Suffolk County District Attorney Ray Tierney said in a statement Wednesday evening his office continues to cooperate with the FBI and Suffolk County Police regarding the ongoing criminal investigation. He said no criminal prosecutions were compromised in the attack because his office had “additional internet technology defenses in place.”

The survival of local journalism depends on your support.
We are a small family-owned operation. You rely on us to stay informed, and we depend on you to make our work possible. Just a few dollars can help us continue to bring this important service to our community.
Support RiverheadLOCAL today.