Good essay arguing that open-source software is a critical national-security asset and needs to be treated as such:
Open source is at least as important to the economy, public services, and national security as proprietary code, but it lacks the same standards and safeguards. It bears the qualities of a public good and is as indispensable as national highways. Given open source’s value as a public asset, an institutional structure must be built that sustains and secures it.
This is not a novel idea. Open-source code has been called the “roads and bridges” of the current digital infrastructure that warrants the same “focus and funding.” Eric Brewer of Google explicitly called open-source software “critical infrastructure” in a recent keynote at the Open Source Summit in Austin, Texas. Several nations have adopted regulations that recognize open-source projects as significant public assets and central to their most important systems and services. Germany wants to treat open-source software as a public good and launched a sovereign tech fund to support open-source projects “just as much as bridges and roads,” and not just when a bridge collapses. The European Union adopted a formal open-source strategy that encourages it to “explore opportunities for dedicated support services for open source solutions [it] considers critical.”
Designing an institutional framework that would secure open source requires addressing adverse incentives, ensuring efficient resource allocation, and imposing minimum standards. But not all open-source projects are made equal. The first step is to identify which projects warrant this heightened level of scrutiny—projects that are critical to society. CISA defines critical infrastructure as industry sectors “so vital to the United States that [its] incapacity or destruction would have a debilitating impact on our physical or economic security or public health or safety.” Efforts should target the open-source projects that share those features.
Rapid7 is pleased to announce the release of Velociraptor version 0.6.5 – an advanced, open-source digital forensics and incident response (DFIR) tool that enhances visibility into your organization’s endpoints. This release has been in development and testing for several months now, and we are excited to share its new features and improvements.
Table transformations
Velociraptor collections or hunts are usually post-processed or filtered in Notebooks. This allows users to refine and post-process the data in complex ways. For example, to view only the Velociraptor service from a hunt collecting all services (Windows.System.Services), one would click on the Notebook tab and modify the query by adding a WHERE statement.
Filtering rows with VQL
In our experience, this ability to quickly filter or sort a table is very common, and sometimes we don't really need the full power of VQL. In 0.6.5, we introduced table transformations — simple filtering/sorting operations on every table in the GUI.
Setting simple table transformations
Multi-lingual support
Velociraptor’s community of DFIR professionals is global! We have users from all over the world, and although most users are fluent in English, we wanted to acknowledge our truly international user base by adding internationalization into the GUI. You can now select from a number of popular languages. (Don’t see your language here? We would love additional contributions!)
Select from a number of popular languages
Here is a screenshot showing our German translations:
The Velociraptor interface in German
New interface themes
The 0.6.5 release expanded our previous offering of 3 themes into 7, with a selection of light and dark themes. We even have a retro feel ncurses theme that looks like a familiar terminal…
A stunning retro ‘ncurses’ theme
Error-handling in VQL
Velociraptor is simply a VQL engine – users write VQL artifacts and run these queries on the endpoint.
Previously, it was difficult to tell when VQL encountered an error. Sometimes a missing file is expected, and other times it means something went wrong. From Velociraptor’s point of view, as long as the VQL query ran successfully on the endpoint, the collection was a success. The VQL query can generate logs to provide more information, but the user had to actually look at the logs to determine if there was a problem.
For example, in a hunt parsing a file on the endpoints, it was difficult to tell which of the thousands of machines failed to parse a file. Previously, Velociraptor marked the collection as successful if the VQL query ran – even if it returned no rows because the file failed to parse.
In 0.6.5, there is a mechanism for VQL authors to convey more nuanced information to the user by way of error levels. The VQL log() function was expanded to take a level parameter. When the level is ERROR the collection will be marked as failed in the GUI.
A failed VQL queryQuery Log messages have their own log level
Custom time zone support
Timestamps are a central part of most DFIR work. Although it is best practice to always work in UTC times, it can be a real pain to have to convert from UTC to local time in your head! Since Velociraptor always uses RFC3389 to represent times unambiguously but for human consumption, it is convenient to represent these times in different local times.
You can now select a more convenient time zone in the GUI by clicking your user preferences and setting the relevant timezone.
Selecting a custom timezone
The preferred time will be shown in most instances in the UI:
Time zone selection influences how times are shown
A new MUSL build target
On Linux Go binaries are mostly static but always link to Glibc, which is shipped with the Linux distribution. This means that traditionally Velociraptor had problems running on very old Linux machines (previous to Ubuntu 18.04). We used to build a more compatible version on an old Centos VM, but this was manual and did not support the latest Go compiler.
In 0.6.5, we added a new build target using MUSL – a lightweight Glibc replacement. The produced binary is completely static and should run on a much wider range of Linux versions. This is still considered experimental but should improve the experience on older Linux machines.
Try it out!
If you’re interested in the new features, take Velociraptor for a spin by downloading it from our release page. It’s available for free on GitHub under an open source license.
As always, please file bugs on the GitHub issue tracker or submit questions to our mailing list by emailing velociraptor-discuss@googlegroups.com. You can also chat with us directly on our Discord server.
Learn more about Velociraptor by visiting any of our web and social media channels below:
In this episode of Security Nation, Jen and Tod chat with Steve Micallef about SpiderFoot, the open-source intelligence tool of which he is the creator and founder. He tells us how the platform went from a passion project to a fully fledged open-source offering, with a SaaS option to boot, and how it can help security engineers automate tasks and focus on finding the major threats in their data.
Stick around for our Rapid Rundown, where Tod chats with producer Jesse about a new paper that reveals all is not as it seems with CVSS scores.
Steve Micallef
Steve Micallef is the author of SpiderFoot (www.spiderfoot.net), an open-source OSINT automation platform. You can follow him @binarypool on Twitter.
Like the show? Want to keep Jen and Tod in the podcasting business? Feel free to rate and review with your favorite podcast purveyor, like Apple Podcasts.
Want More Inspiring Stories From the Security Community?
In this episode of Security Nation, Jen and Tod chat with Phillip Maddux about his project HoneyDB, a site that pulls data together from honeypots around the world in a handy, open-source format for security pros and researchers. He details how his motivations for creating HoneyDB derived from his time in application security and why he thinks open source is such a great format for this kind of project.
No Rapid Rundown this week, since RSAC 2022 has Tod tied up (and several time zones farther from Jen than usual). If you’re in San Francisco for the conference, stop by the Rapid7 booth and say hi!
Phillip Maddux
Phillip Maddux is a staff engineer on the Detection and Response Engineering team at Compass. He has over 15 years of experience in information security, with the majority of that time focused on application security in the financial services sector. Throughout his career, Phillip has been a honeypot enthusiast and is the creator of HoneyDB.io.
Interested in participating in the project? Head to the HoneyDB Agent Docs.
Like the show? Want to keep Jen and Tod in the podcasting business? Feel free to rate and review with your favorite podcast purveyor, like Apple Podcasts.
Want More Inspiring Stories From the Security Community?
In this episode of Security Nation, Jen and Tod sit down with Jim O’Gorman and Ben “g0tmi1k” Wilson of Offensive Security to chat about Kali Linux. They walk our hosts through the vision behind Kali and how they understand the uses, advantages, and challenges of open-source security tools.
Stick around for our Rapid Rundown, where producer Jesse joins Tod to talk about an upcoming change in security protocols across the internet that might make passwords obsolete (eventually).
Jim O’Gorman
Jim O’Gorman (Elwood) began his tech career as a network administrator with a particular talent for network intrusion simulation, digital investigations, and malware analysis. Jim started teaching for OffSec in 2009 as an instructor for the Penetration Testing with Kali (PWK) course — a role he still enjoys. He went on to co-author Metasploit: The Penetration Tester’s Guide and Kali Linux: Revealed, and has developed and curated a number of OffSec courses. As the Chief Content and Strategy officer, he currently oversees the open source Kali Linux development project and participates with OffSec’s Penetration Testing Team.
Ben “g0tmi1k” Wilson
Ben "g0tmi1k" Wilson has been in the information security world for nearly two decades. Since joining Offensive Security nine years ago, he has applied his experience in a number of roles including live instructor, content developer, and security administrator. He is currently managing the day-to-day activity as well as developing Kali Linux, pushing it forward. He has worked on various vulnerabilities, which are published on Exploit-DB that he also works on. Furthermore he created and still runs VulnHub, allowing for hands-on experience.
Like the show? Want to keep Jen and Tod in the podcasting business? Feel free to rate and review with your favorite podcast purveyor, like Apple Podcasts.
Want More Inspiring Stories From the Security Community?
Rapid7 is pleased to announce the release of Velociraptor version 0.6.4 – an advanced, open-source digital forensics and incident response (DFIR) tool that enhances visibility into your organization’s endpoints. This release has been in development and testing for several months now and has a lot of new features and improvements.
The main focus of this release is in improving path handling in VQL to allow for more efficient path manipulation. This leads to the ability to analyze dead disk images, which depends on accurate path handling.
Path handling
A path is a simple concept – it’s a string similar to /bin/ls that can be used to pass to an OS API and have it operate on the file in the filesystem (e.g. read/write it).
However, it turns out that paths are much more complex than they first seem. For one thing, paths have an OS-dependent separator (usually / or \). Some filesystems support path separators inside a filename too! To read about the details, check out Paths and Filesystem Accessors, but one of the most interesting things with the new handling is that stacking filesystem accessors is now possible. For example, it’s possible to open a docx file inside a zip file inside an ntfs drive inside a partition.
Dead disk analysis
Velociraptor offers top-notch forensic analysis capability, but it’s been primarily used as a live response agent. Many users have asked if Velociraptor can be used on dead disk images. Although dead disk images are rarely used in practice, sometimes we do encounter these in the field (e.g. in cloud investigations).
Previously, Velociraptor couldn’t be used easily on dead disk images without having to carefully tailor and modify each artifact. In the 0.6.4 release, we now have the ability to emulate a live client from dead disk images. We can use this feature to run the exact same VQL artifacts that we normally do on live systems, but against a dead disk image. If you’d like to read more about this new feature, check out Dead Disk Forensics.
Resource control
When collecting artifacts from endpoints, we need to be mindful of the overall load that collection will cost on endpoints. For performance-sensitive servers, our collection can cause operational disruption. For example, running a yara scan over the entire disk would utilize a lot of IO operations and may use a lot of CPU resources. Velociraptor will then compete for these resources with the legitimate server functionality and may cause degraded performance.
Previously, Velociraptor had a setting called Ops Per Second, which could be used to run the collection “low and slow” by limiting the rate at which notional “ops” were utilized. In reality, this setting was only ever used for Yara scans because it was hard to calculate an appropriate setting: Notional ops didn’t correspond to anything measurable like CPU utilization.
In 0.6.4, we’ve implemented a feedback-based throttler that can control VQL queries to a target average CPU utilization. Since CPU utilization is easy to measure, it’s a more meaningful control. The throttler actively measures the Velociraptor process’s CPU utilization, and when the simple moving average (SMA) rises above the limit, the query is paused until the SMA drops below the limit.
Selecting resource controls for collections
The above screenshot shows the latest resource controls dialog. You can now set a target CPU utilization between 0 and 100%. The image below shows how that looks in the Windows task manager.
CPU control keeps Velociraptor at 15%
By reducing the allowed CPU utilization, Velociraptor will be slowed down, so collections will take longer. You may need to increase the collection timeout to correspond with the extra time it takes.
Note that the CPU limit refers to a percentage of the total CPU resources available on the endpoint. So for example, if the endpoint is a 2 core cloud instance a 50% utilization refers to 1 full core. But on a 32 core server, a 50% utilization is allowed to use 16 cores!
IOPS limits
On some cloud resources, IO operations per second (IOPS) are more important than CPU loading since cloud platforms tend to rate limit IOPS. So if Velociraptor uses many IOPS (e.g. in Yara scanning), it may affect the legitimate workload.
Velociraptor now offers limits on IOPS which may be useful for some scenarios. See for example here and here for a discussion of these limits.
The offline collector resource controls
Many people use the Velociraptor offline collector to collect artifacts from endpoints that they’re unable to install a proper client/server architecture on. In previous versions, there was no resource control or time limit imposed on the offline collector, because it was assumed that it would be used interactively by a user.
However, experience shows that many users use automated tools to push the offline collector to the endpoint (e.g. an EDR or another endpoint agent), and therefore it would be useful to provide resource controls and timeouts to control Velociraptor acquisitions. The below screenshot shows the new resource control page in the offline collector wizard.
Configuring offline collector resource controls
GUI changes
Version 0.6.4 brings a lot of useful GUI improvements.
Notebook suggestions
Notebooks are an excellent tool for post processing and analyzing the collected results from various artifacts. Most of the time, similar post processing queries are used for the same artifacts, so it makes sense to allow notebook templates to be defined in the artifact definition. In this release, you can define an optional suggestion in the artifact yaml to allow a user to include certain cells when needed.
The following screenshot shows the default suggestion for all hunt notebooks: Hunt Progress. This cell queries all clients in a hunt and shows the ones with errors, running and completed.
Hunt notebooks offer a hunt status cellHunt notebooks offer a hunt status cell
Multiple OAuth2 authenticators
Velociraptor has always had SSO support to allow strong two-factor authentication for access to the GUI. Previously, however, Velociraptor only supported one OAuth2 provider at a time. Users had to choose between Google, Github, Azure, or OIDC (e.g. Okta) for the authentication provider.
This limitation is problematic for some organizations that need to share access to the Velociraptor console with third parties (e.g. consultants need to provide read-only access to customers).
In 0.6.4, Velociraptor can be configured to support multiple SSO providers at the same time. So an organization can provide access through Okta for their own team members at the same time as Azure or Google for their customers.
The Velociraptor login screen supports multiple providers
The Velociraptor knowledge base
Velociraptor is a very powerful tool. Its flexibility means that it can do things that you might have never realized it can! For a while now, we’ve been thinking about ways to make this knowledge more discoverable and easily available.
Many people ask questions on the Discord channel and learn new capabilities in Velociraptor. We want to try a similar format to help people discover what Velociraptor can do.
The Velociraptor Knowledge Base is a new area on the documentation site that allows anyone to submit small (1-2 paragraphs) tips about how to do a particular task. Knowledge base tips are phrased as questions to help people search for them. Provided tips and solutions are short, but they may refer users to more detailed information.
If you learned something about Velociraptor that you didn’t know before and would like to share your experience to make the next user’s journey a little bit easier, please feel free to contribute a small note to the knowledge base.
Importing previous artifacts
Updating the VQL path handling in 0.6.4 introduces a new column called OSPath (replacing the old FullPath column), which wasn’t present in previous versions. While we attempt to ensure that older artifacts should continue to work on 0.6.4 clients, it’s possible that the new VQL artifacts built into 0.6.4 won’t work correctly on older versions.
To make migration easier, 0.6.4 comes built in with the Server.Import.PreviousReleases artifact. This server artifact will load all the artifacts from a previous release into the server, allowing you to use those older versions with older clients.
Importing previous versions of core artifacts
Try it out!
If you’re interested in the new features, take Velociraptor for a spin by downloading it from our release page. It’s available for free on GitHub under an open source license.
As always, please file bugs on the GitHub issue tracker or submit questions to our mailing list by emailing velociraptor-discuss@googlegroups.com. You can also chat with us directly on our discord server.
Learn more about Velociraptor by visiting any of our web and social media channels below:
A Russian bank tells its customers to stop installing security updates, an Apple employee ends up in hot water, and learn our tips to avoid being virtually kidnapped.
All this and much more is discussed in the latest edition of the award-winning "Smashing Security" podcast by computer security veterans Graham Cluley and Carole Theriault, joined this week by Anna Brading.
![[Security Nation] Steve Micallef of SpiderFoot on Open-Source Intelligence](https://blog.rapid7.com/content/images/2022/06/security_nation_logo-1.jpg){kind=logo}
![[Security Nation] Steve Micallef of SpiderFoot on Open-Source Intelligence](https://blog.rapid7.com/content/images/2022/06/75g5hFQF_400x400.jpeg)
![[Security Nation] Phillip Maddux on HoneyDB, the Open-Source Honeypot Data Project](https://blog.rapid7.com/content/images/2022/06/security_nation_logo.jpg){kind=logo}
![[Security Nation] Phillip Maddux on HoneyDB, the Open-Source Honeypot Data Project](https://blog.rapid7.com/content/images/2022/06/image1-1.png)
![[Security Nation] Jim O’Gorman and g0tmi1k on Kali Linux](https://blog.rapid7.com/content/images/2022/05/security_nation_logo--1-.jpg){kind=logo}
![[Security Nation] Jim O’Gorman and g0tmi1k on Kali Linux](https://blog.rapid7.com/content/images/2022/05/Jim-800x800.jpg)
![[Security Nation] Jim O’Gorman and g0tmi1k on Kali Linux](https://blog.rapid7.com/content/images/2022/05/image1.jpg)
