{"id":8036,"date":"2025-10-30T04:03:39","date_gmt":"2025-10-30T04:03:39","guid":{"rendered":"https:\/\/serisec.com\/index.php\/2025\/10\/30\/32432\/"},"modified":"2025-10-30T04:03:39","modified_gmt":"2025-10-30T04:03:39","slug":"32432","status":"publish","type":"post","link":"https:\/\/serisec.com\/index.php\/2025\/10\/30\/32432\/","title":{"rendered":"How to collect memory-only filesystems on Linux systems, (Wed, Oct 29th)"},"content":{"rendered":"<p>    How to collect memory-only filesystems on Linux systems, (Wed, Oct 29th)<br \/>\n \t<BR><br \/>\n<BR><\/BR><br \/>\n    <!-- no image --><br \/>\n \t<BR><br \/>\n<BR><\/BR><\/p>\n<div>\n<p>I&#8217;ve been doing Unix\/Linux IR and Forensics for a long time. I logged into a Unix system for the first time in 1983. That&#8217;s one of the reasons I love teaching <a href=\"https:\/\/www.sans.org\/cyber-security-courses\/linux-threat-hunting-incident-response\">FOR577<\/a>[1], because I have stories that go back to before some of my students were even born that are still relevant today.<\/p>\n<p>In recent years, I&#8217;ve noticed a lot of attackers try to hide their tools or stage their data exfiltration in memory-only filesystems like <span style=\"font-family:Courier New,Courier,monospace;\">\/dev\/shm<\/span> or other <span style=\"font-family:Courier New,Courier,monospace;\">tmpfs<\/span><span style=\"font-family:Lucida Sans Unicode,Lucida Grande,sans-serif;\"> <\/span>locations.<\/p>\n<p><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/isc.sans.edu\/diaryimages\/images\/2025-10-29%252012_37_22-newton-ovpn%2520-%2520SecureCRT.png?ssl=1\" style=\"width: 1471px; height: 524px;\"><\/p>\n<p>Unfortunately, you can&#8217;t just <span style=\"font-family:Courier New,Courier,monospace;\">dd<\/span> these tmpfs filesystems. There is no block device backing it that will let you take a forensically sound image. So, if I want to get all of the metadata and the contents of any files the attacker may have stashed there, I&#8217;m going to need to try something else. Fortunately, after thinking about it a bit, I came up with a method that worked for me. I even talked it over briefly with Hal Pomeranz and we couldn&#8217;t come up with anything better. When I was thinking about this about a year ago, I did a quick Google search and didn&#8217;t see anyone else having talked about this, but I&#8217;d be surprised if others haven&#8217;t come up with the same idea.<\/p>\n<p>The basic idea is to first collect the metadata (inode contents), then collect the file contents, since doing it in the other order would cause the access timestamp in the inode to be updated. Since I came up with this technique, I&#8217;ve used it on dozens (probably 100+) of systems with pretty good success. I have run into a handful that didn&#8217;t have the <span style=\"font-family:Courier New,Courier,monospace;\">stat<\/span> command, so I could only collect the contents, but not the inode metadata. You deal with what the system has available.<\/p>\n<p style=\"margin-left: 40px;\"><code>#\u00a0find \/dev\/shm -exec $(which stat) -c '0|%N|%i|%A|%u|%g|%s|%X|%Y|%Z|%W' {} ; | sed -e 's\/|W$\/|0\/' -e s'\/|?$\/|0\/' | ssh foo@system \"cat - &gt; $(hostname)-dev-shm-bodyfile\"<\/code><\/p>\n<p>This commandline\u00a0will use the <span style=\"font-family:Courier New,Courier,monospace;\">find<\/span> command to walk the <span style=\"font-family:Courier New,Courier,monospace;\">tmpfs<\/span> filesystem (in this example, <span style=\"font-family:Courier New,Courier,monospace;\">\/dev\/shm<\/span>) and run the <span style=\"font-family:Courier New,Courier,monospace;\">stat<\/span> command against every file and directory it finds. The <span style=\"font-family:Courier New,Courier,monospace;\">sed<\/span> command is to compensate for versions of the <span style=\"font-family:Courier New,Courier,monospace;\">stat<\/span> command that don&#8217;t understand the <span style=\"font-family:Courier New,Courier,monospace;\">%W<\/span> format string to print out the creation time (b-time). Then, I pipe the output to <span style=\"font-family:Courier New,Courier,monospace;\">ssh<\/span> to send to a remote system where I collect the evidence (you could, of course, save it off to a USB drive or some other mounted filesystem if you wish).\u00a0Versions of the Linux\u00a0<span style=\"font-family:Courier New,Courier,monospace;\">coreutils<\/span> package prior to, I believe, version 8.32 don&#8217;t have\/use the <span style=\"font-family:Courier New,Courier,monospace;\">statx()<\/span> system call to access the creation timestamp even though it is usually in the inode on recent systems. The output is in bodyfile format which can then be fed to the <span style=\"font-family:Courier New,Courier,monospace;\">mactime<\/span> program from <a href=\"https:\/\/sleuthkit.org\/\">The Sleuth Kit (TSK)<\/a>[2] that can convert it into the classic filesystem timeline format.<\/p>\n<p>Once the metadata is collected, it is safe to collect the file contents (understanding that some file contents may have changed in the interval between the commands). The command that I usually use is as follows.<\/p>\n<p style=\"margin-left: 40px;\"><code>#\u00a0<\/code>find \/dev\/shm\u00a0-type f -print | tar czvO -T &#8211; | ssh foo@system &#8220;cat &#8211; &gt; $(hostname)-dev-shm-fs.tgz&#8221;<\/p>\n<p>This will use the <span style=\"font-family:Courier New,Courier,monospace;\">find<\/span> command to print out the names of all of the regular files (I&#8217;ve never found a need to collect any other type) and pipe them to the <span style=\"font-family:Lucida Sans Unicode,Lucida Grande,sans-serif;\">tar<\/span> command which will collect the contents (printing out the file names to stderr so I can see the progress) and output the (compressed because of the <span style=\"font-family:Courier New,Courier,monospace;\">z<\/span> switch) tarball to stdout to be piped to <span style=\"font-family:Courier New,Courier,monospace;\">ssh<\/span> to be collected on the same collection system. I then hash the tarball on the receiving system. I could, maybe should, hash the files on the live system I&#8217;m collecting from, but to this point, it hasn&#8217;t been an issue.<\/p>\n<p>As mentioned above, I&#8217;ve used this technique and many systems and even had it work on other Unix-ish systems like Juniper routers (that run FreeBSD under the hood) and even an ancient Solaris 9 system or two.<\/p>\n<p>My last run of FOR577 for the year is in a few weeks at <a href=\"https:\/\/www.sans.org\/cyber-security-training-events\/dfircon-miami-2025\">DFIRCon<\/a>[3] in Miami where we will talk about this technique and lots of others. Join me.<\/p>\n<p>References:<\/p>\n<p>[1]\u00a0<a href=\"https:\/\/www.sans.org\/cyber-security-courses\/linux-threat-hunting-incident-response\">https:\/\/www.sans.org\/cyber-security-courses\/linux-threat-hunting-incident-response<\/a><\/p>\n<p>[2]\u00a0<a href=\"https:\/\/sleuthkit.org\/\">https:\/\/sleuthkit.org\/<\/a><\/p>\n<p>[3]\u00a0<a href=\"https:\/\/www.sans.org\/cyber-security-training-events\/dfircon-miami-2025\">https:\/\/www.sans.org\/cyber-security-training-events\/dfircon-miami-2025<\/a><\/p>\n<p>&#8212;&#8212;&#8212;&#8212;&#8212;<br \/>\nJim Clausing, GIAC GSE #26<br \/>\njclausing &#8211;at&#8211; isc [dot] sans (dot) edu<\/p>\n<p> (c) SANS Internet Storm Center. https:\/\/isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.<\/p><\/div>\n<p> \t<BR><br \/>\n <BR><\/BR><\/p>\n<p> \t<BR><br \/>\n<BR><\/BR><br \/>\n<a href=\"https:\/\/isc.sans.edu\/diary\/rss\/32432\">Go to isc.sans.edu<\/a><br \/>\n \t<BR><br \/>\n <BR><\/BR><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to collect memory-only filesystems on Linux systems, (Wed, Oct 29th) I&#8217;ve been doing Unix\/Linux IR and Forensics for a long time. I logged into a Unix system for the first time in 1983. That&#8217;s one of the reasons I love teaching FOR577[1], because I have stories that go back to before some of my [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[56],"tags":[69],"class_list":["post-8036","post","type-post","status-publish","format-standard","hentry","category-isc-sans-edu","tag-isc-sans-edu"],"_links":{"self":[{"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/posts\/8036"}],"collection":[{"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/comments?post=8036"}],"version-history":[{"count":0,"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/posts\/8036\/revisions"}],"wp:attachment":[{"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/media?parent=8036"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/categories?post=8036"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/serisec.com\/index.php\/wp-json\/wp\/v2\/tags?post=8036"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}