Shibe Drill's Website

Cybersecurity Is Dead (For Now)

shibedrill1@gmail.com — Sat, 13 Dec 2025 05:00:00 GMT

Cybersecurity is dead- well, has been dead for a while- and will stay that way until the way we design and maintain our favorite software changes fundamentally.

So What Do I Mean By This?

Cybersecurity is a field which, at least in the industry, consists of learning a few very basic concepts, applying them to your employer's existing tech in a test environment, and then spending a few months engaging in bureaucratic sparring matches with your superiors until your hardening proposals are either rejected or used as an impetus to fire you. So despite the fact that all our favorite software and hardware is deeply flawed, very little is even done to keep it as secure as it presently is- much less to harden it beyond its current state. Security is highly technical, difficult, and expensive, so nobody wants to acknowledge the work it takes to make a secure system, and people would rather talk about vibes-based solutions or trivial changes like security baselines.

The majority of the platforms and design patterns we use today have ancient lineages, many of which lead back to the 1960s or even earlier. The PC platform is a Frankenstein's Monster of modern monkey-wrench solutions, bolted onto the cold, bloated corpse of the IBM PC from 1981. The Linux Kernel began in 1991, when Linus Torvalds rewrote the Unix Kernel, which was invented in 1969 as a major improvement of Multics which began development in 1964. The NT kernel's ancestor, QDOS, is a distant child of the OS/8 operating system from 1971. Everything we use is written in C, a language from 1969, whose design roots date back to CPL in 1963. MacOS also uses a Unix-like kernel with similarly outdated roots.

Huge amounts of modern digital infrastructure were built on the concept of Linux "distributions", where a bunch of unrelated and disconnected parties maintain totally separate software projects which are then shipped downstream to another set of incommunicado developers who "package" said software for use in an OS installation. The interfaces between these programs are poorly defined, often archaic, and frequently buggy to the point of insecurity. The disconnect between developers, packagers, end users, and other developers ensures that solving major bugs with security implications take more time than necessary. Meanwhile, what remains is based on Windows, and Microsoft is too busy trying to eke out a little profit from its AI expenditure to bother making security improvements to their terribly dated OS paradigm. Their goal is to monetize the Windows platform by selling data to brokers and subscriptions to end users, and any kind of API breakage makes the lives of developers harder- which narrows the profit margins, as less software becomes available for the already struggling platform as it continues to change.

So instead of doing anything that would majorly improve the world of cybersecurity, a whole industry was built on identifying the weaknesses in the tech we already have, and figuring out ways to hammer them flat without addressing the systemic issues that underly them. We already have Windows and Linux, so what point is there in making something better? The sunk cost fallacy rears its ugly head at the slightest hint of leaving tradition behind. Software, hardware, and platforms must be profitable before they can be allowed to exist, which obviously presents a big chicken-and-egg problem. Something which doesn't exist cannot be profitable (unless it has shit-loads of funding and hype from venture capital investors, like AI).

Notice how good things cannot exist because they're not profitable until they're mature, but the tech industry is bending over backwards to squeeze profits out of a technology that hasn't proven a single iota of viability but has been the subject of an economic bubble the likes of which hasn't been since since 2001?

And this is the crux of our problem.

How Our Current Systems Fall Short

Legacy compatibility is a big part of why technology today lacks security, due to the effort involved in porting anything to newer systems. The proper way around this is to use forwards-compatible API structures and adapters, but the APIs weren't designed to be forward-compatible in the first place, so we're back at square 1.

Kernel Design

Today's kernels are majority monolithic kernel designs. This means that tons of functionality runs at the highest privilege level in the processor, which is Ring 0 on x86. Things like networking, filesystem drivers, and graphics all run within the kernel, either hardcoded or loaded dynamically as kernel modules. However, this design introduces far too much attack surface within the most privileged part of any computer system!!! Why is that?

Processors, as commanded to do so by the operating system, use special hardware such as the memory management unit (MMU) to enforce hard boundaries between processes. For example, one process may not access another process's address space unless it's explicitly mapped by the kernel. This is great, since each process can be distrusted around the others, and operate in isolation. But adding features directly to the kernel, so that any piece of code can influence any other piece of code, defeats this entirely!!! Why would we design such excellent isolation systems, only to throw it all away by running millions of lines of user-facing code outside the security boundaries the kernel is supposed to enforce?

The opposite of a monolithic kernel is a microkernel. In a microkernel, only things such as scheduling and memory/process management run in the kernel. The rest of the OS's functionality runs unprivileged as a process, with memory access fully mediated by the MMU. Now, the GPU driver cannot access all of physical memory- only the pages that actually lead to the GPU! Microkernels also have a far smaller maintenance burden, and are more flexible, as more software can be modified without a reboot. seL4 is a formally verified microkernel that meets modern security requirements, and exceeds every other extant kernel in almost every category.

Access Control Models

Our current systems rely on old design patterns and paradigms that were created long before some of the most important security concepts of our day were formulated. For example, every operating system kernel available for general desktop, laptop, handheld, or mobile use is based on a kernel that uses a role-based access control model (Mach, Linux, and NT). This means that access to resources by some entity or process is specifically governed by a set of quantitative attributes ascribed to that object, and modulated through things such as access control lists (which provide their own quantitative restrictions). In all these systems, if you create a process as user_A, then that process inherits all the rights that user_A has, even if it's only meant to access one file. So you can run into things such as the Confused Deputy problem, which occurs when a privileged process is tricked into misusing its privilege. One applicable example is that time a Steam install script malfunctioned and wiped a user's whole home directory, because it had all the same rights as the user and failed to check if a certain path was empty.

The proper way around this unsolvable problem with role-based access control is to use capabilities, which are fine-grain "keys" to individual objects, rather than ranks or roles which signify groups of objects. In a capability system, you tell a process to open a door and hand it a key. In a role system, you tell a process to open a door and hand it a fully loaded breaching shotgun.

Packaging

Like I mentioned earlier, another major issue with today's software is the way it's distributed and packaged. Linux software developers rely on third parties to integrate their software with the rest of the software available within the ecosystem. If there's a security misconfiguration or an API incompatibility, there's always some degree of back-and-forth between the packagers and developers until it's even addressed. Hence, there's constant clashing between software developers and third parties who package the software improperly. Just look at the fiasco regarding OBS Studio and Fedora's packaging team.

Not only is the way it's distributed and packaged directly damaging to security, but the way software gets installed on Linux is also a major hole in its nonexistent security model. Every single program and package in the system is part of the same package state, so any modifications require the highest privilege level possible, since dependency conflicts or transaction requests could even modify the kernel or the package manager itself. Things have improved with Flatpak, but Flatpak's sandboxing is questionable and overly permissive, and the base system is still an amalgam of packages which requires root to modify in a useful way.

Userspace software suites and kernels should be developed together, as one project, tightly coupled, and shipped as one single immutable product. This can be easily enforced using harware for a completely locked-down experience which prohibits a broken or invalid system state, including malicious tampering or accidental damage from an overly privileged user. User software should be forced to rely on a single, well-known, highly restrictive API which is enforced by more privileged software. I talked about this at length in another post, about how these ideas could be better applied to Linux for more secure distributions.

Hardware Security

Think of how stupid it would be if, despite all these mitigations, someone could just modify your PC's kernel image or BIOS to include malicious code. Secure Boot helps, if your system is properly designed, manufactured, and configured. But if any kind of privileged code is unsigned or is not checked for integrity- such as your processor's built-in firmware or your UEFI firmware- the whole security model falls apart. This is why I can't stand people who willingly disable their system's security coprocessors or flash custom firmware in the name of "ditching proprietary software" or security, as it often makes security far, far worse.

Ultimately, the x86 boot chain is poorly secured (see my above remark on the dead, yet still beaten, horse of the IBM PC). Most firmware implementations are only secured by a password, which is trivial to lift from NVRAM and modify. Trusted Platform Modules often transmit data on unencrypted buses. Most systems don't disable writing to sensitive regions containing important security firmware. And because securing a system properly is very expensive, almost no OEMS bother to do so. The Linux Vendor Firmware Service reports that very few laptops or PCs available on the market today meet even the most basic security requirements, with trivial, glaring issues such as unsigned BIOS firmware or no support for Secure Boot. The only regions where hardware security is being even slightly innovated are the mobile and console industries. So far, though, only Google and their Pixel phones have proven themselves as secure platforms. Even Nintendo, with all their might, can't figure out how to deterr custom firmware enthusiasts.

The Ways People Think They're Helping

None of these issues are actively talked about in mainstream cybersecurity discourse. For the most part, two things are mentioned: Tracking and "Free Open-Source Software". This seems to apply solely to the topic of surveillance and data exfiltration: "Anything that tracks me is bad, so I should make it impossible to track me. Anything closed-source could track me without knowing so it all must be bad!". This is obviously not true, on both fronts, but it's used as a talking point by open-source pundits who shill for false solutions such as "de-Googling" and using strictly open source software.

Tracking is a recurring topic in security discourse, and it mostly boils down to "I don't want anyone to even know I visited their website". Modern anti-fingerprinting and tracking resistance techniques used by actually secure software, such as the Vanadium browser from GrapheneOS, minimize identifying data in interactions with untrusted servers. But often, "privacy enthusiasts" will go out of their way to strip their outgoing requests of more data than is needed, which can make them stand out like a sore pimple on an itchy back. The truth of the matter is that your goal is to blend in as well as possible with the people around you. ISPs and web hosts are going to notice that some weirdo is browsing their site using Tor and running a billion extensions. They're not going to pay much attention to someone who uses a fairly common user-agent and exhibits no special or stand-out behavior. Blending in is a matter of being uninteresting. Don't buy a sports car if you don't want the cops to pull you over.

With regards to closed source software, the concern appears to be auditability. People believe that if they cannot understand what the software is doing, then it must be evil. However, auditability with regards to software behavior is highly limited. You can't say for sure what a program will do, even if it is open source, because that's the fucking Halting Problem which is proven to be impossible to solve. The only exception is if the program is formally, mathematically verified to be correct, and most software does not use verified compilers. In fact, because of this, your compiler could be backdoored and infecting everything you've ever built from source! Take that, Gentoo nerds!

Furthermore, some people extend this distrust to the hardware, and go as far as to disable hardware features such as the Intel Management Engine or the AMD Platform Security Processor since these "could be spyware!" and run closed-source firmware. But of course, this is fruitless, as it is impossible to know what the processor itself is doing- unless you design and build one yourself, without a single integrated circuit or chip, or you're able to probe literally anywhere within any circuit within the silicon die at any given time. Every single processor on Earth runs some amount of closed-source firmware or microcode. It's impossible to escape.

Computing is now a matter of social trust. Before you use any piece of software or hardware, you must trust the entity that produced it. Which is why things like cryptographic signing and attestation matter now- while you cannot be sure of what the code will do, you have a promise from its producer that it will act within a certain specification. The rest is up to God, or whoever wrote your C compiler.

Ultimately, you can't avoid closed source software or tracking. Securing these elements has become a game of minimization and optimization. Some telemetry is good, some is bad. Some closed source software is good, even necessary. The trick is not to eliminate either, it's to find the configuration that maximizes how much you can reasonably assert about the system's correctness and behavior.

Final Thoughts

How do we fix this catastrophe?

Software product development should be centralized, not decentralized.
Software should be modular, but developed as one product.
Software should rely heavily on well-defined, secure standards.
Software should be made by nonprofits with clear goals and strong governance.
Companies should be more willing to pay up for good security.
End users and developers should be more ready to embrace change.
People should be willing to admit that our current security concepts are outdated, and in dire need of an overhaul.

If technology in the industry cannot evolve to meet the demands of the times, and if people cannot learn that the way we're doing things is royally broken, there is no hope for cybersecurity. As it is, cybersecurity is dead and buried.

It's up to us- the newest generation of freaks, perverts, and computer jerks- to make it right.

Rust & OS Dev: Part 0

shibedrill1@gmail.com — Tue, 08 Jul 2025 05:00:00 GMT

The Rust language is, without a doubt, my favorite programming language of all time. Strongly typed, great build system, memory safe, all that jazz. But how good is it for building operating systems? Only one way to find out.

Recently, I resumed work on Gila, a microkernel OS I began writing in Rust as I was finishing my Operating Systems course in college. It's not in a useful state yet, but regardless, I've learned a lot about Rust's limitations as a language and an ecosystem. In this post, I'll talk about the areas where Rust excels, and the areas that it lacks in- whether that's because of the application of the language, or the language itself.

Build System Features

Cargo, Rust's build system, is the single most important reason I adopted Rust as my go-to programming language. Dependency management is stupidly easy, and I have never had to fiddle around with build scripts or linker arguments. This is obviously complicated when building an OS kernel, as certain linker settings need to be tweaked for the kernel to function with the bootloader, but I was able to find resources that detailed specifically what needed to be done.

Additionally, Cargo's conditional compilation system allows me to gate code behind a #[cfg] statement, which I can use to compile certain architecture specific backends to support an architecture agnostic interface. This makes for far less complexity in the central calling code, as nothing changes if the code is compiled for a different architecture. I also made certain parts of the OS kernel into "features", which could be deactivated at compile-time to save space and reduce attack surface. There's no point in compiling UEFI support into a kernel that will only be booted on BIOS-only systems. (I did this because I wanted to port Gila to IA32 some day, but couldn't due to a core library not supporting it.) In fact, you can even avoid compiling in dependencies for inactive features, by marking them as optional.

Memory Safety

Rust's memory safety model ensures that no two parts of a program may simultaneously obtain mutable access to the same value, and there cannot be a mix of mutable and immutable references to the same value. Also, Rust does not permit mutable static values. Allowing for this would cause race conditions, use-after-free bugs, and TOCTOU bugs. However, in some parts of my OS, I did need multiple parts of the code to access and mutate the same static value- for example, the logger is a global lazy static struct which needs to be mutably referenceable by every part of the kernel to log data to the serial terminal. To get around the limitations of the borrow checker, I made the logger into a mutex: a structure that can be "locked" to temporarily perform some mutation, and then "unlocked" once it goes out of scope.

Unfortunately, widespread use of mutexes can present another problem: deadlocks. If a piece of code attempts to gain lock on another resource, but that resource is being held by some other part of the code which won't release the lock, the section that's trying to lock it will never gain lock & continue executing. To avoid this, I have to be careful about where I lock mutexes, and whether it's possible for a piece of code to attempt to lock a resource that some other part of the code can lock.

We've Been Doing Linux All Wrong

shibedrill1@gmail.com — Mon, 21 Apr 2025 05:00:00 GMT

Ever since I began using Linux, I've always asked myself why there were so many issues with the way software is packaged, delivered, and managed. It's led me to think about the ways people have tried to fix this problem, and here, I provide my own thoughts on what I believe Linux packaging should begin embracing and what it needs to leave behind.

Linux Packaging Sucks

I'd like to ask a rhetorical question to begin this thought exercise: Why do I need to become a local administrator (typically root, which is pretty bad for a number of reasons) to make any kinds of changes to packages on a traditional Linux system, using its native package manager?

The answer: Local admin permissions are needed because package management software- such as dpkg, pacman, and dnf- make zero distinction between software packages associated with the system, and those not associated with the system. To the package manager, there is no difference between the packages spotify (the music player) and linux (the operating system kernel). Hence, the user installing a package must contend with the package manager's solution to their request, possibly influencing system-critical packages in the process. Any transaction has the potential to modify some system component due to a dependency conflict. The result of this is that sometimes, installing strictly user-facing software can cause the package manager to behave unpredictably, and irreversibly damage the system, such as that one time installing Steam on Pop!_OS would uninstall your entire desktop environment.

In response to this, a number of technologies have sprung forth. The most notable ones I can think of are backup systems like Timeshift, or BTRFS snapshots, as well as declarative systems with atomic updates like NixOS. Instead of separating user and system software, these solutions embrace the integration, and seek to allow a means by which to undo an undesirable change resulting in a broken or degraded system.

Maladaptive Solutions

But I do not think that this is the right response to the problem at hand, which is the fact that it makes little sense to allow user software and system software to be a part of the same dependency web, and be handled as one monolithic set of functionalities. It should not be possible for "Please install steam" to have the side effect of "Okay, your entire desktop environment has been destroyed now", much less should the development community be chasing for a way to recover from such a scenario. Installing a refrigerator in your house should not cause a dependency conflict with the load-bearing walls in your kitchen, especially if the only solution involves the mutually exclusive presence of either the fridge or the entire damn first floor. This is an understandable issue if the user was attempting to install a new infrastructural component, such as electrical or HVAC. But the role of the house is to provide a solid, reliable platform, on which additional user-facing accommodations and facilities may be furnished arbitrarily. Similarly, the role of an operating system is to provide a stable underlying system to handle hardware abstraction, user interfacing, and the bare minimum functionality needed to self-modify and self-update. Application software should be very loosely coupled with system software, wherever possible.

Principles

The concepts I'm about to outline are intended for a Linux user who does not often tinker with their system's core components. AKA, someone who needs a stable daily driver system. Once the system is properly configured to work with the hardware, no additional changes should really be needed.

Package management on Linux should define two domains: System software, and user software.

System software consists of a set of packages and configurations necessary to produce a functioning OS with the proper drivers and a desktop environment, as well as some means by which the user may install additional software on top of the OS- not inside or alongside it.
User software consists of anything that is not strictly critical for the abstractions provided by the kernel, desktop, and software management systems to work. This is an effective separation of duties.

Furthermore, the way in which these two domains are managed should differ:

System software should function on versioned sets of specific package versions, where all packages in the set are known to work together without issue.
- Updating should be a single process, whereby all the necessary files to instantiate the newest working set version are downloaded and configured at once.
- This software, being in a known good state (barring machine-specific errata), should be stored in a medium that the user normally does not have direct access to, and the kernel image may assume that this set of software is always valid, and its containing media is always available.
  - In short, it should have a dedicated partition. Since software is almost never added, this partition can be just small enough for the system, and remain that small throughout its lifespan.
User software should function on individual packages (applications or libraries), be available for the user to install, modify, and uninstall.
- It should be user-specific, and should not hinge heavily on libraries provided by the system.
- Libraries and shared content should be managed as additional user software, and dependency conflicts may be resolved by simply installing the fewest number of different versions of the dependency needed to satisfy all constraints- even if that number is greater than one. Modern computers have plenty of power to perform such calculations, and plenty of space to handle duplicate versions of dependencies.
- No amount of corruption or misconfiguration of user software should put the system in an unusable state, and it should always be possible to start fresh simply by purging all user software. This allows for system recovery in the event that the user's partition becomes corrupted or damaged, too.

In this model, user software depends on a well-defined set of interfaces with a small footprint, provided by a known-good system configuration. This is a stark contrast to the current model, where all software depends on any number of poorly defined versioned interfaces within different components of the operating system.

Ultimately, I think this philosophy should serve to guide Linux software development moving forwards. People with vastly more resources than me already have had the same idea- projects like Universal Blue, based on Fedora Atomic, and NixOS, in conjunction with user software technologies like Flatpak- have moved desktop Linux further and further towards a visualization of this set of ideas: an immutable host operating system, wherein the user can trivially install, modify, and uninstall any program not critical to the system's operation. Modifications to the system are versioned and applied atomically, to maintain some known-good configuration and keep the system in a usable state. And to maintain flexibility and compatibility with more standard package management models, all that is needed is a containerized environment in which a typical Linux distro can be installed and modified arbitrarily, without ever modifying the system software.

Emergent Properties

So, let's assume we have a system that subscribes to this philosophy. What kind of interesting properties emerge from this?

Reproducibility And Reliability

Since most systems will be running the same combination of system software, with the exception of hardware-specific software like drivers, it should be relatively easy to identify and trace problems within the system. A configuration can be easily replicated on one machine from another, and a configuration that works on one machine should work on any other- and the inverse would apply to a broken configuration. Preventing the user from directly modifying this configuration maintains this reproducibility.

Security

Disallowing modification of the system in ways that are not well-defined (or disallowing it altogether) prevents the user or any malicious actors from introducing security holes into the software or its configuration. If the system partition is never modified, dm-verity can be configured so that the kernel image will refuse to boot a tampered or corrupted system, and combining this with Secure Boot extends this trust down to the hardware level (on a well-configured, well-made computer). Updates and system images can be signed by a central authority to ensure that no tampering has occurred along the wire. Applications being installed per-user allows for privacy and separation between users, possibly eliminating information side channel attacks.

Flexibility

Separating user software from system software allows the installation of more types of software, as well as versions of applications that may be too old for their native packaging dependencies to exist at the right versions. It increases reliability, in that application functionality is not tied directly to system state. Having applications installed per-user means that different users have fully customized application profiles, including only the software that user needs. Plus, users can locate their home directories on the network, and have the same software selection available anywhere as long as their directory is mounted on a machine running a similarly designed OS.

This Sounds Pretty Cool (Conclusion)

I agree! And like I mentioned, this paradigm isn't nonexistent in the OS space. Some operating systems perform trivial separation of user and system software, like Windows relegating user software to the Program Files directory (Really? Windows got this right? This one stings). Many game consoles employ this paradigm for security and DRM purposes. Universal Blue and other immutable Linux distros embrace this concept, maintaining a stable base system while allowing a large amount of user freedom. Android and its derivatives take this idea to the extreme, with a totally immutable system partition protected by a hardware-based root-of-trust, and applications installed exclusively per-user. And recently, systemd released its own immutable OS, called ParticleOS.

I used to dislike technologies like Flatpak, and be very gung-ho about native packaging being superior. But as the tech has improved, and its upsides became ever more obvious, my opinion has changed wildly. Now, I'm glad to see Linux moving towards a generally more usable, stable, and flexible software management philosophy. I think the issues with the old way of doing things are a big part of what's holding back widespread adoption of Linux on desktops, laptops, and handhelds. And I think immutable systems are a fantastic starting point for introducing a more well-defined security model to desktop Linux- to that end, I started the HalogenOS project to realize my own implementation of an optimally secure system, based on immutability and software management that empowers end users instead of burdening them with the responsibilities of the superuser, or binding them to the archaic rules of typical package managers. I hope one day, systems such as these will allow the average user to run Linux in a way that's stupid easy, secure, reliable, and fun.

If You Want To Try (Further Reading)

Here are a few Linux distributions that employ immutability in the manner I described, while providing a means by which the user can still modify application software:

ParticleOS, a tightly integrated OS based on systemd
Universal Blue, based on Fedora Atomic
Bazzite, based on Universal Blue and specialized for gaming
carbonOS, an independent immutable distro with security in mind
chromeOS, an operating system designed for seamless UX on Chromebooks

Enjoy!

GrapheneOS: the Good, the Bad, and the Ugly

shibedrill1@gmail.com — Mon, 17 Feb 2025 05:00:00 GMT

Recently, I switched my phone- a used Google Pixel 7 Pro- from LineageOS to GrapheneOS, in pursuit of a more secure mobile operating system. I discovered many things in the process, including that I miss Google Pay less than I thought I would.

For a long time, I was running LineageOS 21 on my P7P. I liked the idea of having a nonstandard, fully open source distribution of Android, but I had qualms with it- including the fact that, despite being a relatively trivial task to implement, the bootloader was not re-lockable. Growing slightly paranoid that maybe someone could rootkit my phone over USB without even unlocking it, eventually I decided to try out this operating system I had heard a lot about, called GrapheneOS.

GrapheneOS is a modification of Android that runs Google's Play Services in a sandbox, just like every other app. They don't get privileged access to the operating system, and can be uninstalled at any time. It also includes better permissions handling, including contact scopes and storage scopes, and various hardening features and exploit protections. The features that totally sold me were the lockable bootloader and the "Duress PIN", which wipes the phone if it's input.

The Good

Installing GrapheneOS is actually dead simple. The hardest part was putting my phone in fastboot mode, which I could have accomplished from the Kitsune app, but instead I used the button combination. Once I got it into fastboot mode, I plugged my phone in over USB, unzipped the release folder I had downloaded, and ran the flashing script. It automatically flashed all of the necessary files to the correct partitions and re-locked the bootloader, just like that. It's even simpler if you use Chrome or any browser that supports WebUSB.

Upon reboot, I completed the setup, inlcuding disabling OEM unlocking- and was greeted with a totally blank black home screen, with a very sparse icon selection. I faltered a little bit, because I wasn't totally sure what to do. But after checking the apps drawer, I found an "App Store" app that came installed by default. In it, the Google Play Store and Android Auto were available as downloadable apps. I rely heavily on Android Auto for navigation on my car's head unit, and I get most of my apps from the Play Store, so I opted to install them both. I did have to configure Android Auto's permissions so it could work properly, but after that, everything worked perfectly.

Like I mentioned above, there's not many apps installed by default. There's not many apps in the App Store, or in Accrescent, which is a secure app store for Android devices that can be installed via the App Store. From talking to the GrapheneOS community, I found that it's discouraged to use Play Store alternatives like Aurora Store, or alternate stores like Fdroid. There are security implications to obtaining apps from sources that perform minimal verification of their apps. Hence, the recommendation from them was to use the Play Store when needed, and to use Obtainium for anything with a GitHub release.

GrapheneOS's permissions are distrustful by default, and upon installation, all of an app's permissions are set to the defaults- even if the app was installed previously and granted more permissions. This likely prevents malicious apps from disguising as legitimate apps that were previously installed, and acquiring permissions that the non-malicious version had been granted. Apps still run in isolation, just like stock Android, and certain OS components are hardened to reduce attack surface, such as malloc- which has been replaced by hardened_malloc.

One feature I really like is the "Duress PIN". It's a PIN number or password that can be input at any system-generated PIN or password dialog, and once input, it irreversibly wipes the phone, in a process that cannot be interrupted. I also like that there's a lot of exploit protection features like reboots that trigger if you go a certain time without unlocking the phone. Overall, it makes me fairly confident that if my phone were stolen by some highly skilled and funded adversary, it would be very difficult for them to get any kind of data off of it.

Graphene also includes "profiles"- a way of putting apps into separate "personas". They're not any more isolated from one another, but the idea is to isolate them in the user's mind, and to make it easier to keep "lives" separate- work, school, personal, and online lives, for examples.

The Bad

Since Google's anticompetitive behavior allows them to globally dictate the usability of Android as an operating system, apps that use Google's Play Integrity API will likely not function properly on GrapheneOS. Google Pay is included in this, so tap-to-pay is out of the option for anyone who relies on Google Pay, rather than a regional banking app. Some apps are unaffected, though, like Duo Mobile (a 2FA app), which refused to function under rooted LineageOS but is totally fine under Graphene. The OS produces a notification every time an app uses the API.

And speaking of app compatibility, some apps may crash due to the exploit protection tweaks Graphene uses. Typically, it's for the best that users avoid disabling exploit protection for apps, but in my situation I had no choice- the app that kept crashing was my banking app, and their website is just dogshit and won't let me log in. Disabling hardened malloc fixed the crash for me.

Since, by default, Google Play Services are not granted location permission, all requests to the Play Services API are routed first theough GrapheneOS's own location services- which are less accurate than the Play Services location service. I found that location would often not function if I was indoors, and that I needed line of sight to the sky for satellite pings to register. It is possible and allowed to enable Google's location services, but this obviously means that Google is then able to see your location. The included location services work when I'm in the car, so I don't find myself needing anything more accurate, especially if it would mean Google gets to know where I am.

The Ugly

I like some of the default apps in GrapheneOS. Vanadium, the browser, for instance- it's very well hardened, and employs the secure system webview. I don't use it since I mostly prefer Firefox, but the option is there for people who like Chromium browsers. However, most of the default apps are totally unchanged from the Android Open Source Project, and the UI style is certainly showing its age. The clock, gallery, and messaging apps are real eyesores. The colors are obnoxious and remind me of Android 7.0 Nougat. The files app is just a slight improvement over the Android file picker dialog (which I've always thought looked dated). I've been told that better default apps are on the way, which would be really nice.

Graphene also doesn't include a lot of apps by default. It doesn't come with a music player or a calendar app, for examples. For music, I use Auxio, but I haven't found a nice calendar app so far. I know LineageOS had a nice one, but I'm not sure how much of that is based on the AOSP version, or if there even is an AOSP version. But the developers of GrapheneOS have stood pretty firm on their stance that it's better to include less system apps to reduce attack surface.

Conclusions

I found GrapheneOS easier to use than I thought I would. I don't miss Google Pay as badly as I thought I'd miss it, but erasing the muscle memory that makes me reach for my phone whenever I approach a payment terminal is proving difficult. Overall, I think it's a worthy trade-off for increased security. If I had to do this all over again, I think I would have bought a Pixel 8 Pro instead of the 7, since the 8 series has some cool hardware security features like Memory Tagging Extensions. But I bought this phone as an emergency purchase when I totally destroyed my Moto Z4, so I wasn't thinking too straight. I'm still happy with this setup, and I think I prefer GrapheneOS over Lineage, or stock Android as Google ships it. You have the option to use as much or as little of Google's software as you'd like, but you have significantly improved hardening in almost every respect.

Would I recommend GrapheneOS to someone who's not super into security, or doesn't own an unlockable Pixel? No. But would I recommend it to someone who already owns a compatible device and is willing to sacrifice just a little bit of convenience for some really cool features? Absolutely. If you own a Google Pixel phone with a bootloader that can be unlocked, and you think these features sound like something you want in a mobile OS, I think it's worth checking out for a little bit. And if you don't like it, you can always reflash stock Android.

Operational Security For The New-Age Antifascist

shibedrill1@gmail.com — Thu, 06 Feb 2025 05:00:00 GMT

With the rising threats of state-actor-backed censorship, surveillance, and exploitation, right now is a fantastic time to lean into a more cybersecurity focused mindset with regards to the tech we use every single day. But where do you start if you're not the typical Linux power-user?

Consider downloading this post, using the Print button or Ctrl+P and Print to PDF.

A quick foreword: This guide does not assume any preexisting knowledge with regards to Linux, cybersecurity, cryptography, or any other fields. This is strictly a layman's handbook. You do not need to install any specific operating system or buy any specific hardware for this guide to be useful to you. If you wanted a QubesOS review, maybe look elsewhere.

So. The United States elected a rather unsavory (cough, cough, FASCIST) figure to the Presidency, and his billionaire handler has begun literally breaking into government offices to install his own lackeys and backdoor federal data-handling systems. Websites about critical information are being taken down overnight, without warning. Who's to say this isn't the start of a larger campaign of censorship and information control that might eventually (directly) target civilians? There's already been reports that spyware vendors like Paragon do serve the United States as a customer. Us USAmericans need to step up our cybersecurity game in the face of the new age of fully-automated computerized fascism.

Prologue

This guide goes out to a lot of people. Namely, my friend Dev, who taught me how to use and embrace cryptography. It also goes out to some real life friends who wanted me to tell them more about cybersecurity, but weren't familiar with all the jargon.

This guide is all about reducing attack surface. The more things that can go wrong, the more things that can be leaked or lost or tampered, the greater your attack surface is. The recommendations here are not all-or-nothing. I intentionally kept things tame, because I want this to be accessible to people who have NO IDEA what the fuck security is all about, and aren't as willing to sacrifice conveniences as I am. Take whichever steps feel right to you, don't take the ones you don't feel confident about. Always consult the manual, don't do anything rash. Try to understand how everything affects your personal attack surface, and take steps to reduce it or map it out.

DISCLAIMER: This guide is not perfect! Some of the information in this guide is going to be wrong. I do not claim to be an expert, and I acknowledge that I have the capacity to make mistakes, even big ones. To the best of my knowledge at the time of writing, this guide is accurate, and representative of consensus best practices in the security world.

Some practical advice for you before we begin:

Use strong, unique passwords wherever possible. If you can't remember them, use a password manager. I use Bitwarden.
Do not ever download software from sources you do not trust.
When adopting new software, try to go for options that are open source.
Do not ever follow advice from strangers on the internet telling you to modify your system somehow. If you are uncomfortable, stop, step back, and look at where you are and what you've done. Take instructions only from sources you trust.
Security is a journey, not a destination. Do not chase "perfect". Anything that makes an adversary's life harder is worthwhile.
Never plug in random USBs. Never use airport phone chargers.

Part 1: Threat Models

The first step to a more secure lifestyle in the digital age is to identify the things you want to protect. Typically, this includes things like:

Chat logs and emails
Real-world identifying data
Photos and videos
Passwords, usernames, other credentials
Activities
Purchase histories
Location and location history
Credit cards and financial data
Web traffic and history
Search engine queries and history
Controlled data from a job or institution
Social media posts, likes, follows, and replies

And if you're one of the people who's a little less... "orthodox" in their Internet usage, or you happen to be some kind of journalist or whistleblower, your list might be a little longer:

Cryptographic keys
Cryptocurrency wallets and transaction logs
Encryption keys
Controlled or illegal data
Evidence or incriminating data
Censored information
Activities on illegal or monitored forums
Product invoices

Look at your digital life and create a list of the other things you want to protect- make sure to consider the services you use, and what kind of data they might have from you, both stored & temporarily. Think about how important each item and piece of data is, and begin brainstorming the possible consequences for the data being lost or leaked, what entities pose threats to this data, as well as measures which could possibly protect that data.

So what can be done? To determine what we can do to protect a piece of data, we have to look at the nature of that data. Do you continuously produce this data? Does this data ever travel across the internet? Looking at these factors, data can be classified into two types: Data at rest, and data in motion. Data at rest typically stays in one place, and does not travel across the internet frequently. Data in motion is produced constantly, and does often travel across the internet.

Part 2: Data At Rest

Protecting User Data

To protect data at rest, the best method is via encryption. Encryption can be used to obscure and protect the contents of an entire hard drive on a computer, the contents of a USB drive, or even individual files. When you access data on an encrypted drive, that drive must first be mounted & decrypted, and all the data on the drive is directly available in the OS as long as that drive remains mounted. For a full-disk encryption setup, this means that all the data on the drive is vulnerable once the machine is booted and the password is input. A good practice might be to secure highly sensitive data in an encrypted file or partition which is only ever mounted when the data inside needs to be accessed.

Most Android and iOS devices are encrypted by default. Your specific make/model might not support encryption, but almost all modern phones do. This encryption is backed by your phone's password or PIN, not your biometrics, so make sure you choose a strong password. Same as your computer, this encryption is rendered useless as soon as the phone boots and is unlocked for the first time, which is when the user data partition is decrypted.

Side note: You may want to consider disabling biometric login on your devices, as you can be legally compelled to give your biometrics over to the authorities.

To encrypt your computer's hard drive, you have quite a few options available to you. Windows can use BitLocker or VeraCrypt, Linux uses the Linux Unified Key Setup (LUKS), and macOS uses FileVault. Most of these solutions also allow you to create encrypted container files, which are individual files that act like encrypted disk volumes. The options for doing this on mobile phones are virtually nonexistant, though, so you will probably need a PC or mac for that. Apps that claim to encrypt or hide files are typically either falsely advertising or poorly implemented to the point where you might as well not be encrypting the data at all.

Tip: If you regularly use a USB drive to transfer data between your devices... Please encrypt it. And if you have a USB (or any kind of) drive that you've previously stored sensitive data on, it would be a good idea to securely erase it, as files are typically recoverable after deleting until all of the used blocks have been overwritten. Linux offers the shred command, as well as the good ol' dd if=/dev/urandom, but I am unfamiliar with whatever Windows or macOS alternatives might exist.

Securing data on a booted operating system is a little harder. It's important to set proper access controls on your files, so other users on a multi-user system cannot access it. Only give permissions to programs and users when it is strictly necessary. Use an antivirus such as Windows Defender (Windows) or ClamAV (Linux) to keep malware from spreading on a booted system.

Lastly, like I mentioned, encryption is useless once your system is booted. As such, you should never leave a booted system unattended, for any length of time. Running to the bathroom at a coffee shop? Shut it down fully. Put it in your bag if you want to prevent tampering or theft. Leaving your system running (and especially unlocked) while you're away from it leaves too much of an opportunity for nefarious deeds and defeats the point of encryption.

Tamper-Proofing Bootable Code

While encryption may protect data on the user partition of a computer, the kernel still needs to boot first to mount and decrypt that data. Hence, the operating system code itself is still vulnerable to modification, even on an encrypted machine. Secure Boot is a technology present in newer UEFI firmwares that aims to combat this by "signing" the bootable code, and refusing to boot if the signature is incorrect. This is mainly a tamper-evident mechanism to prevent users from unknowingly booting a kernel that is tainted with malware.

Again, mobile phone operating systems have their own implementation of this. Android has dm-verity, and iOS likely has something similar. The phone will simply refuse to boot if the system partition has been modified without the permission of the operating system, which would trigger the re-signing of the boot image.

Cold Storage

Another way to protect infrequently used data is via secure backups and airgapping. If you almost never access the data, you might want to put it into "cold storage", so to speak. The best ways to do this (in decreasing order of reliability) are tape drives, archival grade CDs, hard disk drives, and flash storage like USB flash drives and SSDs.

Method	Cost	Complexity	Reliability	Notes
Tape drives	$$$	High	High	Requires good climate control, and one extra tape for accounting and metadata.
Compact disks	$$	Low	High	Requires a CD drive that can burn. Also, CDs are only writable once.
Hard disk drives	$$	High	Medium	Hard drives have a habit of failing due to their mechanical nature. You also cannot ever drop them.
Flash cells	$	Low	Low	Flash cells naturally leak charge over time and have a very limited shelf life.

Before archiving or backing up a piece of data, consider: Do you need it? What would happen if you were to lose it? It is unlikely that you will need to store your full rip of every Mythbusters episode, but you might want to archive important financial records or ownership documents so you can reference them down the line. If this data is sensitive, you may consider encrypting it. To protect it against physical damage or theft, you might want to store it in a safe or a fireproof, floodproof document box or a safe. Make sure it's certified by a reputable organization, like the Underwriter's Laboratories.

Part 3: Data In Transit

Encryption In Motion

Encryption isn't purely for data at rest, though. Using public-key cryptography, all sorts of data can be encrypted in transit. You unknowingly use this technology every single day in the form of HTTPS, a public-key protocol used to encrypt web traffic. You can use PKC to encrypt emails, chat communications, files to transfer, and text messages sent over insecure channels. You can also use it to prove the identity of an individual, or the integrity of a file, via signing. Pretty Good Privacy, PGP, is one of the more commonly used cryptographic standards, and there does exist cross-platform software to handle it. The Mozilla Thunderbird email client supports creating and using PGP encryption within the application itself. Consider learning how to use PKC, and using services that implement their own cryptography, such as Signal.

Public-Key Cryptography is a means by which data is encrypted by a "public" key, and decrypted with a "private" key. This means you can share data across an insecure channel, without having to communicate a password across that channel. Data intended for Bob is encrypted using his public key, which results in it only being able to be decrypted using his corresponding private key, which only Bob will ever see.

Signing data cryptographically means using your private key to produce a value that can be verified using your public key. It would be impossible to produce a working signature for Jill's public key, without being in posession of Jill's private key as well. Therefore, a working signature proves that Jill is the real owner of the private key... all without divulging its actual contents. It also proves that the signed data originated from Jill.

If you're interested in using PKC for ensuring confidentiality or integrity of data in transport, you can either use Thunderbird as mentioned earlier, or use Kleopatra (available on Linux, as well as on Windows via GPG4win) to generate a keypair. Then you can send your public key to trusted parties, and import their public keys, to begin sending encrypted or signed data between each other.

Obscuring Internet Traffic

And now that we're talking about data in transit, it's time to discuss the matter of web browsing and web services. Search engines like Google, and websites like Facebook, collect and aggregate terabytes of data about their users, and it is well known that this data is not private. You can reduce your data footprint by using alternative services for searching, as well as refraining from using sites that do not respect user privacy. You can use extensions to block cookies and trackers, and alternative search engines like DuckDuckGo may provide marginally better privacy than Google or Bing. For the utmost privacy, you may want to consider using Tor Browser, which anonymizes network traffic by splitting it across many different network nodes. VPNs may also increase your privacy by obscuring your IP address and its location, but a VPN does not provide any more security with regards to the contents of your web traffic, unless you browse exclusively HTTP-only sites. If you are especially worried about your traffic being monitored, use public Wi-Fi networks with passwords set, in conjunction with Tor.

If you are looking for a privacy-respecting VPN, I suggest Mullvad.

Method	How It Works	Drawbacks
HTTPS	Encrypts web traffic	Domain names visible to ISP, network
VPN	Tunnels traffic elsewhere	Most VPN providers keep logs, not much more secure
Tor	Splits connection and routes elsewhere	Very, very slow

You Are The Data

You produce a lot of data as you move through the world. Your phone, your laptop, your credit card, even your car transmit data that can be used to identify you. Facial recognition technology is advanced enough to pick you out from a crowd of hundreds, and America's highway system is littered with automatic license plate readers which transmit your car's whereabouts to huge databases, timestamps included. There is data literally pouring off of you, all the time, and there's always ears listening for it.

One thing you can look at is what kinds of wireless technologies your devices use, and what kinds of information they transmit. Does your device work with Apple's tracker network? Do you have Wi-Fi roaming enabled? Does your phone have Bluetooth searching on all the time? These technologies, while convenient, make it easy for other entities to pinpoint your whereabouts at any given time. Wireless-capable devices have unique identifiers, called addresses, that can possibly identify their holders. It also goes without saying that if your phone automatically connects to a wireless network, that event is recorded somewhere. This applies to Wi-Fi, Bluetooth, and cellular data. If you want to be certain that your presence in a location is mostly undetected, either put your phone on airplane mode and disable all its wireless connectivity, or shut it down entirely. Your laptop may also transmit information while sleeping, so keep it shut down as long as it's not in use.

Which Means Your Face Is Data, Too

Have you ever been to an ATM? A self-checkout kiosk? A Coca-Cola Freestyle soda fountain? Have you ever seen a screen displaying ads outside a store, or displaying drinks on a drink cooler door? Well, let me tell you something. Anywhere there's a screen, there's a camera, and wherever there's a camera, it's pointed at your face. They have many purposes- cameras on ATMs and self-checkouts are for monitoring potential criminal use, and cameras on advertising displays and drink coolers gauge your reactions to the content on the screen to more finely-tune ad content. But regardless, the abundance of cameras ensures that your face is being perpetually transmitted to some unknown location, for uses which are not immediately clear or disclosed. One way to throw a wrench in the works is to wear a mask when in public. N95 masks and surgical masks work, but in the winter and fall, you can get away with cold-weather face coverings, such as scarves, balaclavas, ski masks, and neck gaiters. Just be careful not to run into a police officer who might shoot you dead because he felt "threatened" by you.

Dirty Money

It should go without saying that almost any form of electronic payment is traceable and trackable. If you intend to avoid leaving a paper trail while moving around in an urban area, then abstain from using cards, aside from cards that can be purchased anonymously with cash (such as Metro cards). With cards such as those, only buy in small amounts, and use and dispose of the entire thing before you turn towards home. If you're super paranoid, use gloves to ensure your fingerprints aren't left behind on the card or any point of payment.

One last thing: Any machine that is designed to dispense bills likely also keeps track of the serial numbers of every bill inserted or dispensed. Self checkout machines also keep a record of your face in conjunction with your purchase receipt, and whichever membership cards, credit cards, gift cards, or paper bills you used to pay for your items. Keep this in mind while buying items that may be potentially regulated in the future. Purchasing things such as birth control, Plan B, basically any other contraceptive, pregnancy tests, prescribed or OTC hormone medication, prepaid phones or SIM/eSIM cards, and banned or challenged books, might produce transaction metadata that could be used to identify you in the future. When buying these items, use cash that you've obtained from a cash drawer as change for a larger bill, and avoid using gift/membership/reward/credit/debit cards. If you want to be extra careful, only handle your "laundered" bills with gloves, and make your purchases at stores you don't usually visit, in conjunction with facial coverings to make you harder to recognize.

Social Media And Web Services

Social media is a huge dumping ground for tons of data that we produce. Snapchat harvests users' contacts, location, browsing habits, posting habits, and more. Almost all social media tracks its userbase. So reducing your data footprint is very much a matter of "how much do I want to give up?" and "how transparent is this platform?". Consider deleting posts from, and even deactivating/deleting, old social media profiles. Also consider auditing content posted to profiles you still use or don't intend to delete, to remove possibly identifying information. Keep in mind that images taken with your phone's camera can possibly contain the exact GPS coordinates of where they were taken, when they were taken, and what model of phone was used. Social media users may want to take care to strip the EXIF metadata from images before posting them online, and scour their profiles for images that do have such data. And lastly, think of what pieces of information tie your social profiles to your real-life profile. Could somebody find your real name or address from your social profile or vice versa? Is your account connected to another service, to which you're publicly registered under your full name?

When using a social platform, consider who owns it, and what their interests are. What do the owners of Facebook have to gain from handing your information to nefarious actors? Reddit? TikTok? GitHub? Think carefully before posting something that could be misinterpreted as against terms of service, or against the grain of that platform's sociocultural attitudes, or could serve as evidence of an illegal act at the state or federal level. Don't stand out if that's not your aim, and never miss a good chance to shut up. Never post photos or videos from protests or other acts of civil disobedience. Do not mention your presence, or the presence of others. If you are dead set on uploading media regarding a protest or demonstration, ensure the photos and videos have zero metadata attached, and blur the faces and identifying features of anyone in them. Do not post anything until long after it has concluded and everybody involved is safely home.

What types of web services do you use every day? What kind of data passes through servers that you do not control? Phone calls? Texts? Emails? Calendar events? Boarding passes? Location data? Are these services things you absolutely need? What kinds of benefits and drawbacks would there be to removing these services from your workflow or replacing them with something more transparent? What would happen if this data was sensitive, or leaked? Try to avoid using a service unless you really need it. The more services you use, the less control you have over your data, as it's less centralized.

Thank You

I must give you my most heartfelt thanks for reading this. If I've piqued your interest in operational security, and you want to learn more advanced topics, please contact me and let me know. I'll be more than happy to write a Part 2!

I hope you thorougly enjoyed this guide, and learned a thing or two from it along the way. Be safe.

Revision: 0.2.0
Last modified: 2025-02-06

Learning NixOS: Part 0

shibedrill1@gmail.com — Tue, 31 Dec 2024 05:00:00 GMT

I began learning NixOS recently, as I have been quickly losing my patience for Windows and its bullshit. Diving in headfirst, these are my notes from my initial investigations.

I'd heard about NixOS in the past, from youtubers like No Boilerplate, and I had tried it once in a VM- but never really caught onto it. However, Windows was proving to be a less-than-optimal OS for software development work, and I wanted to finally move to an environment where I wasn't constantly fucking around with library paths and broken installers. WSL is nice, but it can only get you so far when you're trying to work on complex projects.

The first thing I learned about NixOS is that it has a very loyal fanbase. People love it. It's just like Rust, in that while not many people use it, the people that do use it are religiously enthusiastic about it. That said, I can see why. In development, you really need reproducibility. "It works on my machine" is the whole reason we have Docker, and I dislike Docker so much that I put off learning it for years. Nix seems to be the stopgap solution for reproducible environments anywhere, both on the server and on the desktop. It's versatile, and I can respect that.

The second thing I learned is that NixOS has a pretty active community, and there's a lot of open-source tools and tech built on top of the Nix language, allowing the user to make even more precise and powerful changes to their systems. Home manager is one such example, but there's also things like disko. Installing NixOS in a virtual machine to get started with my first flake, I tried to be future-proof about it- I wanted to write a configuration set that I wouldn't have to massively refactor to include something useful. So, I wanted to write a single flake that included all the bells and whistles. This was my first mistake- I was immediately in over my head, utterly lost, and unsure of what I wanted to even accomplish because I was so caught up in thinking about how I wanted to accomplish it. Analysis Paralysis and bikeshedding are what I do better than any other girl in the techsphere, baby!!!

The third thing I learned: The Nix language has a syntax that I, at the time of writing, hate. It's not that I think it's objectively bad, or poorly designed. It just feels utterly antithetical to anything I've ever used before, and I find that my instincts are often in contradiction with the language's intended syntax. (For context, I primarily use Rust and Java in my day-to-day programming work.) Having to explicitly declare which variables are carried into a local scope seems... wrong. Especially using Nix's let in syntax. Every guide I find for the language just leaves me feeling bewildered, and I wonder how the Nix language might look if it were designed to feel more familiar to users of other languages.

Other things that befuddle me about the Nix language:

No static or strong typing.
Functions produce lambdas instead of values.
Functions lack names.

But maybe it's just me coming from Rust, where everything is needlessly typesafe.

Lastly, I felt overwhelmed when looking for functions and keys to configure programs like Git. I know there's websites that have information on the options Nix offers, but I just wish there was an IDE plugin that would autocomplete them for me. It would be much less of a headache.

My goals as of right now are to create a flake that I can use to configure roughly the same user environment on any PC or laptop, with very minor alterations depending on the system. The configurations I had put together beforehand were done in WSL, but the configurations I found in the VM's configuration.nix were far more varied and detailed. So at some point, I need to look at both configs, and think of a way to merge some of the necessary bare-metal configuration options into my incomplete WSL flake. I have a spare laptop to test it on now, though, which should help- since I can try using it for school work and get used to the environment.

My current NixOS files are available here if you're interested!

RSS Is Now Available!

shibedrill1@gmail.com — Mon, 16 Dec 2024 05:00:00 GMT

RSS is now available for this website! The XML file is located at shibedrill.site/rss.xml. The basic functionality is all there, but later, I intend to add stuff like Markdown parsing to HTML for the description and including the post's full content.

Enjoy!

A Real POS: Upsettingly Hacky Architecture

shibedrill1@gmail.com — Sun, 15 Dec 2024 05:00:00 GMT

I work at a grocery store. As an unfortunate result, I'm rather familiar with the software my job uses, both for the normal checkout terminals and the self-checkouts. During my time here, I've made some... uncomfortable realizations about the way the systems are architectured. And since they didn't make me sign any NDAs, I'm here to share it with a tiny portion of the Internet.

First of all, the normal checkouts run 4690 OS, also known as TCx Sky- a Toshiba product, previously developed by IBM. It's just Linux, with a Java application handling the actual point-of-sale operations. Different stores can use different applications for the top layer, but my job uses something called "Store Configurator ACE GUI". I suspect it runs X11 due to the cursor and the screensaver, and I know it runs Java 8 from the POS application's "Java Version" button. Needless to say, this is a nearly 10 year old ecosystem. If Java 8 released in 2014, the underlying Linux system is likely based on the 3.1x.x series kernel.

At one point, the store suffered a partial power outage due to faulty wiring, and all the checkouts and servers briefly lost power and rebooted. I find it likely that the checkouts came online while the DHCP server or router was still offline, because every terminal failed to boot on account of not having an IP address. Which dropped them directly to root debug shells. I certainly hope they aren't running any kind of disk encryption, because being able to obtain a root shell by unplugging the network cable and rebooting is an awfully stupid security hole that could be patched with a single configuration change.

The self-checkout systems are another story. They run Windows 10, possibly one of the extended release versions. They're also manufactured by NCR Corporation (National Cash Register), both the hardware and the software. But despite this incongruency with the software running on the normal checkouts, transactions can be easily suspended on any self-checkout and resumed on any normal checkout. Also, "overrides" on the self-checkouts are triggered by the same conditions as they are on the normal checkouts, and require the same authentication via barcode IDs. So, logically, they must share some central infrastructure for transactions and authentication. I also know that other establishments, using different regular checkouts, also use the same NCR checkouts with the same software. So there must be some way that the self-checkouts are able to interface with the preexisting infrastructure, which is likely done to save costs. Now, I'm nosy, so let's find out how it works!

The self-checkouts have several glaring problems and strange behaviors which we, the employees, are naturally forced to work around. Sometimes, they will lock up completely, and declare themselves to be "out-of-sync"- a fatal error which will eventually result in the whole system crashing and rebooting. In the reboot process, terminals and management windows briefly pop up displaying information about the status of the software bringup, including individual "devices" involved in the system. One day I was bored, and watched one boot up, and saw it activating devices one by one: Scanners, scales, cash changers, cameras, screens... And "Virtual Simulated Retail Platform Software"?

And then a console window pops up, titled:

4690 Console

And the icon is the fucking Java cup.

The self-checkouts don't merely share the same API and infrastructure as the regular checkouts, no. They don't even use some kind of translation layer at the server level. They run a simulated copy of the TCx Sky Java components, hook into the JVM using C++ code, and manually poke memory values and call functions to translate the customer's actions on NCR's UI into simulated actions inside this simulated copy of 4690 OS. This allows the self-checkouts to use the same infrastructure as the normal checkouts without overhead, but with the added bonus of everything being perpetually broken. Because this is quite literally the hackiest thing I have ever seen get deployed anywhere, for any purpose. You can't even remove produce items from the cart, because the removal function searches for the items by value instead of using a reference, and the searches never return true because the software subtracts the weight of a produce bag automatically.

Oh, also, every so often the regular checkout terminals just throw random null pointer exceptions, and a little icon in the corner starts flashing that says "Check logs". Now pray tell, Toshiba, what am I (a cashier) supposed to do when the software YOU maintain dereferences a nullptr? I mean, I know I'm overqualified for my position. But I'm not exactly in any place to be helping you out. Mostly on account of the fact that I hate Java. I'm morally opposed to working with it, right along with the NSA and DOD.

Cheesecake: Get to know her!

shibedrill1@gmail.com — Mon, 09 Dec 2024 05:00:00 GMT

This is Cheesecake. Everybody say "Hi, Cheesecake!"

Cheesecake is a Wild Republic Cuddlekins Fawn (not sponsored) that was gifted to me for Christmas one year, by my friend Lilu.

I love plushies and all things deer-related (especially white-tailed deer), so ever since I got her, she has been one of my most prized belongings. And just recently, I decided to make her the mascot of my website. So make sure to say hi to her when you see her.

Her favorite hobby? Zeroing the partition tables on people's hard drives.

Tracking down an obscure Linux Kernel regression

shibedrill1@gmail.com — Fri, 29 Nov 2024 05:00:00 GMT

This post has been copied from my Tumblr blog. If you've seen it before, this is why.

So I use a shitty ancient laptop. A Dell Latitude E6420 from 2012: i5-2520m CPU, 16gb of RAM, 512gb SSD. Good enough for me. It's slow, it's thick, it's heavy, but it's got a lot of ports, it's sturdy, and it's repairable. I tried to get a newer laptop, one with a faster processor and bigger screen, but it was so thin and flimsy that it crashed if I looked at it wrong or picked it up while it was running. It was entirely plastic and shattered as soon as I dropped my backpack. With the laptop in the padded laptop compartment. And somehow, it had a maximum RAM capacity of 12gb- smaller than my shitty Latitude. So I switched back, and I still daily drive this hunk of shit business laptop.

I use Linux on all my laptops and mobile machines, and I run EndeavourOS on my laptop right now. Windows is too slow and bloated for my taste, and its drivers for this laptop are kind of broken due to its age. I should also mention that I use my laptop for school, and it's imperative that it boots reliably every single time I pull it out of my backpack. So when I upgraded my kernel from 6.5.9 to 6.6.1, and rebooted to discover that the system would hang before it could even boot the kernel, you can imagine how stressed I was trying to get it working again. Luckily I have a Medicat USB with an Arch ISO on it, so I was able to live boot into Arch, chroot into my installation, and install the linux-lts kernel, which booted without issue. This also gave me the first clue as to what the fuck was happening.

After reinstalling the OS a couple of times as a sanity check (sometimes I break shit by accident) and testing out the linux-zen kernel, I was able to determine the following:

The boot hang didn't occur with linux-lts, but did occur with linux and linux-zen. Which meant that the issue wasn't with packaging, but it was upstream, in the kernel source tree.
The system was not secretly booting without me realizing. I knew this because the hard drive indicator and the CPU fan didn't actually turn on during these stuck boots, whereas normally, the HDD indicator would start flashing two seconds after GRUB booted the kernel.
The problem wasn't anything to do with graphics drivers. I was able to reproduce it with both Nouveau and Nvidia's proprietary drivers, as well as with nomodeset enabled on both of them.
The hang only happened roughly half the time. Sometimes it worked, sometimes it didn't. The bug had something to do with some type of randomized function, or it was related to some type of non-deterministic hardware failure or errata.
The issue occurred somewhere between GRUB starting the kernel, and whatever happens after that. I was able to ascertain this by enabling debug messages from GRUB, and it showed that the hang would occur at "Starting image at 0xdeadbeef...".
The issue was not architecture or microarchitecture specific. I determined this by performing a fresh installation in QEMU, with the CPU topology configured to match the host CPU, and the issue couldn't be replicated.
The issue was not related to GRUB, since it persisted even using systemd-boot. I managed to brick my system in the process of attempting to switch back to GRUB, and had to reinstall.

With all this information in mind, I filed my first ever bug report on the Kernel Bugzilla page. I ended up finding one or two other people who had the same exact issue, after the same exact update, all on older Sandy Bridge processors. I attempted to do a git bisect to figure out the responsible commit, but this is not something that's easy or fast on a 2012 laptop. One compilation took over seven hours. Times fourteen or so steps in the bisection between 6.5.9 and 6.6.1, that's... Four straight days of compilation. Ninety eight hours. And I lost my progress about three compilations in, after reinstalling my system again due to broken graphics drivers.

Thankfully, like I said, I was not alone. Another user on the bug tracker who had the issue was able to perform a successful bisection of the kernel, and determined the offending commit. As luck would have it, the code changes related to the EFI stub and the kernel's interaction with the system EFI firmware, specifically the code responsible for decompressing the kernel image into memory. The users on the bug tracker also surmised that the problem had to do with where the kernel was getting loaded into memory, and wrote up a few patches to display some useful debug information regarding that.

The Linux kernel, in version four point something, had support for KASLR merged. KASLR, or Kernel Address Space Layout Randomization, is a functionality that makes it harder for hackers to call a kernel function with a known address or offset by loading the kernel into a random physical address with each boot. ASLR, Address Space Layout Randomization, already randomizes virtual address space layouts for user space applications, but it's performed optionally for the kernel as well. One person on the forum suggested that KASLR was the issue with the boot failures, and since the boot hang was occurring randomly, I had a feeling that they were right. Disabling KASLR through the kernel command line arguments, I confirmed that the latest kernel (6.6.5 at the time) booted four out of four times. Re-enabling KASLR, it hung on the first boot. So there was our culprit. Everyone else confirmed that KASLR being turned off would resolve the hang.

It was brought up that the Arch wiki had a short note about buggy Dell EFI firmware, which linked to a single patch diff in the 5.10-5.11 kernel update. The EFI firmware from Dell is really fucked up, and hands off more information than it needs to to the kernel, which results in some problems without custom handling code or a specialized EFI shim. That code got merged in the 5.11 update, like I mentioned, and at that time I was using BIOS to boot my laptop so I never noticed the issue.

So at long last, a patch was posted to the bug tracker, wherein it would detect older versions of EFI firmware and disable KASLR accordingly by setting the randomization seed to zero. I recompiled the kernel with the patch, and the system booted without issue three out of three times. I was ecstatic, obviously. The patch subsequently got merged into Linux v6.7-rc6, and was mainlined into Linux v6.7 soon after. I'm so happy I got to contribute to the project, even if it was as simple as reporting a bug and doing some quick testing.

Such are the joys of owning a clunker craptop from 2012. Hope you enjoyed the story of my first contribution to the Kernel Project. I don't know how many people are going to benefit from this patch, but I hope that it's gonna be more than just me.

The Sins of Cloudflare

shibedrill1@gmail.com — Thu, 28 Nov 2024 05:00:00 GMT

Cloudflare is a large enterprise web service provider, which offers services such as DNS, DDoS protection, content delivery, reverse proxying, WAF, and many other things that I am not quite qualified to talk about. In short, though, Cloudflare runs the internet. So much so, in fact, that there has been three major outages in 2024 alone, affecting up to hundreds of thousands of customers each time.

This isn't isolated to 2024, though. Almost every year, something happens with Cloudflare, rendering significant portions of the Internet entirely unusable. And yet, despite this, basically every company with a website uses them. So my question is, why?

Why does half the Internet depend on a company that consistently leaves them hanging?

This is more of a placeholder post than anything. I don't have any legitimate gripes with Cloudflare, other than their high prices and large market share, which kinda pushes out any competition. I would appreciate a lower tier subscription, or a one-time-payment, strictly for proxying Minecraft, SSH, and other common protocols that might get DDoSed. As a hobbyist, my services don't make me any money. So spending very real funds on DDoS protection is not an especially enticing idea to me, at least while the prices are as high as they are.

I do, however, appreciate that they're able to provide free proxying and DNS for every customer. It's made my infrastructure significantly easier to extend and maintain.