Table of contents for January/February 2020 in ADMIN Network & Security

The idea of block I/O caching isn’t revolutionary, but it still is an extremely complex topic. Technically speaking, caching as a whole is complicated and a very difficult solution to implement. It all boils down to the I/O profile of the ecosystem or server on which it is being implemented. Before I dive right in, I want to take a step back, so you understand what I/O caching is and what it is intended to address. What Is I/O Caching? A computer cache is a component (typically leveraging some sort of performant memory) that temporarily stores data for current write and future read I/O requests. In the event of write operations, the data to be written is staged and will eventually be scheduled and flushed to the slower device intended…

When it comes to leveraging the full Office 365 feature set, migrating mailboxes to Exchange Online is one of the greatest challenges. Unlike migrating within an organization, moving to Exchange Online is problematic, because mailboxes are shifted between two separately managed organizations. This connection between an on-premises Exchange instance and Exchange Online is known as a hybrid connection. Microsoft refers to this connection as the Exchange Modern Hybrid and has extended its Hybrid Configuration Wizard (HCW) with Hybrid Agent (Figure 1) to facilitate the connection. With HCW, Hybrid Agent establishes a connection between the local Exchange and Exchange Online, reducing the requirements for external DNS records, certificate updates, and incoming firewall network connections – all of which made the task complex in the past. Multiple Choices Hybrid Agent does not…

One important paradigm in software development, especially web applications, is careful validation of user input before allowing further processing. Much can go wrong if the input is not carefully checked. SQL injection and cross-site scripting attacks are just two of the most common examples of exploitation. Regular expressions are useful for checking user input, but even they are vulnerable to attacks. In this article, I show you how to check your regular expressions for vulnerabilities. Regular Expressions Regular expressions (regex) have become an established method of validating user input before processing to describe and check for permitted entries and to prevent certain characters (e.g., those with a special function in an application) from being entered. Unfortunately, some regular expressions can still cause unwanted behavior with certain types of input and…

For a number of reasons, it makes sense to use today’s cloud-native infrastructure to run software without employing servers; instead, you can use an arms-length, abstracted serverless platform such as AWS Lambda. For example, when you create a Lambda function (source code and a run-time configuration) and execute it, the AWS platform only bills you for the execution time, also called the “compute time.” Simple tasks usually book only hundreds of milliseconds, as opposed to running an Elastic Compute Cloud (EC2) server instance all month long along with its associated costs. In addition to reducing the cost and removing the often overlooked administrative burden of maintaining a fleet of servers to run your tasks, AWS Lambda also takes care of the sometimes difficult-to-get-right automatic scaling of your infrastructure. With Lambda,…

In the past, spinning hard disks were often a potential bottleneck for fast data processing, but in the age of hybrid and all-flash storage systems, the bottlenecks are shifting to other locations on the storage area network (SAN). I talk about where it makes sense to influence the data stream and how possible bottlenecks can be detected at an early stage. To this end, I will be determining the critically important performance parameters within a Fibre Channel SAN and showing optimization approaches. The Fibre Channel (FC) protocol is connectionless and transports data packets in buffer-to-buffer (B2B) mode. Two endpoints, such as a host bus adapter (HBA) and a switch port, negotiate a number of FC frames, which are added to the input buffer as buffer credits at the other end,…

Mapping internal IP addresses to external IP addresses is essential for Voice over IP (VoIP) communications through network address translation (NAT) gateways and firewalls. Session Initiation Protocol (SIP) is the signaling protocol for establishing VoIP connections; however, SIP-based communications have problems working through firewalls and session border controllers, and all too often, VoIP calls or some unified communications functions fail because of NAT. In this article, I show you how IT managers can resolve these issues with the session traversal utilities for NAT (STUN), traversal using relays around NAT (TURN), and Interactive Connectivity Establishment (ICE) techniques to ensure transparent transitions and improve overall SIP security. NAT Characteristics Some years ago, the limited availability of IP addresses led to the development of various strategies by the Internet Engineering Task Force (IETF)…

Hardly a year goes by that does not see some kind of disruptive technology unfold and existing traditions swept away. That’s what the two technologies discussed in this article have in common. Ceph captured the storage solutions market in a flash. Meanwhile, Kubernetes shook up the market for virtualization solutions, not only grabbing market share off KVM and others, but also off industry giants such as VMware. When two disruptive technologies such as containers and Ceph are mixing up the same market, collisions can hardly be avoided. The tool that brings Ceph and Kubernetes together is Rook, which lets you roll out a Ceph installation to a cluster with Kubernetes and offers advantages over a setup where Ceph sits “under” Kubernetes. The most important advantage is undoubtedly that Ceph integrated…

In popular clouds, the providers usually roll out standard distribution images. SUSE, Red Hat, and Canonical offer these explicitly, and there is no reason why you should not use them. However, these images may have one or two annoying features, such as missing packages, wrong configurations, or other everyday difficulties. Changing a finished image is not trivial. Instead, many admins start rebuilding from the source and, sooner or later, give up. In most cases it is not possible to achieve the same image quality as that of the distributors. Either the DIY images are bulky and far too big or they don’t work well. This problem is exactly what FAI.me addresses: The tool is an extension of the Fully Automated Installer (FAI; see also the “FAI Review” box) [1] that…

AWS introduced several new storage services and databases at re:Invent 2018, including new storage classes for Amazon Simple Storage Service (S3). In the meantime, new releases (S3 Intelligent-Tiering and S3 Glacier Deep Archive) have become available that quickly boost the number of storage classes in the oldest and most popular of all AWS services from three to six. In this article, I present the newcomers and their characteristics. Amazon’s Internet storage has always supported storage classes, between which users can choose when uploading an object and which they can also switch automatically later using lifecycle guidelines. The individual storage classes have different price models and availability classes, each of which optimally addresses a different usage profile. So, if you know the most common access patterns to your data stored in…

The Linux kernel already contains a variety of packet filters, starting with ipfwadm and followed by ipchains and iptables. Kernel 3.13 saw the introduction of nftables [1], which uses the nft tool to create and manage rules. With the help of its own virtual machine, nftables ensures that rulesets are converted into bytecode, which is then loaded into the kernel. Not only does it improve performance, but it also allows administrators to enable new rules dynamically without having to reload the entire ruleset. Parts of the old Netfilter framework use nftables, removing the need to develop new hooks, which are nothing more than certain points in the network stack of the Linux kernel at which a packet is inspected and, in the case of a match, one or more actions…

Companies often do not exclusively use public cloud services such as Amazon Web Services (AWS) [1], Microsoft Azure [2], or Google Cloud [3]. Instead, they rely on a mix, known as a hybrid cloud. In this scenario, you connect your data centers (private cloud) with the resources of a public cloud provider. The term “private cloud” is somewhat misleading, in that the operation of many data centers has little to do with cloud-based working methods, but it looks like the name is here to stay. The advantage of a hybrid cloud is that companies can use it to absorb peak loads or special requirements without having to procure new hardware for five-or six-digit figures. In this article, I show how you can add a cloud extension to an Ansible [4]…

After moving your hard disk to a new system, the Linux system suddenly fails to boot. Often this happens because of missing drivers in the ramdisk, which the kernel needs to boot the system. In this article, I take a closer look at the handling of the initramfs file and introduces dracut [1] as a practical helper. Many users only see the initramfs (initial random access memory filesystem) archive file as yet another file in the boot directory. It is automatically created when a new kernel is installed and deleted again when the kernel is removed from the system. But this initial ramdisk plays an important role, since it ensures that the root filesystem can be accessed after the computer has been restarted, to be able to access all the…

Canonical Now Offers an Ubuntu Pro Image for AWS Ubuntu rules the cloud. According to The Cloud Market (https://thecloudmarket.com/stats#/by_platform_definition), Ubuntu is the most widely used cloud image used on the Amazon Elastic Compute Cloud (with nearly 370K images deployed). Not one to be satisfied with being at the top of the digital heap, Canonical (https://canonical.com/) – the company behind Ubuntu (https://ubuntu.com/ – has released a new version of their venerable Ubuntu platform. Ubuntu Pro was created specifically for Amazon Web Services. This new image ships with the Canonical standard Ubuntu Amazon Machine Image and layers on top of that security and compliance subscriptions. Specifically, Ubuntu Pro includes: Up to 10 years of package and security updates for Ubuntu 18.04, and up to eight years for 14.04 and 16.04 Kernel Livepatch…

SchedViz [1] is one of a variety of open source tools recently released by Google that allows you to visualize how your programs are being handled by Linux kernel scheduling. The tool allows you to see exactly how your system is treating the various tasks it is running and allows you to fine-tune the way resources are allotted to each task. SchedViz is designed to overcome a specific problem: The basic Linux tools available for scheduling [2] don’t allow you to see very much. In practice, this means that most people guess how to schedule system resources, and given the complexity of modern systems, these guesses are often wrong. Multiprocessing Modern operating systems (OSs) execute multiple processes simultaneously by splitting the computing load across multiple cores and running each process…

To help understand how Rook is deployed in the overall context of Ceph (Figure 1), I will review only the most important details here. Ceph is based on the principle of object storage. It treats any content that a user uploads to Ceph as a binary object. Advantageously, objects can be separated, distributed, and reassembled at will, as long as everything happens in the correct order. In this way, object storage bypasses the limitations of traditional block storage devices, which are always inextricably linked to the medium to which they belong. The object store at the heart of Ceph is RADOS, which implicitly provides redundancy and multiplies every piece of uploaded information as often as the administrative policy defines. Moreover, RADOS has self-healing capabilities. For example, if a single hard…

Hearing that an external, independent organization has been commissioned to spend time actively attacking the cloud estate you have been tasked with helping to secure can be a little daunting – unless, of course, you are involved with a project at the seminal greenfield stage, and you have yet to learn what goes where and how it all fits together. To add to the complexity, if you are using Amazon Web Services (AWS), AWS Organizations can segregate departmental duties and, therefore, security controls between multiple accounts; commonly this might mean the use of 20 or more accounts. With these concerns and, if you blink a little too slowly, it’s quite possible that you will miss a new AWS feature or service that needs to be understood and, once deployed, secured.…

In conventional setups of the past, admins had to troubleshoot fewer nodes per setup and fewer technologies and protocols than is the case today in the cloud, with its hundreds and thousands of technologies and protocols for software-defined networking, software-defined storage, and solutions like OpenStack. In the worst case, network nodes also need to be checked separately. If you are searching for errors in this kind of environment, you cannot put the required logfiles together manually. The Elasticsearch, Logstash, and Kibana (ELK) team has demonstrated its ability to collect logs continuously from affected systems, store them centrally, index the results, and thus make them searchable. However ELK and its variations prove to be complex beasts. Getting ELK up and running is no mean achievement, and once it is finally running,…

To improve the performance of your applications, you need to conduct some kind of dynamic (program, software, code) analysis, also called profiling, to measure metrics of interest. A key metric for developers is time (i.e., where is the code spending most of its time?), because it allows you to focus on areas, or hotspots, that can be made to run faster. And, this might seem obvious, but if you don’t profile for code optimization, you could flounder all over the code improving sections you think might be bottlenecks. I have seen people spend hours working a particular part of their code when a simple profile showed that portion of the code contributed very little to the overall run time. I admit that I have also done this; however, once I…

At the Dragon Propulsion Laboratory, we are partial to using the simplest tool that will do the job at hand – particularly when dealing with the inherent complexity that performance measurement (and tuning) brings to the table. Yet that same complexity often requires advanced tooling to resolve the riddles posed by performance questions. I will examine my current benchmarking tool favorites from the simplest to the more sophisticated. Tempus Fugit The benchmark archetype is time: simple, easy to use, and well understood by most users. In its purest form, time takes a command as a parameter and times its execution in the real world (real), as well as how much CPU time was allocated in user and kernel (sys) modes: What not everyone knows is that the default time command…