Blog O' Matty


Adding a new Operating System to the cobbler signature file

This article was posted by Matty on 2017-12-16 10:07:01 -0500 EST

This has been an exciting week of code analysis and debugging. As I mentioned previously cobbler doesn’t currently have support for Fedora 27. It it turns out adding a new Operating System signature is super easy. For my specific situation I forked the cobbler git repo and copied the Fedora 26 signature line and changed the 6 to a 7. Once I updated the signature file and restarted cobblerd everything imported without issue:

$ cobbler import --name=fedora27 --arch=x86_64 --path=/mnt

task started: 2017-12-16_095536_import
task started (id=Media import, time=Sat Dec 16 09:55:36 2017)
Found a candidate signature: breed=redhat, version=rhel6
Found a candidate signature: breed=redhat, version=rhel7
Found a candidate signature: breed=redhat, version=fedora21
Found a candidate signature: breed=redhat, version=fedora20
Found a candidate signature: breed=redhat, version=fedora23
Found a candidate signature: breed=redhat, version=fedora22
Found a candidate signature: breed=redhat, version=fedora25
Found a candidate signature: breed=redhat, version=fedora24
Found a candidate signature: breed=redhat, version=fedora27
Found a matching signature: breed=redhat, version=fedora27
Adding distros from path /var/www/cobbler/ks_mirror/fedora27-x86_64:
creating new distro: fedora27-x86_64
trying symlink: /var/www/cobbler/ks_mirror/fedora27-x86_64 -> /var/www/cobbler/links/fedora27-x86_64
creating new profile: fedora27-x86_64
associating repos
checking for rsync repo(s)
checking for rhn repo(s)
checking for yum repo(s)
starting descent into /var/www/cobbler/ks_mirror/fedora27-x86_64 for fedora27-x86_64
processing repo at : /var/www/cobbler/ks_mirror/fedora27-x86_64
need to process repo/comps: /var/www/cobbler/ks_mirror/fedora27-x86_64
looking for /var/www/cobbler/ks_mirror/fedora27-x86_64/repodata/*comps*.xml
Keeping repodata as-is :/var/www/cobbler/ks_mirror/fedora27-x86_64/repodata
*** TASK COMPLETE ***

Now back to our regularly scheduled programming. Time for some eBPF testing!

Dealing With Cobbler Signature Errors

This article was posted by Matty on 2017-12-13 12:51:50 -0500 EST

I wanted to test out a couple of Brendan Gregg’s amazing eBPF scripts scripts this week. My home lab uses cobbler and ansible to provision systems and I was hoping to use Fedora 27 since it has a recent kernel. When I tried to import the Fedora 27 server DVD I received the following error:

$ cobbler import --name=fedora27 --arch=x86_64 --path=/mnt

task started: 2017-12-13_124734_import
task started (id=Media import, time=Wed Dec 13 12:47:34 2017)
Found a candidate signature: breed=redhat, version=rhel6
Found a candidate signature: breed=redhat, version=rhel7
Found a candidate signature: breed=redhat, version=fedora21
Found a candidate signature: breed=redhat, version=fedora20
Found a candidate signature: breed=redhat, version=fedora23
Found a candidate signature: breed=redhat, version=fedora22
Found a candidate signature: breed=redhat, version=fedora25
Found a candidate signature: breed=redhat, version=fedora24
Found a candidate signature: breed=redhat, version=fedora26
Found a candidate signature: breed=redhat, version=cloudlinux6
Found a candidate signature: breed=redhat, version=fedora16
Found a candidate signature: breed=redhat, version=fedora17
Found a candidate signature: breed=redhat, version=fedora18
Found a candidate signature: breed=redhat, version=fedora19
No signature matched in /var/www/cobbler/ks_mirror/fedora27-x86_64
!!! TASK FAILED !!!

Cobbler keeps a set of distribution signatures in /var/lib/cobbler/distro_signatures.json which you can view with the cobbler signature report option:

$ cobbler signature report --name=redhat

Currently loaded signatures:
redhat:
	cloudlinux6
	fedora16
	fedora17
	fedora18
	fedora19
	fedora20
	fedora21
	fedora22
	fedora23
	fedora24
	fedora25
	fedora26
	rhel4
	rhel5
	rhel6
	rhel7

Each signature in this file is a JSON object defining how a given distribution is laid out. To update the contents of this file (which typically fixes the missing signature issue) you can use cobbler signature update:

$ cobbler signature update

task started: 2017-12-13_132701_sigupdate
task started (id=Updating Signatures, time=Wed Dec 13 13:27:01 2017)
Successfully got file from https://cobbler.github.io/signatures/2.8.x/latest.json
*** TASK COMPLETE ***

Updating the signature file unfortunately didn’t fix my issue. A quick review of the official cobbler distro_signatures.json file shows that Fedora 27 hasn’t been added yet so it looks like I will be putting my JSON skills to good use tonight. I also filed a feature enhancement on github to get this added.

Speeding Up Frequently Accessed System Calls With VDSO

This article was posted by Matty on 2017-12-05 18:34:13 -0500 EST

While perusing the latest articles on hacker news I came across an amazing debugging article by Hector Martin. He recently tracked down a nasty segmentation violation and explained step-by-step how he did it. The article dicusses the VDSO (Virtual ELF Dynamic Shared Object) which is an efficient way to access frequently used system calls. Here is a nice description of this feature from the vdso(7) manual page:

Why does the vDSO exist at all? There are some system calls the kernel provides that user-space code ends up using frequently, to the point that such calls can dominate overall performance. This is due both to the frequency of the call as well as the context-switch overhead that results from exiting user space and entering the kernel.

If you run ldd on a program you can see that linux-vdso.so.1 is linked to the executable:

$ ldd /bin/zsh

	linux-vdso.so.1 (0x00007ffc509d6000)
	libgdbm.so.4 => /lib64/libgdbm.so.4 (0x00007fa6c5cab000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fa6c5a38000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fa6c5834000)
	libncursesw.so.6 => /lib64/libncursesw.so.6 (0x00007fa6c55fc000)
	libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007fa6c53d0000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fa6c51c8000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fa6c4eb2000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fa6c4add000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa6c48be000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa6c6192000)

The VDSO is resident in the kernel and mapped to the process at runtime. If you check the memory map for a process you can see where VDSO is mapped into the address space:

$ grep vdso /proc/$$/maps

7ffc616ec000-7ffc616ee000 r-xp 00000000 00:00 0                          [vdso]

VDSO is extremely small and takes a measly 8K of memory on x86/X64 architectures:

$ echo $((0x7ffc616ee000-0x7ffc616ec000))

8192

$ grep -A 1 vdso /proc/$$/smaps

7ffc616ec000-7ffc616ee000 r-xp 00000000 00:00 0                          [vdso]
Size:                  8 kB

The vdso manual page lists 4 system calls that are implemented on x86/X64 architectures. We can verify this with nm (or by reading ARCHITECTURE-SPECIFIC NOTES in vdso(7)):

$ strings vdso64.so | grep '__vdso'

__vdso_clock_gettime
__vdso_gettimeofday
__vdso_time
__vdso_getcpu

Huge thanks to Hector Martin for the amazing article. I learned a lot by reading his article and digging through the VDSO documentation. Adding this here for reference.

UPDATED 12/06/2017

Hector responded to a twitter thanks and mentioned that the VDSO is resident in the kernel and mapped into the process address space when the process is initialized. The object code doesn’t come from disk like I initially thought.

Formatting CSS From The Linux Command Line

This article was posted by Matty on 2017-12-02 08:38:01 -0500 EST

This morning I needed a way to pretty print a CSS style sheet. After poking around the web I came across the cssbeautify-cli utility which takes a CSS file and produces pretty printed output. To use this super useful tool you will first need to install and configure the node package manager (npm):

$ apt install npm

To verify the module is available we can run npm with the search option:

$ npm search cssbeautify

NAME                     DESCRIPTION                                     AUTHOR      DATE       VERSION KEYWORDS
@types/cssbeautify       TypeScript definitions for CSS Beautify v0.3.1  =types      2016-10-03 0.3.1           
cssbeautify              Reindent and reformat CSS.                      =senchalabs 2013-10-09 0.3.1           
cssbeautify-cli          CLI for cssbeautify                             =cyberskunk 2016-12-21 0.5.3           
grunt-cssbeautifier      cssbeautify.com for grunt                       =douzi      2014-07-02 0.1.2           
gulp-cssbeautify         Reindent and reformat CSS                       =jonkemp    2014-03-29 0.1.3           
npmdoc-gulp-cssbeautify  #### api documentation for [gulp-cssbeautify…   =npmdoc     2017-04-19 0.0.3           

To install a module in $HOME/node_modules you can use the npm install option:

$ npm install cssbeautify-cli

If everything worked as expected a symbolic link to the executable should be created in $HOME/node_modules/.bin:

$ $HOME/node_modules/.bin/cssbeautify-cli -h

Usage:
	/home/matty/node_modules/.bin/cssbeautify-cli [options] -f filename
	/home/matty/node_modules/.bin/cssbeautify-cli [options] -s

Options:
  -a, --autosemicolon  insert a semicolon after the last ruleset                                              [default: false]
  -c, --config         json config file to use                                                              
  -f, --file           file to beautify or glob pattern                                                     
  -h, --help           show this help message                                                               
  -i, --indent         string used for the indentation of the declaration (spaces, tabs or number of spaces)  [default: "    "]
  -o, --openbrace      the placement of open curly brace, either end-of-line or separate-line                 [default: "end-of-line"]
  -s, --stdin          use stdin as input                                                                   
  -v, --version        Display program version                                                              
  -w, --writefile      write output to file    

To format a CSS file you can pass the file to the scripts “-f” option and redirect the pretty printed output to a new file (or a pager):

$ ~/node_modules/.bin/cssbeautify-cli -f myCSSFile.css > myCSSFile.css.new

This tool makes viewing and interpreting CSS a snap!

Making sense of Javascript execution contexts and the scope Chain

This article was posted by Matty on 2017-11-30 09:50:43 -0500 EST

Last night I spent a good deal of time studying javascript execution contexts and scoping rules. Most of my programming experience has been with shell, C, Python and Go. Most of these languages use lexical scoping but do so by the code block being executed. Javascript is a different animal when it comes to scoping. When I ran the following code to better understand scoping:

var foo = "String1";
console.log("Global: " + foo);
func1();

function func1() {
    var bar = "String2";
    console.log("Func1: " + foo + bar);
    func2();

    function func2() {
        var fmep = "String3";
        console.log("Func2: "  + foo + bar + fmep);
        func3();
    }
}

function func3() {
    var gorp = "String 4";
    console.log("Func3:" + foo + bar + fmep + gorp);
}

I originally thought node would spit out an error when func2() was executed. This was definitely not the case:

$ node test.js

Global: String1
Func1: String1String2
Func2: String1String2String3
/home/matty/test.js:19
	console.log("Func3:" + foo + bar + fmep + gorp);
	                             ^

ReferenceError: bar is not defined
    at func3 (/home/matty/test.js:19:31)
    at func2 (/home/matty/test.js:13:3)
    at func1 (/home/matty/test.js:8:2)
    at Object.<anonymous> (/home/matty/test.js:3:1)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)

In this case func2() ran just fine and func3() generated an exception when it tried to access the variable bar. But why is this? David Shariff wrote an excellent post describing scoping and javascript execution contexts which made my bug stick out like a sore thumb. The crux of the issue is that func3 is on the top of the execution context stack but it’s SCOPE is limited to the global execution context (see David’s awesome post for an illustrated example). I’m glad I figured this out early on in my javascript adventures. Adding this as a reference for myself.