I wanted to test out a couple of Brendan Gregg’s amazing eBPF scripts scripts this week. My home lab uses cobbler and ansible to provision systems and I was hoping to use Fedora 27 since it has a recent kernel. When I tried to import the Fedora 27 server DVD I received the following error:
$ cobbler import --name=fedora27 --arch=x86_64 --path=/mnt
task started: 2017-12-13_124734_import
task started (id=Media import, time=Wed Dec 13 12:47:34 2017)
Found a candidate signature: breed=redhat, version=rhel6
Found a candidate signature: breed=redhat, version=rhel7
Found a candidate signature: breed=redhat, version=fedora21
Found a candidate signature: breed=redhat, version=fedora20
Found a candidate signature: breed=redhat, version=fedora23
Found a candidate signature: breed=redhat, version=fedora22
Found a candidate signature: breed=redhat, version=fedora25
Found a candidate signature: breed=redhat, version=fedora24
Found a candidate signature: breed=redhat, version=fedora26
Found a candidate signature: breed=redhat, version=cloudlinux6
Found a candidate signature: breed=redhat, version=fedora16
Found a candidate signature: breed=redhat, version=fedora17
Found a candidate signature: breed=redhat, version=fedora18
Found a candidate signature: breed=redhat, version=fedora19
No signature matched in /var/www/cobbler/ks_mirror/fedora27-x86_64
!!! TASK FAILED !!!
Cobbler keeps a set of distribution signatures in /var/lib/cobbler/distro_signatures.json which you can view with the cobbler signature report option:
$ cobbler signature report --name=redhat
Currently loaded signatures:
redhat:
cloudlinux6
fedora16
fedora17
fedora18
fedora19
fedora20
fedora21
fedora22
fedora23
fedora24
fedora25
fedora26
rhel4
rhel5
rhel6
rhel7
Each signature in this file is a JSON object defining how a given distribution is laid out. To update the contents of this file (which typically fixes the missing signature issue) you can use cobbler signature update:
$ cobbler signature update
task started: 2017-12-13_132701_sigupdate
task started (id=Updating Signatures, time=Wed Dec 13 13:27:01 2017)
Successfully got file from https://cobbler.github.io/signatures/2.8.x/latest.json
*** TASK COMPLETE ***
Updating the signature file unfortunately didn’t fix my issue. A quick review of the official cobbler distro_signatures.json file shows that Fedora 27 hasn’t been added yet so it looks like I will be putting my JSON skills to good use tonight. I also filed a feature enhancement on github to get this added.
While perusing the latest articles on hacker news I came across an amazing debugging article by Hector Martin. He recently tracked down a nasty segmentation violation and explained step-by-step how he did it. The article dicusses the VDSO (Virtual ELF Dynamic Shared Object) which is an efficient way to access frequently used system calls. Here is a nice description of this feature from the vdso(7) manual page:
Why does the vDSO exist at all? There are some system calls the kernel provides that user-space code ends up using frequently, to the point that such calls can dominate overall performance. This is due both to the frequency of the call as well as the context-switch overhead that results from exiting user space and entering the kernel.
If you run ldd on a program you can see that linux-vdso.so.1 is linked to the executable:
$ ldd /bin/zsh
linux-vdso.so.1 (0x00007ffc509d6000)
libgdbm.so.4 => /lib64/libgdbm.so.4 (0x00007fa6c5cab000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fa6c5a38000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fa6c5834000)
libncursesw.so.6 => /lib64/libncursesw.so.6 (0x00007fa6c55fc000)
libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007fa6c53d0000)
librt.so.1 => /lib64/librt.so.1 (0x00007fa6c51c8000)
libm.so.6 => /lib64/libm.so.6 (0x00007fa6c4eb2000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa6c4add000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa6c48be000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa6c6192000)
The VDSO is resident in the kernel and mapped to the process at runtime. If you check the memory map for a process you can see where VDSO is mapped into the address space:
$ grep vdso /proc/$$/maps
7ffc616ec000-7ffc616ee000 r-xp 00000000 00:00 0 [vdso]
VDSO is extremely small and takes a measly 8K of memory on x86/X64 architectures:
$ echo $((0x7ffc616ee000-0x7ffc616ec000))
8192
$ grep -A 1 vdso /proc/$$/smaps
7ffc616ec000-7ffc616ee000 r-xp 00000000 00:00 0 [vdso]
Size: 8 kB
The vdso manual page lists 4 system calls that are implemented on x86/X64 architectures. We can verify this with nm (or by reading ARCHITECTURE-SPECIFIC NOTES in vdso(7)):
$ strings vdso64.so | grep '__vdso'
__vdso_clock_gettime
__vdso_gettimeofday
__vdso_time
__vdso_getcpu
Huge thanks to Hector Martin for the amazing article. I learned a lot by reading his article and digging through the VDSO documentation. Adding this here for reference.
UPDATED 12/06/2017
Hector responded to a twitter thanks and mentioned that the VDSO is resident in the kernel and mapped into the process address space when the process is initialized. The object code doesn’t come from disk like I initially thought.
This morning I needed a way to pretty print a CSS style sheet. After poking around the web I came across the cssbeautify-cli utility which takes a CSS file and produces pretty printed output. To use this super useful tool you will first need to install and configure the node package manager (npm):
$ apt install npm
To verify the module is available we can run npm with the search option:
$ npm search cssbeautify
NAME DESCRIPTION AUTHOR DATE VERSION KEYWORDS
@types/cssbeautify TypeScript definitions for CSS Beautify v0.3.1 =types 2016-10-03 0.3.1
cssbeautify Reindent and reformat CSS. =senchalabs 2013-10-09 0.3.1
cssbeautify-cli CLI for cssbeautify =cyberskunk 2016-12-21 0.5.3
grunt-cssbeautifier cssbeautify.com for grunt =douzi 2014-07-02 0.1.2
gulp-cssbeautify Reindent and reformat CSS =jonkemp 2014-03-29 0.1.3
npmdoc-gulp-cssbeautify #### api documentation for [gulp-cssbeautify… =npmdoc 2017-04-19 0.0.3
To install a module in $HOME/node_modules you can use the npm install option:
$ npm install cssbeautify-cli
If everything worked as expected a symbolic link to the executable should be created in $HOME/node_modules/.bin:
$ $HOME/node_modules/.bin/cssbeautify-cli -h
Usage:
/home/matty/node_modules/.bin/cssbeautify-cli [options] -f filename
/home/matty/node_modules/.bin/cssbeautify-cli [options] -s
Options:
-a, --autosemicolon insert a semicolon after the last ruleset [default: false]
-c, --config json config file to use
-f, --file file to beautify or glob pattern
-h, --help show this help message
-i, --indent string used for the indentation of the declaration (spaces, tabs or number of spaces) [default: " "]
-o, --openbrace the placement of open curly brace, either end-of-line or separate-line [default: "end-of-line"]
-s, --stdin use stdin as input
-v, --version Display program version
-w, --writefile write output to file
To format a CSS file you can pass the file to the scripts “-f” option and redirect the pretty printed output to a new file (or a pager):
$ ~/node_modules/.bin/cssbeautify-cli -f myCSSFile.css > myCSSFile.css.new
This tool makes viewing and interpreting CSS a snap!
Last night I spent a good deal of time studying javascript execution contexts and scoping rules. Most of my programming experience has been with shell, C, Python and Go. Most of these languages use lexical scoping but do so by the code block being executed. Javascript is a different animal when it comes to scoping. When I ran the following code to better understand scoping:
var foo = "String1";
console.log("Global: " + foo);
func1();
function func1() {
var bar = "String2";
console.log("Func1: " + foo + bar);
func2();
function func2() {
var fmep = "String3";
console.log("Func2: " + foo + bar + fmep);
func3();
}
}
function func3() {
var gorp = "String 4";
console.log("Func3:" + foo + bar + fmep + gorp);
}
I originally thought node would spit out an error when func2() was executed. This was definitely not the case:
$ node test.js
Global: String1
Func1: String1String2
Func2: String1String2String3
/home/matty/test.js:19
console.log("Func3:" + foo + bar + fmep + gorp);
^
ReferenceError: bar is not defined
at func3 (/home/matty/test.js:19:31)
at func2 (/home/matty/test.js:13:3)
at func1 (/home/matty/test.js:8:2)
at Object.<anonymous> (/home/matty/test.js:3:1)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
In this case func2() ran just fine and func3() generated an exception when it tried to access the variable bar. But why is this? David Shariff wrote an excellent post describing scoping and javascript execution contexts which made my bug stick out like a sore thumb. The crux of the issue is that func3 is on the top of the execution context stack but it’s SCOPE is limited to the global execution context (see David’s awesome post for an illustrated example). I’m glad I figured this out early on in my javascript adventures. Adding this as a reference for myself.
This week I got to debug a fun Javascript bug. I was using the double equal signs to compare two arguments and under certain cases the code would produce an error. After a bit of digging I learned about Javascript type coercion and the differences between double equals and triple equals. To illustrate the issue lets fire up node and create two variables:
$ node
> var i = 23;
> var j = "23";
The first variable listed above is an integer with the value 23. The second variable is a string with the number 23. If we compare these two values with double equals we get a successful comparison:
> i == j
true
If we use triple equals we get a very different result:
> i === j
false
Why is this? In the first example javascript will convert the values to the same type and then compare the result. This is called type coercion. In the second example javascript will compare the two variables AS IS resulting in a failed comparison operation. This was an interesting finding and I’m sure this has bit more than one developer. This definitely gives me a huge appreciation for statically typed languages.