I suspected some firmware file to be encrypted/obfuscated with a simple XOR bit operation, and needed a command line tool for it.

In the end I found xortool-xor that is part of the pip-installable xortool. It can be used as follows:

$ echo "foo bar baz" | xortool-xor -r secretkey --no-cycle -f - > test.enc
$ xortool-xor --no-cycle -r secretkey -f test.enc
foo bar baz


$

I used the --no-cycle parameter because the length of the encrypted data need to be a multiple of the key length. This is not the case in my example, and I wanted no strange data appended at the end.

When I'm setting up a MariaDB or MySQL server, there is no GUI - at least at first. But it is still the time to add new users and check why the login for that certain user fails.

The mysql.user table is the right place, but with its 46 columns the mysql cli client's output is unreadable.

But there is a trick: MySQL's command line client has a special "ego" command:

ego, \G

Send the current statement to the server to be executed and display the result using vertical format.

Let's try it:

$ mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 50
Server version: 10.1.37-MariaDB-3 Debian buildd-unstable

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SELECT * FROM mysql.user LIMIT 1\G
*************************** 1. row ***************************
                  Host: localhost
                  User: root
              Password: 
           Select_priv: Y
           Insert_priv: Y
           Update_priv: Y
           Delete_priv: Y
           Create_priv: Y
             Drop_priv: Y
           Reload_priv: Y
         Shutdown_priv: Y
          Process_priv: Y
             File_priv: Y
            Grant_priv: Y
       References_priv: Y
            Index_priv: Y
            Alter_priv: Y
          Show_db_priv: Y
            Super_priv: Y
 Create_tmp_table_priv: Y
      Lock_tables_priv: Y
          Execute_priv: Y
       Repl_slave_priv: Y
      Repl_client_priv: Y
      Create_view_priv: Y
        Show_view_priv: Y
   Create_routine_priv: Y
    Alter_routine_priv: Y
      Create_user_priv: Y
            Event_priv: Y
          Trigger_priv: Y
Create_tablespace_priv: Y
              ssl_type: 
            ssl_cipher: 
           x509_issuer: 
          x509_subject: 
         max_questions: 0
           max_updates: 0
       max_connections: 0
  max_user_connections: 0
                plugin: unix_socket
 authentication_string: 
      password_expired: N
               is_role: N
          default_role: 
    max_statement_time: 0.000000
1 row in set (0.00 sec)

MariaDB [(none)]>

During the move to our new server, I needed to test if IMAP and SMTP authentication work. Using a mail client is tedious, so I used the awesome curl.

IMAP

I already wrote about using curl for IMAP login a year ago, but here it is again:

$ curl -k imaps://alice%40cweiske.de:password@mail.cweiske.de/

SMTP

You need a mail.txt file that you want to send:

$ curl -vk smtp://mail.cweiske.de:587/ \
  --mail-from alice@cweiske.de'\
  --mail-rcpt bob@cweiske.de'\
  --upload-file mail.txt\
  --user 'alice@cweiske.de:password' --ssl

When running a shell in a docker container, you only see random hashes as hostname:

$ docker exec -it project_backend_1 bash
root@112adda3eb64:/#

Now imagine having a dozen of terminals open, and then you run ./vendor/bin/phpunit in container 71f68dcd5379. The first thing that the PHPUnit bootstrap script does is emptying the database and then running all migrations and seeds.

Unfortunately, you intendet to run that command in 112adda3eb64, your local development container. Let's just say that 71f68dcd5379 was not the the local dev one, but on a server in a data center, and the data thrown away were kind of important.

Show the environment

To prevent such mistakes in the future, the shell shall clearly show which environment you are in - local development, testing, staging or production.

APP_ENV

This environment is available in our Laravel .env file, but it's not so easy to access in the terminal. So the first step is to add the current environment in the docker-compose.yml file:

---
version: "3"

services:

  backend:
    image: docker-hub.example.org/project/backend-dev:latest
    environment:
      - APP_ENV=local

Now we can access this variable in our shell via $APP_ENV.

Bash prompt

The bash prompt $PS1 is set in two places in the Ubuntu 16.04 images that we used:

/etc/bash.bashrc

Loaded when bash is used, no matter which user

/root/.bashrc

Is loaded after the /etc/ version when the user is root.

Both files define $PS1, so we have to load our bash-coloring file in both of them:

FROM ubuntu:xenial
ADD bash.colorprompt /etc/bash.colorprompt
RUN echo '. /etc/bash.colorprompt' >> /etc/bash.bashrc\
 && echo '. /etc/bash.colorprompt' >> /root/.bashrc

Now the only thing left is to write that file that sets the prompt:

# color the prompt according to $APP_ENV variable
case "$APP_ENV" in
    production)
        PS1='\e[41m\n=== $APP_ENV ===\e[m\n\u@\h:\w\$ '
        ;;
    testing)
        PS1='\e[43m$APP_ENV\e[m \u@\h:\w\$ '
        ;;
    local)
        PS1='\e[42m$APP_ENV\e[m \u@\h:\w\$ '
        ;;
esac

Screenshots

PHPUnit?

The obvious question is why PHPUnit was available on that system in the first place.

Our CI server runs unit/integration tests on every deployment, no matter which environment is being deployed to:

1. Build container with environment-specific configuration
2. Run tests in container
3. Deploy and start container on server

While this is in general a good idea, running the tests on the deployment to every environment is something we later stopped doing.

It turned out to be hard to make sure that every single configuration variable is overwritten in phpunit.xml. And if you can't be sure of this, your tests suddenly use some obscure production service that you forgot to stub out.

Sometimes I need to remove tags HTML page that I fetched with curl on the command line. It's pretty easy to do with html2text:

$ curl -s example.org | html2text

Today I had two (planned) power outages that I forgot about, and when I came home my home server would not boot up anymore. I could not ssh into it, and it is located in the server room without a display.

I wanted to connect the home server with a serial cable to my laptop and see what happened, but the serial console was not enabled :/

Enabling

Here is what I had to do to activate the serial console on Debian 9:

1. Edit /etc/default/grub and change the GRUB_CMDLINE_LINUX_DEFAULT line to:

   GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0 console=tty0"

   It enables the serial console in the linux kernel. Note that the default speed is 9600 baud.

2. Run update-grub. /boot/grub/grub.cfg will contain the console kernel parameters now.
3. Reboot. The serial console is activated.

Connect

Before rebooting, I connected my laptop via a USB-to-serial adapter and a serial cable to the home server. Then I started minicom:

$ minicom -D /dev/ttyUSB0

Once started, I changed the speed to 9600:

1. Ctrl + A
2. p (connection settings)
3. c (selects 9600)
4. Return (close dialog)

(Yes, I tried setting the speed with a cli parameter, but that did not work.)

Then I rebooted the home server and saw the kernel messages flowing into minicom. Then I could login as root and examine my system.

$ systemctl enable serial-getty@ttyS0.service

A relative deleted unneeded files on internal memory of their Android 6 phone (Samsung Galaxy S5 mini), only to discover later that all camera images had been wiped as well. I was asked to recover them.

Recovering files on a harddisk or SD card is easy; simply start PhotoRec and let it scan the partition. I wanted to do the same with the phone's internal data partition.

Partition data

Since Android 3 or 4, it is not possible to mount the partitions via USB anymore because that meant that the phone itself could not acccess the partition during that time. Instead MTP was invented which abstracts from the file system.

Another way to get the data is to use adb shell to stream the user data partition file /dev/block/mmcblk0p21 via USB into a local file. At first you need root access, which I got with CF-Auto-Rootö

At first I used a simple cat to get the data:

$ adb shell su -c "cat /dev/block/mmcblk0p21" | pv > mmcblk0p21.raw

At first I thought that the data were encrypted, but Adebar told me that encryption was not enabled (docs/deviceInfo.md).

Unfortunately, adb shell replaces newlines with windows newlines which breaks binary data. Several pages on the internet recommended using sed to strip those carriage returns out, but that also breaks legitimate CRs. Adebar and others used stty raw to switch to binary-friendly output:

$ adb shell "su -c 'stty raw; cat /dev/block/mmcblk0p21'" | pv > mmcblk0p21.raw

But data were broken again; file did not recognize the ext4 partition and photorec did only find text files. Using hexdump -C mmcblk0p21.raw showed me my problem:

sush: stty: not found

It seems on Android 6 there is no stty anymore :/ Fortunately, adb has an undocumented command exec-out that does binary-safe data transfer:

$ adb exec-out "su -c 'cat /dev/block/mmcblk0p21'" | pv > execout-mmcblk0p21.raw

With that, file recognized the ext4 fs and photorec did at least find the images that were not deleted.

TRIM and deleted files

photorec was not able to recover any deleted image files. This is when I learned about TRIM:

Flash storage devices can only write data into empty blocks. Before overwriting a block, they must empty it first.

This slows down the process, so the operating system tells the flash storage devices This area is empty when a file gets deleted logically. The flash device will then fully empty the blocks when it has nothing else to do.

This is why the recovery software could not restore anything; the flash storage had fully deleted the data in the meantime. Maybe there would have been a chance if my relative would have immediately switched off the phone when recognizing the error. But so those data are lost forever.

Other notes

One Way to Use a Linux Computer to Recover Files from an Android Device is outdated. It describes Android 4 which did not implement TRIM.

I believe that TRIM is not used on SD cards. The photos thus would have been recoverable when they had been stored on a SD card.

To decide if your shell application should output ANSI color codes, check if stdout is a tty.

PHP-SQLlint syntax highlighting

PHP-SQLlint does not only check the syntax of SQL files but can format them, syntax-highlighting included.

Outputting ANSI color codes makes only sense when a human is going to see them, not when php-sqllint is used inside a pipe command chain.

To pipe or not to pipe

Shell programs today often check if the stdout pipe of the current process is a tty, which means it is interactive.

If it's interactive, a user is sitting in front of it and we can use colors. If not, it's a pipe or redirection and no colors should be used.

PHP

PHP's POSIX extension has a method for that: posix_isatty. Because it's not available everywhere you have to check if is there:

]]>

You could use the Console_Color2 library to generate ANSI color codes.

Highlighting algorithm

tty-autodetection may not work (e.g. on Windows with ANSI.SYS), or you want to pipe the output to less -R, which is able to handle ANSI escape sequences.

Because of that uncertainties I added a command line option --highlight to php-sqllint:

--highlight=auto

Autodetection via isatty. Default if no option is given.

--highlight=ansi

If you want to force ANSI colors, for use with e.g. less -R.

--highlight=none

Disable highlighting, for easier copy+pasting.

for file in `grep -l 'rel="shadowbox' raw/*.htm`; do echo $file; for imgsrc in `xmlstarlet sel -q -t -v '//_:a[@rel and not(@data-size)]/@href' "$file"`; do size=`exiftool -T -Imagesize raw/$imgsrc`; echo $imgsrc $size; xmlstarlet ed --inplace -P --append "//_:a[@href='$imgsrc' and not(@data-size)]" --type attr -n data-size --value "$size" "$file"; done; done

For this blog I wanted to have an image gallery that works on mobile devices. I found the open source PhotoSwipe library, and after some days I had it integrated in my blog.

PhotoSwipe requires you to specify the full image size when initializing; it does not auto-detect it. I had 29 blog posts with image galleries, and over a hundred images in them - adding the image sizes manually was not an option.

I opted for a HTML5 data attribute on the link to the large image:

<a href="image.jpg" data-size="1200x800">..

What I had to do:

1. Find all files with galleries

   $ grep -l 'rel="shadowbox' raw/*.htm

2. Extract image paths from the HTML files

   $ xmlstarlet sel -q -t -v '//_:a[@rel and not(@data-size)]/@href' "$file"

3. Extract the image size

   $ exiftool -T -Imagesize "raw/$imgsrc"

4. Add the data-size attribute to the link tags which link to the image:

   $ xmlstarlet ed --inplace -P --append "//_:a[@href='$imgsrc' and not(@data-size)]" --type attr -n data-size --value "$size" "$file"

And this all into one nice shell script:

for file in `grep -l 'rel="shadowbox' raw/*.htm`
do
    echo $file
    for imgsrc in `xmlstarlet sel -q -t -v '//_:a[@rel and not(@data-size)]/@href' "$file"`
    do
        size=`exiftool -T -Imagesize raw/$imgsrc`
        echo $imgsrc $size
        xmlstarlet ed --inplace -P --append "//_:a[@href='$imgsrc' and not(@data-size)]" --type attr -n data-size --value "$size" "$file"
    done
done

This all did only work because I my blog posts are XHTML.

You can see the new galleries in e.g. Kinderzimmerlampe im Eigenbau and Playing Tomb Raider 1 on OUYA.

When working with Android key layout files you need to know the (USB) vendor and product ID to place your map file at the correct path of /system/usr/keylayout/Vendor_XXXX_Product_XXXX.kl.

This does not only work for USB but also for Bluetooth devices.

dmesg on the Android phone will contain lines like that when an USB controller is connected:

input: OUYA Game Controller as /devices/virtual/misc/uhid/input14
hid-generic 0005:2836:0001.0002: input: BLUETOOTH HID v1.04 Gamepad [OUYA Game Controller] ...

This lines contain all you need:

0005

Device type

2836

Vendor ID

0001

Product ID

But you don't always want to watch dmesg. An input device directory was created; the path given in dmesg is relative to /sys - /sys/devices/virtual/misc/uhid/input14/.

It contains files that give us the same information as the dmesg line above.

$ cat /sys/devices/virtual/misc/uhid/input14/name
OUYA Game Controller
$ cat /sys/devices/virtual/misc/uhid/input14/id/bustype
0005
$ cat /sys/devices/virtual/misc/uhid/input14/id/vendor
2836
$ cat /sys/devices/virtual/misc/uhid/input14/id/product
0001
$ cat /sys/devices/virtual/misc/uhid/input14/id/version
0104

The possible numbers in the bustype file are enumerated in /usr/include/linux/input.h:

$ grep BUS_ /usr/include/linux/input.h
#define BUS_PCI         0x01
#define BUS_ISAPNP      0x02
#define BUS_USB         0x03
#define BUS_HIL         0x04
#define BUS_BLUETOOTH   0x05
#define BUS_VIRTUAL     0x06
#define BUS_ISA         0x10
#define BUS_I8042       0x11
#define BUS_XTKBD       0x12
#define BUS_RS232       0x13
#define BUS_GAMEPORT    0x14
#define BUS_PARPORT     0x15
#define BUS_AMIGA       0x16
#define BUS_ADB         0x17
#define BUS_I2C         0x18
#define BUS_HOST        0x19
#define BUS_GSC         0x1A
#define BUS_ATARI       0x1B
#define BUS_SPI         0x1C

So if you want to list all bluetooth input devices on your Android phone from the adb shell, iterate over /sys/devices/virtual/misc/uhid/input*/ and check which bustype file contains 5.

Own bluetooth MAC address

To obtain the phone's own bluetooth MAC address, use the following command:

$ settings get secure bluetooth_address
22:22:CE:82:A8:C5