The latest posts in full-text for feed readers.
I suspected some firmware file to be encrypted/obfuscated with a simple XOR bit operation, and needed a command line tool for it.
In the end I found xortool-xor that is part of the pip-installable xortool. It can be used as follows:
$ echo "foo bar baz" | xortool-xor -r secretkey --no-cycle -f - > test.enc $ xortool-xor --no-cycle -r secretkey -f test.enc foo bar baz $
I used the --no-cycle parameter because the length of the encrypted data need to be a multiple of the key length. This is not the case in my example, and I wanted no strange data appended at the end.
Published on 2020-11-01 in linux, shell
When I'm setting up a MariaDB or MySQL server, there is no GUI - at least at first. But it is still the time to add new users and check why the login for that certain user fails.
The mysql.user table is the right place, but with its 46 columns the mysql cli client's output is unreadable.
But there is a trick: MySQL's command line client has a special "ego" command:
- ego, \G
- Send the current statement to the server to be executed and display the result using vertical format.
Let's try it:
$ mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 50 Server version: 10.1.37-MariaDB-3 Debian buildd-unstable Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> SELECT * FROM mysql.user LIMIT 1\G *************************** 1. row *************************** Host: localhost User: root Password: Select_priv: Y Insert_priv: Y Update_priv: Y Delete_priv: Y Create_priv: Y Drop_priv: Y Reload_priv: Y Shutdown_priv: Y Process_priv: Y File_priv: Y Grant_priv: Y References_priv: Y Index_priv: Y Alter_priv: Y Show_db_priv: Y Super_priv: Y Create_tmp_table_priv: Y Lock_tables_priv: Y Execute_priv: Y Repl_slave_priv: Y Repl_client_priv: Y Create_view_priv: Y Show_view_priv: Y Create_routine_priv: Y Alter_routine_priv: Y Create_user_priv: Y Event_priv: Y Trigger_priv: Y Create_tablespace_priv: Y ssl_type: ssl_cipher: x509_issuer: x509_subject: max_questions: 0 max_updates: 0 max_connections: 0 max_user_connections: 0 plugin: unix_socket authentication_string: password_expired: N is_role: N default_role: max_statement_time: 0.000000 1 row in set (0.00 sec) MariaDB [(none)]>
Published on 2019-01-10 in shell, sql
During the move to our new server, I needed to test if IMAP and SMTP authentication work. Using a mail client is tedious, so I used the awesome curl.
I already wrote about using curl for IMAP login a year ago, but here it is again:
$ curl -k imaps://alice%40cweiske.de:password@mail.cweiske.de/
You need a mail.txt file that you want to send:
$ curl -vk smtp://mail.cweiske.de:587/ \ --mail-from alice@cweiske.de'\ --mail-rcpt bob@cweiske.de'\ --upload-file mail.txt\ --user 'alice@cweiske.de:password' --ssl
Published on 2018-12-20 in mail, shell
When running a shell in a docker container, you only see random hashes as hostname:
$ docker exec -it project_backend_1 bash
root@112adda3eb64:/#
Now imagine having a dozen of terminals open, and then you run ./vendor/bin/phpunit in container 71f68dcd5379. The first thing that the PHPUnit bootstrap script does is emptying the database and then running all migrations and seeds.
Unfortunately, you intendet to run that command in 112adda3eb64, your local development container. Let's just say that 71f68dcd5379 was not the the local dev one, but on a server in a data center, and the data thrown away were kind of important.
To prevent such mistakes in the future, the shell shall clearly show which environment you are in - local development, testing, staging or production.
This environment is available in our Laravel .env file, but it's not so easy to access in the terminal. So the first step is to add the current environment in the docker-compose.yml file:
---
version: "3"
services:
backend:
image: docker-hub.example.org/project/backend-dev:latest
environment:
- APP_ENV=local
Now we can access this variable in our shell via $APP_ENV.
The bash prompt $PS1 is set in two places in the Ubuntu 16.04 images that we used:
Both files define $PS1, so we have to load our bash-coloring file in both of them:
FROM ubuntu:xenial
ADD bash.colorprompt /etc/bash.colorprompt
RUN echo '. /etc/bash.colorprompt' >> /etc/bash.bashrc\
&& echo '. /etc/bash.colorprompt' >> /root/.bashrc
Now the only thing left is to write that file that sets the prompt:
# color the prompt according to $APP_ENV variable
case "$APP_ENV" in
production)
PS1='\e[41m\n=== $APP_ENV ===\e[m\n\u@\h:\w\$ '
;;
testing)
PS1='\e[43m$APP_ENV\e[m \u@\h:\w\$ '
;;
local)
PS1='\e[42m$APP_ENV\e[m \u@\h:\w\$ '
;;
esac
The obvious question is why PHPUnit was available on that system in the first place.
Our CI server runs unit/integration tests on every deployment, no matter which environment is being deployed to:
While this is in general a good idea, running the tests on the deployment to every environment is something we later stopped doing.
It turned out to be hard to make sure that every single configuration variable is overwritten in phpunit.xml. And if you can't be sure of this, your tests suddenly use some obscure production service that you forgot to stub out.
Published on 2018-11-22 in linux, mogic, php, shell
Sometimes I need to remove tags HTML page that I fetched with curl on the command line. It's pretty easy to do with html2text:
$ curl -s example.org | html2text
Published on 2018-03-06 in linux, shell, web
Today I had two (planned) power outages that I forgot about, and when I came home my home server would not boot up anymore. I could not ssh into it, and it is located in the server room without a display.
I wanted to connect the home server with a serial cable to my laptop and see what happened, but the serial console was not enabled :/
Here is what I had to do to activate the serial console on Debian 9:
Edit /etc/default/grub and change the GRUB_CMDLINE_LINUX_DEFAULT line to:
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0 console=tty0"
It enables the serial console in the linux kernel. Note that the default speed is 9600 baud.
Before rebooting, I connected my laptop via a USB-to-serial adapter and a serial cable to the home server. Then I started minicom:
$ minicom -D /dev/ttyUSB0
Once started, I changed the speed to 9600:
(Yes, I tried setting the speed with a cli parameter, but that did not work.)
Then I rebooted the home server and saw the kernel messages flowing into minicom. Then I could login as root and examine my system.
systemd also offers serial console support, but enabling it manually via
$ systemctl enable serial-getty@ttyS0.service
only led to the issue that I could not see system service startup messages anymore when a display is connected.
But it's also automatically enabled when the console kernel parameter is detected.. Strange.
Published on 2018-03-05 in linux, shell
A relative deleted unneeded files on internal memory of their Android 6 phone (Samsung Galaxy S5 mini), only to discover later that all camera images had been wiped as well. I was asked to recover them.
Recovering files on a harddisk or SD card is easy; simply start PhotoRec and let it scan the partition. I wanted to do the same with the phone's internal data partition.
Since Android 3 or 4, it is not possible to mount the partitions via USB anymore because that meant that the phone itself could not acccess the partition during that time. Instead MTP was invented which abstracts from the file system.
Another way to get the data is to use adb shell to stream the user data partition file /dev/block/mmcblk0p21 via USB into a local file. At first you need root access, which I got with CF-Auto-Rootö
At first I used a simple cat to get the data:
$ adb shell su -c "cat /dev/block/mmcblk0p21" | pv > mmcblk0p21.raw
At first I thought that the data were encrypted, but Adebar told me that encryption was not enabled (docs/deviceInfo.md).
Unfortunately, adb shell replaces newlines with windows newlines which breaks binary data. Several pages on the internet recommended using sed to strip those carriage returns out, but that also breaks legitimate CRs. Adebar and others used stty raw to switch to binary-friendly output:
$ adb shell "su -c 'stty raw; cat /dev/block/mmcblk0p21'" | pv > mmcblk0p21.raw
But data were broken again; file did not recognize the ext4 partition and photorec did only find text files. Using hexdump -C mmcblk0p21.raw showed me my problem:
sush: stty: not found
It seems on Android 6 there is no stty anymore :/ Fortunately, adb has an undocumented command exec-out that does binary-safe data transfer:
$ adb exec-out "su -c 'cat /dev/block/mmcblk0p21'" | pv > execout-mmcblk0p21.raw
With that, file recognized the ext4 fs and photorec did at least find the images that were not deleted.
photorec was not able to recover any deleted image files. This is when I learned about TRIM:
Flash storage devices can only write data into empty blocks. Before overwriting a block, they must empty it first.
This slows down the process, so the operating system tells the flash
storage devices This area is empty
when a file gets deleted logically.
The flash device will then fully empty the blocks when it has nothing
else to do.
This is why the recovery software could not restore anything; the flash storage had fully deleted the data in the meantime. Maybe there would have been a chance if my relative would have immediately switched off the phone when recognizing the error. But so those data are lost forever.
One Way to Use a Linux Computer to Recover Files from an Android Device is outdated. It describes Android 4 which did not implement TRIM.
I believe that TRIM is not used on SD cards. The photos thus would have been recoverable when they had been stored on a SD card.
Published on 2017-04-11 in android, shell
To decide if your shell application should output ANSI color codes, check if stdout is a tty.
PHP-SQLlint does not only check the syntax of SQL files but can format them, syntax-highlighting included.
Outputting ANSI color codes makes only sense when a human is going to see them, not when php-sqllint is used inside a pipe command chain.
Shell programs today often check if the stdout pipe of the current process is a tty, which means it is interactive.
If it's interactive, a user is sitting in front of it and we can use colors. If not, it's a pipe or redirection and no colors should be used.
PHP's POSIX extension has a method for that: posix_isatty. Because it's not available everywhere you have to check if is there:
]]>
You could use the Console_Color2 library to generate ANSI color codes.
tty-autodetection may not work (e.g. on Windows with ANSI.SYS), or you want to pipe the output to less -R, which is able to handle ANSI escape sequences.
Because of that uncertainties I added a command line option --highlight to php-sqllint:
Published on 2016-12-28 in php, shell
for file in `grep -l 'rel="shadowbox' raw/*.htm`; do echo $file; for imgsrc in `xmlstarlet sel -q -t -v '//_:a[@rel and not(@data-size)]/@href' "$file"`; do size=`exiftool -T -Imagesize raw/$imgsrc`; echo $imgsrc $size; xmlstarlet ed --inplace -P --append "//_:a[@href='$imgsrc' and not(@data-size)]" --type attr -n data-size --value "$size" "$file"; done; done
For this blog I wanted to have an image gallery that works on mobile devices. I found the open source PhotoSwipe library, and after some days I had it integrated in my blog.
PhotoSwipe requires you to specify the full image size when initializing; it does not auto-detect it. I had 29 blog posts with image galleries, and over a hundred images in them - adding the image sizes manually was not an option.
I opted for a HTML5 data attribute on the link to the large image:
<a href="image.jpg" data-size="1200x800">..
What I had to do:
Find all files with galleries
$ grep -l 'rel="shadowbox' raw/*.htm
Extract image paths from the HTML files
$ xmlstarlet sel -q -t -v '//_:a[@rel and not(@data-size)]/@href' "$file"
Extract the image size
$ exiftool -T -Imagesize "raw/$imgsrc"
Add the data-size attribute to the link tags which link to the image:
$ xmlstarlet ed --inplace -P --append "//_:a[@href='$imgsrc' and not(@data-size)]" --type attr -n data-size --value "$size" "$file"
And this all into one nice shell script:
for file in `grep -l 'rel="shadowbox' raw/*.htm`
do
echo $file
for imgsrc in `xmlstarlet sel -q -t -v '//_:a[@rel and not(@data-size)]/@href' "$file"`
do
size=`exiftool -T -Imagesize raw/$imgsrc`
echo $imgsrc $size
xmlstarlet ed --inplace -P --append "//_:a[@href='$imgsrc' and not(@data-size)]" --type attr -n data-size --value "$size" "$file"
done
done
This all did only work because I my blog posts are XHTML.
You can see the new galleries in e.g. Kinderzimmerlampe im Eigenbau and Playing Tomb Raider 1 on OUYA.
Published on 2016-05-20 in html, shell, xml
When working with Android key layout files you need to know the (USB) vendor and product ID to place your map file at the correct path of /system/usr/keylayout/Vendor_XXXX_Product_XXXX.kl.
This does not only work for USB but also for Bluetooth devices.
dmesg on the Android phone will contain lines like that when an USB controller is connected:
input: OUYA Game Controller as /devices/virtual/misc/uhid/input14 hid-generic 0005:2836:0001.0002: input: BLUETOOTH HID v1.04 Gamepad [OUYA Game Controller] ...
This lines contain all you need:
But you don't always want to watch dmesg. An input device directory was created; the path given in dmesg is relative to /sys - /sys/devices/virtual/misc/uhid/input14/.
It contains files that give us the same information as the dmesg line above.
$ cat /sys/devices/virtual/misc/uhid/input14/name
OUYA Game Controller
$ cat /sys/devices/virtual/misc/uhid/input14/id/bustype
0005
$ cat /sys/devices/virtual/misc/uhid/input14/id/vendor
2836
$ cat /sys/devices/virtual/misc/uhid/input14/id/product
0001
$ cat /sys/devices/virtual/misc/uhid/input14/id/version
0104
The possible numbers in the bustype file are enumerated in /usr/include/linux/input.h:
$ grep BUS_ /usr/include/linux/input.h
#define BUS_PCI 0x01
#define BUS_ISAPNP 0x02
#define BUS_USB 0x03
#define BUS_HIL 0x04
#define BUS_BLUETOOTH 0x05
#define BUS_VIRTUAL 0x06
#define BUS_ISA 0x10
#define BUS_I8042 0x11
#define BUS_XTKBD 0x12
#define BUS_RS232 0x13
#define BUS_GAMEPORT 0x14
#define BUS_PARPORT 0x15
#define BUS_AMIGA 0x16
#define BUS_ADB 0x17
#define BUS_I2C 0x18
#define BUS_HOST 0x19
#define BUS_GSC 0x1A
#define BUS_ATARI 0x1B
#define BUS_SPI 0x1C
So if you want to list all bluetooth input devices on your Android phone from the adb shell, iterate over /sys/devices/virtual/misc/uhid/input*/ and check which bustype file contains 5.
To obtain the phone's own bluetooth MAC address, use the following command:
$ settings get secure bluetooth_address
22:22:CE:82:A8:C5
Published on 2016-03-21 in android, shell