Sudo cp path to profit usr local bin là gì

Hi! I’m writing this article as a mini-HOWTO on how to setup a btrfs-raid1 volume on encrypted disks [luks]. This page servers as my personal guide/documentation, althought you can use it with little intervention.

Disclaimer: Be very careful! This is a mini-HOWTO article, do not copy/paste commands. Modify them to fit your environment.

$ date -R
Thu, 03 Dec 2020 07:58:49 +0200

Prologue

I had to replace one of my existing data/media setup [btrfs-raid0] due to some random hardware errors in one of the disks. The existing disks are 7.1y WD 1TB and the new disks are WD Purple 4TB.

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

This will give me about 3.64T [from 1.86T]. I had concerns with the slower RPM but in the end of this article, you will see some related stats.

My primarly daily use is streaming media [video/audio/images] via minidlna instead of cifs/nfs [samba], although the service is still up & running.

Disks

It is important to use disks with the exact same size and speed. Usually for Raid 1 purposes, I prefer using the same model. One can argue that diversity of models and manufactures to reduce possible firmware issues of a specific series should be preferable. When working with Raid 1, the most important things to consider are:

Geometry [size]
RPM [speed]

and all the disks should have the same specs, otherwise size and speed will downgraded to the smaller and slower disk.

Identify Disks

the two [2] Western Digital Purple 4TB are manufacture model: WDC WD40PURZ

The system sees them as:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

try to identify them from the kernel with list block devices:

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

verify it with hwinfo

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

with list hardware:

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

Luks

Create Random Encrypted keys

I prefer to use random generated keys for the disk encryption. This is also useful for automated scripts [encrypt/decrypt disks] instead of typing a pass phrase.

Create a folder to save the encrypted keys:

$ sudo mkdir -pv /etc/crypttab.keys/

create keys with dd against urandom:

WD40PURZ-85A

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

WD40PURZ-85T

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

verify two [2] 4k size random keys, exist on the above directory with list files:

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

Format & Encrypt Hard Disks

It is time to format and encrypt the hard disks with Luks

Be very careful, choose the correct disk, type uppercase YES to confirm.

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Verify Encrypted Disks

print block device attributes:

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Open and Decrypt

opening encrypted disks with luks

WD40PURZ-85A

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

WD40PURZ-85T

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Verify Status

WD40PURZ-85A

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

WD40PURZ-85T

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

BTRFS

Current disks

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

There are a lot of write/read errors :[

btrfs version

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Create BTRFS Raid 1 Filesystem

by using mkfs, selecting a disk label, choosing raid1 metadata and data to be on both disks [mirror]:

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

or in one-liner [as-root]:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

format output

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Notice that both disks have the same UUID [Universal Unique IDentifier] number:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Verify block device

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

once more, be aware of the same

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

04 on both disks!

Mount new block disk

create a new mount point

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

append the below entry in /etc/fstab [as-root]

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

and finally, mount it!

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Disk Usage

check disk usage and free space for the new encrypted mount point

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

btrfs filesystem disk usage

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

btrfs filesystem show

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

stats

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

btrfs fi disk usage

btrfs filesystem disk usage

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

Speed

Using hdparm to test/get some speed stats

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

These are the new disks with 5400 rpm, let’s see what the old 7200 rpm disk shows here:

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

So even that these new disks are 5400 seems to be faster than the old ones !! Also, I have mounted as read-only the problematic Raid-0 setup.

Rsync

I am now moving some data to measure time

Folder-A

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

Folder-B

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

Control and Monitor Utility for SMART Disks

Last but not least, some smart info with smartmontools

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

result :

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

details

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

Second disk

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

selftest results

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

details

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

that’s it !

-ebal

Tag: btrfs, raid, raid1, luks

Network Booting into Graphical Linux, before it was cool!

Posted by ebal at 15:50:56 in blog

back in ~2001 I was working part time in my uni lab for some extra cash and a chance to gain some additional knowledge on hardware & linux. I feel that I need to make a disclaimer here and share that prior to christmas of ‘99, I did not own a personal computer or a PC as it is better known.

Our tech lab had to format and repair/clone/restore hard disks on a daily basis, as the majority of PCs were failing on a regular interval. That was the result of having 80 to 160 students on 10/15 PCs per lab, running 10/12 hours a day. No one had a dedicated PC/seat. Hard disks were failing left and right. Tech lab had to format/restore them or in case of total failure, order a replacement disk. We had to make these orders in bulk.

So, we had to investigating this issue and report back with a solution as the backlog and cost was notable from our uni. From what we have noticed, over 50% of students did not want to wait for logout and safely shutdown their machines. After saving homework on their floppy disks, most of them, just turned off the power button on the back of the tower power supply. Imaging doing that, on hard disk 20years ago, for about 10 times a day, every day on a common lab.

After brainstorming about possible solutions and workarounds, finding ways to restore faster these hard disks, making poster with “Please safely shutdown this Computer”, an idea was introduced as an experiment.

How about completely remove hard disks from the PCs?

We knew it was possible to boot linux from the local network, but can we boot into a graphical environment? We also had to convince lab professors that some of the lab courses can run under linux like Pascal/C/Fortran/Lisp etc. In the end we had to have a dedicated lab for those diskless PCs.

Challenge was indeed accepted, and we had to provide and present a PoC [proof of concept]. The project was well known as LTSP aka the Linux Terminal Server Project. Half of the team was looking into the server part; the other half was working on the client part.

Let us recap for a moment. A PC without a hard disk/operating system, had to boot from network and start into graphical environment. To boot from the network, your bios should have this option. Not an option back then. Then your network card should get some information from the DHCP about tftpd server. Not an option back then. And last your PC should have a proper graphical card that can indeed run X11 [not Xorg, as Xorg was not yet released back then] and remote login to an X server! On top of that CPU & memory was not so “powerful” back then.

I will not get into tech details, this is not that kind of post, but I will share a few details. First, we had to tell the bios to boot from the network card and then we had to program the ROM of the network card. That was not so easy back then. Mostly because not all network cards had a programmable ROM, those were the expensive ones. After that we had to get the boot image from the server, boot into RAM and load the boot menu. Selecting the proper OS, get the kernel and initramfs [if needed for extra drivers] and [pivot] boot into a linux operating system. Finally, we had to auto start in X11 remote login client that was configured to connect to the X11 Server.

diskless boot, etherboot and EPROM configuration, was the game and floppies were the solution! If not mistaken [I may be, don’t remember everything] I think we had to tell the bios to boot from floppy and set the EPROM configuration of the network card and then boot from network to get the initial boot menu. Get the kernel and run X11, login into the X server and run the lab courses to prove that this is working. That was what indeed do! And for another year and a half, my daily computer in the uni was a diskless PII that was booting from network via a floppy disk!

Nowadays you can remote install/format/repair your laptop through the internet, effortless. And my PC had a Turbo button also!!

Tag: ltsp

VMs on KVM with Terraform

many thanks to erethon for his help & support on this article.

Working on your home lab, it is quiet often that you need to spawn containers or virtual machines to test or develop something. I was doing this kind of testing with public cloud providers with minimal VMs and for short time of periods to reduce any costs. In this article I will try to explain how to use libvirt -that means kvm- with terraform and provide a simple way to run this on your linux machine.

Be aware this will be a [long] technical article and some experience is needed with kvm/libvirt & terraform but I will try to keep it simple so you can follow the instructions.

Terraform

Install Terraform v0.13 either from your distro or directly from hashicopr’s site.

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

Libvirt

same thing for libvirt

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

verify that you have access to libvirt

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

Terraform Libvirt Provider

To access the libvirt daemon via terraform, we need the terraform-libvirt provider.

Terraform provider to provision infrastructure with Linux’s KVM using libvirt

The official repo is on GitHub - dmacvicar/terraform-provider-libvirt and you can download a precompiled version for your distro from the repo, or you can download a generic version from my gitlab repo

ebal / terraform-provider-libvirt · GitLab

These are my instructions

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

Terraform Init

Let’s create a new directory and test that everything is fine.

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

$ hwinfo --short --disk
disk:
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T
$ hwinfo --block --short
  /dev/sde             WDC WD40PURZ-85A
  /dev/sdc             WDC WD40PURZ-85T

everything seems okay!

We can verify with tree or find

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

Provider

but did we actually connect to libvirtd via terraform ? Short answer: No.

We just told terraform to use this specific provider.

How to connect ? We need to add the connection libvirt uri to the provider section:

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

so our Provider.tf looks like this

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

Libvirt URI

libvirt is a virtualization api/toolkit that supports multiple drivers and thus you can use libvirt against the below virtualization platforms

LXC - Linux Containers
OpenVZ
QEMU
VirtualBox
VMware ESX
VMware Workstation/Player
Xen
Microsoft Hyper-V
Virtuozzo
Bhyve - The BSD Hypervisor

Libvirt also supports multiple authentication mechanisms like ssh

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

so it is really important to properly define the libvirt URI in terraform!

In this article, I will limit it to a local libvirt daemon, but keep in mind you can use a remote libvirt daemon as well.

Volume

Next thing, we need a disk volume!

Volume.tf

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

I have already downloaded this image and verified its checksum, I will use this local image as the base image for my VM’s volume.

By running

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

05 we will see this output:

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

What we expect is to use this source image and create a new disk volume [copy] and put it to the default disk storage libvirt pool.

Let’s try to figure out what is happening here:

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

and

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

Volume Size

BE Aware: by this declaration, the produced disk volume image will have the same size as the original source. In this case ~2G of disk.

We will show later in this article how to expand to something larger.

destroy volume

$ sudo lshw -short | grep disk
/0/100/1f.5/0        /dev/sdc   disk           4TB WDC WD40PURZ-85T
/0/100/1f.5/1        /dev/sde   disk           4TB WDC WD40PURZ-85A
$ sudo lshw -class disk -json | jq -r .[].product
WDC WD40PURZ-85T
WDC WD40PURZ-85A

verify

$ sudo mkdir -pv /etc/crypttab.keys/

reminder: always destroy!

Domain

Believe it or not, we are half way from our first VM!

We need to create a libvirt domain resource.

Domain.tf

$ sudo mkdir -pv /etc/crypttab.keys/

Apply the new tf plan

$ sudo mkdir -pv /etc/crypttab.keys/

$ sudo mkdir -pv /etc/crypttab.keys/

Verify via virsh:

$ sudo mkdir -pv /etc/crypttab.keys/

Destroy them!

$ sudo mkdir -pv /etc/crypttab.keys/

That’s it !

We have successfully created a new VM from a source image that runs on our libvirt environment.

But we can not connect/use or do anything with this instance. Not yet, we need to add a few more things. Like a network interface, a console output and a default cloud-init file to auto-configure the virtual machine.

Variables

Before continuing with the user-data [cloud-init], it is a good time to set up some terraform variables.

$ sudo mkdir -pv /etc/crypttab.keys/

We are going to use this variable within the user-date yaml file.

Cloud-init

The best way to configure a newly created virtual machine, is via cloud-init and the ability of injecting a user-data.yml file into the virtual machine via terraform-libvirt.

user-data

$ sudo mkdir -pv /etc/crypttab.keys/

cloud init disk

Terraform will create a new iso by reading the above template file and generate a proper user-data.yaml file. To use this cloud init iso, we need to configure it as a libvirt cloud-init resource.

Cloudinit.tf

$ sudo mkdir -pv /etc/crypttab.keys/

and we need to modify our Domain.tf accordingly

$ sudo mkdir -pv /etc/crypttab.keys/

Terraform will create and upload this iso disk image into the default libvirt storage pool. And attach it to the virtual machine in the boot process.

At this moment the tf_libvirt directory should look like this:

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

To give you an idea, the abstract design is this:

apply

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

Lots of output , but let me explain it really quick:

generate a user-data file from template, template is populated with variables, create an cloud-init iso, create a volume disk from source, create a virtual machine with this new volume disk and boot it with this cloud-init iso.

Pretty much, that’s it!!!

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

destroy

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

Most important detail is:

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

cloud-init.iso
ubuntu-2004-vol
ubuntu-2004-vm

Console

but there are a few things still missing.

To add a console for starters so we can connect into this virtual machine!

To do that, we need to re-edit Domain.tf and add a console output:

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

the full file should look like:

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

Create again the VM with

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

And test the console:

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

wow!

We have actually logged-in to this VM using the libvirt console!

Virtual Machine

some interesting details

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

Do not forget to destroy

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

extend the volume disk

As mentioned above, the volume’s disk size is exactly as the origin source image. In our case it’s 2G.

What we need to do, is to use the source image as a base for a new volume disk. In our new volume disk, we can declare the size we need.

I would like to make this a user variable: Variables.tf

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

Arithmetic in terraform!!

So the Volume.tf should be:

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

base image –> volume image

test it

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85A bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.00015914 s, 25.7 MB/s

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

10G !

destroy

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

Swap

I would like to have a swap partition and I will use cloud init to create a swap partition.

modify user-data.yml

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

test it

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

success !!

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

Network

How about internet? network? Yes, what about it ?

I guess you need to connect to the internets, okay then.

The easiest way is to add this your Domain.tf

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

This will use the default network libvirt resource

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

if you prefer to directly expose your VM to your local network [be very careful] then replace the above with a macvtap interface. If your ISP router provides dhcp, then your VM will take a random IP from your router.

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

test it

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

destroy

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

SSH

Okay, now that we have network it is possible to setup ssh to our virtual machine and also auto create a user. I would like cloud-init to get my public key from github and setup my user.

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

Notice, I have added a new variable called

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

Variables.tf

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

and do not forget to update your cloud-init tf

Cloudinit.tf

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

Update VM

I would also like to update & install specific packages to this virtual machine:

$ sudo ls -l /etc/crypttab.keys/WD40PURZ-85*
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85A
-rw-r--r-- 1 root root 4096 Dec  3 08:00 /etc/crypttab.keys/WD40PURZ-85T

Yes, I prefer to uninstall cloud-init at the end.

user-date.yaml

the entire user-date.yaml looks like this:

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Output

We need to know the IP to login so create a new terraform file to get the IP

Output.tf

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

but that means that we need to wait for the dhcp lease. Modify Domain.tf to tell terraform to wait.

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Plan & Apply

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Verify

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

what !!!!

awesome

destroy

$ sudo dd if=/dev/urandom of=/etc/crypttab.keys/WD40PURZ-85T bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes [4.1 kB, 4.0 KiB] copied, 0.000135452 s, 30.2 MB/s

Custom Network

One last thing I would like to discuss is how to create a new network and provide a specific IP to your VM. This will separate your VMs/lab and it is cheap & easy to setup a new libvirt network.

Network.tf

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

and replace

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

08 in Domains.tf

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Closely look, there is a new terraform variable

Variables.tf

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Terraform files

you can find every terraform example in my github repo

tf/0.13/libvirt/0.6.2/ubuntu/20.04 at master · ebal/tf · GitHub

That’s it!

If you like this article, consider following me on twitter ebalaskas.

Tag: libvirt, kvm, cloud-init, terraform, ubuntu, qemu

Curse of knowledge

[Original Published at Linkedin on October 28, 2018]

The curse of knowledge is a cognitive bias that occurs when an individual, communicating with other individuals, unknowingly assumes that the others have the background to understand.

Let’s talk about documentation

This is the one big elephant in every team’s room.

TLDR; Increment: Documentation

Documentation empowers users and technical teams to function more effectively, and can promote approachability, accessibility, efficiency, innovation, and more stable development. Bad technical guides can cause frustration, confusion, and distrust in your software, support channels, and even your brand—and they can hinder progress and productivity internally

so to avoid situations like these:

xkcd - wisdom_of_the_ancients

or/and

Optipess - TitsMcGee4782

documentation must exist!

Myths

Self-documenting code
No time to write documentation
There are code examples, what else you need?
There is a wiki page [or 300.000 pages].
I’m not a professional writer
No one reads the manual

Problems

Maintaining the documentation [up2date]
Incomplete or Confusing documentation
Documentation is platform/version oriented
Engineers who may not be fluent in English [or dont speak your language]
Too long
Too short
Documentation structure

Types of documentation

Technical Manual [system]
Tutorial [mini tutorials]
HowTo [mini howto]
Library/API documentation [reference]
Customer Documentation [end user]
Operations manual
User manual [support]
Team documentation
Project/Software documentation
Notes
FAQ

Why Documentation Is Important

Communication is a key to success. Documentation is part of the communication process. We either try to communicate or collaborate with our customers or even within our own team. We use our documentation to inform customers of new feautures and how to use them, to train our internal team [colleagues], collaborate with them, reach-out, help-out, connect, communicate our work with others.

When writing code, documentation should be the “One-Truth” instead of the code repository. I love working with projects that they will not deploy a new feature before updating the documentation first. For example I read the ‘Release Notes for Red Hat’ and the ChangeLog instead of reading myriads of code repositories.

Know Your Audience

Try to answer these questions:

Who is reading this documentation ?
Is it for internal or external users/customers ?
Do they have a dev background ?
Arey they non-technical people ?

Use personas to create diferrent material. Try to remember this one gold rule:

Audidence should get value from documentation [learning or something].

Guidelines

Here are some guidelines:

Tell a story
Use a narative voice
Try to solve a problem
Simplify - KISS philosophy
Focus on approachability and usability

Even on a technical document try to:

Write documentation agnostic - Independent Platform
Reproducibility
Not writing in acronyms and technical jargon [explain]
Step Approach
Towards goal achievement
Routines

UX

A picture is worth a thousand words

so remember to:

visual representation of the information architecture
use code examples
screencasts
CLI –help output
usage
clear error messages

Customers and Users do want to write nothing. Reducing user input, your project will be more fault tolerant.

Instead of providing a procedure for a deploy pipeline, give them a

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

09, a

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

10 Gui/Web User-Interface and focus your documentation around that.

Content

So what to include in the documentation.

How technical should be or not ?
Use cases ?
General-Purpose ?
Article size [small pages are more manageable set to maintain].
Minimum Viable Content Vs Too much detail
Help them to understand

imagine your documentation as microservices instead of a huge monolith project.

Usally a well defined structure, looks like this:

Table of Contents [toc]
Introduction
Short Description
Sections / Modules / Chapters
Conclusion / Summary
Glossary
Index
References

Tools

I prefer wiki pages instead of a word-document, because of the below features:

Version
History
Portability
Convertibility
Accountability

btw if you are using Confluence, there is a Markdown plugin.

Measurements & Feedback

To understand if your documentation is good or not, you need feedback. But first there is an evaluation process of Review. It is the same thing as writing code, you cant review your own code! The first feedback should come within your team.

Use analytics to measure if people reads your documentation, from ‘Hits per page’ to more advance analytics as Matomo [formerly Piwik]. Profiling helps you understand your audience. What they like in documentation?

Customer satisfaction [CSat] are important in documentation metrics.

Was this page helpful? Yes/No
Allowing customers to submit comments.
Upvote/Downvote / Like
or even let them to contribute in your documentation

make it easy for people to share their feedbak and find a way to include their comments in it.

FAQ

Frequently Asked Questions should answering questions in the style of:

What would customers ask ?
What if
How to

FAQ or QA should be really straight forward, short and simple as it can be. You are writing a FAQ because you are here to help customers to learn how to use this specific feature not the entire software. Use links for more detail info, that direct them to your main documentation.

Conclusion

Sharing knowledge & shaping the culture of your team/end users. Your documentation should reflect your customers needs. Everything you do in your business is to satisfy your customers. Documentation is one way to communicate this.

So here are some final points on documentation:

User Oriented
Readability
Comprehensive
Keep it up to date Tag: knowledge

How to build a SSH Bastion host

[this is a technical blog post, but easy to follow]

recently I had to setup and present my idea of a ssh bastion host. You may have already heard this as jump host or a security ssh hoping station or ssh gateway or even something else.

The main concept

Disclaimer: This is just a proof of concept [PoC]. May need a few adjustments.

The destination VM may be on another VPC, perhaps it does not have a public DNS or even a public IP. Think of this VM as not accessible. Only the ssh bastion server can reach this VM. So we need to first reach the bastion.

SSH Config

To begin with, I will share my initial sshd_config to get an idea of my current ssh setup

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

I only allow, user ebal to connect via ssh.
I do not allow the root user to login via ssh.
I use ssh keys instead of passwords.

This configuration is almost identical to both VMs

bastion [the name of the VM that acts as a bastion server]
VM [the name of the destination VM that is behind a DMZ/firewall]

~/.ssh/config

I am using the ssh config file to have an easier user experience when using ssh

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Create a new user to test this

Let us create a new user for testing.

User/Group

Western Digital Green  1TB, about  70€ each, SATA III [6 Gbit/s], 7200 RPM, 64 MB Cache
Western Digital Purple 4TB, about 100€ each, SATA III [6 Gbit/s], 5400 RPM, 64 MB Cache

Perms

Copy .ssh directory from current user [ ssh ubuntu1804

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

reboot is necessary.

update

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

upgrade

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

reboot

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Do release upgrade

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

help

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

do-release-upgrade

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

server

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

at this moment, we will switch to a gnu/screen session

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Press: y

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Press Enter

update repos

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

…

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Press y

[or review by pressing d ]

Fetching packages

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

services

at some point a question will pop:

Restart services during package upgrade without asking ?

I answered Yes but you should answer this the way you prefer.

patience is a virtue

Get a coffee or tea. Read a magazine.

Patience is a virtue

till you see a jumping animal.

resolved

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

I answered this Y, I will change it later.

vim

same here

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

ssh conf

Remove obsolete packages

and finally

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Press y to continue

Restart

are you ready to restart your machine ?

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Press y to restart

LTS 20.04

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

41 Tag: ubuntu, 18.04, 20.04, LTS

Using LibreDNS with dnscrypt-proxy

Using DNS over HTTPS aka DoH is fairly easy with the latest version of firefox. To use libredns is just a few settings in your browser, see here. In libredns’ site, there are also instructions for DNS over TLS aka DoT.

In this blog post, I am going to present how to use dnscrypt-proxy as a local dns proxy resolver using DoH the LibreDNS noAds [tracking] endpoint. With this setup, your entire operating system can use this endpoint for everything.

Disclaimer: This blog post is about dnscrypt-proxy version 2.

dnscrypt-proxy

dnscrypt-proxy 2 - A flexible DNS proxy, with support for modern encrypted DNS protocols such as DNSCrypt v2, DNS-over-HTTPS and Anonymized DNSCrypt.

Installation

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Verify Package

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Disable systemd-resolved

if necessary

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Configuration

It is time to configure dnscrypt-proxy to use libredns

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

In the top of the file, there is a server_names section

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Resolv Conf

We can now change our resolv.conf to use our local IP address.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Systemd

start & enable dnscrypt service

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Verify

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Dig

asking our local dns [proxy]

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

That’s it !

Yoursystem is now using LibreDNS DoH noads endpoint.

Manual Steps

If your operating system does not yet support dnscrypt-proxy-2 then:

Latest version

You can always download the latest version from github:

To view the files

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

To extrace the files

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Prepare the configuration

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

In the top of the file, there is a server_names section

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Run as root

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Check DNS

Interesting enough, first time is 250ms , second time is zero!

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

That’s it

Tag: LibreDNS, dnscrypt-proxy

Tools I use daily the Win10 edition

almost three [3] years ago I wrote an article about the Tools I use daily. But for the last 18 months [or so], I am partial using windows 10 due to my new job role, thus I would like to write an updated version on that article.

I’ ll try to use the same structure for comparison as the previous article, keep in mind this a nine to five setup [work related]. So here it goes.

NOTICE beer is just for decor ;]

Operating System

I use Win10 as my primary operating system in my worklaptop. I have a couple of impediments that can not work on a linux distribution but I am not going to bother you with them [it’s webex and some internal internet-explorer only sites].

We used to use webex as our primary communication tool. We are sharing our screen and have our video camera on, so that everybody can see each other.Working with remote teams, it’s kind of nice to see the faces of your coworkers. A lot of meetings are integrated with the company’s outlook. I use OWA [webmail] as an alternative but in fact it is still difficult to use both of them with a linux desktop.

We successful switched to slack for text communications, video calls and screen sharing. This choice gave us a boost in productivity as we are now daily using slack calls to align with each other. Although still webex is in the mix. Company is now using a newer webex version that works even better with browser support so that is a plus. It’s not always easy to get everybody with a webex license but as long as we are using slack it is okay. Only problem with slack in linux is when working with multiple monitors, you can not choose which monitor to share.

I have considered to use a VM [virtual machine] but a win10 vm needs more than 4G of RAM and a couple of CPUs just to boot up. In that case, it means that I have to reduce my work laptop resources for half the day, every day. So for the time being I am staying with Win10 as the primary operating system. I have to use the winVM for some other internal works but it is limited time.

Desktop

Default Win10 desktop

I daily use these OpenSource Tools:

AutoHotkey for keyboard shortcut [I like switching languages by pressing capslock]
Ditto as clipboard manager
Greenshot for screenshot tool

and from time to time, I also use:

X-Mouse Controls [window focus with mouse]
Always on Top to Keep Any Window Visible Always
Plumb a Tiling Window Manager

except plumb, everything else is opensource!

So I am trying to have the same user desktop experience as in my Linux desktop, like my language swith is capslock [authotkey], I dont even think about it.

Disk / Filesystem

Default Win10 filesystem with bitlocker. Every HW change will lock the entire system. In the past this happened twice with a windows firmware device upgrade. Twice!

Dropbox as a cloud sync software, with EncFSMP partition and syncthing for secure personal syncing files.

[same setup as linux, except bitlocker is luks]

Mail

OWA for calendar purposes and … still Thunderbird for primary reading mails.

Thunderbird 68.6.0 AddOns:

TbSync
Provider for CalDAV & CardDAV
ExQuilla for Exchange
CompactHeader
Toggle Headers
ConfigDate
Duplicate Contacts Manager
Remove Duplicates
Lightning
Expression Search / GMailUI
Mail Redirect
Markdown Here
Open With

[same setup as linux]

Shell

Windows Subsystem for Linux aka WSL … waiting for the official WSLv2 ! This is a huge HUGE upgrade for windows. I have setup an Arch Linux WSL environment to continue work on a linux environment, I mean bash. I use my WSL archlinux as a jumphost to my VMs.

Terminal Emulator

Mintty The best terminal emulator for WSL. Small, not to fancy, just works, beautiful, love it.

Editor

Using Visual Studio Code for scripting. vim within WSL and notepad for temporary text notes. I have switched to Boostnote for markdown and as my primary note editor.

[same setup as linux]

Browser

Multiple Instances of Firefox, Chromium, Tor Browser and brave

Primary Browser: Firefox Primary Private Browsing: Brave

[same setup as linux]

Communication

I use mostly Slack and Signal Desktop. We are using webex but I prefer Zoom. Riot/Matrix for decentralized groups and IRC bridge. To be honest, I also use Viber & messanger [only through webbrowser].

[same setup as linux - minus the Viber client]

Media

VLC for windows, what else ? Also GIMP for image editing. I have switched to Spotify for music and draw io for diagrams. Last, I use CPod for podcasts. Netflix [sometimes].

[same setup as linux]

In conclusion

I have switched to a majority of electron applications. I use the same applications on my Linux boxes. Encrypted notes on boostnote, synced over syncthing. Same browsers, same bash/shell, the only thing I dont have on my linux boxes are webex and outlook. Consider everything else, I think it is a decent setup across every distro.

Thanks for reading my post.

Tag: win10

restic with minio

restic is a fast, secure & efficient backup program.

I wanted to test restic for some time now. It is a go backup solution, I would say similar to rclone but it has a unique/different design. I prefer having an isolated clean environment when testing software, so I usually go with a VΜ. For this case, I installed elementary OS v5.1, an ubuntu LTS based distro focus on user experience. As backup storage solution, I used MinIO an S3 compatible object storage on the same VM. So here are my notes on restic and in the end of this article you will find how I setup minion.

Be aware this is a technical post!

restic

Most probably your distro package manager has already restic in their repositories.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

download latest version

But just in case you want to install the latest binary version, you can use this command

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

or if you are already root

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

we can see the latest version

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

autocompletion

Enable autocompletion

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

restart your shell.

Prepare your repo

We need to prepare our destination repository. This is our backup endpoint. restic can save multiple snapshots for multiple hosts on the same endpoint [repo].

Apart from the files stored within the keys directory, all files are encrypted with AES-256 in counter mode [CTR]. The integrity of the encrypted data is secured by a Poly1305-AES message authentication code [sometimes also referred to as a “signature”].

To access a restic repo, we need a key. We will use this key as password [or passphrase] and it is really important NOT to lose this key.

For automated backups [or scripts] we can use the environmental variables of our SHELL to export the password. It is best to export the password through a script or even better through a password file.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

eg.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

We can also declare the restic repository through an environmental variable

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Local Repo

An example of local backup repo should be something like this:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

minio S3

We are going to use minio as an S3 object storage, so we need to export the Access & Sercet Key in a similar way as for amazon S3.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

The S3 endpoint is

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

60 so a full example should be:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

source the config file into your shell:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Initialize Repo

We are ready to initialise the remote repo

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Be Careful if you asked to type a password, that means that you did not use a shell environmental variable to export a password. That is fine, but only if that was your purpose. Then you will see something like that:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

backup

We are ready to take our first snapshot.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

You can exclude or include files with restic, but I will not get into this right now. For more info, read Restic Documentation

standard input

restic can also take for backup:

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Check

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Take another snapshot

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

List snapshots

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Remove snapshot

as you can see, I had one more snapshot before my home dir and I want to remove it

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

list again

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Compare snapshots

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Mount a snapshot

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

open another terminal

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

So as we can see, snapshots are based on time.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

be aware as far as we have mounted the restic backup, there is a lock on the repo. Do NOT forget to close the mount point when finished.

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Check again

you may need to re-check to see if there is a lock on the repo

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Restore a snapshot

Identify which snapshot you want to restore

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

create a folder and restore the snapshot

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

List files from snapshot

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

keys

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

restic rotate snapshot policy

a few more words about forget

Forget mode has a feature of keep last

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

61 snapshots, where

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

62 can be

number of snapshots
hourly
daily
weekly
monthly
yearly

and makes restic with local feature an ideally replacement for rsnapshot!

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

Appendix - minio

MinIO is a s3 compatible object storage.

install server

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

run server

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

browser

create demo bucket

install client

$ sudo find /sys/devices -type f -name model -exec cat {} ;
WDC WD40PURZ-85A
WDC WD40PURZ-85T

configure client

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

run mc client

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

mc autocompletion

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

you need to restart your shell.

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc            8:32   0   3.6T  0 disk
sde            8:64   0   3.6T  0 disk

That’s It!

Tag: restic, minio

The story of my first job in Tech Industry

The other day I was thinking about my first ever job in this industry as a junior software engineer at the age of 20. I was doing okay with my studies at the Athens university of applied sciences but I was working outside of this industry. I had to gain some working experience in the field, so I made a decision to find part time work in a small software house. The [bad] experience and lessons learned in those couple weeks are still with me till this day … almost 20 years after!

Introductions

I got a flyer from the job board at school and I walked a couple of kilometers to the address of the place. I didn’t have a car back then [or for the next 7 years], so I had to use public transportation [bus] or walk wherever I wanted to go. I rang the doorbell around noon and went up on the second floor. There I introduced myself and asked for an opportunity to work with them. The owner/head of software team asked me a few things and got to the technical parts of the job.

We are working with visual studio, but we are using HTML pages as forms for our product. In a sense we have copied the Amazon model!

Impressed, that I was going to work with the next amazon, I immediately said Yes to the offer.

HTML4

Do you know HTML ?
No, but I am a quick study.

He smiled at me and gave me [I think] this 800 pages book to read about HTML4.

HTML-4-Bible-Bryan-Pfaffenberger

He then told me:

Read this book and come back when you finished it.

That was Friday noon.

I spent 10 hours quickly reading the book and keeping notes. Then I made a static demo site about Milos Island, where I had spent two weeks in the summer with my girlfriend. I had photos and material to write about, so I did that as an exercise.

Monday morning, I was presenting him with my homework. He didn’t believe me and spent a couple of hours talking about HTML4, just to prove that I had made the site, reading the book he gave me. In the end he was convinced.

Visual Studio

My next assignment was to learn about Visual Basic and Visual Studio. I had a basic idea about this but I had never worked as a professional programmer, so he prepared a few coding exercises to get familiar with the codebase. This was my onboarding period.

Take this exercise and come back when you finish it. It will take you about a week.

Next day, I was again first in the office.

So you came back to ask for help. That is okay. You should ask for help but you need to make an effort to do it yourself.
I finished it, it was easy.
Really? Then here is your next assignment. This is more difficult. Come back when you finished it.

Next day … I was back in the office.

I finished it, what is next ?
Okay, read this today and come back tomorrow.

Read it, returned the next day.

Done
Okay, I need you to sit here and work on the next assignments. I want to see how you are working on these coding exercises for myself.
Okay.

Next two days, worked there on coding exercises to get familiar with their codebase. He was impressed and I was very happy.

QA

Next day [Friday]:

You now have access to our production code. Here are your tasks, whenever your finish something I want to see it. But before all that, here is a copy of our product. Today you will test it and report any bugs that you think we need to fix.

I took this task as my personal goal to prove myself. Worked ten hours that day and made a few comments on how to improve customer experience.

I asked if I can take the CD back with me at home and tested it on my personal computer.

It was a windows executable and the installer was pretty decent.

Next, next, install, done.

My windows 98 second edition didn’t have enough free space on my hard disk, and I needed to also install oracle to work on my semester lab exercises. My 8G hard disk and the gazillion of floppy disks around my home office on my Pentium III was my entire kingdom back then. So I uninstalled the application and rebooted my computer.

Then something horrible happened. My computer could not start the operating system. There were indications of missing DLLs.

I re-installed [repair] windows and was curious about what happened.

I re-installed the application and re-uninstalled it once more. Reboot Windows and again missing DLLs.

First Conflict

I returned on Monday morning at the office and explained in details the extreme bug I had found. When a customer removes our software, they would corrupt their operating system. The majority of our customers didn’t have the technical experience to fix this problem. So I made it very clear that this is something we need to fix ASAP and we should inform every customer not to remove our application and reboot their machine. I was really proud that I had found this super bug and that we were going to save our company.

And then the owner told me:

Our customers are paying us for installation of our software application. They are not paying us for fixing their computer problems.
But this is something we introduced.
Do not be silly, we are professionals, we do not make mistakes.
But …
No butts, this is not our problem.

Whatttt ?

First business lesson was:

We do not make mistakes, customers should pay us for fixing our bugs!

Fixing Bugs

The next thing was to check the installer. We’ve noticed that they had marked a few windows DLLs as important to be there for our application to run. To avoid any mistakes we copied these DLLs from the application’s CD to our customer’s windows. The uninstallation process, was removing everything that installed so … the windows DLLs were gone! It was a simple mistake and easy to fix. Click on the correct checkbox for those files, not to be removed during the uninstallation process.

Distribution

We needed to distribute our application to all 2.000 customers all over Greece. We had to burn 2.000 physical CD’s, print 2.000 CD covers, compile 2.000 CD cases and put them in 2.000 envelopes and write 2.000 addresses on the envelopes. Then visit the local post office, pay for stamps etc and mail 2.000 CDs to our customer’s snail addresses.

We also had to provide letters of instructions:

Uninstall the previous version
Install the new version

in any circumstance do not reboot your PC till the new version is up and running. Then copy your license key into the program and connect to the internet to upload your contracts/data or sync your data from the central database to your laptop/desktop.

Money

For every patch [that meant a new CD to sent] our business model was to get money from our customers for our work and any expenses for distributing these CDs around Greece. That was the business deal with our customers. Customers were paying us, for our mistakes and could also take a week or so to get the fix. Depending on the post office delays. License keys were valid [I am not sure but I believe] for a year and then there was a subscription model for the patches. If customers wanted to subscribe. then they should pay us for every CD, for every patch, for every mistake. Our business model depended on that.

Second Conflict

For some reasons I had opinions about this effort. I made a suggestion to use our web server [web site] to provide the patch, so the customers can download from the internet and install it immediately without waiting for weeks till we sent the next CD with the latest version. Also ,no need of extra money for the post office or CDs or burning 2.000 CDs through the weekend. Customers should pay for the patch [our work] so this way would be best for everybody.

The owner replied to me, that they made more money with the current system, so no need of making things easier or cheaper for customers and I should keep this innovated ideas to myself.

At that point, the thought that I wasn’t working for the next amazon came in mind. They would put this extra profit on top of their customer’s needs.

Coding style

Finally, after my first week as an employee, I was now writing code as a software engineer. I did an impressive work of fixing bugs and refactoring code and in a sense made our product better, faster and safer. I had ideas and worked closely with the senior programmer on a few things. I was doing good, working fast, learning and providing value.

I’ve noticed a specific coding style so I kept it. The senior programmer could read my code and comments [I wrote a lot of comments] and vice versa. Finally I had joy from my work as a programmer.

Third Conflict

I vividly remember a specific coding issue, even 20 years after it happened. There was a form with 10 buttons. 10 clicks were the maximum possible events on this form. So I wrote a case statement of 9 events and one default. I submitting the code and the owner/head software programmer came to the office yelling at me.

I’ve started reviewing your code and I can not read it. Why you are writing code like this. this is shit code. Case statements!!! No no no no. I want from you to write the same code as I write, so I can read/review it.
But your example is a nested if-then-else for 11 events and we only have 10 events there. I made a case statement of 9 events and a default. It’s better.
No, this is not better, it’s shit. I can not review your code. I want you to delete everything and start from the beginning. I want to read your code and think that I was writing this code instead of you.
I am sorry, but I think your are wrong on this. This is better, trust me. I worked closely with our senior programmer and we believe this is better.
No, remove everything.

Final Discussion

after a couple of hours

So I need to talk with you.
Sure, what can I do for you?
I think this collaboration is not working between you and us.
okay, I am really sorry about that. Can I please ask what are the problems so that I can improve in the future. This is my first job.

The truth bomb:

You have all these new ideas to disturb our business model and cash flow. Using the web server to publish and distribute patches? Come on, you are very young to give me advice on how to run my business. you do not know anything.
You made a lot of comments and suggestions about what we are doing wrong. This should never be the case, especially if you are talking to customers. We never make mistakes and we need to be paid for every customer request. I never make mistakes. I have a master’s degree in computer science and you are still a student. If something is wrong, customers should make a request and we are going to make a patch. That’s it.
Finally ,you are writing code that I can not read/review. I am the head software engineer and I need from you to write code as I write code. You should never introduce anything new that I can not read.

Exit

Two weeks, I felt like really shit. I felt like I didn’t know anything about business but he paid me for the whole month.

After all these years, I now believe that he was afraid of my ideas. Of using the internet to help our business and reduce customer’s costs but the most important was he was afraid that new people came to his business and wrote code that he could not understand.

I made a promise that day to myself, that last Friday from my very first job:

I will try always to do my best in this industry.

Almost 20 years have past from those two weeks, I never worked as a programmer, I chose to work as a sysadmin, mostly doing operations.

Thankfully I think I am doing well. So here, to the next 20 years ahead.

Thank you for reading my story.

The importance of culture

Origin Post on LinkedIn, Published on January 6, 2020

Being abroad in Japan the last couple weeks, I’ve noticed that the high efficiency -from crossing roads to almost everything- they do [cooking/public transportation/etc] is due to the fact of using small queues for every step of the process. Reaching to a maximum throughout with small effort.

The culture of small batches/queues reminds me the core principles of

DevOps as they have identified in the book “The Goal: A Process of Ongoing Improvement” by Eli Goldratt and of course in “Theory of Constraints”.

Imagine this culture to everything you do in your life. From work to your personal life. Reducing any unnecessary extra cost, reducing waste by performing Kata. Kata is about form, from dancing to creating your cloud infrastructure with reproducible daily work or routines that are focusing in the process for reaching your business goals.

This truly impresses me in Japanese culture among with the respect they are showing to each other. You may of course notice the young people riding their bicycles in the middle of the street, watching their smartphone instead of the road 😀but the majority of people bow their head to show respect to other people and other people’s work or service.

We, sometimes forget this simple rule in our work. Sometimes the pressure, the deadlines or the plethora of open tickets in our Jira board [or boards] makes us cranky with our colleagues. We forget to show our respect to other people work. We forget that we need each other for reaching to our business values as a team.

We forget to have fun and joy. To be productive is not about closing tickets is about using your creativity to solve problems or provide a new or improve an old feature that can make your customers happy.

Is about the feedback you will get from your customers and colleagues, is about the respect to your work. Is about being happy.

For the first time in my life, I took almost 30days out of work, to relax, to detox [not having a laptop with me] to spend some time with family and friends. To be happy. So if any colleague from work is reading this article: