Seasoned Technical Staff Software Engineer and Secure Systems Architect with 14+ years in the semiconductor industry. Specializes in high performance System-on-Chip Software, System Security & Device Drivers. Adept at Python, System Integration, DevOps, and Cloud Architectures. Proven track record leading cross-functional teams from conception to implementation and customer delivery. Strong communicator and proactive team player dedicated to delivering top-notch results.
Master of Science, Digital Design and Embedded System
Manipal Academy of Higher Education, Manipal, India
January 2011 - February 2013
Bachelor of Technology, Electronics and Communication Engineering
Amrita Vishwa Vidyapeetham, Kollam, India
May 2005 - May 2009
Microsemi Storage Solutions Ltd - A Microchip Company, Burnaby, British Columbia, Canada
April 2022 - Present
RESPONSIBILITIES
Microchip Technology India Pvt. Ltd., Bangalore, India
February 2015 - April 2022
RESPONSIBILITIES
Cisco Systems India Pvt Ltd, Bangalore, India
August 2011 - January 2015
RESPONSIBILITIES
Sasken Technologies Ltd, Bangalore, India
March 2010 - January 2011
RESPONSIBILITIES
Amrita University
August 2009 – March 2010
RESPONSIBILITIES
Seek challenging and interesting technical jobs that can be done within a small period utilizing personal free time resulting in technical growth.
Applications of TinyML
HarvardX
Issued Jan 2022
View Certificate
Deploying TinyML
HarvardX
Issued Jan 2022
View Certificate
Fundamentals of TinyML
HarvardX
Issued Jan 2022
View Certificate
Professional Certificate in Tiny Machine Learning (TinyML)
HarvardX
Issued Jan 2022
View Certificate
Trust Platform Design Suite v2
Microchip Technology Inc.
Issued Dec 2021
View Certificate
AWS IoT: Developing and Deploying an Internet of Things
Coursera
Issued Oct 2021
View Certificate
Microchip Technology, Canada, 2024
Led the architectural design and implementation of system firmware security solutions to safeguard platform security within the innovative First-Generation RISC-V based HPSC system. This project necessitated close collaboration between design, product engineering, and application teams, ensuring cohesive integration of security measures throughout the system’s development lifecycle.
Achievements
Microchip Technology, Canada, 2023
Pioneered the development of the inaugural QEMU model for a heterogeneous RISC-V based HPSC system, comprising an octo-core Application complex, system controller, and secure controller. This endeavor marked a significant milestone in enabling early customer engagement and expediting application development for the HPSC platform.
Achievements
Microchip Technology, Canada, 2023
Conceptualized, designed, and executed a robust early-boot architecture tailored to the specific requirements of RISC-V based Linux and application software systems within the Multicore High-Performance Computing (HPSC) project. Integrated a dedicated boot manager subsystem and implemented early-boot functionality on QEMU and Protium emulation systems. Spearheaded the development of a new boot media and configuration controller, laying the groundwork for future enhancements in boot architecture.
Achievements
Microchip Technology, Canada, 2024
Designed and developed a robust and secure ROM for the system controller of the High-Performance Computing System (HPSC). Implemented a comprehensive security infrastructure to safeguard the integrity and confidentiality of the ROM.
Achievements
Microchip Technology, Canada, 2023
Led the design and development efforts in creating a buildroot-based Linux system tailored specifically for the RISC-V based HPSC platform. This project encompassed the seamless integration of essential components and configurations to ensure optimal performance and compatibility with the unique architecture of the HPSC system. Additionally, spearheaded the successful implementation of Linux bring-up on Protium and QEMU-based emulation systems, laying the foundation for comprehensive testing and validation procedures.
Achievements
Microchip Technology, Canada, 2023
Executed the project to develop a custom RISC-V Debian distribution, “shellfire,” exclusively tailored for the HPSC Application complex. Additionally, implemented a private package repository to streamline the release of HPSC packages for the shellfire distribution, ensuring efficient access to essential software components.
Achievements
Microchip Technology, Canada, 2022
Led the architecture and implementation of a complete software release infrastructure and pipeline from the ground up for the first-generation HPSC project collateral. This initiative aimed to establish a seamless and efficient system for software release management, integrating DevOps practices and ensuring optimal accessibility and version control of project documentation.
Achievements
Microchip Technology (India) Private Limited, Bangalore, 2021
Played a pivotal role as the IoT device security and Cloud unification architect, overseeing the development and implementation of robust security infrastructure for wireless IoT products. Led technical program management initiatives to ensure the security of both new and existing products, guiding developers and customers in secure product development practices and spearheading the design of secure cloud-connected systems.
Achievements
Microchip Technology (India) Private Limited, Bangalore, 2021
Served as the Applications Lead for the new product development of first-generation wireless IoT silicon products, driving the development of innovative silicon features tailored for the IoT market. Led post-silicon bring-up and validation efforts while actively engaging with customers to address field issues and complex design requirements. Conducted competitor analysis and collaborated with cross-functional teams to ensure product competitiveness and market relevance.
Achievements
Microchip Technology (India) Private Limited, Bangalore, 2021
Led the architecture and implementation of a distributed serverless secure cloud model and firmware for application deployment of voice-controlled systems. Designed to enable seamless integration of voice control using Amazon Alexa and Google Home interfaces, this project aimed to create a low-cost, distributed cloud ecosystem for efficient product deployment.
Achievements
Microchip Technology (India) Private Limited, Bangalore, 2021
As the architect and team manager, led the development of an advanced “hardware in the loop” automated test framework project. This involved architecting a comprehensive test harness using modern web-based tools and managing the implementation team to ensure timely delivery of releases to internal teams.
Achievements
CISCO, Bangalore, 2015
Involved in the development of custom STB middleware modules and facilitated their seamless integration with Linux drivers (CDI) to achieve end-to-end integration. This initiative aimed to enhance IPTV services by implementing personalization features in multi-CPE-households and optimizing system-level performance.
Achievements
Sasken communication Technologies ltd, Bangalore, 2011
Involved in the development and maturation of low-level device drivers, protocol layers, and hardware abstraction layer (HAL) for USB on Nokia S40 running on NOS RTOS. This project aimed to enhance the functionality and reliability of USB communication on Nokia S40 devices.
Achievements
Led the design effort for the FPGA-based architecture of a P-Radar controller, aimed at implementing a standalone version of an existing mixed-mode, PC-based PR weather radar transceiver controller. This project focused on developing a robust micro-architecture design to integrate new and existing discrete modules provided by the client.
Achievements
Led the translation of Matlab-based digital subsystems into a real hardware implementation for digital control of a voltage inverter circuit. This project focused on converting theoretical models into practical, hardware-based solutions for efficient voltage control.
Achievements
Developed a proof of concept (PoC) system for patient orientation monitoring utilizing Linux-based software, MPU6050 sensors, and a web-based visualization interface. This project aimed to provide real-time monitoring of patient orientation and visualization of data for healthcare professionals.
Achievements
Led the implementation of MicroBlaze soft processor-based systems on Xilinx Spartan 6 FPGA, specifically the lx9 micro board. This project involved board bring-up, custom Linux kernel bring-up, designing and implementing various peripherals for the System-on-Chip (SoC) in Verilog, and developing drivers and software adaptation layers for the system.
Achievements
Led the design and implementation of a USB-based authentication token for custom software, incorporating PIC 18f4550 microcontroller for end-to-end (E2E) USB hardware design of the authentication token. Developed a Visual Basic-based software design for a simple authenticated notepad demo application, incorporating two-phase security with passcode and hardware-based authentication.
Achievements
Led the development of messaging mechanisms for digital radar control systems, implementing XML-based messaging for multi-location radar controllers. Enhanced XMPP for collaborative networks and incorporated encrypted message bodies for heightened security over TLS, serving as a stepping stone to the Internet of Things (IoT) for radar systems.
Achievements
Led the end-to-end design and fabrication of a custom daughter-card for Raspberry Pi, incorporating a PIC microcontroller-based PCB. Established I2C and GPIO-based communication with the daughter-card and developed a custom Linux build with minimal BusyBox Root File System (RFS). Designed and implemented custom kernel modules for daughter-card control and proposed an I2C bootloader and Linux drivers for firmware upgrade to eliminate ICSP.
Achievements
Undertook a postgraduate project focusing on the hardware-software co-design of a web interface for FPGA-based radar control systems. This project aimed to design and implement a standalone radar control configuration unit based on a MicroBlaze soft-core processor, utilizing LWIP and Xilkernel for implementing HTTP over TCP/IP. Principles of hardware-software co-design were applied to ensure parallel and efficient development of both hardware and software components.
Achievements
Collaborated with the Department of Electrical and Computer Engineering, University of Texas, Austin (USA), to design and implement an educational biomedical kit for the One Laptop Per Child (OLPC) XO platform. This project aimed to develop a comprehensive educational tool for students, incorporating in-house developed sensors such as ECG, Blood Oximeter, and Body Temperature sensors. Vital signals were digitized, collected, and transmitted over USB to the OLPC XO for processing. The graphical user interface (GUI) was developed using Adobe Flex, providing an interactive and user-friendly experience for students. Embedded error control mechanisms were implemented to ensure reliable data acquisition and processing, enhancing the educational value of the kit.
Achievements
This is not one of my typical technical deep dives. This is a post about why I blog, based on a conversation I had with my nephew. I hope you find it interesting.
I recently had a long chat with my nephew, who was about to start his Bachelor’s in Engineering. We discussed the advice I wish I had received when I was in his shoes. One key piece of learning that emerged was the value of documenting what you learn. My nephew, like many, viewed writing as a purely theoretical exercise, detached from the practical world of Engineering. I understood his perspective, as I, too, held this belief in my youth. However, I realized that writing about my learning experiences was a crucial tool for effective learning. This post delves into why this practice is essential and how it has personally benefited me.
I have always been an active learner. I need to read a lot, watch videos, and do a lot of practical work. It is difficult for my brain to internalize concepts by just reading or watching. I need to do something with the knowledge to understand it. So, for the better part of my college and early career years, I always found projects to work on and built some cool stuff. But I never wrote about what I learned. I would write code, build circuits, do a lot of practical work and move on to the next project. Over time, I realized that while the core concepts were clear, I could have articulated them better. I struggled to explain the concepts to others, and I often needed to remember details of the projects I had worked on. I realized that I was retaining the knowledge I gained at a macro level, but I was sometimes unable to recall the details. More importantly, though I was touching upon many exciting concepts, I was learning enough to finish the work. I needed to learn to apply the concepts’ broader aspects to other problems. I often found myself in a situation where I would have to relearn concepts I had already learned and apply them to new problems.
Today’s learning happens through YouTube videos, online courses and even AI assistants. But, when I was getting started, google searches about niche topics often led to simple HTML pages written by enthusiasts. These pages are very detailed and straightforwardly explain the concepts. I often found myself going back to these pages to refer to the concepts and realizing that I would learn something new each time I went back. This meant I could only internalize the concepts partially the first time I read them. I realized that I needed to be learning more effectively. This was humbling and inspiring at the same time. This is what led me to start writing about what I learned.
As a start, I wrote about something I (thought I) knew well at the time - PIC 8 Microcontrollers. During my university days, I spent considerable time writing complex assembly code for PIC8 microcontrollers, and I thought I knew the architecture well. I wrote a blog post about the architecture and the instruction set. I was surprised that I had forgotten a few concepts and missed others. I realized this only when I tried to explain the concepts to a virtual audience. A few years later, when I started working for Microchip, I learned even more gaps in my understanding, which triggered quite a bit of my imposter syndrome. But that’s a story for another day. I have re-homed the post from its original Google Sites location to embeddedinn.com.
This first experience was eye-opening, and I realized that writing about what I learn is a great way to learn effectively. Since then, whenever I do a professional or hobby project, I try to write about it at embeddedinn.com. Though it started with articles about the projects themselves, I eventually started including a lot of articles about the concepts I learned while working on the projects.
A happy side-effect of writing about what I learned is that it has helped me professionally. I have been able to articulate my thoughts better, and I have been able to explain concepts to my colleagues and customers better. It has also triggered a lot of exciting side conversations with people who read my articles. Some readers have extreme opinions about my take on things, and often, these conversations lead to more exciting projects and learning opportunities.
Today, the list of things I have written about is an excellent resource for me. I often return to my articles to refer to concepts I have written about. Some of these articles have also become inspiration and reference materials for other like-minded people learning about the same concepts. The challenge now is balancing the backlog of items I have to write about from past projects and completing new exciting projects I take on. But it’s a good problem to have.
]]>Device trees are typically used in Linux systems to describe the hardware components of a system. Lately, RTOSes like Zephyr have made it more popular among developers of systems with lesser complexity. Device trees can be used in any bare-metal system to describe a system’s hardware components and pass run-time parameters (as opposed to compile-time parameters). This approach offers a great deal of interoperability and portability. It also allows for a clear separation of hardware and software, making it easier to maintain and update the software. This article describes how to use device trees in bare-metal systems.
We will follow the latest 0.4 version of the Device Tree Specification.
A Device tree is a hierarchical data structure often used to describe the hardware components and their configuration in a system. A device tree source (DTS) is a human-readable representation of the tree. The device tree compiler (DTC) is used to convert the DTS to a binary device tree blob (DTB) that can be used by software like a bootloader or the kernel.
Device tree bindings are similar to a schema in XML. It describes the properties and nodes used to describe a hardware component. The bindings are used to validate the device tree source and to generate documentation.
Device tree overlays (dtbo
) are used to modify the device tree at runtime. This is useful for adding or removing hardware components from the device tree without modifying the device tree source. For example, when a beaglebone cape is added to the system, the device tree overlay can be used to add the cape to the device tree without modifying the device tree source.
Systems like Zephyr primarily convert device tree sources and bindings to produce a generated C header. However, in some systems, like an embedded system running Linux, the DTB is stored in a known location and is loaded into the kernel by the bootloader.
Before we get into the details of using device trees in bare-metal systems, let’s review the basics of the device tree syntax.
A property is a key-value pair that describes a hardware component. A property is defined using the following syntax:
property-name = <value>;
The property name is a unique identifier for the property. The value can be a number, a string, or a list of numbers.
compatible = "vendor,device";
reg = <0x12340000 0x1000>;
interrupts = <0 10 0>;
In this example, the compatible
property is a string, the reg
property is a list of numbers, and the interrupts
property is a list of numbers.
Some of the standard properties from the specification are tabulated below:
Property | Description |
---|---|
compatible |
A list of strings that describe the hardware component. |
reg |
A list of numbers describing the hardware component’s address and size. |
virtual-reg |
specifies an effective address that maps to the first physical address specified in the reg property of the device node. |
interrupts |
A list of numbers that describe the interrupt number and type of the hardware component. |
clocks |
A list of phandles that describe the clocks used by the hardware component. |
resets |
A list of phandles that describe the resets used by the hardware component. |
gpio |
A list of numbers that describe the GPIO pins used by the hardware component. |
pinctrl-0 |
A phandle describing the pin configuration the hardware component uses. |
model |
A string that describes the model of the hardware component. |
phandle |
A number that references a node in the device tree. |
status |
A string that describes the status of the hardware component. Examples include “okay”, “reserved”, “disabled” etc. |
#address-cells and #size-cells |
Numbers that describe the number of cells used to describe the address and size of the hardware component. |
ranges |
provides a means of defining a mapping or translation between the address space of the bus (the child address space) and the address space of the bus node’s parent (the parent address space). |
A node is a collection of properties that describe a hardware component. A node is defined using the following syntax:
node-name {
property-name = <value>;
...
};
The node name is a unique identifier for the node. The properties are a collection of key-value pairs that describe the hardware component. The value can be a number, a string, or a list of numbers.
A path is a unique identifier for a node in the device tree. It is a collection of node names separated by slashes. For example, the path /soc/uart@12340000
refers to the node with the name uart@12340000
, which is a child of the node with the name soc.
The #include
directive includes other device tree sources in the current device tree source. This is useful for reusing common device tree definitions across multiple sources.
#include "common.dtsi"
Comments in the device tree source are similar to comments in the C programming language. They start with //
and continue to the end of the line.
// This is a comment
Practically, a device tree corresponds to a hardware component. For example, consider an embedded system with a UART, a GPIO, and a SPI controller. The SPI controller is connected to a Flash memory, a temperature sensor, and an accelerometer.
The DTS for this system would look something like this:
Note: The following example is a simplified version of the device tree. The device tree would be more complex and include additional properties and nodes.
/dts-v1/;
/ {
compatible = "vendor,device";
#address-cells = <1>;
#size-cells = <1>;
soc {
compatible = "vendor,soc";
#address-cells = <1>;
#size-cells = <1>;
uart@800000 {
compatible = "vendor,uart";
reg = <0x800000 0x100>;
interrupts = <10 0>;
};
gpio@800100 {
compatible = "vendor,gpio";
reg = <0x800100 0x100>;
interrupts = <11 0>;
};
spi@800200 {
compatible = "vendor,spi";
reg = <0x800200 0x100>;
interrupts = <12 0>;
#address-cells = <1>;
#size-cells = <1>;
flash@0 {
compatible = "vendor,flash";
chip-select = <0>;
flash-size=<0x100000>;
};
temp-sensor@1 {
compatible = "vendor,temp-sensor";
chip-select = <1>;
};
accel@2 {
compatible = "vendor,accel";
chip-select = <2>;
};
};
};
};
The idea is to change the hardware configuration without changing the software. This is especially useful in systems with many hardware components and systems that are expected to be used in different configurations. For example, we should be able to use the same software on a system with the chip select lines for the Flash and Temperature sensor swapped without changing the software.
To make this possible, the Flash and Temperature sensor device driver should not hardcode the chip select lines. Instead, it should use the device tree to find the chip select lines. This way, the device driver can be used on different systems without modification. The device tree will be loaded from a known location in the system, and the device driver will use the device tree to find the chip select lines.
First, we must compile the device tree into a binary device tree blob (DTB). Store the device tree from the example above in a file called soc.dts
. Then, use the device tree compiler to compile the device tree into a binary device tree blob:
dtc -I dts -O dtb -o soc.dtb soc.dts
Since this is a simplified, non-standard example, you will see warnings about missing properties and nodes. You can safely ignore these warnings for now.
Note
dtc
can also decompile a device tree blob into a device tree source. This is useful for debugging and understanding how the device tree compiler works. To decompile a device tree blob, use the-I dtb
and-O dts
options to specify the input and output formats, respectively.
The output of the device tree compiler is a binary device tree blob, often called the flat device tree (FDT). It is referred to as a flat device tree because it is a flat memory image that contains the entire device tree that also brings in the includes.
The Devicetree .dtb
Structure is depicted in the following diagram:
The fdt_header
is a fixed-size header that contains information about the device tree. The layout of the header for the device tree is defined by the following C
structure.
struct fdt_header {
uint32_t magic; //0xd00dfeed
uint32_t totalsize; //total size of the device tree in bytes
uint32_t off_dt_struct; //offset in bytes of the structure block
uint32_t off_dt_strings; //offset in bytes of the strings block
uint32_t off_mem_rsvmap; //offset in bytes of the memory reservation block
uint32_t version; //format version
uint32_t last_comp_version; //last compatible version
uint32_t boot_cpuid_phys; //physical ID of the boot CPU. Not applicable in non-standard systems
uint32_t size_dt_strings; //size of the strings block in bytes
uint32_t size_dt_struct; //size of the structure block in bytes
};
Please refer to the Device Tree Specification for a full description of the header fields and the blob structure.
We will be using the bare-metal libfdt
implementation from within uboot to parse the device tree. While the full libfdt
is part of the Linux kernel, the uboot version is a stripped-down version that provides essential capabilities.
Note The ubo0t libfdt supports FDT manipulation and hence contains a lot of code that is not needed for a bare-metal read-only system. For production use, this can be stripped down to the bare minimum.
libfdt
Clone the latest version of uboot from the official repository:
git clone https://github.com/u-boot/u-boot.git
Change to the uboot directory and checkout the latest stable release:
cd u-boot
git checkout v2024.01
copy the libfdt
directory to a new directory called libfdt-uboot
:
mkdir ../libfdt-uboot
cp -r scripts/dtc/libfdt/ ../libfdt-uboot
cd ../libfdt-uboot/libfdt
Create a makefile for the libfdt
:
touch Makefile
Add the following content to the makefile:
include Makefile.libfdt
# Name of the library to create
LIBFDT_LIB = libfdt.a
# Rule to compile each source file into an object file
%.o: %.c
$(CC) -c $< -o $@ -I . # override CC with your system toolchain
# Rule to create the library from the object files
$(LIBFDT_LIB): $(LIBFDT_OBJS)
ar rcs $@ $^
ranlib $@
# Default rule
all: $(LIBFDT_LIB)
test: $(LIBFDT_LIB)
$(CC) -o main main.c $(LIBFDT_LIB) -I .
# Clean rule
clean:
rm -f $(LIBFDT_OBJS) $(LIBFDT_LIB)
Write a simple C program to parse the device tree:
#include <libfdt.h>
#include <stdio.h>
#include <stdlib.h>
// function to read the dtb file
static int read_dtb(char *dtbPath, void **fdt_blob) {
FILE *fp = fopen(dtbPath, "rb");
if (fp == NULL) {
fprintf(stderr, "Error: Unable to open file\n");
return 1;
}
fseek(fp, 0, SEEK_END);
long fsize = ftell(fp);
fseek(fp, 0, SEEK_SET);
*fdt_blob = malloc(fsize);
if (*fdt_blob == NULL) {
fprintf(stderr, "Error: Unable to allocate memory\n");
return 1;
}
fread(*fdt_blob, fsize, 1, fp);
fclose(fp);
return 0;
}
int main(int argc, char *argv[]) {
const void *fdt = NULL;
int err;
void *fdt_blob = NULL;
if (argc != 2) {
fprintf(stderr, "Usage: %s <dtb file>\n", argv[0]);
return 1;
}
read_dtb(argv[1], &fdt_blob);
// check if the file is a valid fdt
if (fdt_check_header(fdt_blob) != 0) {
fprintf(stderr, "Error: Invalid device tree\n");
return 1;
}
// get the /soc/spi tree and print the properties
fdt = fdt_blob;
int offset = fdt_path_offset(fdt, "/soc/spi");
if (offset < 0) {
fprintf(stderr, "Error: Unable to find /soc/spi\n");
return 1;
}
int len;
const char *prop = fdt_getprop(fdt, offset, "compatible", &len);
if (prop == NULL) {
fprintf(stderr, "Error: Unable to find compatible property\n");
return 1;
}
printf("compatible: %s\n", prop);
prop = fdt_getprop(fdt, offset, "reg", &len);
if (prop == NULL) {
fprintf(stderr, "Error: Unable to find reg property\n");
return 1;
}
// print the reg address and size
int i;
for (i = 0; i < len / sizeof(uint32_t); i++) {
printf("reg[%d]: 0x%x\n", i, fdt32_to_cpu(((fdt32_t *)prop)[i]));
}
printf("\n");
// get all the spi nodes and print the properties
int node;
for (node = fdt_next_node(fdt, offset, NULL); node >= 0;
node = fdt_next_node(fdt, node, NULL)) {
char reg[32];
const char *name = fdt_get_name(fdt, node, &len);
if (name == NULL) {
fprintf(stderr, "Error: Unable to find node name\n");
return 1;
}
printf("node: %s\n", name);
prop = fdt_getprop(fdt, node, "compatible", &len);
if (prop == NULL) {
fprintf(stderr, "Error: Unable to find compatible property\n");
return 1;
}
printf("\tcompatible: %s\n", prop);
// if compatible to "vendor,flash" print the flash-size property.
if (strcmp(prop, "vendor,flash") == 0) {
prop = fdt_getprop(fdt, node, "flash-size", &len);
if (prop == NULL) {
fprintf(stderr, "Error: Unable to find flash-size property\n");
return 1;
}
// Assuming flash-size is a 32-bit integer
int flash_size = fdt32_to_cpu(*(fdt32_t *)prop);
printf("\tFlash size: 0x%x\n", flash_size);
}
prop = fdt_getprop(fdt, node, "chip-select", &len);
if (prop == NULL) {
fprintf(stderr, "Error: Unable to find chip-select property\n");
return 1;
}
// convert property to string and print
int cs = fdt32_to_cpu(*(fdt32_t *)prop);
printf("\tChip Select: %d\n\n", cs);
}
free(fdt_blob);
return 0;
}
Compile and run the program:
make test
./main soc.dtb
The program should print the compatible and reg properties for the /soc/spi node and for each of the spi nodes.
compatible: vendor,spi
reg[0]: 0x800200
reg[1]: 0x100
node: flash@0
compatible: vendor,flash
Flash size: 0x100000
Chip Select: 0
node: temp-sensor@1
compatible: vendor,temp-sensor
Chip Select: 1
node: accel@2
compatible: vendor,accel
Chip Select: 2
The program reads the device tree from the file soc.dtb
and prints the compatible and reg properties for the /soc/spi
node and each of the spi nodes.
Device trees are typically used in Linux systems to describe the hardware components of a system. However, device trees can be used in any bare-metal system to describe the hardware components of a system. This approach offers a great deal of interoperability and portability. It also allows for a clear separation of hardware and software, making it easier to maintain and update the software. This article showed how to use device trees in bare-metal systems.
]]>In this article we will go over the steps to build a bespoke Debian distro for RISC-V. We will use QEMU to emulate the RISC-V architecture and build a Debian distro for it. A bespoke OS is a custom OS that is built for a specific purpose. In this case, we will build a Debian distro that is built for RISC-V architecture. This is especially useful for embedded systems where we need full control over the OS and the packages that are installed on it.
Debian is a very popular Linux distro that is used in many embedded systems. It is also the base for many other Linux distros like Ubuntu. Debian is also very popular in the RISC-V community. Debian provides a very robust package management system that allows us to install and manage packages on the OS. Debian also provides a very robust build system that allows us to build packages for the OS.
lb
)Debian Live Build (lb
) is a tool that allows us to build a Debian distro from scratch. It is a very powerful tool that allows us to customize the distro to our needs. It also allows us to build a distro for a specific architecture. In this article we will use lb
to build a Debian distro for RISC-V architecture. Live Build uses a configuration directory to completely automate and customize all aspects of building a Live image.
I am using a clean debian system running on an x86 machine to build the distro.
Since we are creating a Debian distro for the RISC-V architecture on a x86 machine, we will use the qemu-user-static
tool to emulate the RISC-V architecture.
Install the following dependencies:
sudo apt install qemu-user qemu-user-static qemu-system-riscv64 binutils-riscv64-linux-gnu-dbg binutils-riscv64-linux-gnu \
libguestfs-tools binfmt-support build-essential git vim debian-archive-keyring debootstrap
In case of a non-systemd system (like WSL2), we need to enable binfmt support for qemu. This can be done by running the following command:
# running this explicitly since systemd does not start this on non-systemd systems like WSL2
sudo /usr/lib/systemd/systemd-binfmt
# these two steps are to check if it is enabled. It should return `enabled` for the qemu-riscv64 format
sudo update-binfmts --enable qemu-riscv64
sudo cat /proc/sys/fs/binfmt_misc/status
A side note on binfmt
binfmt
lets us configure additional binary formats for executables at boot. It uses the Kernel Support for miscellaneous Binary Formatsbinfmt_misc
that allows us to register an intrepreter to use for a specific binary format. This is useful when we want to run binaries for a different architecture on our host machine.
In our case, we are usingqemu-user-static
to emulate the RISC-V architecture on our x86 machine. We will usebinfmt
to registerqemu-user-static
as the interpreter for the RISC-V binary format. This will allow us to run RISC-V binaries on our x86 machine.
The magic number and other format elements used to identify the interpretor is stored in the following files/etc/binfmt.d/*.conf
/run/binfmt.d/*.conf
/usr/lib/binfmt.d/*.conf
thesystemd-binfmt
command registers these formats with the kernel.
You can also register a format manually like this:
echo ':qemu-riscv64:M::\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/libexec/qemu-binfmt/riscv64-binfmt-P:OCPF' > /proc/sys/fs/binfmt_misc/register
Set up packages for risc-v with the following steps:
sudo dpkg --add-architecture riscv64
sudo apt-get install gcc-riscv64-linux-gnu g++-riscv64-linux-gnu
Then do a sudo apt update
to update the package list.
We will fetch the latest version of live-build from git and build it from source.
git clone https://salsa.debian.org/live-team/live-build.git
cd live-build
mkdir install
# temp fix for missing manpages
mkdir -p manpages/fr manpages/ja
touch manpages/fr/temp manpages/ja/temp
sudo make install
Check the installation with
lb --version
Running lb
without any arguments will create a default configuration directory in the current directory. Hpwever, we want to customize the configuration at creation time. We will use the following command to create a semi-customized configuration directory. Run this command in a directory of your choice.
export LB_BOOTSTRAP_INCLUDE="apt-transport-https gnupg"
lb config \
--apt-indices false \
--apt-secure false \
--architectures riscv64 \
--distribution unstable \
--mirror-bootstrap http://deb.debian.org/debian/ \
--mirror-binary http://deb.debian.org/debian/ \
--keyring-packages "debian-archive-keyring debian-archive-keyring" \
--security false \
--archive-areas 'main contrib non-free' \
--apt-options '--yes -oAPT::Get::AllowUnauthenticated=true' \
--binary-filesystem ext4 \
--binary-images tar \
--bootappend-live "hostname=embeddedinn username=embeddedinn" \
--bootstrap-qemu-arch riscv64 \
--bootstrap-qemu-static /usr/bin/qemu-riscv64-static \
--cache true \
--firmware-binary false \
--firmware-chroot false \
--apt-source-archives false \
--chroot-filesystem none \
--compression gzip \
--debootstrap-options "--keyring=/usr/share/keyrings/debian-archive-keyring.gpg --variant=minbase --include=apt-transport-https,apt-utils,gnupg,ca-certificates,ssl-cert,openssl" \
--distribution unstable \
--gzip-options '-9 --rsyncable' \
--iso-publisher 'embeddedinn; https://embeddedinn.com/; vysakhpillai@embeddedinn.com' \
--iso-volume 'embeddedinn riscv64 $(date +%Y%m%d)' \
--linux-flavours none \
--linux-packages none \
--mode debian \
--system normal \
--updates false
Each of the options we passed are explained in the table below:
Option | Description |
---|---|
--apt-indices false |
Do not download apt indices |
--apt-secure false |
Do not use apt secure |
--architectures riscv64 |
Build for the riscv64 architecture |
--distribution unstable |
Build for the unstable distribution. This is because we want to build for the latest version of Debian. |
--mirror-bootstrap http://deb.debian.org/debian/ |
Use the Debian ports mirror for the bootstrap. Bootstrap is the first stage of the build process where we build the base system which is further used to build the final system. |
--mirror-binary http://deb.debian.org/debian/ |
Use the Debian ports mirror for the binary. Binary is the second stage of the build process where we build the final system. |
--keyring-packages "debian-archive-keyring debian-archive-keyring" |
Use the Debian ports keyring for the build. |
--security false |
Do not use security updates. We are ignoring security updates for now. |
--archive-areas 'main contrib non-free' |
Use the main, contrib, and non-free archives to pull packages from. |
--apt-options '--yes -oAPT::Get::AllowUnauthenticated=true' |
Use the --yes option for apt and allow unauthenticated packages. |
--binary-filesystem ext4 |
Use the ext4 filesystem for the binary. Other options are squashfs and tar. |
--binary-images tar |
Use the tar image for the binary. Other options are iso and netboot. |
--bootappend-live "hostname=embeddedinn username=embeddedinn" |
Append the hostname and username to the boot command line. |
--bootstrap-qemu-arch riscv64 |
Use the riscv64 architecture for the bootstrap. Qemu will use this architecture to emulate the bootstrap machine to build the base system. |
--bootstrap-qemu-static /usr/bin/qemu-system-riscv64 |
Use the qemu-system-riscv64 binary to emulate the bootstrap machine. We installed this binary in the dependencies section. An explicit path lets us use custom QEMU binaries. |
--cache true |
Use the cache to speed up the build process. This includes the apt cache and the package cache. |
--firmware-binary false |
Do not include firmware in the binary. Firware refers to components like the kernel and the initrd. We will build these components separately. here we are just building the OS rootfs. |
--firmware-chroot false |
Do not include firmware in the chroot. chroot is the root directory of the OS during the build process. |
--apt-source-archives false |
Do not include apt source archives. We are not interested in the apt source archives since we are building a distro for an embedded system. We will talk about packages in a separate article. |
--chroot-filesystem none |
Do not use a chroot filesystem. Instead, use the host filesystem. We do this because we are building for a different architecture than the host. |
--compression gzip |
Use gzip compression for the output. |
--debootstrap-options |
debootstrap is the tool that is used to bootstrap the base system. We are passing the following options to debootstrap: --keyring=/usr/share/keyrings/debian-archive-keyring.gpg - Use the Debian ports keyring for the bootstrap. --variant=minbase - Use the minbase variant of debootstrap. This variant installs only the essential packages. --include=apt-transport-https,apt-utils,gnupg,ca-certificates,ssl-cert,openssl - Include the following packages in the bootstrap: apt-transport-https, apt-utils, gnupg, ca-certificates, ssl-cert, openssl. |
--distribution unstable |
Build for the unstable distribution. This is because we are building for the latest version of Debian. |
--gzip-options '-9 --rsyncable' |
Use the -9 option for gzip compression and use the --rsyncable option to make the output file rsync friendly. |
--iso-publisher |
Set the publisher of the iso. |
--iso-volume |
Set the volume of the iso. |
--linux-flavours none |
Do not include linux flavours. |
--linux-packages none |
Do not include linux packages. |
--mode debian |
Build a Debian distro. Other options are live and netboot . |
--system normal |
Build a normal system. Other options are chroot and live . |
--updates false |
Do not use updates. We will pull the latest packages from the Debian ports mirror. |
As a result of the command, the following files and directories will be created:
. ├── auto ├── config │ ├── apt │ ├── archives │ ├── binary │ ├── bootloaders │ ├── bootstrap │ ├── chroot │ ├── common │ ├── debian-installer │ ├── hooks │ │ ├── live │ │ │ ├── 0010-disable-kexec-tools.hook.chroot -> /usr/share/live/build/hooks/live/0010-disable-kexec-tools.hook.chroot │ │ │ └── 0050-disable-sysvinit-tmpfs.hook.chroot -> /usr/share/live/build/hooks/live/0050-disable-sysvinit-tmpfs.hook.chroot │ │ └── normal │ │ ├── 1000-create-mtab-symlink.hook.chroot -> /usr/share/live/build/hooks/normal/1000-create-mtab-symlink.hook.chroot │ │ ├── 1010-enable-cryptsetup.hook.chroot -> /usr/share/live/build/hooks/normal/1010-enable-cryptsetup.hook.chroot │ │ ├── 1020-create-locales-files.hook.chroot -> /usr/share/live/build/hooks/normal/1020-create-locales-files.hook.chroot │ │ ├── 5000-update-apt-file-cache.hook.chroot -> /usr/share/live/build/hooks/normal/5000-update-apt-file-cache.hook.chroot │ │ ├── 5010-update-apt-xapian-index.hook.chroot -> /usr/share/live/build/hooks/normal/5010-update-apt-xapian-index.hook.chroot │ │ ├── 5020-update-glx-alternative.hook.chroot -> /usr/share/live/build/hooks/normal/5020-update-glx-alternative.hook.chroot │ │ ├── 5030-update-plocate-database.hook.chroot -> /usr/share/live/build/hooks/normal/5030-update-plocate-database.hook.chroot │ │ ├── 5040-update-nvidia-alternative.hook.chroot -> /usr/share/live/build/hooks/normal/5040-update-nvidia-alternative.hook.chroot │ │ ├── 8000-remove-adjtime-configuration.hook.chroot -> /usr/share/live/build/hooks/normal/8000-remove-adjtime-configuration.hook.chroot │ │ ├── 8010-remove-backup-files.hook.chroot -> /usr/share/live/build/hooks/normal/8010-remove-backup-files.hook.chroot │ │ ├── 8020-remove-dbus-machine-id.hook.chroot -> /usr/share/live/build/hooks/normal/8020-remove-dbus-machine-id.hook.chroot │ │ ├── 8030-truncate-log-files.hook.chroot -> /usr/share/live/build/hooks/normal/8030-truncate-log-files.hook.chroot │ │ ├── 8040-remove-mdadm-configuration.hook.chroot -> /usr/share/live/build/hooks/normal/8040-remove-mdadm-configuration.hook.chroot │ │ ├── 8050-remove-openssh-server-host-keys.hook.chroot -> /usr/share/live/build/hooks/normal/8050-remove-openssh-server-host-keys.hook.chroot │ │ ├── 8060-remove-systemd-machine-id.hook.chroot -> /usr/share/live/build/hooks/normal/8060-remove-systemd-machine-id.hook.chroot │ │ ├── 8070-remove-temporary-files.hook.chroot -> /usr/share/live/build/hooks/normal/8070-remove-temporary-files.hook.chroot │ │ ├── 8080-reproducible-glibc.hook.chroot -> /usr/share/live/build/hooks/normal/8080-reproducible-glibc.hook.chroot │ │ ├── 8090-remove-ssl-cert-snakeoil.hook.chroot -> /usr/share/live/build/hooks/normal/8090-remove-ssl-cert-snakeoil.hook.chroot │ │ ├── 8100-remove-udev-persistent-cd-rules.hook.chroot -> /usr/share/live/build/hooks/normal/8100-remove-udev-persistent-cd-rules.hook.chroot │ │ ├── 8110-remove-udev-persistent-net-rules.hook.chroot -> /usr/share/live/build/hooks/normal/8110-remove-udev-persistent-net-rules.hook.chroot │ │ ├── 9000-remove-gnome-icon-cache.hook.chroot -> /usr/share/live/build/hooks/normal/9000-remove-gnome-icon-cache.hook.chroot │ │ ├── 9010-remove-python-pyc.hook.chroot -> /usr/share/live/build/hooks/normal/9010-remove-python-pyc.hook.chroot │ │ └── 9020-remove-man-cache.hook.chroot -> /usr/share/live/build/hooks/normal/9020-remove-man-cache.hook.chroot │ ├── includes │ ├── includes.binary │ ├── includes.bootstrap │ ├── includes.chroot_after_packages │ ├── includes.chroot_before_packages │ ├── includes.installer │ ├── includes.source │ ├── package-lists │ ├── packages │ ├── packages.binary │ ├── packages.chroot │ ├── preseed │ ├── rootfs │ └── source └── local └── bin 25 directories, 30 files
As you would have noticed, a lot of the hooks are pointing to defaults. These hooks are used to customize the build process. We will talk about some of the hooks we would want to add and/or replace with custom hooks in order to customize our image.
There are a lot of customiations that can be done to the distro. We will just focus on a few examples here to get started.
These files are typically stored in a seperate customization folder and then copied into the autogenerated configuration directory. This allows us to keep the autogenerated configuration directory clean and easy to maintain.
All the files below are created within the config folder.
This is a relatively random list of packages that I want to install in the distro. This is not a complete list and is just for demonstration purposes.
filename: package-lists/embeddedinn-sid-server.list.chroot
mkdir -p "./config/package-lists"
cat >"./config/package-lists/embeddedinn-sid-server.list.chroot" <<"EOM"
2ping
accountsservice
apparmor
apt-utils
arping
arpwatch
at
attr
bash-completion
bc
bcache-tools
bonnie++
bridge-utils
buffer
bzip2
build-essential
ca-certificates
cifs-utils
console-data
console-setup
cpio
cron
cryptsetup-bin
curl
dnsutils
dbus
debian-ports-archive-keyring
debootstrap
dmsetup
dnsutils
dosfstools
dselect
dump
ed
eject
elinks
etherwake
ethtool
exfat-fuse
exfatprogs
f2fs-tools
file
finger
fsarchiver
ftp
fuse3
gawk
gdbserver
gdisk
gdu
genromfs
git
gpart
gpg-agent
groff-base
hdparm
hexedit
htop
iftop
inetutils-telnet
info
iotop
ipcalc
iperf3
ipmitool
iptables
iptraf-ng
iputils-ping
iputils-tracepath
irqbalance
isc-dhcp-client
isc-dhcp-common
iso-codes
isomd5sum
kbd
keyboard-configuration
keyutils
less
lftp
lldpd
lm-sensors
locales
lrzsz
lsb-release
lshw
lsof
lvm2
lz4
lzip
lzop
makedev
man-db
manpages
mbr
memtester
mime-support
minicom
mtools
mtr-tiny
ncat
netcat-openbsd
netcat-traditional
net-tools
nicstat
nmap
ntfs-3g
ntpdate
ntpsec-ntpdate
ntpsec-ntpdig
nvme-cli
nwipe
openssh-client
perl
perl-modules
openssh-server
openssl
openvpn
p7zip
partclone
parted
patch
pciutils
perl
perl-modules
policykit-1
powermgmt-base
procinfo
psmisc
python3
python3-dbus
python3-distutils
python3-gdbm
python3-gi
resolvconf
rdate
rdiff-backup
rename
rlwrap
rmlint
rsync
rsyslog
screen
sdparm
setserial
sipcalc
smbclient
socat
squashfs-tools
sshfs
ssl-cert
strace
stress-ng
stunnel4
sudo
telnet
time
tcpdump
time
tmux
tofrodos
traceroute
ucf
udftools
ufw
unp
unzip
usbutils
uuid-runtime
vim
vlan
w3m
wakeonlan
wget
whois
wipe
xorriso
xxd
xxhash
xz-utils
zerofree
EOM
filename: includes.chroot/root/.profile
mkdir -p "./config/includes.chroot/root"
cat >"./config/includes.chroot/root/.profile" <<"EOM"
export LD_LIBRARY_PATH=$LIB:$LIBUSR:$LD_LIBRARY_PATH
EOM
filename: includes.chroot/root/.bashrc
mkdir -p "./config/includes.chroot/root"
cat >"./config/includes.chroot/root/.bashrc" <<"EOM"
export LD_LIBRARY_PATH=$LIB:$LIBUSR:$LD_LIBRARY_PATH
EOM
filename: includes.chroot/etc/resolv.conf
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/resolv.conf" <<"EOM"
nameserver 8.8.8.8
EOM
filename: includes.chroot/etc/rc.local
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/rc.local" <<"EOM"
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
FLAG="/var/log/firstboot.log"
if [ ! -f ${FLAG} ]
then
echo "First boot detected" >> ${FLAG}
echo "Performing initial setup tasks." >> ${FLAG}
echo "Creating ssh host keys..." >> ${FLAG}
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -q -N ""
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -q -N ""
fi
ntpdate -ub 0.pool.ntp.org > /dev/null
exit 0
EOM
chmod +x "./config/includes.chroot/etc/rc.local"
filename: includes.chroot/etc/os-release
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/os-release" <<"EOM"
NAME="embeddedinn RISC/V "
VERSION="13.0"
ID=embeddedinn
ID_LIKE=debian
PRETTY_NAME="embeddedinn RISC-V"
VERSION_ID="11.0"
HOME_URL="https://embeddedinn.com/"
SUPPORT_URL="https://embeddedinn.com/"
BUG_REPORT_URL="https://embeddedinn.com/"
EOM
filename: includes.chroot/etc/lsb-release
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/lsb-release" <<"EOM"
DISTRIB_ID=embeddedinn
DISTRIB_RELEASE=13
DISTRIB_CODENAME=Sid
DISTRIB_DESCRIPTION="embeddedinn Sid"
EOM
filename: includes.chroot/etc/legal
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/legal" <<"EOM"
The programs included with the Embeddedinn system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Embeddedinn comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
EOM
filename: includes.chroot/etc/issue.net
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/issue.net" <<"EOM"
Embeddedinn Sid riscv64
EOM
filename: includes.chroot/etc/issue
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/issue" <<"EOM"
Embeddedinn Sid RISC/V \n \l
EOM
filename: includes.chroot/etc/hosts
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/hosts" <<"EOM"
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.1.1 embeddedinn
EOM
filename: includes.chroot/etc/hostname
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/hostname" <<"EOM"
embeddedinn
EOM
filename: includes.chroot/etc/fstab
mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/fstab" <<"EOM"
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
tmpfs /run tmpfs defaults 0 0
tmpfs /var/log tmpfs defaults 0 0
tmpfs /var/tmp tmpfs defaults 0 0
tmpfs /var/spool tmpfs defaults 0 0
tmpfs /var/cache tmpfs defaults 0 0
EOM
filename: includes.chroot/etc/update-motd.d/10-help-text
mkdir -p "./config/includes.chroot/etc/update-motd.d"
cat >"./config/includes.chroot/etc/update-motd.d/10-help-text" <<"EOM"
#!/bin/sh
#
# 10-help-text - print the help text associated with the distro
# Copyright (C) 2009-2010 Canonical Ltd.
#
# Authors: Dustin Kirkland <kirkland@canonical.com>,
# Brian Murray <brian@canonical.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
[ -r /etc/lsb-release ] && . /etc/lsb-release
if [ -z "$DISTRIB_RELEASE" ] && [ -x /usr/bin/lsb_release ]; then
# Fall back to using the very slow lsb_release utility
DISTRIB_RELEASE=$(lsb_release -sr)
fi
URL="https://embeddedinn.com/"
printf "\n * Documentation: %s\n" "$URL"
EOM
chmod +x "./config/includes.chroot/etc/update-motd.d/10-help-text"
filename: includes.chroot/etc/skel/.profile
mkdir -p "./config/includes.chroot/etc/skel"
cat >"./config/includes.chroot/etc/skel/.profile" <<"EOM"
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
# if running bash
if [ -n "$BASH_VERSION" ]; then
# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
fi
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
export LD_LIBRARY_PATH=$LIB:$LIBUSR:$LD_LIBRARY_PATH
EOM
filename: includes.chroot/etc/skel/.bashrc
mkdir -p "./config/includes.chroot/etc/skel"
cat >"./config/includes.chroot/etc/skel/.bashrc" <<"EOM"
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth
# append to the history file, don't overwrite it
shopt -s histappend
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000
# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize
# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar
# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"
# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
debian_chroot=$(cat /etc/debian_chroot)
fi
# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
xterm-color) color_prompt=yes;;
esac
# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes
if [ -n "$force_color_prompt" ]; then
if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
# We have color support; assume it's compliant with Ecma-48
# (ISO/IEC-6429). (Lack of such support is extremely rare, and such
# a case would tend to support setf rather than setaf.)
color_prompt=yes
else
color_prompt=
fi
fi
if [ "$color_prompt" = yes ]; then
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt
# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
;;
*)
;;
esac
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
#alias dir='dir --color=auto'
#alias vdir='vdir --color=auto'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
fi
# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'
# Add an "alert" alias for long running commands. Use like so:
# sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'
# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
if [ -f /usr/share/bash-completion/bash_completion ]; then
. /usr/share/bash-completion/bash_completion
elif [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
fi
export LD_LIBRARY_PATH=$LIB:$LIBUSR:$LD_LIBRARY_PATH
EOM
filename: includes.chroot/etc/network/interfaces.d/eth0
mkdir -p "./config/includes.chroot/etc/network/interfaces.d"
cat >"./config/includes.chroot/etc/network/interfaces.d/eth0" <<"EOM"
auto eth0
allow-hotplug eth0
iface eth0 inet dhcp
EOM
filename: includes.chroot/etc/default/locale
mkdir -p "./config/includes.chroot/etc/default"
cat >"./config/includes.chroot/etc/default/locale" <<"EOM"
LANG=en_US.UTF-8
EOM
Files in the hooks/live
directory are executed during the live boot process. We will add a few hooks to customize the live boot process. filenames ending with .chroot
are executed in the chroot environment.
filename: hooks/live/98-update_password.chroot
mkdir -p "./config/hooks/live"
cat >"./config/hooks/live/98-update_password.chroot" <<"EOM"
#!/bin/sh
echo "I: update password"
echo "root:embeddedinn" | chpasswd
EOM
chmod +x "./config/hooks/live/98-update_password.chroot"
The full list of customization files we created is shown below:
. ├── hooks │ └── live │ └── 98-update_password.chroot ├── includes.chroot │ ├── etc │ │ ├── default │ │ │ └── locale │ │ ├── fstab │ │ ├── hostname │ │ ├── hosts │ │ ├── issue │ │ ├── issue.net │ │ ├── legal │ │ ├── lsb-release │ │ ├── network │ │ │ └── interfaces.d │ │ │ └── eth0 │ │ ├── os-release │ │ ├── rc.local │ │ ├── resolv.conf │ │ ├── skel │ │ └── update-motd.d │ │ └── 10-help-text │ └── root └── package-lists └── embeddedinn-sid-server.list.chroot 12 directories, 15 files
Now that we have a customized configuration directory, we can build the distro with the following command:
sudo lb build
The debian live build goes through the following high level stages:
debootstrap
is the tool that is used to bootstrap the base system. It brings up a minimal system that is used to build the final system.This command will take a while to complete. Once it is done, the main file that we are interested in is the live-image-riscv64.tar.tar.gz
file and its source - the chroot
folder. They contain the rootfs of the distro. We can create an ext2
filesystem with this file and use it to boot the distro on a RISC-V machine.
IMAGENAME=embeddedinn-sid-riscv64-$(date +%Y%m%d)
sudo virt-make-fs --partition=gpt --type=ext4 --size=10G ./chroot $IMAGENAME.img;
sudo chmod a+rwx $IMAGENAME.img;
As you would have noticed, we just created the rootfs of the distro. We still need to create the kernel and the initrd. I have covered this extensively in other articles on bringing up Linux on RISCV. So, for the time being, we will quickly build a vanilla kernel and use the bootloader that comes packaged with QEMU.
NOTE Livebuild has the capability to pull in and package one or more kernel flavours into the image by default, depending on the architecture. You can choose different flavours via the
--linux-flavours
option. We chosenone
since a custom kernel is ofetn practical for embedded systems.
We will build a kernel and initrd/rootfs with buildroot to make it easier to setup the system. you cna also just compile the kernel directly.
sudo apt install flex bison bc unzip rsync libncurses-dev
wget https://buildroot.org/downloads/buildroot-2023.02.8.tar.gz
tar -xvf buildroot-2023.02.8.tar.gz
cd buildroot-2023.02.8
make qemu_riscv64_virt_defconfig
make
Before booting the debian distro, we can check if the kernel and initrd are working by booting the kernel and initrd with QEMU built by Buildroot. Buildroot provides a startup script that can be executed with:
output/images/start-qemu.sh
To boot into the bespoke Debian disk image we built, instead of the buildroot disk , we will first coy the .img
we created into the output folder and then modify the startup script to use the new image as below.
exec qemu-system-riscv64 -M virt -bios fw_jump.elf -kernel Image -append "rootwait root=/dev/vda1 rw" -m 4G -drive file=embeddedinn-sid-riscv64-20231228.img,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -netdev user,id=net0 -device virtio-net-device,netdev=net0 -nographic ${EXTRA_ARGS}
Note that QEMU is using virt-io
block device emulation to mount the disk image. In a real system, we would use a real block device like an SD card or an eMMC. The bootloader or the system ROM would initialise the hardwre interface and get it ready to be mounted by the kernel.
Looking at the boot and kernel logs , the high leve steps involved in juming into the debian system are:
/sbin/init
which is a symbolic link to /lib/systemd/systemd
. This was installed into the rootfs by livebuild when we added the systemd-sysv
package to the package list.getty
service starts the getty
process on the console. This is the process that allows us to login to the system.I am not going to re-iterate what Git is and why it is so popular. There are plenty of articles and videos out there that do a great job of explaining that. I am going to assume that you already know what Git is and how to use it. If you don’t, I would recommend you to go through the official Git documentation and Pro Git book before reading this article.
The target audience for this article is the curious ones who wants to waddle in the internals of Git and learn how it works under the hood. For instance, we will be creating git objects with Python code and creating commits with git plumbing commands that are not meant to be used by end users. If you are not comfortable with git in the first place, this article is not for you.
When you run git init
in a directory, Git creates a .git
directory in that directory. This .git
directory is where Git stores all the data related to the repository. Let’s take a look at the contents of the .git
directory.
.git/
├── branches
├── config
├── description
├── HEAD
├── hooks
│ ├── applypatch-msg.sample
│ ├── commit-msg.sample
│ ├── fsmonitor-watchman.sample
│ ├── post-update.sample
│ ├── pre-applypatch.sample
│ ├── pre-commit.sample
│ ├── pre-merge-commit.sample
│ ├── prepare-commit-msg.sample
│ ├── pre-push.sample
│ ├── pre-rebase.sample
│ ├── pre-receive.sample
│ ├── push-to-checkout.sample
│ └── update.sample
├── info
│ └── exclude
├── objects
│ ├── info
│ └── pack
└── refs
├── heads
└── tags
The .git
directory contains a bunch of files and directories. Let’s take a look at each of them.
File/Dir | Contents |
---|---|
branches | directory that contains the list of branches in the repository. The branch info is stored in the form of a file with the branch name as the filename and the contents of the file being the commit hash of the latest commit in that branch. |
config | file that contains the configuration of the repository. This can be used to override the global configuration. this file also contains the remote repository information. |
description | file that contains the description of the repository. This is used by GitWeb to display the description of the repository. |
HEAD | file that contains the reference to the current branch. We will talk more about what a reference is later in this article. |
hooks | directory that contains the hooks that can be used to trigger custom actions at various stages of the Git workflow. We will not go into the details of hooks in this article. The .sample files are the sample hooks that can be used as a starting point for writing custom hooks. |
info | directory that contains the global exclude file. This file contains the list of files that should be ignored by Git. As the repository grows, additional information like the list of alternates and the list of grafts are stored in this directory. |
objects | directory that contains the actual data of the repository. This is where Git stores all the commits, trees, blobs, and tags. |
refs | directory that contains the references to the commits. We will talk more about what a reference is later in this article. |
We will use a non-conventional approach to committing a file into git to understand how Git stores commits.
First, lets create a new file with some content in it.
echo "Hello World" > hello.txt
Now, we will use the git hash-object
command to create a blob object from the file.
$ git -w hash-object hello.txt
557db03de997c86a4a028e1ebd3a1ceb225be238
The -w
flag tells Git to write the object to the object database. The hash-object
command returns the SHA-1 hash of the object that was created. The SHA-1 hash of the object is the name of the file that is created in the object database. Let’s take a look at the contents of the object database.
.git/
├── branches
├── config
├── description
├── HEAD
├── hooks
├── info
│ └── exclude
├── objects
│ ├── 55
│ │ └── 7db03de997c86a4a028e1ebd3a1ceb225be238
│ ├── info
│ └── pack
└── refs
├── heads
└── tags
Git stores contents in an object using its own custom format that includes a zlib compressed version of the contents, the type of the object, and the size of the contents. The type of the object is stored as a header in the object. some key type of the object are as follows (not all object types are covered).
Type | Description |
---|---|
blob | A blob object represents the contents of a file. |
tree | A tree object represents the contents of a directory. |
commit | A commit object represents a commit. |
tag | A tag object represents a tag. |
A python
routine to read the contents of the object is shown below.
import zlib
import hashlib
def read_object(sha):
with open('.git/objects/' + sha[:2] + '/' + sha[2:], 'rb') as f:
raw = zlib.decompress(f.read())
return raw
print(read_object('557db03de997c86a4a028e1ebd3a1ceb225be238')) # use .decode() to print the contents of the object
The oputput of the above program is shown below.
b'blob 12\x00Hello World\n'
We have just created a loose object that is not part of anything tracked by Git. But we can use teh git show
or git cat-file -p
command to view the contents of the object.
$ git cat-file -p 557db03d
Hello World
We can even use a python routine to create a loose object.
import zlib
import hashlib
import os
def write_object(data, type):
header = type + ' ' + str(len(data)) + '\x00'
store = header.encode() + data
sha = hashlib.sha1(store).hexdigest()
if not os.path.exists('.git/objects/' + sha[:2]):
os.makedirs('.git/objects/' + sha[:2])
with open('.git/objects/' + sha[:2] + '/' + sha[2:], 'wb') as out:
out.write(zlib.compress(store))
return sha
print(write_object(b'Hello New World\n', 'blob'))
The program will create a loose object in the object database and return the SHA-1 hash of the object.
d9786ef99a397ad94795405041cb9590712053f6
.git/
├── branches
├── config
├── description
├── HEAD
├── hooks
├── info
│ └── exclude
├── objects
│ ├── 55
│ │ └── 7db03de997c86a4a028e1ebd3a1ceb225be238
│ ├── d9
│ │ └── 786ef99a397ad94795405041cb9590712053f6
│ ├── info
│ └── pack
└── refs
├── heads
└── tags
Contents of this new file can be viewed using the git cat-file -p
command.
$ git cat-file -p d9786ef9
Hello New World
Though there is a new object, there is no file in our working directory with this contents.
$ ls
hello.txt
As you would have noticed, there is no information about the file name or the directory structure in the object. This is because Git does not store the file name or the directory structure in the object. Git only stores the contents of the file in the object. The file name and the directory structure is stored in a tree object. Lets create a tree object and see how it is stored in the object database.
Git stores a group of files and directories in a tree object. A tree object is a essentially a directory that contains other trees and blobs. Lets create a tree object that contains the two objects we created earlier.
For this, we need to first stage the two files in the index. We can do this using the git update-index
command.
$ git update-index --add --cacheinfo 100644 557db03d hello.txt
$ git update-index --add --cacheinfo 100644 d9786ef9 hello2.txt
In this case, you’re specifying a mode of 100644
, which means it’s a normal file. Other options are 100755
, which means it’s an executable file; and 120000
, which specifies a symbolic link. The mode is taken from normal UNIX modes but is much less flexible.
Now you can see that there is a new index
file in the .git
directory.
The index
file has a git internal format that is documented in the git documentation. We can use the git ls-files --stage
command to view the contents of the index file.
$ git ls-files --stage
100644 557db03de997c86a4a028e1ebd3a1ceb225be238 0 hello.txt
100644 d9786ef99a397ad94795405041cb9590712053f6 0 hello2.txt
Issuing a git status
will show that there are two files that are available to commit, even though there is no second file in the working directory.
$ git status
On branch main
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: hello.txt
new file: hello2.txt
Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
deleted: hello2.txt
Since the hello2.txt
file is not present on disk, git sees it as a delete operation. We can use the git restore hello2.txt
command that will create the hello2.txt
file in the working directory with contents from the index
and the object
we created earlier. But this step is not necessary for us to create the tree object.
When we issue the git write-tree
command, git will create a tree object that contains the two files we added to the index. The tree object will be stored in the object database and the SHA-1 hash of the tree object will be returned.
$ git write-tree
60fdbb80045aca16edfa035e7a4b7b2ce5ebe5aa
We can use the git cat-file
command to view the contents of the tree object.
$ git cat-file -t 60fdbb80
tree
$ git cat-file -p 60fdbb80
100644 blob 557db03de997c86a4a028e1ebd3a1ceb225be238 hello.txt
100644 blob d9786ef99a397ad94795405041cb9590712053f6 hello2.txt
The tree object contains the file mode, the type of the object (blob or tree) and the SHA-1
hash of the object.
The git read-tree
command can be used to read the contents of a tree object into the index. This is useful when you want to checkout a commit. The git read-tree
command will read the contents of the tree object into the index and the git checkout-index
command will create the files in the working directory.
$ git read-tree 60fdbb80
$ git checkout-index -a
A commit object is a git object that contains the commit message, the author, the committer and the tree object that represents the contents of the commit. Lets create a commit object that contains the tree object we created earlier.
$ git commit-tree 60fdbb80 -m "Initial commit"
efb4ebf62f7ec3e9e078f232ef0f00a175140046
Ans the commit object contents can be viewed using the git cat-file
command.
$ git cat-file -p efb4ebf6
tree 60fdbb80045aca16edfa035e7a4b7b2ce5ebe5aa
author vpillai <vysakhpillai@embeddedinn.xyz> 1686972765 -0700
committer vpillai <vysakhpillai@embeddedinn.xyz> 1686972765 -0700
Initial commit
We can also use the git log
command to view the commit history.
$ git log --stat efb4ebf6
Author: vpillai <vysakhpillai@embeddedinn.xyz>
Date: Fri Jun 16 20:32:45 2023 -0700
Initial commit
hello.txt | 1 +
hello2.txt | 1 +
2 files changed, 2 insertions(+)
But git status
still reports that there are no commits.
$ git status
On branch main
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: hello.txt
new file: hello2.txt
This is because the HEAD
reference is still pointing to the main
branch that does not have a corresponding ref entry. We can use the git update-ref
command to update the HEAD
reference to point to the commit object we created earlier.
$ git update-ref refs/heads/main efb4ebf6
This creates a new file in the .git/refs/heads
directory that contains the SHA-1
hash of the commit object.
$ cat .git/refs/heads/main
efb4ebf62f7ec3e9e078f232ef0f00a175140046
An entry is also created in the logs/refs/heads
directory that contains the SHA-1
hash of the commit object and the commit message.
$ cat .git/logs/refs/heads/main
0000000000000000000000000000000000000000 efb4ebf62f7ec3e9e078f232ef0f00a175140046 vpillai <vysakhpillai@embeddedinn.xyz> 1686973167 -0700
This entry says that main moved from 00
to efb4ebf6
. The 00
is the SHA-1
hash of the empty tree object. The git log
command will now show the commit we created earlier.
$ git log
commit efb4ebf62f7ec3e9e078f232ef0f00a175140046 (HEAD -> main)
Author: vpillai <vysakhpillai@embeddedinn.xyz>
Date: Fri Jun 16 20:32:45 2023 -0700
Initial commit
The refs
directory contains the references to the commit objects. The HEAD
reference is a symbolic reference that points to the current branch. The HEAD
reference is stored in the .git/HEAD
file.
$ cat .git/HEAD
ref: refs/heads/main
The refs/heads
directory contains the references to the branches. The refs/tags
directory contains the references to the tags. The refs/remotes
directory contains the references to the remote branches.
A branch is a reference to a commit object. When a new branch is created, a new file is created in the .git/refs/heads
directory that contains the SHA-1
hash of the commit object. Lets create a new branch called dev
and checkout the branch using low level git commands.
$ git update-ref refs/heads/dev efb4ebf6
A new branch is now created
$ git branch
dev
* main
This creates a new file in the .git/refs/heads
directory and the .git/log/refs/heads
directory that contains the SHA-1
hash of the commit object.
.git/
├── branches
├── config
├── description
├── HEAD
├── hooks
├── index
├── info
│ └── exclude
├── logs
│ ├── HEAD
│ └── refs
│ └── heads
│ ├── dev
│ └── main
├── objects
│ ├── 55
│ │ └── 7db03de997c86a4a028e1ebd3a1ceb225be238
│ ├── 60
│ │ └── fdbb80045aca16edfa035e7a4b7b2ce5ebe5aa
│ ├── d9
│ │ └── 786ef99a397ad94795405041cb9590712053f6
│ ├── ef
│ │ └── b4ebf62f7ec3e9e078f232ef0f00a175140046
│ ├── info
│ └── pack
└── refs
├── heads
│ ├── dev
│ └── main
└── tags
At this stage, main and dev are pointing to the same commit object. Checking out the new branch simply means updating the HEAD
reference to point to the new branch. Instead of using git update-ref HEAD refs/heads/dev
, we can simply update the contents of the .git/HEAD
file to point git to the new branch.
$ echo "ref: refs/heads/dev" > .git/HEAD
$ git branch
* dev
main
Now, we can create a new commit on the dev
branch, but stil using the low level git commands.
#update the file
$ echo "Hello World Uno" > hello.txt
# create a new object for the file
$ git hash-object -w hello.txt
2a323159bea5a5bf98c0ccaef350cd6141f0f3df
$ git update-index --add --cacheinfo 100644 2a323159 hello.txt
$ git status
On branch dev
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: hello.txt
# add sub tree
$ git write-tree
e0aefbba82dd2e7653ae6d46f00bbed584fac52f
# commit tree with parent
$ git commit-tree e0aefbba -p efb4ebf6 -m "Commit to dev"
152b7866c2e126eec65bafec327d9d760bef99c7
# make dev point to the new commit
$ git update-ref refs/heads/dev 152b7866
# check status
$ git status
On branch dev
nothing to commit, working tree clean
# check log
$ cat .git/logs/refs/heads/dev
0000000000000000000000000000000000000000 efb4ebf62f7ec3e9e078f232ef0f00a175140046 vpillai <vysakhpillai@embeddedinn.xyz> 1686974500 -0700
efb4ebf62f7ec3e9e078f232ef0f00a175140046 152b7866c2e126eec65bafec327d9d760bef99c7 vpillai <vysakhpillai@embeddedinn.xyz> 1686974696 -0700
Lets look a how Git handles incremental changes. For this, we will start with a clean repository.
$ git init
$ echo "Hello World" > hello.txt
$ git add hello.txt
$ git commit -m "Initial commit"
.git/
├── branches
├── COMMIT_EDITMSG
├── config
├── description
├── HEAD
├── hooks
├── index
├── info
│ └── exclude
├── logs
│ ├── HEAD
│ └── refs
│ └── heads
│ └── main
├── objects
│ ├── 3c
│ │ └── 80f66564c9ee69d0987e48a545becc3025deb1
│ ├── 55
│ │ └── 7db03de997c86a4a028e1ebd3a1ceb225be238
│ ├── 97
│ │ └── b49d4c943e3715fe30f141cc6f27a8548cee0e
│ ├── info
│ └── pack
└── refs
├── heads
│ └── main
└── tags
# check the tree contents
$ git cat-file -p 97b49d4c
100644 blob 557db03de997c86a4a028e1ebd3a1ceb225be238 hello.txt
Now, lets update the file and commit the changes.
$ echo "Hello World Uno" > hello.txt
$ git add hello.txt
$ git commit -m "Update hello.txt"
This created 3 new objects:
Type | Object hash |
---|---|
blob | 2a323159 |
tree | 106ed651 |
commit | 06a73f25 |
The blob contains the entire contents of the updated file (not just the diff). The tree contains the updated blob and the commit contains the updated tree.
$ git cat-file -p 557db03d
Hello World
$ git cat-file -p 2a323159
Hello World Uno
git log
shows the commit history including the two commit objects. they are linked by the parent
field in the commit object.
$ git cat-file -p 06a73f25
tree 106ed65198ebfbde9f4e7e8bd6ceb2dd2e5268ce
parent 3c80f66564c9ee69d0987e48a545becc3025deb1
author vpillai <vysakhpillai@embeddedinn.xyz> 1686976022 -0700
committer vpillai <vysakhpillai@embeddedinn.xyz> 1686976022 -0700
This tree information can be visualized using standard git commands.
$ git log --graph --oneline --decorate --all
* 06a73f2 (HEAD -> main) Update hello.txt
* 3c80f66 Initial commit
This article is a brief introduction to the internals of Git. It is by no means a complete guide. The goal was to understand the basic concepts and to get a feel for how basic Git operations works. The next article will cover the internals of the git add
command and how it interacts with the index
and the working tree.
Digital signatures are a fundamental building block of modern cryptography. We often use digital signatures without realizing it in our day to day lives. For example, when you visit a website, your browser verifies the authenticity of the website using a digital signature. In this article, we’ll explore how digital signature works, how to generate and verify digital signatures, and how to use digital signatures to sign and verify files.
My original motivation to write this article is to develop some fundamental understanding of digital signatures for the embedded systems and system engineering community. I have often seen that digital signatures are seen as a mystery by many developers. I hope that this article will help demystify digital signatures and help developers understand how digital signatures work and how to use them in their projects.
While there is an abundance of digital signature usage in the cryptocurrency world, I will not be covering that in this article.
I have built a web-tool to play around with digital signatures at https://vppillai.github.io/cryptoScript/FileSigner.html. You can use this tool to generate and verify digital signatures for files. The tool is written in Javascript and uses the WebCrypto API to generate and verify digital signatures. The source code for the tool and a few other useful crypto scripts are available at https://github.com/vppillai/cryptoScript
We will cover the details of the tool in the later sections of this article. For now, let’s start with the basics.
I would recommend that you also read my articles on Introduction to digital certificates and Understanding X.509 Certificate Structure before continuing with this article if you are new to cryptography.
A digital signature is a mathematical scheme for verifying the authenticity of digital messages or documents. A valid digital signature gives a recipient reason to believe that the message was created by a known sender, and that the message was not altered in transit. Digital signatures are commonly used for software distribution, financial transactions, and in other cases where it is important to detect forgery or tampering.
Now, what does it look like ?
The short / incomplete answer - something like this (in base64)
``bash DzjGzKR5LZgAdU9ObP/bt9LFf5/+YtlTHJNtWZH8kAOQE5cTW0lE/Q9jQcdScVsNuvktdhfaBZhowgoY08S8Fw==
Or, this (in `hex`)
```bash
0f 38 c6 cc a4 79 2d 98 00 75 4f 4e 6c ff db b7 d2 c5 7f 9f fe 62 d9 53 1c 93 6d 59 91 fc 90 03
90 13 97 13 5b 49 44 fd 0f 63 41 c7 52 71 5b 0d ba f9 2d 76 17 da 05 98 68 c2 0a 18 d3 c4 bc 17
To understand how a digital signature is generated and how its useful in verifying the authenticity of data, we need to understand the underlying cryptographic primitives that are used to generate a digital signature. Most importantly, we need to understand the concept of a cryptographic hash function.
A cryptographic hash function is a mathematical algorithm that maps data of arbitrary size to a bit string of a fixed size (a hash) and is designed to be a one-way function, that is, a function which is non-feasible to invert. The only way to recreate the input data from an ideal cryptographic hash function’s output is to attempt a brute-force search of possible inputs to see if they produce a match, or use a rainbow table of matched hashes. (rainbow tables are pre-computed lookup tables where the hash of every possible text input combination is stored. This allows for quick lookup of data given the hash).
Since the hashing function converts an input of arbitrary size to a fixed size output, it is often called a Digest
function. The length of the hash determines how easy or difficult it is to find a hash collision for a given input. The longer the hash, the more difficult it is to find a collision. For example, the SHA-1 hash function produces a 160-bit hash. This means that there are 2^160 possible hashes. This is a very large number. If you were to try to find a collision for a given input, you would have to try 2^160 different inputs to find a collision. This is not feasible. However, if the hash function produces a 32-bit hash, then there are only 2^32 possible hashes. This is a much smaller number. If you were to try to find a collision for a given input, you would only have to try 2^32 different inputs to find a collision. This is feasible given the computational capability we have today.
A cryptographic hash function is different from a regular hash function in that it has to satisfy the following properties:
An example of sha256
hash of a string is shown below:
$ echo -n "Hello World" | sha256sum
a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e -
$ echo -n "Hello World!" | sha256sum
7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069 -
Simply put, a digital signature is an “Encrypted hash” of a message. This implies that an encryption algorithm is involved in the process of creating a digital signature. However, the encryption algorithm used to create a digital signature is different from the encryption algorithm used to encrypt data. The encryption algorithm used to create a digital signature is called a Signing Algorithm
. The encryption algorithm used to encrypt data is called an Encryption Algorithm
. The signing algorithm is a part of a larger class of algorithms called Public Key Cryptography
algorithms.
Some examples of Digital Signature Algorithms are:
RSA
(Rivest-Shamir-Adleman): RSA
is an asymmetric cryptographic algorithm widely used for encryption, digital signatures, and key exchange. It relies on the difficulty of factoring large prime numbers.DSA
(Digital Signature Algorithm): DSA
is a United States Federal Government standard for digital signatures. It is based on the mathematical concept of modular exponentiation and discrete logarithm problem.ECDSA
(Elliptic Curve Digital Signature Algorithm): ECDSA
is an asymmetric cryptographic algorithm based on elliptic curve mathematics. It provides strong security with shorter key lengths compared to RSA and DSA
.EdDSA
(Edwards-curve Digital Signature Algorithm): EdDSA
is an elliptic curve digital signature algorithm designed for high performance and security. It offers faster signing and verification compared to ECDSA
.We will focus on the ECDSA
algorithm to look at how a digital signature is generated.
We will not go into the details of key generation and the mathematics behind the ECDSA
algorithm. We will focus on the steps involved in generating a digital signature.
The steps involved in generating a digital signature are as follows:
Note: These steps provide a simplified overview of the ECDSA process. In practice, there may be additional considerations such as encoding, padding, and other security measures that need to be taken into account for proper implementation.
A private key and a corresponding public key. The private key (signing key
) is kept secret, while the public key (verification key
) can be shared with others in order to verify the signature.
ephemeral key
or nonce
. This is a one-time use number.k-coordinate
by multiplying the generator point on the elliptic curve by the nonce.r-coordinate
by taking the x
-coordinate of the resulting point modulo the order of the curve.s-coordinate
by taking the modular inverse of the nonce and multiplying it with the sum of the hash of the message and the product of the private key and r-coordinate, again modulo the order of the curve.The resulting signature is a pair of 32 byte values (r, s)
. The signature is 64 bytes in total. Note that the signature is also dependent on the hash algorithm used. typically sha256
is used with ECDSA
.
My web-tool available here can be used to generate a signature using the ECDSA
algorithm. The tool also provides the option to generate a key pair and verify a signature.
You can also try this out using the openssl
command line tool. With the commands below, you can generate a key pair, sign a message, and view the signature. note the hashing algorithm used is sha256
. Openssl lets you use any of the following hashing algorithm selectors here : -sha1
, -sha224
, -sha256
, -sha3-224
, -sha3-256
, -sha3-384
, -sha3-512
, -sha384
, -sha512
, -sha512-224
, -sha512-256
, -shake128
, -shake256
.
$ openssl ecparam -name secp256k1 -genkey -noout -out private.pem
$ openssl ec -in private.pem -pubout -out public.pem
$ echo -n "Hello World" | openssl dgst -sha256 -sign private.pem > signature.bin
$ hexdump -C signature.bin
The signature is generated in the DER
format. The hexdump
command can be used to view the contents of the signature file. You would notice that the output is larger than 64 bytes (32 bytes for r
and 32 bytes for s
). This is because the signature is encoded in the DER
format. To see the actual r
and s
values, you can use the asn1parse
command:
$ openssl asn1parse -inform DER -in signature.bin
A sample of the output, while using my temporary keys looks like this:
0:d=0 hl=2 l= 69 cons: SEQUENCE
2:d=1 hl=2 l= 33 prim: INTEGER :D4EC3929BF67345CA1442AE9B145BA1B50550C5DD1851A2EFE91D23F26CE012C
37:d=1 hl=2 l= 32 prim: INTEGER :4B0C79E4AFBED5C3158968C0398D3910948D103E101B720F8ABC2B12A1A275E0
The DER encoding for storing ECDSA signatures is defined in RFC 3279.
Its interesting to note that the signature changes every time you sign the same message. This is because the nonce is chosen randomly every time. This is a security feature to prevent the private key from being exposed. You can opt out of this feature by using a deterministic nonce. An example is Deterministic ECDSA
RFC 6979. This is not supported by openssl
yet.
s-coordinate
.w-coordinate
by multiplying the inverse of s
with the hash of the message.u-coordinate
by multiplying the inverse of s
with the signature’s r-coordinate
.v-coordinate
by multiplying the generator point on the elliptic curve by u-coordinate
and the public key by w-coordinate
, and summing them.x-coordinate
of v-coordinate
modulo the order of the curve matches the signature’s r-coordinate
, the signature is considered valid.Following the steps in the signature generation section, you can verify the signature using the openssl
command line tool:
$ echo -n "Hello World" | openssl dgst -sha256 -verify public.pem -signature signature.bin
Any change to the message will result in a different signature, and the verification will fail.
$ echo -n "Hello World!" | openssl dgst -sha256 -verify public.pem -signature signature.bin
The digital signature algorithms do not specify how the generated digital signatures can be clubbed with the original data. This is left to the implementer. The simplest form of transmitting the signature is one where the r
and s
values are concatenated together. This is called the raw
format. The resulting signature is 64 bytes in length. There is no reference to the signer or the message that was signed. This is not a recommended format for transmitting signatures unless it is part of a larger, well defined protocol.
DER
formatted signatures as defined in RFC 3279 are just a step up from the raw format. The DER
format is a binary format that is not human readable. The base64
encoding of the DER
format is 88
bytes in length and can be appended to the original message and transmitted over the network. The receiver can then extract the signature and verify it.
$ echo -n "Hello World" | openssl dgst -sha256 -sign private.pem | base64
Unlike other PEM
format headers specified in specifications like RFC 7468, there is no header defined to store just the base64 encoded signature since the signature alone cannot be used without additional data about the signer or the message. However, in all these methods standalone signature transmission is possible since the process of signature verification recomputes the hash of the message and compares it with the hash in the signature.
Note that the key used to verify the signature must be stored securely in a root of trust.
CMS, also commonly referred to as PKCS#7
(Public Key Cryptography Standards #7), is a standard syntax for securely exchanging signed or encrypted messages. The ECDSA digital signature is typically included as part of a SignedData
structure, along with the original message, the signer’s certificate, and any necessary cryptographic information. CMS allows for more advanced features like including multiple signers, certificates, and timestamps. (Dat ain CMS is often referred to as CMS container or PKCS#7 container)
PKCS#7 has an embedded mode where the original message is included in the signature. It also has a detached mode where the original message is not included in the signature. The detached mode is typically used when the original message is too large to be included in the signature or is wrapped in another data format like PDF already.
The openssl
command line tool can be used to generate a CMS container. However, a certificate is required to hold the public key of the signer. The certificate can be self-signed or issued by a certificate authority. An ECC certificate can be generated using the openssl
command line tool as follows:
$ openssl req -new -x509 -key private.pem -out cert.pem -days 365 -subj '/CN=embeddedinn.ca/O=embeddedinn/C=IN'
Now, we can generate a CMS container using the openssl
command line tool as follows:
$ echo -n "Hello World" | openssl cms -sign -signer cert.pem -inkey private.pem -outform PEM -out signedData.cms
The generated data is in DER format. You can see the contents using the command
$ openssl asn1parse -in signedData.cms
0:d=0 hl=4 l= 941 cons: SEQUENCE
4:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-signedData
15:d=1 hl=4 l= 926 cons: cont [ 0 ]
19:d=2 hl=4 l= 922 cons: SEQUENCE
23:d=3 hl=2 l= 1 prim: INTEGER :01
26:d=3 hl=2 l= 13 cons: SET
28:d=4 hl=2 l= 11 cons: SEQUENCE
30:d=5 hl=2 l= 9 prim: OBJECT :sha256
41:d=3 hl=2 l= 11 cons: SEQUENCE
43:d=4 hl=2 l= 9 prim: OBJECT :pkcs7-data
54:d=3 hl=4 l= 462 cons: cont [ 0 ]
58:d=4 hl=4 l= 458 cons: SEQUENCE
62:d=5 hl=4 l= 367 cons: SEQUENCE
66:d=6 hl=2 l= 3 cons: cont [ 0 ]
68:d=7 hl=2 l= 1 prim: INTEGER :02
71:d=6 hl=2 l= 19 prim: INTEGER :258BB8CE183F0F0AB4BD0D5119E69397ABFD72
92:d=6 hl=2 l= 10 cons: SEQUENCE
94:d=7 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256
104:d=6 hl=2 l= 60 cons: SEQUENCE
106:d=7 hl=2 l= 23 cons: SET
108:d=8 hl=2 l= 21 cons: SEQUENCE
110:d=9 hl=2 l= 3 prim: OBJECT :commonName
115:d=9 hl=2 l= 14 prim: UTF8STRING :embeddedinn.ca
131:d=7 hl=2 l= 20 cons: SET
133:d=8 hl=2 l= 18 cons: SEQUENCE
135:d=9 hl=2 l= 3 prim: OBJECT :organizationName
140:d=9 hl=2 l= 11 prim: UTF8STRING :embeddedinn
153:d=7 hl=2 l= 11 cons: SET
155:d=8 hl=2 l= 9 cons: SEQUENCE
157:d=9 hl=2 l= 3 prim: OBJECT :countryName
162:d=9 hl=2 l= 2 prim: PRINTABLESTRING :IN
166:d=6 hl=2 l= 30 cons: SEQUENCE
168:d=7 hl=2 l= 13 prim: UTCTIME :230609171437Z
183:d=7 hl=2 l= 13 prim: UTCTIME :240608171437Z
198:d=6 hl=2 l= 60 cons: SEQUENCE
200:d=7 hl=2 l= 23 cons: SET
202:d=8 hl=2 l= 21 cons: SEQUENCE
204:d=9 hl=2 l= 3 prim: OBJECT :commonName
209:d=9 hl=2 l= 14 prim: UTF8STRING :embeddedinn.ca
225:d=7 hl=2 l= 20 cons: SET
227:d=8 hl=2 l= 18 cons: SEQUENCE
229:d=9 hl=2 l= 3 prim: OBJECT :organizationName
234:d=9 hl=2 l= 11 prim: UTF8STRING :embeddedinn
247:d=7 hl=2 l= 11 cons: SET
249:d=8 hl=2 l= 9 cons: SEQUENCE
251:d=9 hl=2 l= 3 prim: OBJECT :countryName
256:d=9 hl=2 l= 2 prim: PRINTABLESTRING :IN
260:d=6 hl=2 l= 86 cons: SEQUENCE
262:d=7 hl=2 l= 16 cons: SEQUENCE
264:d=8 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
273:d=8 hl=2 l= 5 prim: OBJECT :secp256k1
280:d=7 hl=2 l= 66 prim: BIT STRING
348:d=6 hl=2 l= 83 cons: cont [ 3 ]
350:d=7 hl=2 l= 81 cons: SEQUENCE
352:d=8 hl=2 l= 29 cons: SEQUENCE
354:d=9 hl=2 l= 3 prim: OBJECT :X509v3 Subject Key Identifier
359:d=9 hl=2 l= 22 prim: OCTET STRING [HEX DUMP]:04142382890FDDDBBF313C80856D85320FFB674D8D75
383:d=8 hl=2 l= 31 cons: SEQUENCE
385:d=9 hl=2 l= 3 prim: OBJECT :X509v3 Authority Key Identifier
390:d=9 hl=2 l= 24 prim: OCTET STRING [HEX DUMP]:301680142382890FDDDBBF313C80856D85320FFB674D8D75
416:d=8 hl=2 l= 15 cons: SEQUENCE
418:d=9 hl=2 l= 3 prim: OBJECT :X509v3 Basic Constraints
423:d=9 hl=2 l= 1 prim: BOOLEAN :255
426:d=9 hl=2 l= 5 prim: OCTET STRING [HEX DUMP]:30030101FF
433:d=5 hl=2 l= 10 cons: SEQUENCE
435:d=6 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256
445:d=5 hl=2 l= 73 prim: BIT STRING
520:d=3 hl=4 l= 421 cons: SET
524:d=4 hl=4 l= 417 cons: SEQUENCE
528:d=5 hl=2 l= 1 prim: INTEGER :01
531:d=5 hl=2 l= 83 cons: SEQUENCE
533:d=6 hl=2 l= 60 cons: SEQUENCE
535:d=7 hl=2 l= 23 cons: SET
537:d=8 hl=2 l= 21 cons: SEQUENCE
539:d=9 hl=2 l= 3 prim: OBJECT :commonName
544:d=9 hl=2 l= 14 prim: UTF8STRING :embeddedinn.ca
560:d=7 hl=2 l= 20 cons: SET
562:d=8 hl=2 l= 18 cons: SEQUENCE
564:d=9 hl=2 l= 3 prim: OBJECT :organizationName
569:d=9 hl=2 l= 11 prim: UTF8STRING :embeddedinn
582:d=7 hl=2 l= 11 cons: SET
584:d=8 hl=2 l= 9 cons: SEQUENCE
586:d=9 hl=2 l= 3 prim: OBJECT :countryName
591:d=9 hl=2 l= 2 prim: PRINTABLESTRING :IN
595:d=6 hl=2 l= 19 prim: INTEGER :258BB8CE183F0F0AB4BD0D5119E69397ABFD72
616:d=5 hl=2 l= 11 cons: SEQUENCE
618:d=6 hl=2 l= 9 prim: OBJECT :sha256
629:d=5 hl=3 l= 228 cons: cont [ 0 ]
632:d=6 hl=2 l= 24 cons: SEQUENCE
634:d=7 hl=2 l= 9 prim: OBJECT :contentType
645:d=7 hl=2 l= 11 cons: SET
647:d=8 hl=2 l= 9 prim: OBJECT :pkcs7-data
658:d=6 hl=2 l= 28 cons: SEQUENCE
660:d=7 hl=2 l= 9 prim: OBJECT :signingTime
671:d=7 hl=2 l= 15 cons: SET
673:d=8 hl=2 l= 13 prim: UTCTIME :230609171500Z
688:d=6 hl=2 l= 47 cons: SEQUENCE
690:d=7 hl=2 l= 9 prim: OBJECT :messageDigest
701:d=7 hl=2 l= 34 cons: SET
703:d=8 hl=2 l= 32 prim: OCTET STRING [HEX DUMP]:A591A6D40BF420404A011733CFB7B190D62C65BF0BCDA32B57B277D9AD9F146E
737:d=6 hl=2 l= 121 cons: SEQUENCE
739:d=7 hl=2 l= 9 prim: OBJECT :S/MIME Capabilities
750:d=7 hl=2 l= 108 cons: SET
752:d=8 hl=2 l= 106 cons: SEQUENCE
754:d=9 hl=2 l= 11 cons: SEQUENCE
756:d=10 hl=2 l= 9 prim: OBJECT :aes-256-cbc
767:d=9 hl=2 l= 11 cons: SEQUENCE
769:d=10 hl=2 l= 9 prim: OBJECT :aes-192-cbc
780:d=9 hl=2 l= 11 cons: SEQUENCE
782:d=10 hl=2 l= 9 prim: OBJECT :aes-128-cbc
793:d=9 hl=2 l= 10 cons: SEQUENCE
795:d=10 hl=2 l= 8 prim: OBJECT :des-ede3-cbc
805:d=9 hl=2 l= 14 cons: SEQUENCE
807:d=10 hl=2 l= 8 prim: OBJECT :rc2-cbc
817:d=10 hl=2 l= 2 prim: INTEGER :80
821:d=9 hl=2 l= 13 cons: SEQUENCE
823:d=10 hl=2 l= 8 prim: OBJECT :rc2-cbc
833:d=10 hl=2 l= 1 prim: INTEGER :40
836:d=9 hl=2 l= 7 cons: SEQUENCE
838:d=10 hl=2 l= 5 prim: OBJECT :des-cbc
845:d=9 hl=2 l= 13 cons: SEQUENCE
847:d=10 hl=2 l= 8 prim: OBJECT :rc2-cbc
857:d=10 hl=2 l= 1 prim: INTEGER :28
860:d=5 hl=2 l= 10 cons: SEQUENCE
862:d=6 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256
872:d=5 hl=2 l= 71 prim: OCTET STRING [HEX DUMP]:304502207B4407CA32F558874B428B6FE214D213AF62DB5319F2B8551388FD96551B7830022100F8A1A13639D8FD346F10D2AEA42FD644C06B206B09F6D93FF80B6FBD2C495AA5
In this case, the pkcs7-data
is empty. However, if you use the -nodetach
flag when signing, the pkcs7-data
will contain the data that was signed.
The signer information is packaged along with the signature. The signer information includes the signer’s certificate, which contains the signer’s public key. The recipient can use the signer’s public key to verify the signature. The recipient can also use the signer’s certificate to verify the signer’s identity. typically, (unlike this example of a self signed certificate), the signer’s certificate is signed by a trusted certificate authority (CA), which means that the recipient can verify the signer’s identity using its root of trust.
To verify the signature using openssl, we can use the following command:
openssl cms -verify -inform PEM -in signedData.cms -content <(echo -n "Hello World") -CAfile cert.pem
Note that we are passing the CA certificate to the -CAfile
flag. This is because the signer’s certificate is self-signed, so we need to tell openssl to trust it. If the signer’s certificate was signed by a trusted CA, we would not need to pass the CA certificate to the -CAfile
flag.
Commands to generate and verify embedded CMS signatures are shown below:
$ echo -n "Hello World" | openssl cms -sign -signer cert.pem -inkey private.pem -outform PEM -out signedData.cms -nodetach
$ openssl cms -verify -inform PEM -in signedData.cms -CAfile cert.pem
GPG (GNU Privacy Guard) is a widely-used software tool that provides encryption and cryptographic functionality, including the ability to generate and verify digital signatures. GPG is an implementation of the OpenPGP (Pretty Good Privacy) standard, which is a widely-adopted protocol for secure communication and data encryption.
GPG uses the standard algorithms like RSA and ECDSA. But adds some custom layers on top of it. The overall steps to generate a digital signature are very similar and are as follows:
To create a digital signature, the owner of the private key uses GPG to sign a document or a message, generating a unique cryptographic hash of the data and encrypting it with the private key. This creates a digital signature that can be attached to the document.
To verify the digital signature, recipients of the signed document can use GPG and the signer’s public key. GPG performs the necessary cryptographic operations to validate the digital signature and confirm the integrity and authenticity of the document. If the verification process is successful, it means that the document has not been tampered with and was indeed signed by the owner of the corresponding private key.
GPG provides a command-line interface (CLI) as well as integration with various email clients and other software applications. It is widely used for secure communication, file encryption, and digital signatures in both personal and enterprise contexts.
While importing an existing PEM format key into GPG is possible, it need some amount of key manipulation and is outside the scope of this article. Instead, we will generate a new key pair using GPG. The following command will generate a new key pair:
$ gpg --full-generate-key --expert
The --expert
flag is used to enter the expert mode, which allows us to specify the key type and size.
I generated a P-256 ECC key pair. To sign a message, we can use the following command:
$ echo -n "Hello World" > content.txt
$ gpg --armor --detach-sign content.txt
This generates a detached signature file called content.txt.asc
. If the --armor
option is not passed, the signature will be binary.
-----BEGIN PGP SIGNATURE-----
iHUEABMIAB0WIQTp3iRUphvQZqakqnlJeX7QAKUCYQUCZIN6dQAKCRBJeX7QAKUC
YYQSAQCfF7eYS9q7pNCF9uuqC2Ns5xWEARA1dY6oDpSMfY9a7wEAr8F+J3lpNoLV
SuGs9Gz9M4flI+1sxMm2xHTOb6UwhEI=
=zC2Y
-----END PGP SIGNATURE-----
To verify the signature, we can use the following command:
$ gpg --verify content.txt.asc content.txt
gpg: Signature made Fri 09 Jun 2023 12:16:05 PM PDT
gpg: using ECDSA key E9DE2454A61BD066A6A4AA7949797ED000A50261
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Good signature from "vppillai <vppillai@embeddedinn.xyz>" [ultimate]
We can also use the gpg
command to encrypt and decrypt files. The following command will encrypt the file content.txt
using the public key of the recipient:
$ gpg --encrypt --recipient vppillai@embeddedinn.xyz --armor content.txt
The encrypted file will be called content.txt.asc
.
-----BEGIN PGP MESSAGE-----
hH4DdnSk57UrZfYSAgMEAPi09MyRCe5Dq1Cw4Mv9ZZErePUjVThoQ+WymHGocn8F
Es90lp0GVUSkG/7e3fjuUQDpcuDTH7PthurqYwhL8DAjAFFykwnsIrr+SoDtXZ3B
j2FJqeYz+bIPAn2azwq2z2TVSCIy6KxN7alOF2oLfTHSUgE8a6/YFh+nr8ye0WpG
xYGOVE0P+z9ZI5Tx1tXuYxWiTb15bUPXeWdnc5p2FewjT9b88L6Bn7Y05dwXb2gW
Jbg4Spq4jF2fM9YSmuGEWoEulz8=
=JrES
-----END PGP MESSAGE-----
To decrypt the file, we can use the following command. In a real-world scenario, the recipient would use their private key to decrypt the file. Contents of the file cannot be decrypted without the private key.
$ gpg --decrypt content.txt.asc
gpg: encrypted with 256-bit ECDH key, ID 7674A4E7B52B65F6, created 2023-06-09
"vppillai <vppillai@embeddedinn.xyz>"
Hello World
GPG key servers are used to share public keys. The following command can be used to upload the public key to a key server:
gpg --keyserver keyserver.ubuntu.com --send-keys E9DE2454A61BD066A6A4AA7949797ED000A50261
The following command can be used to download the public key from a key server:
gpg --keyserver keyserver.ubuntu.com --recv-keys E9DE2454A61BD066A6A4AA7949797ED000A50261
To delete a key from the key server, we can use the following command:
gpg --keyserver keyserver.ubuntu.com --delete-keys E9DE2454A61BD066A6A4AA7949797ED000A50261
We will now look at some of the common use cases of digital signatures.
X.509 is a standard format for public key certificates. It is used in many applications, including TLS/SSL, email, and code signing. X.509 certificates are used to authenticate the identity of the communicating parties and ensure the integrity of the exchanged messages.
When a certificate is requested from a certificate authority (CA), the CA verifies the identity of the requester and issues a certificate. The certificate contains the requester’s public key and is digitally signed by the CA. The CA’s signature is created using the CA’s private key. The certificate also contains the CA’s public key, which is used to verify the CA’s signature, along with the root of trust present in the client’s system.
you can see the signature algorithm used in the certificate and the signature using the command:
$ openssl x509 -in cert.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
25:8b:b8:ce:18:3f:0f:0a:b4:bd:0d:51:19:e6:93:97:ab:fd:72
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = embeddedinn.ca, O = embeddedinn, C = IN
Validity
Not Before: Jun 9 17:14:37 2023 GMT
Not After : Jun 8 17:14:37 2024 GMT
Subject: CN = embeddedinn.ca, O = embeddedinn, C = IN
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:89:ac:ee:13:35:c5:41:5a:8c:c9:f8:9b:22:ce:
99:8e:0a:29:f9:f5:d2:dc:6e:1f:1f:4b:15:9b:8f:
2a:01:ed:d9:ee:a4:e3:ef:7c:a0:cc:a7:37:73:5a:
47:8a:3b:0e:87:24:56:b3:1c:f0:89:73:54:fd:15:
b9:71:4d:61:a3
ASN1 OID: secp256k1
X509v3 extensions:
X509v3 Subject Key Identifier:
23:82:89:0F:DD:DB:BF:31:3C:80:85:6D:85:32:0F:FB:67:4D:8D:75
X509v3 Authority Key Identifier:
keyid:23:82:89:0F:DD:DB:BF:31:3C:80:85:6D:85:32:0F:FB:67:4D:8D:75
X509v3 Basic Constraints: critical
CA:TRUE
Signature Algorithm: ecdsa-with-SHA256
30:46:02:21:00:af:d2:93:36:10:fe:39:f1:88:9d:b5:0f:37:
a7:f1:db:8e:3c:41:f7:39:42:12:76:93:fb:69:cb:7c:99:11:
f1:02:21:00:d2:b5:63:cf:b7:d9:f6:74:66:69:5a:00:6d:08:
55:f7:cf:31:99:19:ef:c1:65:cc:69:80:55:26:13:29:84:4c
In the TLS 1.2 protocol, digital signatures are primarily used for authentication and integrity verification of data exchanged between the client and server. The key areas where digital signatures are used in TLS 1.2 are:
Handshake Protocol: to authenticate the identities of the communicating parties and ensure the integrity of the exchanged messages.
Server Authentication: When the server presents its digital certificate to the client during the handshake, it includes a digital signature created using the private key of the certificate authority (CA) that issued the certificate. The client verifies this signature using the corresponding CA’s public key to ensure the authenticity of the server’s certificate.
Client Authentication (optional): In case of mutual authentication, the server may request the client to present its digital certificate. Similar to server authentication, the client’s certificate includes a digital signature from a trusted CA, and the server verifies it to authenticate the client.
Key Exchange: The key exchange process in TLS 1.2 ensures that the client and server can establish a shared secret key to encrypt and decrypt data during the secure communication. Digital signatures are used to ensure the integrity of the key exchange process.
Server Key Exchange: In certain cases, the server may use a digital signature to sign the parameters used in the key exchange, ensuring that they have not been tampered with during transmission. An example of this is the Diffie-Hellman
key exchange method, where the server signs the Diffie-Hellman
parameters using its private key.
Client Key Exchange: The client may also employ digital signatures during the key exchange, depending on the key exchange method used. For example, in the Diffie-Hellman
key exchange method, the client signs the Diffie-Hellman
parameters using its private key.
Certificate Revocation: TLS 1.2 supports checking the revocation status of certificates to ensure they have not been revoked or invalidated. Digital signatures are utilized in the certificate revocation process, where revocation information is signed by the CA or certificate issuer.
When a client connects to an SSH server for the first time, the server presents its host key. The client verifies the authenticity of the host key by checking its digital signature, which is typically performed using a public key infrastructure (PKI) scheme. This process involves the following the client sends a challenge to the server, which signs the challenge using its private key and sends the digital signature back to the client. The client verifies the signature using the server’s public key to authenticate the server. This is defined in the SSH Transport Layer Protocol.
If an SSH, users initiates auth using public key cryptography. The challenge response process is similar to host key authentication, but the client and server roles are reversed. The client generates a key pair (public key and private key) and adds the public key to the server’s authorized keys file. When the user attempts to connect, the server sends a challenge to the client, which signs the challenge using its private key and sends the digital signature back to the server. The server verifies the signature using the user’s public key to authenticate the user.
SSH also supports certificate-based authentication. In this case, users have certificates issued by a trusted certificate authority (CA). The certificates contain public keys and digital signatures from the CA. When a user connects to the server, the client sends its certificate to the server. The server verifies the certificate’s digital signature with the CA’s public key to authenticate the user. This also includes the challenge response process to prove access to the private key.
The protocol commonly used for implementing digital signatures in emails is the Secure/Multipurpose Internet Mail Extensions (S/MIME). It operates within the protocol the framework of the MIME standard, which allows the inclusion of non-textual attachments, such as images or documents, in email messages.
S/MIME relies on X.509 digital certificates issued by trusted Certificate Authorities (CAs) and relies on PKI to verify the authenticity of the sender. The sender’s email client uses the private key of the sender’s digital certificate to sign the email message. The recipient’s email client uses the sender’s public key to verify the signature and authenticate the sender. The recipient’s email client also uses the sender’s public key to encrypt the email message, ensuring confidentiality.
The CMS format discussed earlier is used to store and transmit the signatures.
The sender and the receiver have to exchange their public key with each other. This process happens automatically when the sender and recipient exchange emails for the first time with most of the modern email clients.
GPG provides a similar functionality for email signing and encryption. It uses a web of trust model, where users can sign each other’s public keys to verify their authenticity. GPG also supports the use of X.509 certificates for email signing and encryption.
ARC (Authenticated Received Chain) and DKIM (DomainKeys Identified Mail) aim at protecting emails in transit:
ARC is a protocol designed to address the challenges of email authentication when messages pass through intermediaries, such as mailing lists or forwarding services. These intermediaries can modify the message or its headers, potentially breaking the existing email authentication mechanisms like SPF, DKIM, and DMARC.
ARC provides a mechanism to create a chain of authentication results as the email passes through these intermediaries. It adds ARC headers to the message, containing cryptographic seals and signatures that verify the integrity of the email’s original authentication results.
When an email recipient receives a message with ARC headers, they can use these headers to validate the authenticity and integrity of the authentication results. By examining the ARC-Seal and ARC-Message-Signature, the recipient can ensure that the email’s authentication results have not been tampered with during transit.
DKIM is an email authentication method that allows the receiver to verify the authenticity of an email’s domain and ensure that the message hasn’t been altered during transit.
When an email is sent using DKIM, the sending domain adds a digital signature to the email headers using asymmetric cryptography. The signature is generated by the sending domain’s private key, and the recipient can verify it using the domain’s public key published in DNS (Domain Name System) records.
The DKIM-Signature header contains information such as the signing algorithm, domain, selector, timestamp, and a hash value of selected headers. By recalculating the hash and verifying the signature with the public key, the recipient can confirm that the email has indeed been sent by the claimed domain and hasn’t been modified.
DKIM helps combat email spoofing and provides a level of trust in the authenticity of the email’s source. It is one of the components used by email providers to determine the email’s reputation and whether it should be delivered to the recipient’s inbox or treated as suspicious or spam.
Both ARC and DKIM enhance email authentication and integrity. While DKIM focuses on verifying the domain and message integrity, ARC focuses on preserving the authentication results as the email passes through intermediaries. Together, they contribute to a more secure email ecosystem by preventing spoofing, tampering, and enhancing trust in email communication.
Git can be configured to digital signatures to verify the authenticity of commits and tags. Git uses the GPG tool to generate and manage the keys. The following commands can be used to configure Git to use GPG:
$ git config user.signingkey <key-id>
The GPG format can be optionally be set to ssh
or x509
using the following commands:
$ git config gpg.format ssh
$ git config gpg.format x509
To sign commits and tags, use the -S
option with the git commit
and git tag
commands:
$ git commit -S -m "commit message"
$ git tag -s <tag-name>
A signed commit or tag can be verified using the git verify-commit
and git verify-tag
commands:
$ git verify-commit <commit-id>
$ git verify-tag <tag-name>
And, a signed commit can be viewed using the git show
command:
$ git show <commit-id>
A sample output of the git show
command is shown below:
$ git show v1.0.0
tag v1.0.0
Tagger: vppillai <vppillai@embeddedinn.ca>
Date: Fri June 9 20:29:41 2023 -0700
Signed tag
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
dSWsBAADAgAGBQJTZbQlAAoJEF0+sviABDDrZbQH/09PfE51KPVPlanr6q1v4/Ut
LQxAojUWiLQdg2ESJItkcuweYg+kc3HCyFejeDIBw9dpXt00rY26p05qrpnG+85b
hM1/PswpHLuBSr+oCIDj5GMC2r2iEKsfv2fJbNW8iWAXVLoWZRF8B0MfqX/YTMbm
ecorc4iXzQu7tupRihslbNkfvfciMnSDeSvzCpWAHl7h8Wj6hhqePmLm9lAYqnKp
8S5B/1SSQuEAjRZgI4IexpZoeKGVDptPHxLLS38fozsyi0QyDyzEgJxcJQVMXxVi
RUysgqjcpT8+iQM1PblGfHR4XAhuOqN5Fx06PSaFZhqvWFezJ28/CLyX5q+oIVk=
=EFTF
-----END PGP SIGNATURE-----
commit e3cb51b3086661a2876825062c3057d0e851a444
Author: vppillai <vppillai@embeddedinn.ca>
Date: Fri June 9 20:29:41 2023 -0700
As you can see, the signature is stored as part of the commit object.
While source signing as we saw in the Git section is technically a form of code signing, the term code signing is typically used to refer to the signing of compiled code. Code signing is used to verify the authenticity of software and to ensure that the software has not been tampered with. Code signing is commonly used for software distribution, such as software updates, and for mobile apps. Code signing is also used for driver signing in Windows and for kernel modules in Linux.
There is no standard way to store a signature as part of a binary like format like ELF. The signature is typically stored in a separate file, which is then distributed along with the binary. The signature file can be in any format, such as CMS, PGP, or XML. The signature file can also be stored in a separate location, such as a database, and referenced by the binary.
The Linux kernel handles signed modules differently. While the details are available in the Linux documentation, the highlights are outlined below.
CONFIG_MODULE_SIG
configuration option to y
in the kernel configuration file. The certifiate used for verification is stored in the certs
directory in the kernel source tree./proc/keys
scripts/sign-file
tool is used to generate and append the signature to the module. A signed module has a digital signature simply appended at the end. The string ~Module signature appended~
. at the end of the module’s file confirms that a signature is present.Module Signature Block
in the module consists of the following components: struct module_signature {
uint8_t algo; /* Public-key crypto algorithm [0] */
uint8_t hash; /* Digest algorithm [0] */
uint8_t id_type; /* Key identifier type [PKEY_ID_PKCS7] */
uint8_t signer_len; /* Length of signer's name [0] */
uint8_t key_id_len; /* Length of key identifier [0] */
uint8_t __pad[3];
uint32_t sig_len; /* Length of signature data */
};
USE_PKCS7
option is enabled by default. If disabled, a CMS format signature is generated.~Module signature appended~
string is present , the kernel will then read the signature header and signature from the end of the module and validate it. The offset of the signature header is calculated by subtracting the size of the signature header from the size of the module. The signature header is then read from the module and validated. If the signature header is valid, the signature is then read and validated. If the signature is valid, the module is loaded. If the signature is invalid, the module is not loaded and an error message is displayed.JSON Web Token (JWT) is an open standard for securely transmitting information between parties as a JSON object. The information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA. You can read more about JWTs in detail in my previous article on the topic.
By system engineering, I refer to the low level software that is responsible for booting the system and loading the operating system. secure system boot and secure software update are two important aspects of system engineering. Secure boot ensures that the system boots only authorized software. Secure software update ensures that the software is not tampered with during the update process. Image signing is an important aspect of both secure boot and secure software update.
The topics in the following sub sections require a detailed explanation. I will cover them in detail in future posts. For now, I will provide a brief overview of each topic.
UEFI stands for Unified Extensible Firmware Interface. UEFI is a specification that defines a software interface between an operating system and platform firmware. UEFI is a replacement for the legacy BIOS firmware interface. UEFI is a very broad topic that requires its own post. So, in order to avoid mis-representation or over simplification of data, I will just state that when Secure Boot is enabled, the UEFI firmware checks the digital signatures of the bootloader and any subsequent components before allowing them to execute. I will later write a detailed post on UEFI and Secure Boot, going into the details of the signature header, signing and key exchange process etc.
Das U-Boot (Universal Bootloader) is an open source, primary boot loader used in embedded devices to boot the Linux kernel, among other things. U-Boot provides a mechanism known as Verified boot to verify the integrity of the boot image. The boot image is a FIT image that contains the kernel, device tree, and other components. The boot image is signed using a private key. The public key is stored in the U-Boot environment. The signature is verified by U-Boot before booting the image.
U-Boot Flattened Image Tree (FIT) is a very flexible, structured container format which supports multiple kernel images, device trees, ram disks, etc. It includes hashes to verify images. So adding signatures to it is straight forward.
U-Boot can be configured to use a variety of signature formats, such as PKCS7, CMS, and raw signatures. The signature is stored in the FIT image itself. The signature is verified by U-Boot before booting the image. U-Boot can also be configured to use a TPM to store the public key. The TPM can be used to verify the signature.
GRUB (GRand Unified Bootloader) is a popular bootloader used in many Linux-based operating systems. Starting with version 2.02, GRUB introduced support for digital signatures to ensure the authenticity and integrity of its configuration files and executable modules.
The digital signature support in GRUB involves the following key components:
GRUB Configuration File (grub.cfg): The main configuration file for GRUB, which specifies the boot options and configuration settings. The configuration file can be signed using digital signatures to prevent unauthorized modifications.
Executable Modules: GRUB can load and execute various modules during the boot process, such as filesystem drivers or cryptographic libraries. These modules can also be signed to ensure their integrity and authenticity.
Signature Verification: During the boot process, GRUB verifies the digital signatures of its configuration file and executable modules to ensure they haven’t been tampered with or modified by unauthorized sources. The signatures are verified using the corresponding public keys.
The specific signature format used by GRUB is based on the GNU gpg (GNU Privacy Guard) format. GRUB utilizes the GPG toolchain, including GPG keys and the gpg utility, for creating and verifying the digital signatures.
The process generally involves signing the GRUB configuration file and modules using a private key, and the corresponding public key is embedded within the GRUB bootloader or stored in a secure location accessible by GRUB. During the boot process, GRUB verifies the digital signatures using the embedded or trusted public key.
The exact steps and configuration options for setting up and managing digital signatures in GRUB can vary depending on the specific version and distribution of GRUB being used. It’s important to consult the GRUB documentation and distribution-specific guides for detailed instructions on generating keys, signing files, and configuring GRUB to perform signature verification.
Android Verified Boot (AVB) is a security feature implemented in Android devices to ensure the integrity and authenticity of the boot process. AVB utilizes digital signatures to verify the integrity of boot components, including the bootloader, kernel, and device tree.
Here’s an overview of how digital signatures are used in Android Verified Boot:
Boot Image Signatures: Android devices use a verified boot process where each boot image (bootloader, kernel, and device tree) is digitally signed. The digital signature is generated using an asymmetric cryptography algorithm, such as RSA, and is based on a private key held by the device manufacturer or trusted entity.
Signature Verification: During the boot process, the bootloader verifies the digital signatures of the boot images using the corresponding public key. If the signatures are valid and match the computed hash of the boot images, the boot process continues. If the signatures are invalid or the hashes do not match, the bootloader will not proceed, indicating a potential tampering or compromise.
Chain of Trust: AVB establishes a chain of trust by ensuring that each subsequent boot component is verified using its own digital signature. This means that the bootloader verifies the kernel, and the kernel verifies the device tree, ensuring the integrity of each stage.
Key Management: The public key used for verifying the boot image signatures is stored in the device’s trusted hardware, such as a Trusted Execution Environment (TEE), or in a read-only memory location that is tamper-resistant. This ensures that only authorized parties can update the boot components and corresponding digital signatures.
Updates and Rollback Protection: When updating the firmware or boot images, AVB enforces version checks and prevents downgrades to older, potentially vulnerable versions. It ensures that the updated boot images are signed with a newer version of the digital signature.
The exact implementation of Android Verified Boot, including the signature formats and key management mechanisms, can vary across different device manufacturers and Android versions. However, the underlying principle of using digital signatures to verify the integrity and authenticity of boot components remains consistent.
Linux IMA (Integrity Measurement Architecture) and EVM (Extended Verification Module) are security features in the Linux kernel that provide mechanisms for measuring and verifying the integrity of files and data on a Linux system. Here’s a brief overview of IMA and EVM:
Together, IMA and EVM provide a framework for enforcing and verifying the integrity and authenticity of files and data on a Linux system. They are often used in security-sensitive environments where ensuring the integrity of system components and detecting unauthorized changes is crucial.
It’s important to note that the specific implementation and configuration of IMA and EVM may vary across Linux distributions and kernel versions. Detailed documentation and configuration options can be found in the Linux kernel source tree and the documentation provided by the specific distribution or security frameworks built on top of IMA and EVM.
There are so many more places and ways to use digital signatures. This is just a small sample of the many ways digital signatures are used in the real world. Hopefully this gives you a better understanding of how digital signatures are used in practice. Its important to note that the use of digital signatures in any system is only as good as the system design that can store Signer Public Keys in a tamper resistant way. If the Signer Public Keys can be tampered with, then the entire system is compromised.
]]>Transferring files between your host and RISC-V QEMU machine is often a daunting task. typical solutions include using the 9P FS which requires host kernel level modifications, which might not always be practical. In the following sections,we will build a RISC-V Linux image using Buildroot and then boot it using QEMU. We will then configure the Linux image to use sshfs to mount the host filesystem. This will allow us to seamlessly transfer files between the host and the RISC-V QEMU machine.
I am using a clean Windows machine with WSL2 Ubuntu 22.04
as my development environment. Install basic packages with:
sudo apt update && sudo apt -y upgrade
sudo apt install -y build-essential unzip bc libncurses-dev openssh-server
SSH server will not be started by default since WSL does not bring up systemd
. I took the easy route and enabled systemd on the Ubuntu image with the following line in /etc/wsl.conf
:
[boot]
systemd=true
WSL $PATH
includes spaces and special characters that are not compatible with Buildroot and Kconfig. To fix this, we need to add the following line to /etc/wsl.conf
:
[interop]
appendWindowsPath=false
Alternatively, you can also add the following line to your .bashrc
, or run it in your terminal in a new bash session:
export PATH=$(echo $PATH | tr -d '()[:space:]')
Restart WSL by issuing the following command in a PowerShell terminal:
wsl --shutdown
2023.02
for this tutorial. Extract the archive and navigate to the extracted directory. wget https://buildroot.org/downloads/buildroot-2023.02.tar.gz
tar xvf buildroot-2023.02.tar.gz
cd buildroot-2023.02
make qemu_riscv64_virt_defconfig
sshfs
in Buildroot with menuconfig. make menuconfig
Navigate to Target packages -> Filesystem and flash utilities -> sshfs (FUSE)
and enable it. Save the config and exit.
sshfs
requires FUSE
(Filesystem in Userspace) support in the kernel to operate. To enable this, launch the Buildroot Linux configuration menu with: make linux-menuconfig
Navigate to File systems -> FUSE (Filesystem in Userspace) support
and enable it to be built as an inbuilt module (*
instead of M
in menuconfig). Save the config and exit.
make
This will take a while to complete. Once the build is complete, you will find the RISC-V Linux kernel image, RFS and QEMU binaries in output/images/
.
To launch the newly compiled system, execute the output/images/start-qemu.sh
script. This will launch QEMU with the RISC-V Linux kernel image and RFS.
The default username is root
without a password.
Issue the following command in the QEMU shell to mount a host directory with sshfs. Make sure that the ssh server is up and running on the host machine.
sshfs -o allow_other,default_permissions <username>@10.0.2.2:<host path> /mnt
Contents of the host directory will now be available in the QEMU shell at /mnt
.
10.0.2.2
is the default IP address of the host machine in the QEMU network. The Buildroot generated startup script sets up the SLIRP network for QEMU makingthe host accessible over this “special” IP address. You can read more about this here.
In this tutorial, we built a RISC-V Linux image using Buildroot and then booted it using QEMU. We then configured the Linux image to use sshfs to mount the host filesystem. This allowed us to seamlessly transfer files between the host and the RISC-V QEMU machine.
]]>Get ready to dive into the intricacies of Linux boot on a RISC-V machine! This comprehensive guide will walk you through the process of compiling QEMU, the Linux kernel, and the root filesystem from scratch. By the end of this journey, you’ll have a deep understanding of the Linux boot flow and be equipped with the knowledge to write your own Linux bootloaders for RISC-V. So buckle up, grab your coffee, and let’s get started!
In previous blog posts, I’ve gone into the details of compiling QEMU, Linux and bare-metal programs. Here, we will rely on Buildroot to build all the components for us in one shot.
We start with downloading the latest buildroot release from here. I’m using buildroot-2022.11.1
for this post. Extract the tarball and navigate to the buildroot directory.
wget https://buildroot.org/downloads/buildroot-2022.11.1.tar.gz
tar -xvf buildroot-2022.11.1.tar.gz
cd buildroot-2022.11.1
To build the images for RISC-V, we will use the qemu_riscv64_virt_defconfig
configuration file. This would compile QEMU, the Linux kernel and the root filesystem for us. We can also use the make menuconfig
command to customize the build. For this post, we will use the default configuration.
make qemu_riscv64_virt_defconfig
make -j12
At the end of the build, the images will be available in the output/images
directory. We will use the rootfs.ext2
file as the root filesystem for our QEMU machine. A launch script start-qemu.sh
will also be generated in the output/images
directory. We will use this script to launch the QEMU machine.
cd output/images
./start-qemu.sh
OpenSBI v0.9
____ _____ ____ _____
/ __ \ / ____| _ \_ _|
| | | |_ __ ___ _ __ | (___ | |_) || |
| | | | '_ \ / _ \ '_ \ \___ \| _ < | |
| |__| | |_) | __/ | | |____) | |_) || |_
\____/| .__/ \___|_| |_|_____/|____/_____|
| |
|_|
Platform Name : riscv-virtio,qemu
Platform Features : timer,mfdeleg
Platform HART Count : 1
Firmware Base : 0x80000000
Firmware Size : 124 KB
Runtime SBI Version : 0.2
Domain0 Name : root
Domain0 Boot HART : 0
Domain0 HARTs : 0*
Domain0 Region00 : 0x0000000080000000-0x000000008001ffff ()
Domain0 Region01 : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address : 0x0000000080200000
Domain0 Next Arg1 : 0x0000000082200000
Domain0 Next Mode : S-mode
Domain0 SysReset : yes
Boot HART ID : 0
Boot HART Domain : root
Boot HART ISA : rv64imafdcsuh
Boot HART Features : scounteren,mcounteren,time
Boot HART PMP Count : 16
Boot HART PMP Granularity : 4
Boot HART PMP Address Bits: 54
Boot HART MHPM Count : 0
Boot HART MHPM Count : 0
Boot HART MIDELEG : 0x0000000000001666
Boot HART MEDELEG : 0x0000000000f0b509
[ 0.000000] Linux version 5.15.43 (vppillai@BBY-LT-C16658) (riscv64-linux-gcc.br_real (Buildroot 2022.11.1) 11.3.0, GNU ld (GNU Binutils) 2.38) #1 SMP Fri Feb 10 11:58:06 PST 2023
[ 0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[ 0.000000] Machine model: riscv-virtio,qemu
[ 0.000000] efi: UEFI not found.
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000080200000-0x0000000087ffffff]
.
.
.
.
Starting network: udhcpc: started, v1.35.0
udhcpc: broadcasting discover
udhcpc: broadcasting select for 10.0.2.15, server 10.0.2.2
udhcpc: lease of 10.0.2.15 obtained from 10.0.2.2, lease time 86400
deleting routers
adding dns 10.0.2.3
OK
Welcome to Buildroot
buildroot login:
In the boot logs, you would see OpenSBI being loaded and then the Linux kernel being loaded. The kernel would then mount the root filesystem and start the init process. You can login to the machine using the username root
with an empty password.
To exit , press Ctrl+A
and then X
.
Now, that we have a working QEMU machine, let’s dive into the boot process.
The start-qemu.sh
script is generated by the buildroot build process. Let’s take a look at the script.
#!/bin/sh
(
BINARIES_DIR="${0%/*}/"
cd ${BINARIES_DIR}
if [ "${1}" = "serial-only" ]; then
EXTRA_ARGS='-nographic'
else
EXTRA_ARGS=''
fi
export PATH="/home/vppillai/temp/buildroot-2022.11.1/output/host/bin:${PATH}"
exec qemu-system-riscv64 -M virt -bios fw_jump.elf \
-kernel Image -append "rootwait root=/dev/vda ro" \
-drive file=rootfs.ext2,format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-netdev user,id=net0 -device virtio-net-device,netdev=net0 \
-nographic ${EXTRA_ARGS}
We are mainly interested in the exec
command. This is the command that launches the QEMU machine. Let’s do a quick break down the command in the table below. We will go into the details of each option later.
Command | Description |
---|---|
qemu-system-riscv64 |
The QEMU binary to launch. |
-M virt |
The machine type to emulate. We are using the virt machine, a configurable machine model with the basic peripherals for a working machine. |
-bios fw_jump.elf |
The OpenSBI firmware to load. This is the first piece of “user code” that executes on QEMU and is responsible for bootloading Linux. Its built by buildroot |
-kernel Image |
The Linux kernel to load. This is the second “user code” piece that executes on QEMU. It’s built by buildroot |
-append "rootwait root=/dev/vda ro" |
Additional kernel command line argument to use the disk mounted onto /dev/vda as the root file system after kernel init. |
-drive file=rootfs.ext2,format=raw,id=hd0 |
Frontend option to mount a disk image. |
-device virtio-blk-device,drive=hd0 |
Backend option to mount the disk image. We are using virtio-blk-device through the virtio_mmio interface to expose the disk image in the host device into the qemu machine |
-netdev user,id=net0 |
Frontend option to configure the network interface. user initialize the SLiRP network between the host and guest |
-device virtio-net-device,netdev=net0 |
Backend option to configure the network interface. We are using virtio-net-device through the virtio_mmio interface to expose the network interface in the host device into the qemu machine. |
-nographic |
Disable the graphical interface and use the serial console. |
The source code of QEMU used for the buildroot build is located at buildroot-2022.11.1/output/build/host-qemu-7.1.0
. The source code for the virt
machine model is located at buildroot-2022.11.1/output/build/host-qemu-7.1.0/hw/riscv/virt.c
. The corresponding online version is at https://github.com/qemu/qemu/blob/v7.1.0/hw/riscv/virt.c
. Let’s take a look at the source code.
We will touch upon the portions relevant to the boot process. We’ll do a deep dive into creating a custom machine model in a future post.
The default reset vector of the emulated CPUs is defined with DEFAULT_RSTVEC
as 0x1000
in target/riscv/cpu_bits.h:579
. Its set as the default using the resetvec
property if the CPU in target/riscv/cpu.c:951
. It can be altered while creating the machine CPUs using the qdev_prop_set_uint64()
API to set resetvec
property of the CPU. The virt
machine leaves to the default value.
The value of resetvec
is consumed by riscv_cpu_reset()
to set the CPUs PC value at reset, which is in turn used by cpu_loop()
.
In the virt machine, a VIRT_MROM
is emulated at the reset vector. The following section looks at how QEMU populates this region before passing control to the CPU to execute code at the reset vector.
The boot setup of the virt machine starts with a call to the riscv_setup_rom_reset_vec() function. But, before calling this function, a couple of steps are done by QEMU.
The firmware
passed with the -bios
option (OpenSBI in our case) will be parsed and loaded using riscv_find_and_load_firmware()
. If the firmware is not provided, it will load the most relevant one from a set of images built/packaged with QEMU. The buildroot build this can be found at buildroot-2022.11.1/output/host/share/qemu
. In the case of the virt
machine, this is loaded to the VIRT_FLASH
device at 0x20000000
.
Next, the kernel passed with the -kernel
option (Linux in our case) will be parsed and loaded using riscv_load_kernel()
. The kernel must be loaded at a 2MiB
aligned address for a 64-bit machine. The following aligned address after the firmware is used to load the kernel is calculated using riscv_calc_kernel_start_addr()
.
initrd
image is passed with the -initrd
option, it will be parsed and loaded using riscv_load_initrd()
. The initrd image is loaded at an aligned offset kernel in the RAM without overlap with the other entries populated so far. Initrd is typically a compressed CPIO archive and contains minimal files required to boot the kernel. The initrd can also be packaged into the kernel image itself. In our case, the initrd image is not used. Instead, we use the root file system image directly along with the DTS entry to mount the root file system.
-append "rootwait root=/dev/vda ro"
option we passed to QEMU. The rootwait
option tells the kernel to wait for the root file system to be mounted before proceeding with the boot process. The root=/dev/vda
option tells the kernel to mount the root file system on the disk mounted on /dev/vda
. The ro
option tells the kernel to mount the root file system in read-only mode.-dtb
option. If no FDT file is passed, QEMU will generate an FDT based on the machine model. In our case, we are using the default FDT generated by QEMU. Before loading the FDT, the initrd
address and any arguments passed with the -append
option is added to the FDT. The following comment in the QEMU source code summarizes the FDT load requirements:
We should put fdt as far as possible to avoid kernel/initrd overwriting its content. But it should be addressable by 32-bit system as well. Thus, put it at a 2MB aligned address that is less than the fdt size from the end of a dram or 3GB, whichever is lesser.
ZSBL refers to the first piece of code executed by the CPU at reset. In the case of the virt
machine, this is the VIRT_MROM
region. The VIRT_MROM
region is populated by the riscv_setup_rom_reset_vec()
function. In the case of Linux boot, the “firmware” used to boot Linux is OpenSBI. OpenSBI is a RISC-V SBI (Supervisor Binary Interface) implementation. The SBI is a standard interface between the OS and the firmware. The SBI is used to initialize the hardware and provide OS services. Therefore it can also act as the “second stage bootloader” that can boot Linux. Optionally, OpenSBI can launch a more complex bootloader like U-Boot. In our case, we are using OpenSBI as the launcher for Linux.
OpenSBI provides options like fw_jump
, fw_dynamic
, and fw_payload
for the ZSBL to pass the information about the next stage it has to boot. In the case of booting Linux, this includes where the Kernel and DTS files are loaded. OpenSBI also consumes the DTS file and updates it in-memory before passing it on to the kernel if required.
QEMU uses the fw_dynamic
mechanism by default. This involves populating a struct fw_dynamic_info
with the information required to boot the next stage and passing its address in the a2
register of the RISC-V CPU. The address must be aligned to 8 bytes on RV64 and 4 bytes on RV32. Code comments in the excerpt below from the OpenSBI source code summarize the requirements for the struct fw_dynamic_info
:
struct fw_dynamic_info {
/* Expected value of info magic ('OSBI' ASCII string in hex) */
/* #define FW_DYNAMIC_INFO_MAGIC_VALUE 0x4942534f*/
unsigned long magic;
/** Info version. The current latest is 0x2*/
unsigned long version;
/** Next booting stage address */
unsigned long next_addr;
/** Next booting stage mode */
/* FW_DYNAMIC_INFO_NEXT_MODE_S = 0x1 */
unsigned long next_mode;
/** Options for OpenSBI library */
unsigned long options;
/**
* Preferred boot HART id
*
* It is possible that the previous booting stage used the same link
* address as the FW_DYNAMIC firmware. In this case, the relocation
* lottery mechanism can potentially overwrite the previous booting
* stage while other HARTs are still running in the previous booting
* stage leading to a boot-time crash. To avoid this boot-time crash,
* the previous booting stage can specify the last HART that will jump
* to the FW_DYNAMIC firmware as the preferred boot HART.
*
* To avoid specifying a preferred boot HART, the previous booting
* stage can set it to -1UL which will force the FW_DYNAMIC firmware
* to use the relocation lottery mechanism.
*/
unsigned long boot_hart;
}
For Linux boot, next_mode
is set to S
to select a CPU that supports S
mode to execute the Kernel. OpenSBI expects the fdt addr passed via the a1
register. It can also be built into OpenSBI in the case of the fw
or fw_paylaod
modes.
At boot, the RISC-V Linux kernel expects a0
to contain a unique per-hart ID and a1
to contain a pointer to the flattened device tree. OpenSBI ensures this.
We will not go into the details of the OpenSBI source code in this post.
The riscv_setup_rom_reset_vec()
function in QEMU sets up the fw_dynamic_info
.
QEMU generates a ZSBL with all the dynamic information based on user input to perform the steps described in the reset_vec[]
array and loads it into the reset vector.
The entire process can be visualized in the following animation:
]]>As an embedded system developer, you would be very familiar with the use of a debugger to step through code or using UART-based printf()
statements to print to the console. However, what if you want to print to the console or perform file I/O operations from your embedded system? Semihosting lets you do just that. Semihosting lets embedded systems talk to a host computer through a debugger interface to perform tasks such as file I/O, and printing to the console. This article explains how semihosting is implemented on RISC-V and we get hands-on using a RSIC-V QEMU model and some bare metal code.
Semihosting is a mechanism that allows a debugger to communicate with a target system. It uses a clever combination of code running on the target system and a debugger to perform tasks such as file I/O, and printing to the console. Semihosting was first defined by ARM in 1995 and is available today as the ARM Semihosting Specification. It is implemented by many debuggers and is supported in many libraries.
Essentially, semihosting depends on placing some information on a debugger-readable memory location and then executing a breakpoint instruction. The debugger can then read the information and perform the requested task. The target system can then continue execution after the breakpoint instruction. If the debugger wants to provide return information back to the target (like data read from a file), it can do so by writing to the same memory location. When continuing execution, the target system can read the return information from the memory location.
Semihosting is implemented on RISC-V by using the ebreak
instruction. The ebreak
instruction is a breakpoint instruction that is used to halt the processor. The RISC-V spec uses a cleaver trick of “wrapping” the ebreak
instruction with additional instructions that helps the debugger distinguish a “semihosting ebreak
” from a “regular ebreak
”.
Specifically, the following piece of code is used to implement semihosting on RISC-V:
slli x0, x0, 0x1f # Entry NOP
ebreak # Break to debugger
srai x0, x0, 7 # NOP encoding the semihosting call number 7
The data exchange format uses the same ARM semihosting specification.
This is what the RISC-V spec says about semihosting (from Sec 2.8
of Volume I: RISC-V Unprivileged ISA V20191213
):
EBREAK
was primarily designed to be used by a debugger to cause execution to stop and fall back into the debugger.EBREAK
is also used by the standard gcc compiler to mark code paths that should not be executed.<p> Another use ofEBREAK
is to support“semihosting”
, where the execution environment includes a debugger that can provide services over an alternate system call interface built around the EBREAK instruction. Because the RISC-V base ISA does not provide more than oneEBREAK
instruction, RISC-V semihosting uses a special sequence of instructions to distinguish a semihostingEBREAK
from a debugger insertedEBREAK
.<p>slli x0, x0, 0x1f # Entry NOP ebreak # Break to debugger srai x0, x0, 7 # NOP encoding the semihosting call number 7
Note that these three instructions must be 32-bit-wide instructions, i.e., they mustn’t be among the compressed 16-bit instructions described in Chapter 16.<p> The shift NOP instructions are still considered available for use as HINTS.<p> Semihosting is a form of service call and would be more naturally encoded as an
ECALL
using an existing ABI, but this would require the debugger to be able to interceptECALLs
, which is a newer addition to the debug standard. We intend to move over to usingECALLs
with a standard ABI, in which case, semihosting can share a service ABI with an existing standard.<p> We note that ARM processors have also moved to usingSVC
instead ofBKPT
for semihosting calls in newer designs.
The spec defines 24 semihosting calls. The following table shows the semihosting calls and their description:
Category | Semihosting calls | Description |
---|---|---|
File I/O | SYS_OPEN , SYS_WRITE , SYS_CLOSE , SYS_FLEN , SYS_SEEK , SYS_ISTTY , SYS_READ , SYS_REMOVE , SYS_RENAME , SYS_TMPNAM |
Operate on files in the host file system. |
Console I/O | SYS_WRITEC , SYS_WRITE0 , SYS_READC |
Read/Write on the console. The console is implemented by the debugger. For QEMU, it can be routed to STDIO, GDB uses its own console etc. |
Status | SYS_ISERROR , SYS_HEAPINFO , SYS_ERRNO |
Get information about the status of the target system. |
control status | SYS_EXIT , SYS_EXIT_EXTENDED , SYS_GET_CMDLINE |
Control the target system. |
system commands | SYS_SYSTEM |
Execute a system command on the host. |
clock | SYS_CLOCK , SYS_TIME , SYS_ELAPSED , SYS_TICKFREQ |
Get information about the clock on the target system. expect skews since communication is over JTAG |
Each semihosting call is identified by a number. This number will be placed in the a0
register before the semihosting ebreak sequence is performed. If the operation requires arguments, they will be placed in the system memory and the address will be placed in the a1
register. The debugger will place the return value in the a0
register before “un-halting” the CPU from the EBREAK
instruction.
The following diagram shows the semihosting sequence. On the left is the target system and on the right is the debugger:
Lets see semihosting in action
As usual, lets start with a clean ubuntu machine. Install basic tools with the following command:
sudo apt-get install -y build-essential git wget
Then we install the RISC-V elf toolchain (a.k.a the baremetal toolchain). We will use the pre-built toolchain packaged for Ubuntu and install it with the following commands:
sudo apt-get install -y gcc-riscv64-unknown-elf
Just the compiler is not enough to compile baremetal programs. We need a linker script and a startup file along with a library which provides the standard C library functions. You can install these on ubuntu with the following commands. Since our intent is to understand semihosting and not building the baremetal infrastructure, we will use the pre-built libraries and linker scripts provided by the picolibc project. The picolibc
project also has native semihosting support. However, we will not be using it in this tutorial. We will be building our own target semhosting infrastructure.
Install picolibc
with the following command:
sudo apt-get install -y picolibc-riscv64-unknown-elf
In ubuntu, the files are installed under /usr/lib/picolibc/riscv64-unknown-elf/
. We will be using this path in teh next section
(You can find this for any package with dpkg -L <package name>
)
You can compile C code to generate statically linked images for RSICV using this command
riscv64-unknown-elf-gcc -specs=/usr/lib/picolibc/riscv64-unknown-elf/picolibc.specs main.c
However, this generates images targeted for execution from the RAM entry point address 0x10000000
. This might not work out for all machines, especially the RISCV Virt machine that we plan to use.
You can check the ELF header of the generated image with :
riscv64-unknown-elf-readelf a.out -h
In the case using the default spec, you would see
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: RISC-V
Version: 0x1
Entry point address: 0x10000000
Start of program headers: 64 (bytes into file)
Start of section headers: 9224 (bytes into file)
Flags: 0x5, RVC, double-float ABI
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 4
Size of section headers: 64 (bytes)
Number of section headers: 19
Section header string table index: 18
The QEMU RSICV virt model RAM (and execution) starts at 0x80000000
. You can check this in the hw/riscv/virt.c file.
So, in order to re-target the binary, we need to create a linker script with the following contents.
__flash = 0x80000000;
__flash_size = 0x00080000;
__ram = 0x80080000;
__ram_size = 0x40000;
__stack_size = 1k;
INCLUDE picolibc.ld
Now we can compile images that would execute from the required location by passing this linker script using the -T
argument. A sample is given below:
riscv64-unknown-elf-gcc -specs=/usr/lib/picolibc/riscv64-unknown-elf/picolibc.specs main.c -Tvirt.ld
The new ELF header would look like this: (Note the change in the Entry point address
)
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: RISC-V
Version: 0x1
Entry point address: 0x80000000
Start of program headers: 64 (bytes into file)
Start of section headers: 9392 (bytes into file)
Flags: 0x5, RVC, double-float ABI
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 4
Size of section headers: 64 (bytes)
Number of section headers: 19
Section header string table index: 18
For running semihosting on RISC-V with usermode support, we would need QEMU version 7.2 or later Ref. You get the latest source from the QEMU git repository.
To compile from source, you can use the following commands:
First get the build dependencies:
sudo sed -i 's/# deb-src/deb-src/' /etc/apt/sources.list
sudo apt update
sudo apt -y build-dep qemu
Then clone, checkout and build QEMU:
git clone https://git.qemu.org/git/qemu.git
cd qemu
git checkout v7.2.0
./configure --target-list=riscv64-softmmu
make -j8
QEMU will be built in the riscv64-softmmu
directory. You can run the following command to check if QEMU is working:
./build/qemu-system-riscv64 --version
Export the path to the QEMU binary to the PATH
variable:
export PATH=$(pwd)/build:$PATH
Now that we have the toolchain, lets write a simple C program to implement semihosting. We will implement a semihosting_write
function that will write a string to the console and a semihosting_open
, semihosting_write
and semihosting_close
functions that will write a string to a file.
#include <stdio.h>
#include <string.h>
#define SH_FILE_MODE_WPLUS 6
#define APP_SH_TEST_STR_LENGTH 25
enum semihosting_operation_numbers
{
SH_SYS_OPEN = 0x01,
SH_SYS_CLOSE = 0x02,
SH_SYS_WRITE0 = 0x04,
SH_SYS_WRITE = 0x05,
};
/*REF: https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/n-5VQ9PHZ4w/m/KbzH5t9MBgAJ */
static inline int __attribute__((always_inline)) sys_sh(int reason, void *argPack)
{
register int value asm("a0") = reason;
register void *ptr asm("a1") = argPack;
asm volatile(
// Force 16-byte alignment to make sure that the 3 instructions fall
// within the same virtual page.
" .balign 16 \n"
" .option push \n"
// Force non-compressed RISC-V instructions
" .option norvc \n"
// semihosting e-break sequence
" slli x0, x0, 0x1f \n" // # Entry NOP
" ebreak \n" // # Break to debugger
" srai x0, x0, 0x7 \n" // # NOP encoding the semihosting call number 7
" .option pop \n"
/*mark (value) as an output operand*/
: "=r"(value) /* Outputs */
// The semihosting call number is passed in a0, and the argument in a1.
: "0"(value), "r"(ptr) /* Inputs */
// The "memory" clobber makes GCC assume that any memory may be arbitrarily read or written by the asm block,
// so will prevent the compiler from reordering loads or stores across it, or from caching memory values in registers across it.
// The "memory" clobber also prevents the compiler from removing the asm block as dead code.
: "memory" /* Clobbers */
);
return value;
}
// function to write a NULL terminated string to the console
void sh_write0(const char *buf)
{
// Print zero-terminated string
sys_sh(SH_SYS_WRITE0, (void *)buf);
}
// function to open a file.
// A real implementation would keep track of the file handle.
// and also check the validity of the input passed on to the function.
int sh_open(const char *filename, int mode)
{
uintptr_t argPack[3];
argPack[0] = (uintptr_t)filename;
argPack[1] = (uintptr_t)(mode);
argPack[2] = (uintptr_t)strlen(filename);
int file_handle = sys_sh(SH_SYS_OPEN, (void *)argPack);
return file_handle;
}
// function to close a file
int sh_close(int file_handle)
{
uintptr_t argPack[1];
argPack[0] = file_handle; // To prevent compiler warning
int res = sys_sh(SH_SYS_CLOSE, (void *)argPack);
return res;
}
// function to write to a file
// A real implementation would keep track of the number of bytes written and the position.
// and also check the validity of the input passed on to the function.
int sh_write(int file_handle, const char *buf, int len)
{
// Write to file
uintptr_t argPack[3];
argPack[0] = (uintptr_t)file_handle;
argPack[1] = (uintptr_t)buf;
argPack[2] = (uintptr_t)len;
int res = sys_sh(SH_SYS_WRITE, (void *)argPack);
return res;
}
int main()
{
// write a string to the console with semihosting
sh_write0("Hello semihosting-World\r\n");
// open a file and write a string to it
int file_handle = sh_open("test.txt", SH_FILE_MODE_WPLUS);
if (file_handle == -1)
{
goto exit;
}
int retval = sh_write(file_handle, "Hello semihosting-World\r\n", APP_SH_TEST_STR_LENGTH);
if (retval == -1)
{
goto exit;
}
else
{
sh_write0("Wrote text to file\r\n");
}
sh_close(file_handle);
exit:
return 0;
}
We can build the code using the following command:
riscv64-unknown-elf-gcc -specs=/usr/lib/picolibc/riscv64-unknown-elf/picolibc.specs main.c -Tvirt.ld -mcmodel=medany
Notice the additional -mcmodel=medany
flag? This is what the GCC manual says about this:
-mcmodel=medlow
:Generate code for the medium-low code model. The program and its statically defined symbols must lie within a single 2 GiB address range and must lie between absolute addresses -2 GiB and +2 GiB. Programs can be statically or dynamically linked. This is the default code model<p>-mcmodel=medany
:Generate code for the medium-any code model. The program and its statically defined symbols must be within any single 2 GiB address range. Programs can be statically or dynamically linked.<p> The code generated by the medium-any code model is position-independent, but is not guaranteed to function correctly when linked into position-independent executables or libraries.
Now we can run the program using the following command:
qemu-system-riscv64 -bios a.out -M virt -display none \
-chardev stdio,id=stdio0 \
-semihosting-config enable=on,userspace=on,chardev=stdio0
The expected output is:
Hello semihosting-World
Wrote text to file
Now you can see that a file named test.txt
has been created in the current directory in the host machine.
As you would have noticed, the code does not implement a UART driver and QEMU is not instantiating a serial chardev backend
. This print is coming via the semihosting infrastructure and not from the UART.
In a real machine (unlike the emulated one here), When a debugger is connected to the target, the semihosting output is redirected to the debugger console. File I/O is also redirected to the debugger which then performs the file I/O operations on the host machine.
The picolibc library has support for semihosting. This means that we can use the standard C library functions to perform semihosting operations. In order to perform a semihosted printf()
using picoLibc, include the --oslib=semihost
flag when building the code. This will link the semihosting support library to the application.
For instance, consider the following code:
#include <stdio.h>
int main()
{
printf("Hello semihosting-World\r\n");
return 0;
}
You can compile it with
riscv64-unknown-elf-gcc -specs=/usr/lib/picolibc/riscv64-unknown-elf/picolibc.specs -Tvirt.ld -mcmodel=medany --oslib=semihost test.c
and run it with
qemu-system-riscv64 -bios a.out -M virt -display none \
-chardev stdio,id=stdio0 \
-semihosting-config enable=on,userspace=on,chardev=stdio0
To get this output:
Hello semihosting-World
Without the --oslib
flag, it will not be possible to use the printf
function.
/usr/lib/riscv64-unknown-elf/bin/ld: /usr/lib/picolibc/riscv64-unknown-elf/lib/rv64imafdc/lp64d/libc.a(libc_tinystdio_puts.c.o): in function `puts':
./riscv64-unknown-elf/../../../newlib/libc/tinystdio/puts.c:41: undefined reference to `stdout'
collect2: error: ld returned 1 exit status
Similarly, you can use the fopen
, fwrite
and fclose
functions to perform semihosted file I/O.
#include <stdio.h>
#include <stdlib.h>
int main()
{
FILE* fp;
fp=fopen("test.txt", "w");
fprintf(fp, "Hello World!");
fclose(fp);
}
This is done in picolibc
by using the __attribute__((weak))
attribute. This attribute allows the user to override the default implementation of a function. The default implementation of the semihosting functions is provided by the picolibc library. The user can override the default implementation by providing their own implementation.
Semihosting is often performed over JTAG and includes a lot of overhead. This means that semihosting is not suitable for high performance applications. However, semihosting is still useful for debugging and prototyping. It is not expected to be even compiled into a production system.
The debug control and status register in the RISC-V debug specification provides provisions to control wether the target should halt on an EBREAK
instruction or not. This is controlled by the ebreakm
, ebreaks
and ebreaku
bits in the dcsr
register. The ebreakm
bit controls the behavior of EBREAK instructions in machine mode and the ebreaku
bit controls the behavior of EBREAK instructions in user mode etc. There are also mcontrol
registers which can be used to control the address of the instruction to be executed when the target enters debug mode. There are more interesting features in the debug specification which can be used to control the target behavior. You can read more about it here.
While making access-restricted content source code deliveries, it is the norm today to use a private Git repository. In the case of GitHub, it is a good practice to create an organization and create private repos and teams under the organization to handle who can access the contents. This works well for source code and can all be done with free tier resources. However, a git repo is not the best place to host archives. While all popular platforms, including GitHub, provide some free storage space, it is not enough for most use cases. Also, static pages like GitHub pages cannot be access controlled.
In this article, I will show you how to set up GitHub SSO for your website serving developer resources using vouch-proxy. This will allow you to host your archives and other developer resources on your website and control access to them using GitHub SSO.
An overview of how the flow works are shown below:
We need first to set up a Linux instance on AWS EC2. I will be using Ubuntu 20.04 for this article. The instance should have a public IP address and be accessible from the internet. I will be using a t2.micro
instance for this article. I will use Elastic IP addresses to prevent IP addresses from changing across reboots. You can read more about it here. Once a domain is purchased, we will redirect the domain to the public IP address of the instance. We will also need to open ports 80 and 443 on the instance to allow access to the website. You can read more about it here. Also, enable port 22 for SSH access. For security, this can be disabled or restricted once the setup is complete.
Now we need to set up a domain. You can use any domain registrar of your choice. I will be using namecheap for this article. Once you have a domain, you need to set up a DNS record to point to the EC2 instance’s public IP address.
Create the following DNS records:
Type | Host | Value |
---|---|---|
A | @ | Public IP address of the EC2 instance |
A | content | Public IP address of the EC2 instance |
CNAME | www | The AWS dns name e.g. xxx.ca-central-1.compute.amazonaws.com |
We will discuss the content
subdomain later in the article.
We will be using NGINX as a reverse proxy to serve our website. We will also use NGINX to serve static content. To install NGINX, run the following commands:
sudo apt update
sudo apt install nginx
To verify that NGINX is running, run the following command:
sudo systemctl status nginx
You should see the following output:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-09-23 05:17:57 UTC; 18h ago
Docs: man:nginx(8)
Process: 55148 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 55149 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 55150 (nginx)
Tasks: 2 (limit: 1146)
Memory: 4.9M
CPU: 686ms
CGroup: /system.slice/nginx.service
├─55150 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
└─55151 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
Sep 23 05:17:57 ip-192-21-10-255 systemd[1]: nginx.service: Deactivated successfully.
Sep 23 05:17:57 ip-192-21-10-255 systemd[1]: Stopped A high performance web server and a reverse proxy server.
Sep 23 05:17:57 ip-192-21-10-255 systemd[1]: Starting A high performance web server and a reverse proxy server...
Sep 23 05:17:57 ip-192-21-10-255 systemd[1]: Started A high performance web server and a reverse proxy server.
Vouch Proxy is a reverse proxy that authenticates users using a third-party identity provider. It is a Go application and requires golang to be installed before you can build it. To install golang, run the following commands:
sudo apt update
sudo apt install golang
To verify that golang is installed, run the following command:
go version
You should see something like the following output:
go version go1.18.1 linux/amd64
Now we can clone and build vouch-proxy. To do so, run the following commands:
git clone https://github.com/vouch/vouch-proxy.git
cd vouch-proxy
./do.sh goget
./do.sh build
To verify that vouch-proxy is built, run the following command:
./vouch-proxy -version
You should see the following output. The version number may differ depending on where the build was done.
vouch-proxy version 0.37.3
Vouch Proxy requires a configuration file to run. The configuration file is a YAML file. Sample configuration files can be found under the config
directory. You can copy config/config.yml_example_github
to config/config.yml
and edit it to suit your needs. The configuration file is well documented with comments.
The following is a sample configuration file:
vouch:
logLevel: info
testing: false
listen: 0.0.0.0 # VOUCH_LISTEN
port: 9090 # VOUCH_PORT
writeTimeout: 15 # VOUCH_WRITETIMEOUT
readTimeout: 15 # VOUCH_READTIMEOUT
idleTimeout: 15 # VOUCH_IDLETIMEOUT
domains:
- yourdomain.io
cookie:
secure: true
domain: yourdomain.io
teamWhitelist:
- your_org/your_team1
- your_org/your_team2
headers:
jwt: X-Vouch-Token # VOUCH_HEADERS_JWT
querystring: access_token # VOUCH_HEADERS_QUERYSTRING
jwt:
maxAge: 240
compress: true
oauth:
# create a new OAuth application at:
# https://github.com/settings/applications/new
provider: github
client_id: xxxxxxxxxxxxxxxxxxxx
client_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
We have set the configuration to use GitHub as the identity provider and allow only members of the your_org/your_team1
and your_org/your_team2
teams to access the website. You can change the configuration to suit your needs.
To create the client id and client secret, go to https://github.com/settings/applications/new and create a new OAuth application.
Homepage URL
to the domain name you have setup earlier.Authorization callback URL
to https://yourdomain.io/auth
.Vouch-proxy is configured to listen to port 9090. We will be using NGINX to proxy requests to this port. Though this configuration is listening on the 0.0.0.0
interface, we will use the localhost
interface to access the application. Since the 9090
port is not exposed to the internet, this is not a security concern. However, it can be configured to listen on the localhost
interface by setting the listen
configuration.
To run vouch-proxy, run the following command to run it as a daemon:
nohup ./vouch-proxy -loglevel debug > vouch.log 2>&1 &
The vouch-proxy documentation (here)[https://github.com/vouch/vouch-proxy/tree/master/examples/startup] has more information on how to run vouch-proxy as a systemd service.
Note that logs are redirected to the vouch.log
file. You can view the logs and can fill up the disk space. You can use tail -f vouch.log
to view the logs in real-time.
Now that we have vouch-proxy
up and running, we can configure NGINX to proxy requests. We will create two sites in NGINX. One for the authentication and the other for the actual website that serves requests to yourdomain.io
. The actual website will be proxied to the vouch-proxy
server using the auth_request
module of NGINX. So any requests to the actual website will be redirected to the vouch-proxy
server for authentication. This in turn, takes the user to the GitHub login page. After the user logs in, GitHub redirects the user back to the vouch-proxy
server. The vouch-proxy
server then redirects the user back to the actual website.
The two configuration files below should be created under the /etc/nginx/sites-available
directory. Then a soft link should be created under the /etc/nginx/sites-enabled
directory.
The following is the configuration for the authentication site:
upstream vouch {
# set this to the location of the vouch proxy
server localhost:9090;
}
server {
listen 443 ssl http2;
server_name yourdomain.io;
# Certificates set by Let's encrypt certbot
ssl_certificate /etc/letsencrypt/live/yourdomain.io/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.io/privkey.pem;
# This location serves all of the paths vouch uses
location ~ ^/(auth|login|logout|static) {
proxy_pass http://vouch;
proxy_set_header Host $http_host;
}
location = /validate {
# forward the /validate request to Vouch Proxy
proxy_pass http://vouch/validate;
# be sure to pass the original host header
proxy_set_header Host $http_host;
# Vouch Proxy only acts on the request headers
proxy_pass_request_body off;
proxy_set_header Content-Length "";
# optionally add X-Vouch-User as returned by Vouch Proxy along with the request
auth_request_set $auth_resp_x_vouch_user $upstream_http_x_vouch_user;
# these return values are used by the @error401 call
auth_request_set $auth_resp_jwt $upstream_http_x_vouch_jwt;
auth_request_set $auth_resp_err $upstream_http_x_vouch_err;
auth_request_set $auth_resp_failcount $upstream_http_x_vouch_failcount;
}
# if validate returns `401 not authorized`, then forward the request to the error401block
error_page 401 = @error401;
location @error401 {
# redirect to Vouch Proxy for login
return 302 $scheme://$http_host/login?url=$scheme://$http_host$request_uri&vouch-failcount=$auth_resp_failcount&X-Vouch-Token=$auth_resp_jwt&error=$auth_resp_err;
# you usually *want* to redirect to Vouch running behind the same Nginx config protected by HTTPS
# but to get started you can forward the end user to the port that vouch is running on
}
error_page 404 @error404;
location @error404 {
root /var/www/html/404.html;
internal;
}
# proxy pass authorized requests to your service
location / {
# send all requests to the `/validate` endpoint for authorization
auth_request /validate;
# forward authorized requests to your service yourdomain.io
proxy_pass http://127.0.0.1:8080;
# you may need to set these variables in this block as per https://github.com/vouch/vouch-proxy/issues/26#issuecomment-425215810
auth_request_set $auth_resp_x_vouch_user $upstream_http_x_vouch_user;
auth_request_set $auth_resp_x_vouch_idp_claims_groups $upstream_http_x_vouch_idp_claims_groups;
auth_request_set $auth_resp_x_vouch_idp_claims_given_name $upstream_http_x_vouch_idp_claims_given_name;
# set user header (usually an email)
proxy_set_header X-Vouch-User $auth_resp_x_vouch_user;
# optionally pass any custom claims you are tracking
proxy_set_header X-Vouch-IdP-Claims-Groups $auth_resp_x_vouch_idp_claims_groups;
proxy_set_header X-Vouch-IdP-Claims-Given_Name $auth_resp_x_vouch_idp_claims_given_name;
# optionally pass the accesstoken or idtoken
# proxy_set_header X-Vouch-IdP-AccessToken $auth_resp_x_vouch_idp_accesstoken;
# proxy_set_header X-Vouch-IdP-IdToken $auth_resp_x_vouch_idp_idtoken;
}
}
# HTTP redirect to HTTPS
server {
listen 80;
server_name _;
return 301 https://$host$request_uri;
}
The following is the configuration for the actual website hosting the contents running on port 8080
, listening on 127.0.0.1
:
server {
listen 127.0.0.1:8080 default_server;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
}
We will use Let's Encrypt
to issue certificates. First install certbot
sudo apt install certbot python3-certbot-nginx
Then run the following command to see if certificates can be issued for the domain. This is a dry run to see if the domain configurations are in order and if the certificates can be issued.
sudo certbot certonly --nginx -d yourdomain.io -d www.yourdomain.io -d content.yourdomain.io --dry-run
If the above command runs successfully, run the following command to issue the certificates. This step will also update the nginx configuration files to include the certificates. In the configuration in the previous section, the certificates are already set:
sudo certbot certonly --nginx -d yourdomain.io -d www.yourdomain.io -d content.yourdomain.io
Note: we will talk about the
content.yourdomain.io
domain later.
Now that the certificates are issued, we can populate the site contents. All the content should go into /var/www/html
served by the site configured to listen to the 8080
port.
To test if the configurations are correct, run the following command:
sudo nginx -t
If the configurations are correct, then restart Nginx:
sudo systemctl restart nginx
Now that the site is up and running, we can test it. Open the browser and go to https://yourdomain.io
. You should first be redirected to GitHub
to login in case you are not already logged in. Once logged in, you will be asked to allow yourdomain.io
to access your GitHub profile. Once it is allowed to, the team membership will be verified, and you will be redirected to the site contents.
Vouch-proxy is built to work with a web browser. This makes accessing contents behind the proxy via the command line a bit difficult. To solve this, we will provide an alternate access mechanism to the contents via the content.yourdomain.io
domain. This domain will utilize the GitHub API Key and Username passed in the request header to authenticate the user. This makes it easy to frame curl
commands to access contents. We will restrict access using this mechanism to only the items under the content
directory. This is because documentation and other items that are not meant to be accessed via the command line should not be accessible via this mechanism.
To do this, we will create a new site configuration file for the content.yourdomain.io
domain within the yourdomain.io
configuration file with the following changes:
An API key can be created by going to https://github.com/settings/tokens/ne and creating a new token with the read:org
scope. This token will be used to authenticate the user. The username is the GitHub username of the user.
server {
# sideband server to serve contents with GitHub API key-based authentication
listen 443 ssl http2;
server_name content.yourdomain.io;
# certificate placed here by certbot
ssl_certificate /etc/letsencrypt/live/microchip-hpsc.io/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/microchip-hpsc.io/privkey.pem;
# proxy pass authorized requests to your service
location / {
# send all requests to the `/validate` endpoint for authorization
auth_request /keyvalidate;
# unauthorized requests return 404 from Github API. by default, Nginx returns 500 for this case. But we want to return 404.
error_page 500 =401 /error/401;
# forward authorized requests to microchip-hpsc.io/contents
proxy_pass http://127.0.0.1:8080/contents/;
}
# validate the request by checking the membership of the user in the GitHub org
location /keyvalidate {
# use google's DNS server to resolve the GitHub API
resolver 8.8.8.8;
proxy_method GET;
proxy_set_header Authorization "Bearer $http_apikey";
proxy_set_header Accept "application/vnd.github+json";
proxy_pass https://api.github.com/orgs/your_orgName/teams/your_team1/memberships/$http_user;
}
location /error/401 {
return 401;
}
}
The /validate
endpoint will validate any request to the content.yourdomain.io
domain. This endpoint will use the apikey
and user
headers to validate the user. If the user is a member of the your_team1
team in the your_orgNam
organization, then the request will be forwarded to the http://127.0.0.1:8080/contents/
endpoint. This endpoint will serve the contents under the contents
directory.
This means that to access contents under “https://yourdomain.io/contents/yourfile.txt”, using APIKey and Username, the request should be directed to “https://content.yourdomain.io/yourfile.txt” with the apikey
and user
headers set.
An example of such a request is:
curl -H "apikey: <your_apikey>" -H "user: <your_username>" https://content.yourdomain.io/yourfile.txt
In case the membership of the user is not verified, the GitHub request will return a 404
error. Since auth_request
honours only 2xx
and 401
, this will be returned as a 500
to the user. Hence, we do additional configuration to map the 500
error to the 401
error. This will make it easy to handle the error on the client side.
This section was added on 17-Feb–2023, after the original post was published.
The above configuration supports only one team for command line access of content. To support multiple teams, we can build a local authentication server that will validate the user membership in the team. This server would run in parallel to vouch-proxy
and can be used as the proxy_pass
endpoint in the nginx configuration.
The server code would look something like this:
# A fastapi server listening on port 8000
# This server is used to authenticate users and return a 401 if the user is not authenticated
# Sample curl command to test the API is as follows:
# curl -H "apikey: <your_apikey>" -H "user: <your_username>" localhost:8000/auth
# Github teams to check are in the list github_teams.
from fastapi import FastAPI, Header, HTTPException
from fastapi.responses import JSONResponse
import requests
import os
app = FastAPI()
github_teams = ["team1","team2","team3"]
orgName = "your_orgName"
@app.get("/auth")
async def auth_user(apikey: str = Header(...), user: str = Header(...)):
#strip the apikey of any leading or trailing spaces
apikey = apikey.strip()
if check_user(user,apikey):
return JSONResponse(status_code=200, content="")
else:
return JSONResponse(status_code=401, content="")
def check_user(user,apikey):
# check if the user is part of one of the given github teams
for team in github_teams:
url = f"https://api.github.com/orgs/{orgName}/teams/{team}/memberships/{user}"
request_headers = {
'Authorization': 'Bearer ' + apikey,
'Accept': 'application/vnd.github.v3+json'
}
response = requests.get(url, headers=request_headers)
print("response: ", response.status_code)
if response.status_code == 200:
return True
return False
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="localhost", port=8000)
In this post, we have seen how to set up a site accessible only to members of a GitHub organization. We have also seen how to access the site’s contents via the command line using the GitHub API Key and Username.
]]>