Testing

[utterances-bot]

This is a general purpose comment thread for the Testing post.

[leggettc18]

I'm not sure if I did something wrong, but by following along with this guide exactly I was receiving an error about fmt being undeclared after setting up the CompareMessage implementation. I fixed it by adding use core::fmt; and use core::fmt::Write; at the top of the file, and increasing the PANIC_LINE by 2 to compensate. Did I miss a step along the way or is there a missing step in the guide?

[phil-opp]

@leggettc18 You did nothing wrong, I forgot to mention these imports. Fixed in 1f97103. Thanks for reporting!

[Lorithan]

I'm unable to run cargo xtest. Instead I keep keeping the message

error[E0463]: can't find crate for std
|
= note: the x86_64-blog_os-1954185736335484285 target may not be installed

cargo xbuild works fine, though.

[AntoineSebert]

Try cargo xtest --target x86_64-blog_os.json.

I have the same problem but couldn't figure out why the file isn't automatically found.

[phil-opp]

@AntoineSebert Do you have a build.target key defined in your .cargo/config file?

@Beidah Sounds like forgot a #![no_std] attribute (perhaps the conditional attribute in your lib.rs?). Or you accidentally use a dependency that uses std.

[AntoineSebert]

@phil-opp Yes it's there. The problem seems to have disappread today, and anyway it's not a big deal.

[AntoineSebert]

The recent improvements in the test framework now produce text outputs that are not colored in green, thus making the test result less visible within the terminal.
Is there a way to specify a color for the text when calling print/println macros ?

[phil-opp]

Yes, there is! You can use ANSI escape codes for coloring. For example, you can create a type to print something in green:

use core::fmt;

struct Green(&'static str);

impl fmt::Display for Green {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { 
        write!(f, "\x1B[32m")?; // prefix code
        write!(f, "{}", self.0)?;
        write!(f, "\x1B[0m")?; // postfix code
        Ok(())
    }
}

Now you can do serial_println!("{}", Green("[ok]")); at the end of your test.

I opened #603 for adding this to the post.

[bytesnail]

In the process of adding tests, I encountered some strange errors. In this test, I simply created a writer, but failed, and judging from the error message, it seems that it did not execute any tests at all, because I did not see the information printed by test_runner or panic_handler on the console.

lws@lsw:/document/project/flandre-os/FlandreOS$ cargo xtest test_default_writer
   Compiling flandre_os v0.0.1 (/document/project/flandre-os/FlandreOS)
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
     Running target/x86_64-flandre_os/debug/deps/flandre_os-d120248f92b70111
Building bootloader
   Compiling bootloader v0.6.0 (/work/tool/rust/cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/bootloader-0.6.0)
    Finished release [optimized + debuginfo] target(s) in 0.72s
Running: `qemu-system-x86_64 -drive format=raw,file=/document/project/flandre-os/FlandreOS/target/x86_64-flandre_os/debug/deps/bootimage-flandre_os-d120248f92b70111.bin -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none test_default_writer`
qemu-system-x86_64: -display none: drive with bus=0, unit=0 (index=0) exists
error: test failed, to rerun pass '--lib'

[bytesnail]

I did all kinds of printing tests and the printing function seemed to work normally, but when I tried to add some tests to ensure that these functions would not be damaged in future modifications, I got the above errors.

[bjorn3]

I think passing a test name to cargo xtest doesnt work. Try running just cargo xtest.

[phil-opp]

I think passing a test name to cargo xtest doesnt work. Try running just cargo xtest.

Yes, as you see from the "Running: …" output, the test_default_writer is directly passed as an argument to QEMU, which does not understand it and exits with an error. Passing the name of a single test is a feature of the default test runner and not available in our bare-metal environment because our kernel does not get any arguments passed when it's started.

What you can do is to run the tests of the library component through cargo xtest --lib and the tests of the binary through cargo xtest --bin blog_os. You can also run an individual integration test through cargo xtest --test test_name.

[xj42]

Now we can run the test using cargo xtest --test panic_handler. We see that it passes, which means that the reported panic info is correct. If we use a wrong line number in PANIC_LINE or panic with an additional character through panic!("{}x", MESSAGE), we see that the test indeed fails.

I tried adding the "x" as you indicated in the above paragraph. The test did not fail. WIth some more fooling about i see that the fn write_str(&mut self, s: &str) -> fmt::Result {} function is called 4 times. The third time it is called, it sets equals to true. Because there is no else setting false, the test does not work as expected.

I have submitted a pull request #610

[phil-opp]

@xj42 Thanks a lot for reporting! I created #611 to fix this.

[ta5een]

Hi, thank you for taking your time and effort into making this brilliant blog! I've learnt so much about operating systems and the Rust language in general! Can't wait to see how this blog unfolds and what new things you'll bring.

I just wanted to ask if it is the expected behaviour for serial_print! and serial_println! to only work in test mode? With cargo xtest, the serial print macros work as expected; they print to my host's terminal. However, with cargo xrun, nothing is shown in my host's terminal, nor my QEMU instance. Oddly enough, the system hasn't panicked, so it somehow worked (but the bytes don't show up).

It seems to me that perhaps this was the desired behaviour? If so, is there a way for me to send data to the serial port while the QEMU instance is running as normal?

[zhanlidegui]

[package.metadata.bootimage] test-args = ["-device", "isa-debug-exit,iobase=0xf4,iosize=0x04","-serial", "stdio","-display","none"] test-success-exit-code = 33 test-timeout = 120

This setting is only valid for testing

[dedestem]

cargo test gives this: (signal: 11, SIGSEGV: invalid memory reference)

[Fincap]

If anyone else is experience the same issue that I was with cargo test failing, and the posted workaround of removing panic = "abort" from Cargo.toml not working, I have a workaround that worked for me.

Insert into your .cargo/config.toml in the [unstable] section:
panic-abort-tests = true

Make sure you're using the latest nightly compiler.

[delguoqing]

both lib.rs and basic_boot.rs contain a entry point function "_start", but cargo test still compile. Why is that?

[delguoqing]

ok, I think I know why. When we're running test cases for basic_boot.rs, cfg(test) is not used for lib.rs?

[SimonOneNineEight]

I think there's a minor error, in the panic function it said shouldn't return, but there's no semi-column at the end of the function call. I've tried to copy paste the code into my editor and the lsp tells me to add a semi-column.

// in tests/basic_boot.rs

#![test_runner(blog_os::test_runner)]

#[panic_handler]
fn panic(info: &PanicInfo) -> ! {
    blog_os::test_panic_handler(info)
}

[matveitolochko]

You shouldn't need to add the semicolon, because blog_os::test_panic_handler should never return.

[danielmorozoff]

I am receiving a compilation error whenever I attempt to include the x86_64 = "0.14.2" crate. Has anyone come across this and what was the remedy that alleviated this?

error[E0053]: method `steps_between` has an incompatible type for trait
   --> /Users/danielmorozoff/.cargo/registry/src/index.crates.io-6f17d22bba15001f/x86_64-0.14.12/src/addr.rs:382:51
    |
382 |     fn steps_between(start: &Self, end: &Self) -> Option<usize> {
    |                                                   ^^^^^^^^^^^^^ expected `(usize, Option<usize>)`, found `Option<usize>`
    |
    = note: expected signature `fn(&addr::VirtAddr, &addr::VirtAddr) -> (usize, Option<usize>)`
               found signature `fn(&addr::VirtAddr, &addr::VirtAddr) -> Option<usize>`
help: change the output type to match the trait
    |
382 |     fn steps_between(start: &Self, end: &Self) -> (usize, Option<usize>) {
    |                                                   ~~~~~~~~~~~~~~~~~~~~~~

[kingananas20]

i just tried out the latest crate x86_64 v 0.14.13 and it works

0.14.13 doesn't work anymore but 0.14.12 works for me

[ChocolateLoverRaj]

0.14.13 doesn't work anymore but 0.14.12 works for me

@kingananas20 0.14.13 expects a newer Rust nightly version than 0.14.12. So if you want to use 0.14.13 just upgrade your Rust nightly version.

[Zain-Najeeb]

I am getting this same error, not really sure why either. Ive tried both 0.14.12 and 0.14.13 and rustc nightly version on both 1.84.0 and 1.88.0. Here is my Cargo.toml file if it helps

 
[package]
name = "blog_os"
version = "0.1.0"
edition = "2021"

[dependencies]
bootloader = "0.9.22"
volatile = "=0.2.6" 
spin = "0.5.2"
x86_64 = "0.14.12"

[dependencies.lazy_static]
version = "1.4.0"
features = ["spin_no_std"]

[package.metadata.bootimage]
test-args = ["-device", "isa-debug-exit,iobase=0xf4,iosize=0x04"]

[Zain-Najeeb]

I was able to fix it after doing some testing. Being on version 1.88.0 and with x86_64 version 0.15.2 worked for me

[ChocolateLoverRaj]

I was able to fix it after doing some testing. Being on version 1.88.0 and with x86_64 version 0.15.2 worked for me

@Zain-Najeeb note that 0.15.2 could contain breaking changes. If you are following this tutorial I would recommend keeping your Rust version as it is and using 0.14.13

[UnwhiteCat]

I have a error at the first step trying to run the "cargo test" barely...Has the different error as wrote in this book.

UnwhiteCat@uccomputer ~/Codes/rust/blogos $ cargo test
   Compiling scopeguard v1.2.0
   Compiling bootloader v0.9.29
error[E0152]: duplicate lang item in crate `core` (which `rustc_std_workspace_core` depends on): `sized`
  |
  = note: the lang item is first defined in crate `core` (which `bootloader` depends on)
  = note: first definition in `core` loaded from /home/UnwhiteCat/Codes/rust/blogos/target/x86_64-blog_os/debug/deps/libcore-33e3a870bea73245.rmeta
  = note: second definition in `core` loaded from /home/UnwhiteCat/Codes/rust/blogos/target/x86_64-blog_os/debug/deps/libcore-f389adb7b0a7ac1b.rmeta

error[E0152]: duplicate lang item in crate `core` (which `rustc_std_workspace_core` depends on): `sized`
  |
  = note: the lang item is first defined in crate `core` (which `scopeguard` depends on)
  = note: first definition in `core` loaded from /home/UnwhiteCat/Codes/rust/blogos/target/x86_64-blog_os/debug/deps/libcore-33e3a870bea73245.rmeta
  = note: second definition in `core` loaded from /home/UnwhiteCat/Codes/rust/blogos/target/x86_64-blog_os/debug/deps/libcore-f389adb7b0a7ac1b.rmeta

For more information about this error, try `rustc --explain E0152`.
error: could not compile `scopeguard` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `bootloader` (lib) due to 1 previous error

Here is my rustc version.

UnwhiteCat@uccomputer ~/Codes/rust/blogos $ rustc --version
rustc 1.85.0-nightly (6b6a867ae 2024-11-27)

[rctcwyvrn]

#840 (comment)

Delete panic = "abort" from your cargo.toml

[doomed-neko]

for some reason, qemu does not exit after calling exit_qemu(), this only happens when using cargo bootimage but when I use qemu-system-x86_64 and pass the isa-debug-exit,iobase=0xf4,iosize=0x04 argument, it works fine

[doomed-neko]

nevermind, turns out I have set the [package.metadata.bootimage] options in .cargo/config.toml instead of Cargo.toml

[devashish-sood]

When writing the serial_println! macro, why are these cases distinct and more than the cases we had when writing the println! macro?

[RobinToncourt]

If a test fails it prevents the next tests to be executed(by going through the panic handler due to the assert).
To prevent that you can rewrote the assert and assert_eq methods.
Here my solution: https://github.com/RobinToncourt/os/blob/master/src/test_utils.rs
As for the should_panic test, I don't find them relevant as I consider that an operating system should never panic.

[jsanchezpc]

Hi!
I searched and can't find any solutions. Can anybody help me please?

When I try to run cargo test, it doesnt launch qemu, but I can't get to see any errors at all. Here is my code:

#![no_std] // don't link the Rust standard library
#![no_main] // disable all Rust-level entry points
#![feature(custom_test_frameworks)]
#![test_runner(crate::test_runner)]
#![reexport_test_harness_main = "test_main"]

mod vga_buffer;
use core::panic::PanicInfo;

#[cfg(test)]
pub fn test_runner(tests: &[&dyn Fn()]) {
    println!("Running {} tests", tests.len());
    for test in tests {
        test();
    }
}

#[test_case]
fn trivial_assertion() {
    print!("trivial assertion... ");
    assert_eq!(1, 1);
    println!("[ok]");
}

/// This function is called on panic.
#[panic_handler]
fn panic(info: &PanicInfo) -> ! {
    println!("{}", info);
    loop {}
}

#[no_mangle] // don't mangle the name of this function
pub extern "C" fn _start() -> ! {
    // this function is the entry point, since the linker looks for a function
    // named `_start` by default

    println!("Hello World!");

    #[cfg(test)]
    test_main();

    loop {}
}

The output in my console:

> cargo test
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.10s 

All seems good but Qemu wont show up. Why?? Tried to use chatgpt but didnt help at all.

[jsanchezpc]

Ok I just solved the problem. If you have the same issue, try this:
Go to your Cargo.toml file and set the test value to true (in [[bin]] )

[HernanFdz]

First of all, thanks a lot for all this work!! Haven't gotten to the real meat yet I guess, but already enjoying it and learning very much.

Just a tiny detail here: I guess it should be "... all exit codes ..."

The problem is that cargo test considers all error codes other than 0 as failure.

[HernanFdz]

@phil-opp about this:

A significant drawback of this approach is that it only works for a single test function. With multiple #[test_case] functions, only the first function is executed because the execution cannot continue after the panic handler has been called. I currently don’t know of a good way to solve this problem, so let me know if you have an idea!

Here's the best solution I could come up with. It seems to work just fine for me, but please let me know if you find any issue with this approach:

// tests/should_panic.rs

#![no_std]
#![no_main]
#![feature(custom_test_frameworks)]
#![test_runner(test_runner)]
#![reexport_test_harness_main = "test_main"]

use spin::Mutex;

use rust_os::{
    qemu::{types::QemuExitCode, utils::exit_qemu},
    serial_print, serial_println,
};

static TEST_IDX: Mutex<u8> = Mutex::new(0);

#[unsafe(no_mangle)] // don't mangle the name of this function
pub extern "C" fn _start() -> ! {
    test_main();

    loop {}
}

#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    serial_println!("[ok]");

    *TEST_IDX.lock() += 1;

    test_main();

    loop {}
}

pub fn test_runner(tests: &[&dyn Fn()]) {
    let num_tests = tests.len();

    let test_idx = *TEST_IDX.lock() as usize;

    if test_idx < num_tests {
        let test = tests[test_idx];

        test();
        serial_println!("[test did not panic]");
        exit_qemu(QemuExitCode::Failed);
    } else {
        exit_qemu(QemuExitCode::Success);
    }
}

#[test_case]
fn should_fail_0() {
    serial_print!("should_panic::should_fail_0...\t");
    assert_eq!(0, 1);
}

#[test_case]
fn should_fail_1() {
    serial_print!("should_panic::should_fail_1...\t");
    assert_eq!(0, 1);
}

[gcaseres]

I'm kind of a fan of using VSCode Docker devcontainers to avoid populating my computer with millions of dependencies from different languages / code experimentations.

I started this tutorial by building in my devcontainer and then manually running qemu from the host.

This worked until I started the "testing" session, since bootimage doesn't seem to have a way to build the test binaries and use them to create the image.

After some research, I came up with the following script that sort of does the same than bootimage but allows me to create images from my test binaries and see the test results in qemu.

TARGET_NAME=$1
TARGET_KIND=$2

KERNEL_BIN_PATH=$(cargo build --tests --message-format=json | jq -r "select(.target.name == \"$TARGET_NAME\" and (.target.kind | any(index(\"$TARGET_KIND\"))) and .target.test and .profile.test) | .executable | select(. != null)")

BOOTLOADER_MANIFEST_PATH=$(cargo metadata --format-version 1 | jq -r '.packages[] | select(.name == "bootloader") | .manifest_path')
BOOTLOADER_TARGET_PATH=$(dirname $BOOTLOADER_MANIFEST_PATH)/x86_64-bootloader.json

KERNEL=$KERNEL_BIN_PATH KERNEL_MANIFEST=$(pwd)/Cargo.toml cargo build --manifest-path $BOOTLOADER_MANIFEST_PATH -Zbuild-std=core --release --features binary --target $BOOTLOADER_TARGET_PATH --target-dir ./target

rust-objcopy -I elf64-x86-64 -O binary --binary-architecture=i386:x86-64 target/x86_64-bootloader/release/bootloader target/bootimage-$TARGET_KIND-$TARGET_NAME.bin

I figured out the script by looking at the bootimage source and the bootloader README documentation (version 0.9).

I hope it's useful for others

[ThatBlockyPenguin]

Rust newbie here - why do we have to duplicate so much testing code between main.rs and lib.rs? Surely if lib.rs is the testing entrypoint, we should be able to remove all test-related code from main?

[0xPxt]

Hi, this is my solution for running multiple tests that panic. I just tweaked it and shamelessly stole from @HernanFdz the idea of re-calling test_main in the panic handler.

I added an in-panic guard (explained why in the code) and used atomics instead of mutexes because grabbing a spinlock in the panic handler can lead to an endless loop if not carefully done.

Here's what worked for me (some of the boilerplate is different because I am using the latest version of bootloader but you get the idea):

#![feature(custom_test_frameworks)]
#![test_runner(crate::test_runner)]
#![reexport_test_harness_main = "test_main"]
#![no_std]
#![no_main]

use core::panic::PanicInfo;
use core::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use kernel::qemu::{QemuExitCode, exit_qemu};
use log::{error, info};

// Using atomic to avoid using `unsafe`, but it would be sound to use `unsafe` here.
static NEXT: AtomicUsize = AtomicUsize::new(0);
static IN_PANIC: AtomicBool = AtomicBool::new(false);

#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    if IN_PANIC.swap(true, Ordering::Relaxed) {
        // Somehow ended up here without stepping into `test_runner` first.
        // Currently not required, but prevents painful debugging in the future:
        // The way to end up here other than after a test ran by test_runner is that
        // anything between setting `IN_PANIC` and clearing it again in `test_runner`
        // panics, for example, the `info!` call below. If that were the case and
        // there was no IN_PANIC guard, the panic handler would be triggered endlessly
        // until the stack overflows.
        exit_qemu(QemuExitCode::FAILURE);
        loop {}
    }
    info!("[ok]");
    NEXT.fetch_add(1, Ordering::Relaxed);

    test_main();

    loop {}
}

use bootloader_api::BootInfo;
bootloader_api::entry_point!(main);

fn main(boot_info: &'static mut BootInfo) -> ! {
    kernel::init(boot_info);
    test_main();
    loop {}
}

// Re-implement test_runner because the one in lib does not expect panic!
pub fn test_runner(tests: &[&dyn Fn()]) {
    let next = NEXT.load(Ordering::Relaxed);

    if next == 0 {
        info!("Running {} tests", tests.len());
    }

    if next < tests.len() {
        IN_PANIC.store(false, Ordering::Relaxed);
        tests[next]();
        error!("[test did not panic]");
        exit_qemu(QemuExitCode::FAILURE);
    }

    // no more tests to run, all panicked
    exit_qemu(QemuExitCode::SUCCESS);
}

#[test_case]
fn should_fail_0() {
    info!("should_panic::should_fail...");
    assert_eq!(0, 1);
}

#[test_case]
fn should_fail_1() {
    info!("should_panic::should_fail_two...");
    assert_eq!(0, 2);
}