I’ve been experimenting with the performance and stability of the VisionFive 2 by building GCC and LLVM on the Debian Image provided by StarFive (Google Drive link). I found that:
- It is quite fast (relative to my previous toolchain work done under qemu) – I could build LLVM in under 4 hours.
- It seems rock-solid – I have kept all 4 cores at 100% for hours on end, with no apparent issues. Running the GCC testsuite identified no issues.
Overall I’m really impressed with the board and I think it’s a great platform for RISC-V toolchain development.
This post outlines:
- What I did to build the toolchains, and
- Some measurements and observations.
LLVM
I built LLVM first because I’m actually going to need to use it to work on Numba. I used the LLVM 15.0.6 sources (the latest release version at the time of writing). First I installed dependencies:
sudo apt install \
cmake ninja-build chrpath texinfo sharutils libelf-dev \
libffi-dev lsb-release patchutils diffstat xz-utils \
python3-dev libedit-dev libncurses5-dev swig \
python3-six python3-sphinx binutils-dev libxml2-dev \
libjsoncpp-dev pkg-config lcov procps help2man \
zlib1g-dev libjs-mathjax python3-recommonmark \
doxygen gfortran libpfm4-dev python3-setuptools \
libz3-dev libcurl4-openssl-dev libgrpc++-dev \
protobuf-compiler-grpc libprotobuf-dev \
protobuf-compiler
I would usually just use:
sudo apt build-dep llvm-15
(or something like that) but I couldn’t easily work out how to add a deb-src line to the APT sources that actually worked, so I derived that list of packages by looking at the source package instead. Following dependency installation, I configured for RISC-V only (because I am mainly interested in working on JIT compilation) a release build with assertions – I wasn’t sure if I had enough disk space for a debug build, and release with assertions is a reasonable compromise:
mkdir build
cd build
cmake ../llvm -G Ninja \
-DCMAKE_INSTALL_PREFIX=/data/opt/llvm/15.0.6 \
-DLLVM_TARGETS_TO_BUILD=RISCV \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_ENABLE_ASSERTIONS=ON
ninja -j 3
ninja install
I had to use 3 cores because 8GB RAM is not enough to link without running out of RAM when buiding with 4 cores. I was using ld from binutils, and I wonder if using an alternative linker would be more efficient – I need to check whether gold, mold, or lld support RISC-V first.
The build was pretty fast:
real 230m55.985s
user 671m8.705s
sys 20m51.997s
The build completed in just under 4 hours, which for an SBC is pretty good – it’s quick enough for it not to be a complete pain to build toolchains. Compared to an emulated system I was using before where I had to wait all day even though I was emulating 8 cores on a Xeon Gold 6128, it’s blazing fast!
Since I built LLVM I’ve been using it for llvmlite development without issues. llvmlite is primarily for compiling and JITting code – LLVM JIT works “out of the box” in LLVM on RISC-V (HowToUseLLJIT is an example program included with the LLVM source distribution that demonstrates how to use the ORCv2 LLJIT class):
$ ninja HowToUseLLJIT
[1/1] Linking CXX executable bin/HowToUseLLJIT
$ ./bin/HowToUseLLJIT
add1(42) = 43
GCC
Next I tried building GCC 12.2 – this was mainly a curiosity for me because I don’t have a real use for it right now (other than the generally good “principle” of using up-to-date software), but I’ve spent a lot of time working on GCC-based toolchains, particularly for RISC-V, so it seemed interesting. I installed a few dependencies – possibly not all of these are required, but I was pretty certain this would cover everything needed:
sudo apt install \
libc6-dev m4 libtool gawk lzma xz-utils patchutils \
gettext texinfo locales-all sharutils procps \
dejagnu coreutils chrpath lsb-release time pkg-config \
libgc-dev libmpfr-dev libgmp-dev libmpc-dev \
flex yacc bison
Then to configure and build GCC:
mkdir build-gcc-12.2.0
cd build-gcc-12.2.0
../gcc/configure \
--enable-languages=c,c++,fortran \
--prefix=/data/gmarkall/opt/gcc/12.2.0 \
--disable-multilib \
--with-arch=rv64gc \
--with-abi=lp64d
make -j4
There’s one problem building GCC with the Debian image on the VisionFive 2, which is that the GCC RISC-V target seems not to be correctly set up for Debian multiarch – building will eventually fail with a message like:
/usr/include/stdio.h:27:10: fatal error:
bits/libc-header-start.h: No such file or directory
27 | #include <bits/libc-header-start.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
as it fails to find headers because it’s looking in the wrong location – /usr/include instead of the architecture-specific /usr/include/riscv64-linux-gnu.
The stage 1 xgcc doesn’t seem to know about multiarch:
$ ./gcc/xgcc -print-multiarch
# (no output)
A small patch works around this:
diff --git a/gcc/config/riscv/t-linux b/gcc/config/riscv/t-linux
index 216d2776a18..f714026b3cc 100644
--- a/gcc/config/riscv/t-linux
+++ b/gcc/config/riscv/t-linux
@@ -1,3 +1,4 @@
# Only XLEN and ABI affect Linux multilib dir names, e.g. /lib32/ilp32d/
MULTILIB_DIRNAMES := $(patsubst rv32%,lib32,$(patsubst rv64%,lib64,$(MULTILIB_DIRNAMES)))
MULTILIB_OSDIRNAMES := $(patsubst lib%,../lib%,$(MULTILIB_DIRNAMES))
+MULTIARCH_DIRNAME = $(call if_multiarch,riscv64-linux-gnu)
Running the build again after making this change succeeds. Also, xgcc now knows about multiarch:
$ ./gcc/xgcc -print-multiarch
riscv64-linux-gnu
I think I need to report this and maybe submit a proper patch upstream – the above patch is good enough to work around the issue for building now, but isn’t a general solution – for example, it’s guaranteed not to work on a riscv32 system!
The build takes a bit longer than LLVM:
real 331m22.192s
user 1192m59.381s
sys 24m57.932s
At first I was surprised that it took longer than LLVM, but it has the bootstrapping process to go through, which goes some way towards increasing the build time.
I also ran the GCC test suite:
make check -j 4
Which takes a little while, but is worth the wait:
real 245m35.888s
user 773m59.513s
sys 133m17.827s
The results are not 100% passing, but look pretty good:
=== gcc Summary ===
# of expected passes 138016
# of unexpected failures 461
# of unexpected successes 4
# of expected failures 1035
# of unresolved testcases 48
# of unsupported tests 2888
=== g++ Summary ===
# of expected passes 221581
# of unexpected failures 544
# of unexpected successes 4
# of expected failures 1929
# of unresolved testcases 40
# of unsupported tests 10418
=== gfortran Summary ===
# of expected passes 65099
# of unexpected failures 13
# of expected failures 262
# of unsupported tests 165
None of the problems look like any kind of real issue – many asan tests failed, which I guess might be due to lack of support (or it not having been enabled correctly by default) and many other issues are checks that scan the output for particular patterns (which can be fragile) or tests for excess error messages / warnings. I’d have to look in depth to be certain, but I have high confidence that all is well here.
Conclusion
Building GCC and LLVM on the VisionFive 2 was relatively pain-free, especially for a very new SBC with a relatively new software ecosystem. If you’re into RISC-V toolchain development and want to work on a native system (as opposed to e.g. using qemu under linux, or targeting very small embedded systems that need cross compilation anyway), it’s a great choice!
I’m expecting to have a productive time working on Numba and llvmlite for RISC-V with this board.
Graham Markall 