FreeBSD Quarterly Status Report - Fourth Quarter 2018
23 March, 2019 by trasz@freebsd.org | freebsd
FreeBSD Project Quarterly Status Report - 4th Quarter 2018
Since we are still on this island among many in this vast ocean of the
Internet, we write this message in a bottle to inform you of the work
we have finished and what lies ahead of us. These deeds that we have
wrought with our minds and hands, they are for all to partake of - in
the hopes that anyone of their free will, will join us in making
improvements. In todays message the following by no means complete or
ordered set of improvements and additions will be covered:
i386 PAE Pagetables for up to 24GB memory support, Continuous
Integration efforts, driver updates to ENA and graphics, ARM
enhancements such as RochChip, Marvell 8K, and Broadcom support as well
as more DTS files, more Capsicum possibilities, as well as pfsync
improvements, and many more things that you can read about for
yourselves.
Additionally, we bring news from some islands further down stream,
namely the nosh project, HardenedBSD, ClonOS, and the Polish BSD
User-Group.
We would, selfishly, encourage those of you who give us the good word
to please send in your submissions sooner than just before the
deadline, and also encourage anyone willing to share the good word to
please read the section on which submissions we're also interested in
having.
Yours hopefully,
Daniel Ebdrup, on behalf of the status report team.
__________________________________________________________________
FreeBSD Team Reports
* Continuous Integration
* FreeBSD Core Team
* FreeBSD Foundation
* FreeBSD Graphics Team status report
* FreeBSD Release Engineering Team
* Ports Collection
Projects
* amd64 Usermode Protection Keys
* bhyve - Live Migration
* bhyve - Save/Restore
* Capsicum
* Collection of vt(4) color schemes
* i386 PAE Pagetables
* Improving FreeBSD boot security
* pfsync performance improvement
* PWM Kernel API and userland utility
Architectures
* Broadcom ARM64 SoC support
* DTS Update
* ENA FreeBSD Driver Update
* FreeBSD on Power9 (ppc64) Parity
* FreeBSD/RISC-V update
* libvdsk - QCOW2 implementation
* Marvell 8K SoC support
* Pinebook SDCard Image
* RockChip Support
Ports
* FreeBSD KDE status report
Miscellaneous
* BSD PL
Third-Party Projects
* ClonOS: virtualization platform on top of FreeBSD Operating System
* HardenedBSD 2018Q4 Update
* The nosh project
__________________________________________________________________
FreeBSD Team Reports
Entries from the various official and semi-official teams, as found in
the Administration Page.
Continuous Integration
Links
FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org
FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org/
FreeBSD Jenkins wiki URL: https://wiki.freebsd.org/Jenkins
freebsd-testing Mailing List URL:
https://lists.FreeBSD.org/mailman/listinfo/freebsd-testing
freebsd-ci Repository URL: https://github.com/freebsd/freebsd-ci
Tickets related to freebsd-testing@ URL:
https://preview.tinyurl.com/y9maauwg
Hosted CI wiki URL: https://wiki.freebsd.org/HostedCI
Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
The FreeBSD CI team maintains continuous integration system and related
tasks for the FreeBSD project. The CI system regularly checks the
changes committed to the project's Subversion repository can be
successfully built, and performs various tests and analysis over the
results. The results from build jobs are archived in artifact server,
for the further testing and debugging needs.
The members on the CI team examine the failing builds and unstable
tests, and work with the experts in that area to fix the code or build
and test infrastructure, to improve the software quality of the FreeBSD
base system. The CI team member and the FreeBSD foundation staff Li-Wen
is the maintainer of Jenkins and Jenkins related ports.
In this quarter, we worked on extending test executing environment to
improve the coverage, temporarily disabling flakey test cases (and
opening tickets to work with domain experts). Please see
freebsd-testing@ related tickets for more information.
In addition to that, starting from this quarter, we also work on
collaboration with external projects to extend their CI to cover
FreeBSD. See "HostedCI" wiki page for more information.
Work in progress:
* Fixing the failing test cases and builds
* Adding drm ports building test against -CURRENT
* Adding tests for selected project branches, e.g.: clang800-import
* Implementing automatic tests on bare metal hardware
* Planning the embedded testbed
* Planning running ztest and network stack tests
__________________________________________________________________
FreeBSD Core Team
Contact: FreeBSD Core Team <core@FreeBSD.org>
Noteworthy events since the last quarterly report:
* Yuri Pankov (yuripv@) was awarded a source commit bit under the
mentorship of Konstantin Belousov (kib@).
* Core agrees that portmgr@ may enforce a 12-month commit bit
expiration for ports committers.
* Thomas Munro (tmunro@) was awarded a source commit bit under the
mentorship of Mateusz Guzik (mgj@) and co-mentorship of Allan Jude
(allanjude@).
* With the approval of FCP-0101, 10/100 Ethernet drivers will be
deprecated.
* Core approved the promotion of Remko Lodder (remko@) to Deputy
Security Officer.
__________________________________________________________________
FreeBSD Foundation
Contact: Deb Goodkin <deb@FreeBSDFoundation.org>
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated
to supporting and promoting the FreeBSD Project and community
worldwide. Funding comes from individual and corporate donations and is
used to fund and manage software development projects, conferences and
developer summits, and provide travel grants to FreeBSD contributors.
The Foundation purchases and supports hardware to improve and maintain
FreeBSD infrastructure and provides resources to improve security,
quality assurance, and release engineering efforts; publishes marketing
material to promote, educate, and advocate for the FreeBSD Project;
facilitates collaboration between commercial vendors and FreeBSD
developers; and finally, represents the FreeBSD Project in executing
contracts, license agreements, and other legal arrangements that
require a recognized legal entity.
Here are some highlights of what we did to help FreeBSD last quarter:
Partnerships and Commercial User Support
As a 501(c)(3) non-profit, we don't directly support commercial users,
but we do work with them to understand their needs and help facilitate
collaboration with the community. Last quarter, we were able to meet
with a number of FreeBSD users and supporters at the October FreeBSD
Developer Summit and MeetBSD conference in addition to our regular
company meetings. These in-person meetings provide the opportunity to
discuss pain points, identify how they can contribute back to FreeBSD,
talk about what technologies they would like to see supported, and what
can be done to support FreeBSD over more of their technologies and
products.
Fundraising Efforts
By end of last year, we raised over $1.3M and were able to add Juniper,
Netflix and Facebook and Handshake.org to our list of Foundation
Partners. You can view the entire list here
https://www.freebsdfoundation.org/donors/. We are still finalizing
total donations, and will report the final numbers in early February.
Thank you to everyone who supported our efforts in 2018.
OS Improvements
In the fourth quarter of 2018 six authors made a total of 315 commits
to the FreeBSD development tree that were identified as being sponsored
by the FreeBSD Foundation. These included staff members Konstantin
Belousov, Glen Barber, Li-Wen Hsu and Ed Maste, and grant recipients
Mateusz Guzik and Mark Johnston.
Mateusz' work over the quarter consisted of identifying and fixing
bottlenecks in the FreeBSD kernel and system libraries. The FreeBSD
base system build, and ports built via Poudriere, were used as
motivating cases.
Mark added an in-kernel Intel CPU microcode loader. This simplifies and
increases the robustness of microcode updates, which is increasingly
important as mitigations for speculative execution vulnerabilities are
delivered in microcode.
Mark also fixed a number of issues relating to capsicum support in base
system utilities, implemented a number of NUMA enhancements and bug
fixes, and fixed a number of high profile kernel bugs.
Ed committed a large number of tool chain fixes to LLVM's lld linker
and ELF Tool Chain components.
Along with several FreeBSD developers Ed worked on the OpenSSL 1.1.1
import in preparation for FreeBSD 12.0, including incorporating OpenSSH
and ntp changes for compatibility. Ed also added build-time knobs for
to enable userland retpoline and to enable BIND_NOW which can be used
as part of a vulnerability mitigation strategy.
Continuous Integration and Quality Assurance
The Foundation provides a full-time staff member who is working on
improving our automated testing, continuous integration, and overall
quality assurance efforts.
During the fourth quarter of 2018, Foundation employee Li-Wen Hsu
continuously worked on improving the project's CI infrastructure,
examining the failing build and test cases, and work with other teams
in the project for their testing needs. In this period, we also worked
on collaboration with external projects to improve their CI on FreeBSD.
See the FreeBSD CI section of this report for more information.
Release Engineering
The Foundation provides a full-time staff member to lead the release
engineering efforts. This has provided timely and reliable releases
over the last five years. During the fourth quarter of 2018, Glen
Barber led the the FreeBSD Release Engineering team in continuing
working on the 12.0-RELEASE with the official announcement sent
December 11.
See the FreeBSD Release Engineering Team section of this report for
more information.
Supporting FreeBSD Infrastructure
The Foundation provides hardware and support to improve the FreeBSD
infrastructure. Last quarter, we continued supporting FreeBSD hardware
located around the world.
FreeBSD Advocacy and Education
A large part of our efforts are dedicated to advocating for the
Project. This includes promoting work being done by others with
FreeBSD; producing advocacy literature to teach people about FreeBSD
and help make the path to starting using FreeBSD or contributing to the
Project easier; and attending and getting other FreeBSD contributors to
volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD
presentations.
The FreeBSD Foundation sponsors many conferences, events, and summits
around the globe. These events can be BSD-related, open source, or
technology events geared towards underrepresented groups. We support
the FreeBSD-focused events to help provide a venue for sharing
knowledge, to work together on projects, and to facilitate
collaboration between developers and commercial users. This all helps
provide a healthy ecosystem. We support the non-FreeBSD events to
promote and raise awareness of FreeBSD, to increase the use of FreeBSD
in different applications, and to recruit more contributors to the
Project.
Some of the advocacy and education work we did last quarter includes:
* Organized, sponsored, and presented at the October 2018 FreeBSD
Developers Summit in Santa Clara, CA
* Sponsored and exhibited at MeetBSD 2018 in Santa Clara, CA
* Exhibited for the first time at All Things Open in Raleigh, NC
* Exhibited and sponsored as an Industry Partner at LISA' 18 in
Nashville, TN
* Sponsored USENIX OSDI `18 in Carlsbad, CA as an Industry Partner
* Held an Intro to FreeBSD workshop and a "Why You Should Contribute
to FreeBSD" talk at the Rocky Mountain Celebration of Women in
Computing in Lakewood, Colorado
We continued producing FreeBSD advocacy material to help people promote
FreeBSD around the world.
Read more about our conference adventures in the conference recaps and
trip reports in our monthly newsletters:
https://www.freebsdfoundation.org/news-and-events/newsletter/
We help educate the world about FreeBSD by publishing the
professionally produced FreeBSD Journal. We recently announced that the
FreeBSD Journal will become a Free publication with the
January/February 2019 issue.
https://www.FreeBSDfoundation.org/journal/.
You can find out more about events we attended and upcoming events at
https://www.FreeBSDfoundation.org/news-and-events/.
For a look back at all of efforts in 2018, please see the year-end
video at
https://www.freebsdfoundation.org/blog/thank-you-for-supporting-freebsd/.
Legal/FreeBSD IP
The Foundation owns the FreeBSD trademarks, and it is our
responsibility to protect them. We also provide legal support for the
core team to investigate questions that arise. Last quarter, we
approved 6 requests to use the Trademark. Go to
http://www.FreeBSDfoundation.org to find out how we support FreeBSD and
how we can help you!
__________________________________________________________________
FreeBSD Graphics Team status report
Links
Project GitHub page URL: https://github.com/FreeBSDDesktop
Contact: FreeBSD Graphics Team <x11@freebsd.org>
Contact: Niclas Zeising <zeising@freebsd.org>
The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD
graphics stack. This includes graphics drivers, graphics libraries such
as the MESA OpenGL implementation, the X.org xserver with related
libraries and applications, and Wayland with related libraries and
applications.
In the forth quarter, the team focused on stablizing the graphics
drivers and ports for the FreeBSD 12.0 release. The graphics drivers
have been updated with new versions for both FreeBSD 11.2 and FreeBSD
12.0. The ports have been renamed in order to make it clearer which
version of a port runs on which version on FreeBSD. We also created a
new meta port, graphics/drm-kmod, which will install the correct driver
based on FreeBSD version and architecture. Moving forward this is the
recommended way to install the FreeBSD graphics drivers.
The DRM drivers themselves are named graphics/drm-current-kmod and
graphics/drm-fbsd12.0-kmod for CURRENT and 12.0 respectively, both of
which have been updated to use the 4.16 Linux Kernel source. For
FreeBSD 11.2 we have graphics/drm-fbsd11.2-kmod which uses the 4.11
Linux Kernel source. Finally, we created graphics/drm-legacy-kmod,
which works on FreeBSD 12.0 and CURRENT. This is a copy of the legacy
drivers from the FreeBSD base system. This work will make it possible
for us to remove the drm2 code from CURRENT, something we are planning
to do in early February. A remnant of the drm2 code will remain in the
base after this due to an unresolved dependency for the NVIDIA Tegra
ARM chip. Plans for its migration are expected to be finalized in first
quarter in 2019.
Support for i386 and PowerPC 64 has been added to the drm kernel
drivers. This is currently in an alpha state.
Wayland has been enabled by default in the ports tree, meaning that all
packages are build with Wayland support enabled. This makes it much
easier to use and test Wayland.
Support for VMware graphics pass through has been added to the kernel
driver. Support for this is still missing in graphcs/mesa-dri though,
so it currently does not work out of the box.
The input stack has been updated and is now for the most part current
with upstream. Evdev headers were split off from multimedia/v4l_compat
into their own port, devel/evdev-proto. This makes it easier to update
those headers and keep them current with upstream, as needed. The input
stack is still an area where more work needs to be done to make it
easier to use various input devices with X and Wayland on FreeBSD.
Several meetings has been held over the course of the period. Meeting
notes have been sent out to the public x11@FreeBSD.org mailing list.
People who are interested in helping out can find us on the
x11@FreeBSD.org mailing list, or on our gitter chat:
https://gitter.im/FreeBSDDesktop/Lobby. We are also available in
#freebsd-xorg on EFNet.
We also have a team area on GitHub where our work repositories can be
found: https://github.com/FreeBSDDesktop
__________________________________________________________________
FreeBSD Release Engineering Team
Category: team
The FreeBSD Release Engineering Team is responsible for setting and
publishing release schedules for official project releases of FreeBSD,
announcing code freezes and maintaining the respective branches, among
other things.
During the fourth quarter of 2018, the FreeBSD Release Engineering team
continued working on the 12.0-RELEASE. The stable/12 branch was created
on October 19, with the first BETA build being started shortly after.
The release cycle slipped slightly with the addition of 12.0-BETA4,
after which the releng/12.0 branch was created on November 16.
The remainder of the release cycle continued relatively smoothly for
the duration of the release candidate (RC) phase, with the final
release builds starting December 7, and the official announcement sent
December 11.
Throughout the quarter, several development snapshots builds were
released for the head and stable/11 branches.
Much of this work was sponsored by the FreeBSD Foundation.
__________________________________________________________________
Ports Collection
Links
About FreeBSD Ports URL: https://www.FreeBSD.org/ports/
Contributing to Ports URL:
https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html
FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/index.html
Ports Management Team">Ports Management Team URL:
https://www.freebsd.org/portmgr/index.html
Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>
The number of ports in the last quarter shrunk a bit to 32,900. At the
end of the quarter there were 2365 open port PRs of which a small 500
were unassigned. The last quarter saw 7333 commits from 174 committers.
This means that more port PRs were resolved than last quarter and the
number of commits remained approximately the same.
During the last quarter, we welcomed Alexandre C. Guimarães
(rigoletto@) and Vinícius Zavam (egypcio@). The port commit bits of
Alberto Villa (avilla@), Lars Thegler (lth@), Dryice Dong Liu
(dryice@), Ion-Mihai Tetcu (itetcu@), Gabor Pali (pgj@), Tom Judge
(tj@), Ollivier Robert (roberto@), and Maxim Sobolev (sobomax@) were
taken in for safekeeping.
The number of commit bits safekept is higher than usual because for
port commit bits the idle timeout changed from 18 months to 12 months.
Some default versions were changed:
* PHP from 7.1 to 7.2
* Perl5 from 5.26 to 5.28
* Ruby from 2.4 to 2.5
* For LLVM, version 7.0 is now supported as a default version.
Other big changes are:
* info files are stored in the share/info directory just as other
operating systems do.
* PyQt ports can now be installed concurrently.
* As FreeBSD 10 reached its end of life, support for this branch has
been removed from the Ports Collection. People still requiring
FreeBSD 10 support can use the RELEASE\_10\_EOL tag.
* USES=cmake now defaults to outsource
* KDE 4 has reached its end-of-life and has been removed from the
Ports Collection.
Eager as ever, antoine@ ran 36 exp-runs this quarter to ensure major
port upgrades were correct.
__________________________________________________________________
Projects
Projects that span multiple categories, from the kernel and userspace
to the Ports Collection or external projects.
amd64 Usermode Protection Keys
Links
The patch URL: https://reviews.freebsd.org/D18893
Contact: Konstantin Belousov <kib@freebsd.org>
Skylake Xeons have a new feature in 4-level paging implementation
called Usermode Protection Keys. It is a complementary page access
permission management mechanism, which provides very low-overhead
disabling of all accesses or only modifications, on per-page basis.
Each thread of execution gets 16 slots, called protection keys, while
each userspace page mapping is tagged with one key. Processor provides
a new 32bit register PKRU, which holds access and modification disable
bits per key, the PKRU register is automatically context-switched, and
managed by userspace using RDPKRU and WRPKRU instructions. See Intel
SDM rev. 68 Vol 3 4.6.2 Protection Keys for further details.
Since a key index must be always specified, this makes the key zero a
default key, which permissions are tricky to modify without breaking
the process environment. The rest 15 keys are usable, for instance
process might put some sensitive data like decoded private key into the
key protected area, and only enable access on as needed basis, without
issuing costly mprotect(2) syscall. Note that permissions are enforced
even for kernel access, so sneaky read(2) from other thread is subject
to the same permission checks.
I implemented the support for the amd64 pmap and provided convenient
wrappers in libc both for 64bit and 32bit processes. Prototypes for the
API are presented below and their use should be obvious from the
explanation.
int x86_pkru_get_perm(unsigned int keyidx, int access, int modify); int
x86_pkru_set_perm(unsigned int keyidx, int access, int modify); int
x86_pkru_protect_range(void *addr, unsigned long len, unsigned int
keyidx, int flag); int x86_pkru_unprotect_range(void *addr, unsigned
long len);
This project was sponsored by The FreeBSD Foundation.
__________________________________________________________________
bhyve - Live Migration
Links
Github wiki - How to Warm Migrate a bhyve guest URL:
https://github.com/FreeBSD-UPB/freebsd/wiki/Virtual-Machine-Migration-using-bhyve
Github - Warm Migration branch URL:
https://github.com/FreeBSD-UPB/freebsd/tree/projects/bhyve_migration
Github - Live Migration branch URL:
https://github.com/FreeBSD-UPB/freebsd/tree/projects/bhyve_migration_dev
Contact: Elena Mihailescu <elenamihailescu22@gmail.com>
Contact: Darius Mihai <dariusmihaim@gmail.com>
Contact: Sergiu Weisz <sergiu121@gmail.com>
Contact: Mihai Carabas <mihai@freebsd.org>
The Migration feature uses the Save/Restore feature to migrate a bhyve
guest from a FreeBSD host to another FreeBSD host. To migrate a bhyve
guest, one needs to start an empty guest on the destination host from a
shared guest image using the bhyve tool with the -R option followed by
the source host IP and the port to listen to migration request. On the
source host, the migration is started by executing the bhyvectl command
with the --migrate or --migrate-live option, followed by the
destination host IP and the port to send to the messages.
New features added:
* Prove that live migration cannot be implemented using the FreeBSD's
Copy-on-Write mechanism;
* Add --migrate-live option to bhyvectl;
* Add additional message exchange between source and destination host
to establish the migration type and the number of rounds;
* Implement a dirty-bit approach for live migrating the guest's wired
memory;
Future tasks:
* Clear the dirty bit after each migration round;
* Extend live migration to highmem segment;
* Extend live migration to unwired memory;
This project was sponsored by Matthew Grooms.
__________________________________________________________________
bhyve - Save/Restore
Links
Github repository for the save/restore and migration features URL:
https://github.com/FreeBSD-UPB/freebsd/tree/projects/bhyve_migration
Github wiki - How to Save and Restore a bhyve guest URL:
https://github.com/FreeBSD-UPB/freebsd/wiki/Save-and-Restore-a-virtual-machine-using-bhyve
Github wiki - Suspend/resume test matrix URL:
https://github.com/FreeBSD-UPB/freebsd/wiki/Suspend-Resume-test-matrix
Contact: Elena Mihailescu <elenamihailescu22@gmail.com>
Contact: Darius Mihai <dariusmihaim@gmail.com>
Contact: Sergiu Weisz <sergiu121@gmail.com>
Contact: Mihai Carabas <mihai@freebsd.org>
The Save/Restore for bhyve feature is a suspend and resume facility
added to the FreeBSD/amd64's hypervisor, bhyve. The bhyvectl tool is
used to save the guest state in three files (a file for the guest
memory, a file for devices' and CPU's state and another one for some
metadata that are used in the restore process). To suspend a bhyve
guest, the bhyvectl tool must be run with the --suspend
<state_file_name> option followed by the guest name.
To restore a bhyve guest from a checkpoint, one simply has to add the
-r option followed by the main state file (the same file that was given
to the --suspend option for bhyvectl) when starting the VM.
New features added:
* Improve timers' save and restore state feature;
* Fix synchronization issues related to the ahci device save and
restore state feature;
* Add suspend/resume support for Windows guests;
* Refactor save and restore code - save component's state field by
field
Future tasks:
* Open ticket on Phabricator;
* Add suspend/resume support for nvme;
* Add suspend/resume support for virtio-console;
* Add suspend/resume support for virtio-scsi;
* Add TSC offseting for restore for AMD CPUs;
This project was sponsored by Matthew Grooms; iXsystems;.
__________________________________________________________________
Capsicum
Links
Capsicum Wiki Page URL: https://wiki.FreeBSD.org/Capsicum
Contact: Mark Johnston <markj@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Mariusz Zaborski <oshogbo@FreeBSD.org>
The major improvement in Capsicum is introducing a Casper service
fileargs, which is an easy way helps to sandbox the utils which need
access to the filesystem. There are several examples of usage fileargs
in applications like brandelf(1), wc(1), savecore(1), head(1) and
strings(1). The fileargs service also helps to bring new features to
the bhyve like audio device which is secured using Capsicum.
Another big step was introducing a private Casper service and
sandboxing the rtsold(8) and rtsol(8).
Next major improvement, which is still under the review, is rewriting
the sysctl service. The new sysctl service will allow in an easy way to
use cap_sysctl() and cap_sysctlnametomib().
__________________________________________________________________
Collection of vt(4) color schemes
Links
iTerm2-Color-Schemes repository with previews URL:
https://github.com/mbadolato/iTerm2-Color-Schemes
iTerm2-Color-Schemes vt color schemes URL:
https://github.com/mbadolato/iTerm2-Color-Schemes/tree/master/freebsd_vt
Contact: Tobias Kortkamp <tobik@FreeSD.org>
Since 11.2-RELEASE vt(4) supports setting custom color schemes via the
kern.vt.color.X.rgb tunables. This is nice but what was missing were
some ready to use themes.
iTerm2-Color-Schemes is a collection of around 200 color schemes for
various terminals. It has recently gained support for vt(4).
Customizing your console is now as easy as copy and pasting your
favorite theme to /boot/loader.conf or /boot/loader.conf.local.
__________________________________________________________________
i386 PAE Pagetables
Links
https://reviews.freebsd.org/D18894 URL: https://reviews.freebsd.org/D18894
Contact: Konstantin Belousov <kib@freebsd.org>
The i386 architecture (in modern terms, x86 architecture in 32bit
protected mode), has supported hardware execute-disable since early
200x. The only problem preventing the i386 FreeBSD kernel from using it
was that default page table format used by the kernel is 2-level
paging, while nx bit is only available for PAE (2.5 levels) page table
structures. PAE option is too intrusive, it changes both vm_paddr_t and
bus_addr_t to 64bit, which is not too friendly to many drivers.
I tried to provide PAE_PAGETABLES kernel option which only changed page
table format, without affecting vm_paddr_t or bus_addr_t, thus keeping
kernel/driver interfaces intact. But I was not able to make i386
releases carry two kernels, one to support relic hardware which cannot
use PAE pagetables, and another for newer machines.
So I finally did a merge which makes single i386 kernel carry two pmap
modules, one for PAE and one for old two-level paging structures. Also
I did not find a reason to not expand vm_paddr_t, while I have to keep
bus_addr_t at 32bit.
With a single boot-time knob, i386 kernel can now also utilize up to
24G or memory, if drivers correctly use busdma(9). I tried to fix
iflib(4) and ahci(4) so that the most common hardware work, but I
cannot do the pass over the whole tree.
Hopefully, together with earlier 4/4G split work, this gives enough
life for i386 kernel.
This project was sponsored by The FreeBSD Foundation.
__________________________________________________________________
Improving FreeBSD boot security
Links
TPM 2.0 driver URL:
https://svnweb.freebsd.org/base/head/sys/dev/tpm/tpm20.c?revision42084&nview=markup
Loader Secure Boot support URL: https://reviews.freebsd.org/D18798
Secure Boot library URL: https://reviews.freebsd.org/D18797
binsign utility URL: https://reviews.freebsd.org/D18799
Contact: Michal Stanek <mst@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>
Contact: Kornel Duleba <mindal@semihalf.com>
FreeBSD now supports TPM 2.0 devices. TPM (Trusted Platform Module) is
a discrete chip which provides secure computation and secure NVRAM
storage. It is most commonly associated with Measured Boot i.e.
providing hash measurements of boot elements such as firmware images
and boot configuration to the OS. In FreeBSD, the TPM can be used to
strengthen security of services such as Strongswan IPsec, SSH or TLS by
performing cryptographic operations in the TPM chip itself using
embedded keys inaccessible to software. TPM facilities such as secure
NVRAM storage, data sealing, random number generation and others are
also available to any software via the IBM TSS library.
UEFI Secure Boot is a technology which provides authentication of
images that are executed on the host during boot. This prevents
persistence of unauthorized malicious boot code such as rootkits. UEFI
stores a list of allowed and blacklisted certificates and verifies
signatures of all boot images and UEFI applications before they are
executed on the CPU. Semihalf has developed support for X509
certificates and signature verification code in EFI loader with the
help of the minimal BearSSL library. Lists of allowed and forbidden
certificates are retrieved from UEFI environmental variables. This
allows users to sign kernel binaries with a self-signed certificate,
append the signature and let the loader verify its authenticity.
UEFI Secure Boot support code will most likely be integrated with sjg's
Veriexec support which is currently being reviewed on Phabricator.
Semihalf is also working on improving security of Veriexec by moving
manifest signature verification to the kernel.
This project was sponsored by Stormshield.
__________________________________________________________________
pfsync performance improvement
Contact: Kristof Provost <kp@freebsd.org>
While pf itself can operate on multiple states simultaneously (on
different cores), pfsync could not. It used a single PFSYNC_LOCK. This
greatly reduced throughput on multicore systems as soon as pfsync was
loaded.
This was improved by splitting the pfsync queues into buckets, based on
the state ID. This ensures that updates for a given connection always
end up in the same bucket, allowing pfsync to still collapse multiple
updates into one, while allowing multiple cores to proceed at the same
time. The buckets are independently locked, allowing multiple cores to
proceed at once.
The number of buckets is tunable, but defaults to twice the number of
cpus. Benchmarking has shown improvement of 30 to 100% depending on
hardware and setup.
During this effort several vnet-related issues were fixed as well, and
a basic pfsync test case was added.
This was committed into head in r341646, and later merged into
stable/12 and stable/11.
This project was sponsored by Orange Business Services.
__________________________________________________________________
PWM Kernel API and userland utility
Contact: Emmanuel Vadot <manu@FreeBSD.org>
A new subsystem was added to the kernel for PWM drivers to register
themselves. In pair with the kernel subsystem, a pwm(8) utility is also
available so users can configure PWM on their embedded boards. For now
the only PWM driver compatible with this subsystem is for ARM Allwinner
SoCs.
__________________________________________________________________
Architectures
Updating platform-specific features and bringing in support for new
hardware platforms.
Broadcom ARM64 SoC support
Contact: Michal Stanek <mst@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>
Semihalf has recently started work on FreeBSD support for BCM5871X SoC
series.
These are quad-core 64-bit ARMv8 Cortex-A57 communication processors
targeted for networking applications such as 10G routers, gateways,
control plane processing and NAS. Initial support will include iProc
PCIe, internal BNXT Ethernet controller, OTP (One Time Programmable
memory) and crypto engine acceleration for IPsec offloading. This work
is expected to be ready for FreeBSD-HEAD before Q3 2019.
This project was sponsored by Juniper.
__________________________________________________________________
DTS Update
Contact: Emmanuel Vadot <manu@FreeBSD.org>
DTS files (Device Tree Sources) were updated to be on par with Linux
4.20 for head and 4.19 for the 12-STABLE branch.
The DTS are now compiled for some arm64 boards, as the one present in
U-Boot are not always up-to-date.
__________________________________________________________________
ENA FreeBSD Driver Update
Links
ENA README URL:
https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README
Contact: Michal Krawczyk <mk@semihalf.com>
ENA (Elastic Network Adapter) is the smart NIC which is used in the
virtualized environment of Amazon Web Services (AWS). It supports
multiple queues and can handle up to 25 Gb/s, depending on the instance
type on which it is used.
ENAv2 has been under development for FreeBSD, similar to Linux OS and
DPDK. New changes are including:
* Upgrade of the HAL to the version supporting ENAv2
* Optimization of the logging on the Tx path
* LLQ (Low Latency Queue) feature, which is reducing latency on
instances supporting ENAv2
* Optimization of the locks on hot paths by adding Tx queue
management and lockless Rx queue cleanup
* Fixes on the error handling paths
* Use bitfield for tracking device states
* Add additional doorbells on Tx path
* Add queue depth setup in the runtime and allows Rx queue depth to
be configured independently
* And more minor bug fixes and code reorganization
Todo:
* Internal review and validation
* Upstream of the patches
This project was sponsored by Amazon.com Inc.
__________________________________________________________________
FreeBSD on Power9 (ppc64) Parity
* NMI semantics: NMIs need to be emulated by only soft disabling
interrupts, disabling interrupts blocks all interrupts except
machine check exceptions and system resets.
* Superpage support is stable and on by default in the POWER9BSD
staging branch
* NUMA support: Parse OFW and set up appropriate structures for
memory to be allocated from the correct domain and interrupts to be
bound to the correct socket.
* LKPI support for POWER9, Drm-next supports radeonkms. Some
additional big endian changes required for amdgpu.
* Interrupt handling improvements resulting in up to a 10% reduction
in buildkernel time.
* Cached XICS IPI vector
* Added XIVE exploitation mode driver
* Rust support in review.
* Successfully booted an LLVM compiled kernel.
__________________________________________________________________
FreeBSD/RISC-V update
Contact: Ruslan Bukin <br@freebsd.org>
Contact: Mark Johnston <markj@freebsd.org>
FreeBSD/RISC-V is getting more mature during last quarter.
We have optimised RISC-V copyin(9)/copyout(9) routines. They now
support word-sized copies where possible to dramatically increase speed
of copying data between kernel and userspace.
We made a series of improvements and bug fixes to pmap support
(machine-dependent portion of virtual memory subsystem). This part was
not touched during the last years, and is now getting attention.
RISC-V GENERIC kernel gets support for witness(4) (The FreeBSD lock
validation facility).
The British company Embecosm has reported that they were able to boot
FreeBSD on real hardware -- a SiFive Unleashed board. The support is
limited to a single core only. We are expecting patches from them.
__________________________________________________________________
libvdsk - QCOW2 implementation
Links
Github - Libvdsk QCOW2 branch URL:
https://github.com/FreeBSD-UPB/freebsd/tree/projects/bhyve_libvdsk
Contact: Sergiu Weisz <sergiu121@gmail.com>
Contact: Marcelo Araujo <araujo@freebsd.org>
Contact: Mihai Carabas <mihai@freebsd.org>
New features added:
* Extend libvdsk to make it easier to implement new formats;
* Implement read/write/probe functionalities in order to parse QCOW2
image files;
Future tasks:
* Add support for Copy-On-Write;
* Add support for multiple snapshots;
* Integrate libvdsk in bhyve
This project was sponsored by Matthew Grooms.
__________________________________________________________________
Marvell 8K SoC support
Contact: Emmanuel Vadot <manu@FreeBSD.org>
Contact: Luis Octavio O Souza <loos@FreeBSD.org>
Support for booting FreeBSD on Marvell 8K SoC (present on the
MacchiatoBin for example) has been commited. As of today, clocks, gpio,
thermal, sdcard/eMMC drivers has been commited. SATA and USB were
already working.
This project was sponsored by Rubicon Communications, LLC ("Netgate").
__________________________________________________________________
Pinebook SDCard Image
Contact: Emmanuel Vadot <manu@FreeBSD.org>
SDCard image is now produced for the Pinebook. By default the console
is directed in the EFI Framebuffer and the serial console.
__________________________________________________________________
RockChip Support
Contact: Emmanuel Vadot <manu@FreeBSD.org>
Early support for the RockChip RK3399 has been commited. For now it's
only possible to netboot boards (Like the RockPro64). Original patch
was submitted by Greg V <greg@unrelenting.technology>.
Support for the RK805 and RK808 PMIC (Power Management IC) has been
added. This allow changing some regulators voltage such as the cores
one so cpufreq support works. You can change core frequencies with
sysctl or powerd(8).
__________________________________________________________________
Ports
Changes affecting the Ports Collection, whether sweeping changes that
touch most of the tree, or individual ports themselves.
FreeBSD KDE status report
Links
KDE FreeBSD URL: https://freebsd.kde.org/
Contact: Adriaan de Groot <adridg@FreeBSD.org>
Contact: Tobias C. Berner <tcberner@FreeBSD.org>
First of all, we removed KDE 4 from the ports tree this quarter. Qt4
will follow it by the end of march.
Thanks to the update of libinput in ports we could finally update
Plasma Desktop past 5.12, and are now again in sync with the upstream
releases.
KDE Frameworks and Applications were also kept in sync with upstream.
We've also updated Qt5 to 5.12 -- with QtWebEngine still hanging on on
5.9.5 for now, but thanks to a new contributor we should have 5.12 by
the end of Q1.
In the background we changed the default behavior of cmake in the ports
tree to default to outsource builds.
People who are willing to contribute can find us on #kde-freebsd on
freenode, and the kde@FreeBSD.org mailing list. Further we accept
pull-requests and contributions on
github.com/freebsd/freebsd-ports-kde.
__________________________________________________________________
Miscellaneous
Objects that defy categorization.
BSD PL
Links
https://bsd-pl.org/ URL: https://bsd-pl.org/
Contact: Official <meetup@bsd-pl.org>
Contact: Konrad Witaszczyk <def@FreeBSD.org>
Contact: Mariusz Zaborski <oshogbo@FreeBSD.org>
Contact: Jarosl/aw Zurek <contact@zurek.pro>
The Polish BSD User group is an initiative promoting systems from the
BSD family. We organize both meetings and as well as tutorial sessions.
In general, we have three presentations which last around 15 minutes.
Afterwards there's an open discussions about topics related to
operating systems and security. There's something for everybody, and
the first presentation is about something connected to BSD and it's
aimed at beginners. The second presentation is for more advanced BSD
users but the final talk is more general and not connected to BSD.
Usually it covers an interesting topic related to technology. Everyone
can suggest a subject for the presentations and discussions. Some
presentations from the past were about: ZFS checkpoints, GELI, FreeNAS,
PAM, DTrace, Yubikey, Pytest, ZeroTrust, Jenkins and the iocage
training session. Hope to see you there!
__________________________________________________________________
Third-Party Projects
Many projects build upon FreeBSD or incorporate components of FreeBSD
into their project. As these projects may be of interest to the broader
FreeBSD community, we sometimes include brief updates submitted by
these projects in our quarterly report. The FreeBSD project makes no
representation as to the accuracy or veracity of any claims in these
submissions.
ClonOS: virtualization platform on top of FreeBSD Operating System
Links
ClonOS Main Site URL: https://clonos.tekroutine.com
Contact: Oleg Ginzburg <olevole@olevole.ru>
What is ClonOS?
ClonOS is a turnkey open-source platform based on FreeBSD and the CBSD
framework. ClonOS offers a complete web UI for an easy control,
deployment and management of FreeBSD jails containers and bhyve/Xen
hypervisor virtual environments.
ClonOS is currently the only available platform which allows both Xen
and bhyve hypervisors to coexist on the same host. Since ClonOS is a
FreeBSD-based platform, it has the ability to create and manage jails
natively, allowing you to run FreeBSD applications without losing
performance.
Features:
* easy management via web UI interface
* bhyve management (create, delete VM)
* Xen management (create, delete VM) [coming soon, roadmap]
* connection to the "physical" guest console via VNC from the browser
or directly
* real time system monitoring
* access to load statistics through SQLite3 and beanstalkd
* support for ZFS features (cloning, snapshots)
* import/export of virtual environments
* public repository with virtual machine templates
* puppet-based helpers for configuring popular services
ClonOS 2018Q4 Status Report
During this period, work was carried out to:
* implement real-time graph for jail/bhyve based on RACCT statistics
* test bhyve live migration, support live migration in CBSD
* prepare ClonOS 19.01-RELEASE
Open task:
* ClonOS roadmap: https://clonos.tekroutine.com/roadmap.html
* FreeNAS/XigmaNAS or any other NAS integration
* I would like to see ClonOS in real-world use. In this regard, I am
interested in finding more people and companies that use FreeBSD
for vm/jail services.
__________________________________________________________________
HardenedBSD 2018Q4 Update
Links
https://hardenedbsd.org URL: https://hardenedbsd.org
Contact: Shawn Webb <shawn.webb@hardenedbsd.org>
Introduction to HardenedBSD
HardenedBSD is a security-enhanced fork of FreeBSD that aims to provide
the BSD community with a clean-room reimplementation of the
publicly-documented parts of the grsecurity patchset for Linux. We
maintain close compatibility with FreeBSD by syncing every six hours
with FreeBSD.
HardenedBSD Foundation Update
Through a generous donation by DEF CON, the computer security
conference held each year in Las Vegas, and an anonymous member of the
community, the HardenedBSD Foundation was able to provide the
HardenedBSD project with a new Cavium ThunderX2 server. HardenedBSD has
been working closely with FreeBSD's and Cavium's Jayachandran
(jchandra@freebsd) to gain working support for the ThunderX2. As soon
as the ThunderX2 becomes functional, HardenedBSD will be able to
support both 12-STABLE and 13-CURRENT for arm64.
We assisted OPNsense's migration from FreeBSD to HardenedBSD as the
base operating system. OPNsense's January 2019 release (19.1) will
complete the migration. Further work will be done to enable
HardenedBSD's PaX NOEXEC implementation in OPNsense. PaX NOEXEC is a
strong form of W^X, which prevents memory allocations from being both
writable and executable, and toggling between the two.
The HardenedBSD Foundation Corp. is a registered 501(c)(3) tax-exempt
not-for-profit charitable organization in the United States. We look
forward to a productive 2019, with work to support Cross-DSO CFI still
ongoing.
HardenedBSD 12-STABLE Released
In December 2018, HardenedBSD published is first official release of
12-STABLE. From the release announcement:
Improvements in 12-STABLE from 11-STABLE:
* Non-Cross-DSO Control-Flow Integrity (CFI) for applications on
amd64 and arm64. At this time, CFI is not applied to the kernel.
More info on CFI is below.
* Jailed bhyve (upstreamed to FreeBSD)
* Per-jail toggles for unprivileged process debugging (the
security.bsd.unprivileged_process_debug sysctl node. Upstreamed to
FreeBSD.)
* Spectre v2 mitigation with retpoline applied to the entirety of
base and ports (with only a few ports opting out.)
* Symmetric Multi-Threading (SMT) disabled by default (re-enable by
setting machdep.hyperthreading_allowed to 1 in loader.conf(5)).
* Migration of more compiler toolchain components to llvm's
implementations (llvm-ar, llvm-nm, and llvm-objdump).
* Compilation of applications with Link-Time Optimization (LTO).
Non-Cross-DSO CFI
Non-Cross-DSO CFI is an exploit mitigation technique that helps to
prevent attackers from modifying the behavior of a program and jumping
to undefined or arbitrary memory locations. Microsoft has implemented a
variant of CFI, which they term Control Flow Guard, or CFG. The PaX
team has spent the last few years perfecting their Reuse Attack
Protector, RAP. CFI, CFG, and RAP all attempt to accomplish the same
goal, with RAP being the most complete and effective implementation.
Clang's CFI is stronger than Microsoft's CFG and PaX Team's RAP is
stronger than both CFI and CFG. RAP would be a great addition to
HardenedBSD; however, it requires a GPLv3 toolchain and is patented.
Clang's CFI requires a linker that supports Link-Time Optimization
(LTO). HardenedBSD 12-STABLE ships with lld as the default linker. All
CFI schemes have been enabled for nearly all applications in base.
Please note that any application that calls function pointers resolved
via dlopen + dlsym will require the cfi-icall scheme to be disabled.
HardenedBSD is the first enterprise operating system to apply
Non-Cross-DSO CFI broadly to userland.
__________________________________________________________________
The nosh project
Links
Introduction and blurb URL: http://jdebp.eu./Softwares/nosh/
Guide URL: http://jdebp.eu./Softwares/nosh/guide.html
FreeBSD binary packages URL:
http://jdebp.eu./Softwares/nosh/freebsd-binary-packages.html
Installation how-to URL:
http://jdebp.eu./Softwares/nosh/timorous-admin-installation-how-to.html
Roadmap URL: http://jdebp.eu./Softwares/nosh/roadmap.html
Contact: Jonathan de Boyne Pollard
<J.deBoynePollard-newsgroups@NTLWorld.COM>
Background
The nosh project is a suite of system-level utilities for initializing,
running, and shutting down BSD systems; and for managing daemons,
terminals, and logging.
It supersedes BSD init, the Mewburn rc system, and OpenRC, drawing
inspiration from daemontools-encore for service control/status
mechanisms, UCSPI for networked services, Solaris SMF for named
milestones, and IBM AIX for separated service and system management. It
includes a range of compatibility mechanisms, including shims for
familiar commands from other systems, and an automatic import mechanism
that takes existing configuration data from /etc/fstab,
/etc/rc.conf{,.local}, /etc/ttys, and elsewhere, applying them to its
native service definitions and creating additional native services.
It is portable (including to Linux) and composable, it provides a
migration path from the world of systemd Linux, and it does not require
new kernel APIs. It provides clean service environments, has orderings
and dependencies between services, has parallelized startup and
shutdown (including fsck), provides strictly size-capped and
autorotated logging, has the service manager as a "subreaper", provides
per-user service management as well as system-wide, provides user-space
virtual terminals, brings TTY login under the general service
management umbrella, and uses kevent(2) for event-driven parallelism.
For more, see the aforelinked Introduction and blurb, and the nosh
Guide.
News
The project has seen a lot of development since the last status report
in 2017. To briefly touch upon just some of the things that have been
worked on:
* There are several more packages for things like running Bruce
Guenter's bcron, shims for OpenRC's rc-update and rc-service tools,
and shims for portable substitutes for a couple of Linux's
util-linux tools.
* There are quite a lot of new tools, including getuidgid,
userenv-fromenv, setgid-fromenv, envgid, printenv, setlogin,
console-decode-ecma48, console-control-sequence,
console-flat-table-viewer, console-input-method, and
local-stream-socket-connect. To look at just two of these:
* printenv as a built-in allows more convenient use in conjunction
with clearenv. It can also generate output in some additional
formats.
* console-control-sequence also responds to the name setterm, and can
do most of what the non-portable util-linux tool by that name does;
excluding the things that are specific to non-portable Linux
ioctl()s and control codes (such as display adapter power
management), but also including _extra_ standard DEC VT and ECMA-48
things that the util-linux tool does _not_ do (such as turning
strikethrough, calculator keypad application mode, mouse reports,
and the alternative screen buffer on and off).
* There are a lot of new service bundles for more services, too many
to list here. One can find them listed in the 1.37 and 1.38 + 1.39
release announcements.
* There are new chapters in the nosh Guide, on packages and ports, on
resources for terminals such as keyboard maps, input methods, and
fonts, and on how the head0 user-space virtual terminal is
structured. There are also new manual pages - in addition to the
ones for all of the new commands, of course - on the subjects of
the http://jdebp.eu./Softwares/nosh/guide/TerminalCapabilities.html
system. There are also some replacements for some Linux manual
pages that have gone missing over the past decade.
* The external format configuration import subsystem has seen some
major improvements in per-user service configuration. The per-user
service manager itself gained a control FIFO, addressing a
long-standing bug.
A particular area of improvement since the last status report is the
inclusion of input method capabilities in user-space virtual terminals.
The input method mechanism uses the same CIN files as used by several
other softwares, similar to how one can use existing SCO/FreeBSD
keyboard maps and FreeBSD vt fonts. It places a simple textual user
interface on top of a user-space virtual terminal, can switch amongst
multiple input methods on the fly, and responds to both the dedicated
keys on a JIS 106/109-key keyboard or a Korean 103/106-key keyboard and
the conventional keys used on other keyboards. The blurb includes an
example of how this works for a Japanese user, and the virtual terminal
chapters of the nosh Guide now incorporate input methods into the doco.
Another area of work was eliminating Wide NCurses from almost all of
the tools, apart from the one tool that by definition uses it
(console-ncurses-realizer). Wide NCurses has long been a porting
difficulty for several operating systems, including Gentoo Linux and
OpenBSD, and does not really model modern real world terminals and
terminal emulators very well. It has been replaced by a new
TerminalCapabilities library, in conjunction with a library for
handling ECMA-48 character sequence decoding and ECMA-48/DEC VT control
sequence generation. The decoder is the basis for the new
console-decode-ecma48 tool, for example, as well as being the decoder
for terminal input in console-termio-realizer and in full-screen TUI
tools like chkservice and the new console-flat-table-viewer.
The external formats import subsystem will also now make a replacement
/etc/system-control/convert/termcap/termcap.db that one can use, which
includes amongst other things the currently missing teken terminal
type.
Roadmap
In addition to what is on the aforelinked roadmap, several things are
on the cards for forthcoming versions. Tools that can feed the process
table into console-flat-table-viewer in the proper vis(3) form. The
ability to have different keyboard maps for different keyboards if one
has multiple keyboards. A Linux shim for login.conf. Proper handling of
CSI sub-parameters in SoftTerm. A manual page for the CIN file format.
A time-env-next-matching tool.
How you can help
* The Z shell completions now have extensive coverage of the toolset,
but there are no completions for the Bourne Again shell or the
Friendly Interactive shell. Work on such completions would be
welcome. The users who use those shells would welcome it
especially.
* The system-manager already recognizes a -b option for emergency
mode. Work to make the FreeBSD loader and kernel send such an
option to process #1, in response to an additional emergency mode
boot menu choice, would be very welcome.
* The monitor-fsck-progress and monitored-fsck tools stand ready to
work with a -C option to fsck that makes it spit out progress
information to an open file descriptor. Another way to help is to
add this capability to fsck.
* teken needs to be added to base termcap. It was put into NCurses
terminfo back in 2014.
__________________________________________________________________