Figured it out. I added Microsoft.WindowsDesktop.App.WPF as a dependency in my Class Library. In the .csproj file this changed the PropertyGroup from:
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
to:
<PropertyGroup>
<TargetFramework>net6.0-windows</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<UseWPF>true</UseWPF>
</PropertyGroup>
In my ViewModel I can now refer to the System.Windows.Documents namespace through:
using System.Windows.Documents;
and create a FlowDocument object via:
FlowDocument flowDocument = new();