I got a non working copy Shazzam Shader Editor, and there are some Dll imports to some legacy dll functions. So I thought I update it a bit using this instead:
https://msdn.microsoft.com/en-us/library/windows/desktop/hh446872(v=vs.85).aspx[
^]
So I took the original code and modified it to fit the new function:
<Guid("8BA5FB08-5195-40e2-AC58-0D989C3A0102"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)> _
Private Interface ID3DXBuffer
<PreserveSig> _
Function GetBufferPointer() As IntPtr
<PreserveSig> _
Function GetBufferSize() As Integer
End Interface
<PreserveSig> _
<DllImport("d3dcompiler_47.dll", CharSet:=CharSet.Auto, CallingConvention:=CallingConvention.Cdecl)> _
Private Shared Function D3DCompileFromFile(<MarshalAs(UnmanagedType.LPStr)> pFilename As String,
pDefines As IntPtr,
pInclude As IntPtr,
<MarshalAs(UnmanagedType.LPStr)> pEntrypoint As String,
<MarshalAs(UnmanagedType.LPStr)> pTarget As String,
flags1 As Integer,
flags2 As Integer,
ByRef ppCode As ID3DXBuffer,
ByRef ppErrorMsgs As ID3DXBuffer) As Integer
End Function
Public Sub Compile(ByVal File As HLSLFileHelperClass)
Dim pFilename As String = File.GetSourceFileFullName
Dim pDefines As IntPtr = IntPtr.Zero
Dim pInclude As IntPtr = IntPtr.Zero
Dim pEntrypoint As String = File.HLSLEntryPoint
Dim pTarget As String = File.ShaderCompilerVersion.ToString
Dim flags1 As Integer = 0
Dim flags2 As Integer = 0
Dim ppCode As ID3DXBuffer = Nothing
Dim ppErrorMsgs As ID3DXBuffer = Nothing
Dim CompileResult As Integer = 0
CompileResult = D3DCompileFromFile(pFilename,
pDefines,
pInclude,
pEntrypoint,
pTarget,
flags1,
flags2,
ppCode,
ppErrorMsgs)
If CompileResult <> 0 Then
Dim errors As IntPtr = ppErrorMsgs.GetBufferPointer()
Dim size As Integer = ppErrorMsgs.GetBufferSize()
ErrorText = Marshal.PtrToStringAnsi(errors)
IsCompiled = False
Else
ErrorText = ""
IsCompiled = True
Dim psPath = File.GetCompiledFileFullName
Dim pCompiledPs As IntPtr = ppCode.GetBufferPointer()
Dim compiledPsSize As Integer = ppCode.GetBufferSize()
Dim compiledPs = New Byte(compiledPsSize - 1) {}
Marshal.Copy(pCompiledPs, compiledPs, 0, compiledPs.Length)
Using psFile = IO.File.Open(psPath, FileMode.Create, FileAccess.Write)
psFile.Write(compiledPs, 0, compiledPs.Length)
End Using
End If
If ppCode IsNot Nothing Then
Marshal.ReleaseComObject(ppCode)
End If
ppCode = Nothing
If ppErrorMsgs IsNot Nothing Then
Marshal.ReleaseComObject(ppErrorMsgs)
End If
ppErrorMsgs = Nothing
End Sub
However using this code I get the following error:
Additional information: A call to PInvoke function 'WpfAdventuresInPixelShader!WpfAdventuresInPixelShader.ShaderCompiler::D3DCompileFromFile' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.
If there is a handler for this exception, the program may be safely continued.
If I change the attribute from this:
<DllImport("d3dcompiler_47.dll", CharSet:=CharSet.Auto, CallingConvention:=CallingConvention.Cdecl)>
to this:
<DllImport("d3dcompiler_47.dll", CharSet:=CharSet.Auto)>
I get a result from the compiler of Integer.MinValue -2.....
So what do I need to do to get this working?
(oh, there is nothing wrong with the files I want to compile, as I can do that with the fxc.exe without any problems)