mscript - Version 2.0 Adds Error Handling, New Functions, and DLL Integration

Michael Sydney Balloni

5.00/5 (2 votes)

Mar 24, 2022

Apache

5 min read

9603

137

If you were on the fence about adding mscript to your arsenal of system tools, have another look.

Download source - 83.7 KB

Introduction

This article shows improvements to mscript for version 2.0, including how DLL integration works, how JSON support was integrated, and the error handling system.

Before diving into version 2.0, if you are unfamiliar with mscript, check out the CodeProject article about version 1.0.

There is a breaking change for turning ! statements from comments into error logging.

There another breaking change, more a fixing change, to not provide the "magic" of accepting values of variables as function names. Yeah, that was a bad idea. Burned myself. I won't let it happen to you. Sorry, not a functional language, and no function pointers. I think you'll live without.

External Links

mscript is an open source project on GitHub.

For a language reference, check out mscript.io.

Error Handling

In version 1.0, you would call the error() function, and the program would print the message and quit.

In version 2.0, you still call error(). Now you use ! statements to handle those errors.

NOTE: Breaking Change: I stole ! from being used for single-line comments. You now use / for single line comments. You can use // if that makes you feel better. ;-)

Here's an example to demonstrate the error handling system:

~ verifyLow(value)
    ? value >= 10
        * error("Value is too high: " + value)
    }
}

* verifyLow(13)
! err
    > "verifyLow failed: " + err
}

mscript makes the verifyLow(13) function call.
verifyLow() sees that 13 is >= 10.
verifyLow() calls error() with a string.
mscript looks for a ! statement below the error() call.
Not finding a ! statement in verifyLow(), the error bubbles up to the calling code.
mscript scans after the verifyLow() call for a ! statement.
Finding the ! statement, it executes the ! statement with err set to whatever error() was called with.

There are no "try {} catch {}" machinations.

You can pass whatever you want to the error() function, and ! statements can use the getType() function and indexes, so error handling can be as simple or as sophisticated as you want.

With the ! statement, you can name the error handler's variable whatever you like - it doesn't have to be err, name it whatever you'd like.

You can have any number of statements before a ! statement, it's not just a ! per prior statement.

So if you have...

* do1()
* do2()
* do3()
! msg
    > "I did something wrong: " + msg
}

If do1() causes an error, mscript will scan past the do2() and do3() calls to find the ! statement.

New Functions

There are now toJson(mscriptObject) and fromJson(jsonString) functions for turning any mscript object into a JSON string, and for taking a JSON string and creating an mscript object. With the readFile and writeFile functions, you can have a settings file, for example.

There are now getEnv(name) and putEnv(name, value) functions for working with environment variables. Changes made to the environment made by putEnv() are only visible within mscript and visible by programs that mscript spawns with the exec() function. This should be useful for replacing .bat files.

There's sleep(seconds).

JSON Integration

Rather than burning object::toJson() / object::fromJson() into the object class and its header and source files, I chose to develop the new functionality in a separate module, with two functions:

objectFromJson(jsonString), and
objectToJson(object)

The header just includes object.h, no pollution of JSON code outside of this module.

For objectToJson, I hand rolled the code, as it was really simple. The only complications were escape characters in strings. The rest was easy peasy.

Parsing JSON, however, takes a bit of doing.

I chose nlohmann's json.hpp because it's trivial to integrate and I've seen it work well:

#include "../../json/single_include/nlohmann/json.hpp"

I chose the SAX-style parser for simplicity. Getting it right took a bit. The result is clean and I have not had an issue with it.

// We maintain two stacks: m_objStack for objects that can nest, like lists and indexes;
// and m_keyStack for managing the name of the current index key, to handle nested indexes 
class mscript_json_sax
{
public:
    // Start with a stack frame
    mscript_json_sax()
    {
        m_objStack.emplace_back();
    }

    // Finish reading the first frame
    object final() const
    {
        return m_objStack.front();
    }

    //
    // SAX values interface
    // Handle the little easy stuff
    //
    bool null()
    {
        set_obj_val(object());
        return true;
    }

    bool boolean(bool val)
    {
        set_obj_val(val);
        return true;
    }

    bool number_integer(json::number_integer_t val)
    {
        set_obj_val(double(val));
        return true;
    }
    bool number_unsigned(json::number_unsigned_t val)
    {
        set_obj_val(double(val));
        return true;
    }
    bool number_float(json::number_float_t val, const json::string_t&)
    {
        set_obj_val(double(val));
        return true;
    }

    bool string(json::string_t& val)
    {
        set_obj_val(toWideStr(val));
        return true;
    }

    bool binary(json::binary_t&)
    {
        raiseError("Binary values are not allowed in mscript JSON");
    }

    //
    // SAX index interface
    //

    // Push an index onto the stack
    bool start_object(std::size_t)
    {
        m_objStack.push_back(object::index());
        return true;
    }
    
    // on_end takes care of indexes and lists
    bool end_object()
    {
        return on_end();
    }

    //
    // SAX list interface
    //

    // Push a list onto the stack
    bool start_array(std::size_t)
    {
        m_objStack.push_back(object::list());
        return true;
    }

    bool end_array()
    {
        return on_end();
    }

    // Push the current index key name for later retrieval
    bool key(json::string_t& val)
    {
        m_keyStack.push_back(object(toWideStr(val)));
        return true;
    }

    // If there is a problem parsing the JSON, use our exception raising function to punt it
    bool parse_error(std::size_t pos, const std::string&, 
                     const nlohmann::detail::exception& exp)
    {
        raiseError("JSON parse error at " + std::to_string(pos) + 
                   ": " + std::string(exp.what()));
        //return false;
    }

private:
    // When we receive a new object value:
    // it could be a new index value (use the m_keyStack key);
    // it could be a new list value;
    // or it could be the value for the main top-level object
    void set_obj_val(const object& obj)
    {
        object& cur_obj = m_objStack.back();
        if (cur_obj.type() == object::INDEX)
        {
            if (m_keyStack.empty())
                raiseError("No object key in context");
            cur_obj.indexVal().set(m_keyStack.back(), obj);
            m_keyStack.pop_back();
        }
        else if (cur_obj.type() == object::LIST)
            cur_obj.listVal().push_back(obj);
        else
            cur_obj = obj;
    }
    
    // When a list or index ends, we get the top of the stack, back_obj and pop the stack.
    // The new top is top cur_obj.
    // If cur_obj is an index or list, we add back_obj to it, using the key stack for an index.
    //      This would be when the back_obj was some index or list that got started
    //      inside of another list or index, cur_obj.
    //      So when back_obj is popped, it needs to find its home in cur_obj.
    // If cur_obj is not index or list, its value is set to what was popped, back_obj.
    bool on_end()
    {
        object back_obj = m_objStack.back();
        m_objStack.pop_back();

        object& cur_obj = m_objStack.back();
        if (cur_obj.type() == object::INDEX)
        {
            if (m_keyStack.empty())
                raiseError("No object key in context");
            cur_obj.indexVal().set(m_keyStack.back(), back_obj);
            m_keyStack.pop_back();
        }
        else if (cur_obj.type() == object::LIST)
            cur_obj.listVal().push_back(back_obj);
        else
            cur_obj = back_obj;
        return true;
    }

    std::vector<object> m_objStack;
    std::vector<object> m_keyStack;
};

// With the SAX processor complete, implementing objectFromJson is straightforward
object mscript::objectFromJson(const std::wstring& json)
{
    mscript_json_sax my_sax;
    if (!json::sax_parse(json, &my_sax))
        raiseError("JSON parsing failed");
    object final = my_sax.final();
    return final;
}

DLL Integration

With mscript 2.0, developers are invited to write their own DLLs to extend mscript:

Quote:

To develop an mscript DLL...

I use Visual Studio 2022 Community Edition; Visual Studio 2019 and any edition should be fine.

Clone the mscript solution from GitHub

Clone the nlohmann JSON library on GitHub and put it alongside the mscript solution's directory, not inside it, next to it.

Get the mscript solution to build and get the unit tests to pass.

To understand DLL integration, it's best to look at the mscript-dll-sample project.

pch.h:
#pragma once
#include "../mscript-core/module.h"
#pragma comment(lib, "mscript-core")

That alone brings in everything you need for doing mscript DLL work.

Then you write a dllinterface.cpp file to implement your DLL.

Here is mscript-dll-sample's dllinterface.cpp:

#include "pch.h"

using namespace mscript;

// You implement mscript_GetExports to specify which functions you will be exporting.
// Your function names have to be globally unique, 
// and can't have dots, so use underscores and make it unique.
wchar_t* __cdecl mscript_GetExports()
{
    std::vector<std::wstring> exports
    {
        L"ms_sample_sum",
        L"ms_sample_cat"
    };
    return module_utils::getExports(exports);
}

// You need to provide a memory freeing function for strings that your DLL allocates
void mscript_FreeString(wchar_t* str)
{
    delete[] str;
}

// Here's the big one. You get a function name, and JSON for a list of parameters.
// module_utils::getNumberParams keeps this code pristine
// Use module_utils::getParams instead for more general parameter handling.
// module_utils::jsonStr(retVal) turns any object into a JSON wchar_t* 
// to return to mscript.
// module_utils::errorStr(functionName, exp) gives you consolidated error handling,
// returning an error message JSON wchar_t* that mscript expects.
wchar_t* mscript_ExecuteFunction(const wchar_t* functionName, 
                                 const wchar_t* parametersJson)
{
    try
    {
        std::wstring funcName = functionName;
        if (funcName == L"ms_sample_sum")
        {
            double retVal = 0.0;
            for (double numVal : module_utils::getNumberParams(parametersJson))
                retVal += numVal;
            return module_utils::jsonStr(retVal);
        }
        else if (funcName == L"ms_sample_cat")
        {
            std::wstring retVal;
            for (double numVal : module_utils::getNumberParams(parametersJson))
                retVal += num2wstr(numVal);
            return module_utils::jsonStr(retVal);
        }
        else
            raiseWError(L"Unknown mscript-dll-sample function: " + funcName);
    }
    catch (const user_exception& exp)
    {
        return module_utils::errorStr(functionName, exp);
    }
    catch (const std::exception& exp)
    {
        return module_utils::errorStr(functionName, exp);
    }
    catch (...)
    {
        return nullptr;
    }
}

You can tread far off this beaten path.

Process the parameter list JSON and return JSON that maps to an mscript value. That's all that's assumed.

Once you've created your own DLL, in mscript code, you import it with the same + statement as importing mscripts.

DLLs are searched for relative to the folder the mscript EXE resides in and, for security, nowhere else.

That's the overview of mscript DLL development. But how does mscript work with these DLLs?

Let's look at module.h to see what mscript provides to DLL authors.

// This header is big on convenience for the DLL author...
#pragma once

// If on Windows, bring in Windows
#if defined(_WIN32) || defined(_WIN64)
#include <Windows.h>
#endif

// Bring in all the mscript code needed to use objects
// and do JSON serialization
#include "object.h"
#include "object_json.h"
#include "utils.h"

// Define the entry points that mscript DLLs must implement
extern "C"
{
	__declspec(dllexport) wchar_t* __cdecl mscript_GetExports();
	__declspec(dllexport) void __cdecl mscript_FreeString(wchar_t* str);
	__declspec(dllexport) wchar_t* __cdecl mscript_ExecuteFunction
	          (const wchar_t* functionName, const wchar_t* parametersJson);
}

// The module_utils class has all utility code for easy mscript DLL development
// wchar_t* is the output returned from mscript_ExecuteFunction's to mscript,
// so all routines are returning freshly cloned raw character buffers
// mscript receives the buffers, stashes them in objects, 
// and calls the mscript_FreeString
// function to release the memory
namespace mscript
{
	class module_utils
	{
	public:
        // Core string cloning routine
		static wchar_t* cloneString(const wchar_t* str);
        
        // Helper for implementing mscript_GetExports
        // Just pass in your vector of function name strings, return it, 
        // you're all set
		static wchar_t* getExports(const std::vector<std::wstring>& exports);
        
        // Extract parameters from the JSON passed into the function
		static object::list getParams(const wchar_t* parametersJson);
		static std::vector<double> getNumberParams(const wchar_t* parametersJson);
        
        // Turn anything into a JSON string ready to be returned to mscript
		static wchar_t* jsonStr(const object& obj);
		static wchar_t* jsonStr(const std::string& str);
        
        // Handle exceptions cleanly by returning the output of these functions
		static wchar_t* errorStr(const std::wstring& function, 
                                 const user_exception& exp);
		static wchar_t* errorStr(const std::wstring& function, 
                                 const std::exception& exp);

	private:
		module_utils() = delete;
		module_utils(const module_utils&) = delete;
		module_utils(module_utils&&) = delete;
		module_utils& operator=(const module_utils&) = delete;
	};
}

Armed with that module, developing a mscript DLL should be straightforward:

Inside mscript, DLLs are represented by the lib class:

typedef wchar_t* (*GetExportsFunction)();
    typedef void (*FreeStringFunction)(wchar_t* str);
    typedef wchar_t* (*ExecuteExportFunction)(const wchar_t* functionName, 
                       const wchar_t* parametersJson);

class lib
{
public:
    lib(const std::wstring& filePath);
    ~lib();

    const std::wstring& getFilePath() const { return m_filePath; }
    object executeFunction(const std::wstring& name, const object::list& paramList) const;

    static std::shared_ptr<lib> loadLib(const std::wstring& filePath);
    static std::shared_ptr<lib> getLib(const std::wstring& name);

private:
#if defined(_WIN32) || defined(_WIN64)
    HMODULE m_module;
#endif
    std::wstring m_filePath;

    FreeStringFunction m_freer;
    ExecuteExportFunction m_executer;

    std::unordered_set<std::wstring> m_functions;

    static std::unordered_map<std::wstring, std::shared_ptr<lib>> s_funcLibs;
	static std::mutex s_libsMutex;
};

You create a lib with a path to the DLL, then you can execute the lib's functions.

The lib constructor does all the LoadLibrary / GetProcAddress machinations.

lib::lib(const std::wstring& filePath)
    : m_filePath(filePath)
    , m_executer(nullptr)
    , m_freer(nullptr)
    , m_module(nullptr)
{
    m_module = ::LoadLibrary(m_filePath.c_str());
    if (m_module == nullptr)
        raiseWError(L"Loading library failed: " + m_filePath);

    m_freer = (FreeStringFunction)::GetProcAddress(m_module, "mscript_FreeString");
    if (m_freer == nullptr)
        raiseWError(L"Getting mscript_FreeString function failed: " + m_filePath);

    std::wstring exports_str;
    {
        GetExportsFunction get_exports_func = 
           (GetExportsFunction)::GetProcAddress(m_module, "mscript_GetExports");
        if (get_exports_func == nullptr)
            raiseWError(L"Getting mscript_GetExports function failed: " + m_filePath);

        wchar_t* exports = get_exports_func();
        if (exports == nullptr)
            raiseWError(L"Getting exports from function mscript_GetExports failed: " + 
                          m_filePath);
        exports_str = exports;

        m_freer(exports);
        exports = nullptr;
    }

    std::vector<std::wstring> exports_list = split(exports_str, L",");
    for (const auto& func_name : exports_list)
    {
        std::wstring func_name_trimmed = trim(func_name);
        if (!isName(func_name_trimmed))
            raiseWError(L"Invalid export from function mscript_GetExports: " + 
                        m_filePath + L" - "  + func_name_trimmed);

        const auto& funcIt = s_funcLibs.find(func_name_trimmed);
        if (funcIt != s_funcLibs.end())
        {
            if (toLower(funcIt->second->m_filePath) != toLower(m_filePath))
                raiseWError(L"Function already defined in another export: function " + 
                func_name_trimmed + L" - in " + m_filePath + L" - already defined in " + 
                                    funcIt->second->m_filePath);
            else if (m_functions.find(func_name_trimmed) != m_functions.end())
                raiseWError(L"Duplicate export function: " + func_name_trimmed + L" in " + 
                            m_filePath);
            else
                continue;
        }

        m_functions.insert(func_name_trimmed);
    }

    m_executer = (ExecuteExportFunction)::GetProcAddress(m_module, "mscript_ExecuteFunction");
    if (m_executer == nullptr)
        raiseWError(L"Getting mscript_ExecuteFunction function failed: " + m_filePath);
}

Executing a DLL function uses the DLL's functions, and a little hack for dealing with DLL exceptions:

object lib::executeFunction(const std::wstring& name, const object::list& paramList) const
{
    // Turn the parameter list in JSON
    const std::wstring input_json = objectToJson(paramList);

    // Call the DLL function and get a JSON response back
    wchar_t* output_json_str = m_executer(name.c_str(), input_json.c_str());
    if (output_json_str == nullptr)
        raiseWError(L"Executing function failed: " + m_filePath + L" - " + name);

    // Capture the JSON locally, the call the DLL to free the buffer it allocated
    std::wstring output_json = output_json_str;
    m_freer(output_json_str);
    output_json_str = nullptr;

    // Turn the output JSON into an object
    object output_obj = objectFromJson(output_json);

    // A little hack for detecting exceptions
    // Look for an unlikely string prefix, then chop it off to yield a useful error function
    if (output_obj.type() == object::STRING)
    {
        static const wchar_t* expPrefix = L"mscript EXCEPTION ~~~ mscript_ExecuteFunction: ";
        static size_t prefixLen = wcslen(expPrefix);
        if (startsWith(output_obj.stringVal(), expPrefix))
            raiseWError(L"Executing function failed: " + m_filePath + L" - " + 
                        output_obj.stringVal().substr(prefixLen));
    }

    // All done
    return output_obj;
}

Conclusion and Points of Interest

I hope you are eager to try out mscript for the first time, of if you've tried it before, to give it another look.

I look forward to hearing from folks if the new error handling makes sense.

And I definitely want to hear from folks interested in DLL development.

Enjoy!

History

23^rd March, 2022: Initial version