Version 3.0 of mscript adds DLLs for cx_Logging-based logging and SQL and NoSQL database programming. A preprocessor allows for \ line continuations, and // comments that can appear anywhere on a line. The * and & statement prefixes are no longer required. A syntax checker catches many script errors.
Introduction
With the third major version of mscript, I got feedback and guidance that informed my code changes and DLL development. I invite your insights about the good, the bad, and the ugly, and I commit to taking your input into account as I continue to shape mscript to free you from the limits of batch files and the weight of Powershell and Python.
In this article, I will go over the new database and logging DLLs, and the new argument processing function, parseArgs().
Before launching into API and implementation details, let's have a look at a couple sample scripts:
- file_search.ms - load the lines of a text file into a 4db, then supply full-text keyword search against the lines
- file_sizes.ms - load file size information about a directory, then provide searching for file count and space taken
1. file_search.ms - Full-text Keyword Searching of Lines in a File
+ "mscript-db.dll"
/ Define the command line flags we expect as arguments specifications
/ The arguments specifications are represented as a list of indexes
$ arg_specs = \
list \
( \
index \
( \
"flag", "-in", \
"long-flag", "--input_file_path", \
"description", "Path of the input file to process", \
"takes", true, \
"required", true \
), \
index \
( \
"flag", "-db", \
"long-flag", "--database_file_path", \
"description", "Path for the database file", \
"takes", true, \
"required", true \
), \
index \
( \
"flag", "-enc", \
"long-flag", "--text_encoding", \
"description", "The encoding of the input text file", \
"takes", true, \
"required", true, \
"default", "utf8" \
) \
)
$ args = parseArgs(arguments, arg_specs)
$ input_file_path = args.get("-in")
$ database_file_path = args.get("-db")
$ text_encoding = args.get("-enc")
>>
>> Command line arguments:
> "Input file: " + input_file_path
> "Database file: " + database_file_path
> "Text encoding: " + text_encoding
! err
> "Error processing command line parameters: " + err
exit(1)
}
>>
>> Reading lines from input file...
$ file_lines = readFileLines(input_file_path, text_encoding)
! err
> "Error reading input file: " + err
exit(1)
}
> "File Line Count: " + file_lines.length()
>> Getting 4db database up and running...
exec('del "' + database_file_path + '"') msdb_4db_init("db", database_file_path)
! err
> "Error initializing database: " + err
exit(1)
}
>> Importing lines into database...
++ f : 0 -> file_lines.length() -1
/ NOTE: Using line as the primary key did not handle identical lines,
/ hence the line number
$ line = file_lines.get(f)
$ line_number = f + 1
msdb_4db_define("db", "lines", line_number + ": " +
line, index("len", length(line)))
}
! err
> "Error importing lines into database: " + err
exit(1)
}
>> Database loaded, ready to query!
O >>
>> Enter your search criteria:
$ criteria = trimmed(input())
? criteria.length() == 0
^
}
>>
> "Criteria: " + criteria
>>
/ Get the results, using the "value" column, which is the primary key
/ which is the text of the line, to SELECT out, and to MATCH the criteria
/ We sort by id for top-to-bottom file output
/ The criteria is passed into msdb_4db_query as a param->value index
$ results = \
msdb_4db_query \
( \
"db", \
"SELECT value, id FROM lines WHERE value MATCHES @criteria ORDER BY id", \
index("@criteria", criteria) \
)
/ Process the results
$ result_count = results.length()
> "Results: " + (result_count - 1) ++ idx : 1 -> result_count - 1
$ line = results.get(idx)
> line.get(0) }
! err
> "Error getting search results: " + err
}
}
2. file_sizes.ms - Directory Indexing for File Size Stats
This samples demonstrates directory processing, logging, and SQL programming:
+ "mscript-log.dll"
+ "mscript-db.dll"
$ arg_specs = \
list \
( \
index \
( \
"flag", "-dir", \
"long-flag", "--directory", \
"description", "What directory should be processed to get file sizes?", \
"takes", true, \
"default", "." \
), \
index \
( \
"flag", "-db", \
"long-flag", "--database-file-path", \
"description", "Where should the file sizes database be put?", \
"takes", true, \
"default", "file-sizes.db" \
), \
index \
( \
"flag", "-log", \
"long-flag", "--log-file-path", \
"description", "Where should log output be written?", \
"takes", true, \
"default", "file-sizes.log" \
), \
index \
( \
"flag", "-ll", \
"long-flag", "--log-level", \
"description", "What level to log at? (DEBUG, INFO, ERROR, or NONE)", \
"takes", true, \
"default", "INFO" \
), \
index \
( \
"flag", "-rebuild", \
"long-flag", "--rebuild-index", \
"description", "Should the program start over and rebuild the index?", \
"default", false \
) \
)
$ args = parseArgs(arguments, arg_specs)
$ starting_dir_path = args.get("-dir")
$ db_file_path = args.get("-db")
$ log_file_path = args.get("-log")
$ log_level = args.get("-ll")
$ rebuild = args.get("-rebuild")
>>
>> Configuration:
> "Dir Path: " + starting_dir_path
> "DB File Path: " + db_file_path
> "Log File Path: " + log_file_path
> "Log Level: " + log_level
> "Rebuild: " + rebuild
>>
! err
> "Processing command line arguments failed: " + err
exit(1)
}
/ Start logging to a fresh file every time
$ exec_options = index("ignore_errors", true)
exec(fmt('del "{0}"', log_file_path), exec_options)
mslog_start(log_file_path, log_level)
~ logError(msg)
mslog_error(msg)
> "ERROR: " + msg
}
~ logMsg(msg)
mslog_info(msg)
> msg
}
~ logDebug(msg)
mslog_debug(msg)
}
$ did_db_file_exist = false
{
$ exec_result = exec(fmt('dir "{0}"', db_file_path), exec_options)
did_db_file_exist =
firstLocation(trimmed(exec_result.get("output")), "File Not Found") < 0
}
? !did_db_file_exist
rebuild = true
}
? rebuild
exec(fmt('del "{0}"', db_file_path), exec_options)
}
? rebuild
logMsg("Processing files and folders...")
msdb_sql_init("db", db_file_path)
msdb_sql_exec \
( \
"db", \
"CREATE TABLE file_sizes (FilePath STRING NOT NULL,
SizeBytes NUMBER NOT NULL)" \
)
processDir(starting_dir_path)
msdb_sql_close("db")
}
msdb_sql_init("db", db_file_path)
! err
logError("Setting up index failed: " + err)
exit(1)
}
O >>
>> Enter the path pattern to compute stats. Like *.mp3
>>
$ pattern = trimmed(input())
$ sql_pattern = "%" + pattern.replaced("*", "%")
$ sql_query = \
"SELECT COUNT(*), SUM(SizeBytes) FROM file_sizes WHERE FilePath LIKE @like"
$ results = msdb_sql_exec("db", sql_query, index("@like", sql_pattern))
? results.length() <= 1
>> No results
^
}
$ result = results.get(1)
>>
$ file_count = result.get(0)
? file_count = null
file_count = 0
}
logMsg("Count: " + file_count)
/ Size (GB):
$ size_bytes = result.get(1)
? size_bytes = null
size_bytes = 0
}
$ size_str = ""
? size_bytes > 1024 * 1024 * 1024
size_str = "Size (GB): " + round(size_bytes / 1024 / 1024 / 1024, 2)
}
? size_bytes > 1024 * 1024
size_str = "Size (MB): " + round(size_bytes / 1024 / 1024, 2)
}
<>
size_str = "Size (KB): " + round(size_bytes / 1024, 2)
}
logMsg(size_str)
! err
> "Querying interface failed: " + err
^
}
}
/ Unreachable...
msdb_sql_close("db")
mslog_stop()
>> All done.
~ processDir(dirPath)
logDebug("DIR Path: " + dirPath)
> dirPath
$ dir_output_lines = null
{
$ dir_result = exec(fmt('dir "{0}"', dirPath))
$ dir_output = dir_result.get("output")
dir_output_lines = splitLines(dir_output)
}
$ found_dirs = list()
$ found_file_sizes = index()
$ line_pattern = \
"[0-9\/]+" + \
"\s*" + \
"[0-9\:]+\s*(AM|PM)" + \
"\s*" + \
"((\<DIR\>\s*)|([0-9\,]+))" + \
"\s*" + \
"(\S.*)"
@ line : dir_output_lines
/ Skip header lines
? line.firstLocation(" ") == 0
^
}
/ Match up the DIR parts
$ matches = line.getMatches(line_pattern, true)
? matches.length() < 2 ^
}
$ path_stem = trimmed(matches.get(matches.length() - 1))
? path_stem = "." OR path_stem = ".."
^
}
$ full_path = path_stem
? dirPath <> "."
full_path = dirPath + "\" + full_path
}
$ len_str = replaced(trimmed(matches.get(matches.length() - 2)), ",", "")
? len_str.length() == 0
found_dirs.add(full_path)
}
<>
found_file_sizes.add(full_path, number(len_str))
}
}
! err
> fmt('Error processing directory "{0}": {1}', dirPath, err
exit(1)
}
/ load the file size data we found into the DB
$ insert_statement = \
"INSERT INTO file_sizes (FilePath, SizeBytes) VALUES (@filePath, @sizeBytes)"
$ insert_params = index()
@ file_path : found_file_sizes
insert_params.set("@filePath", file_path)
insert_params.set("@sizeBytes", found_file_sizes.get(file_path))
msdb_sql_exec("db", insert_statement, insert_params)
}
/ recurse on the dirs we found
@ dir_path : found_dirs
processDir(dir_path)
}
}
New DLL APIs
Let's have a look under the hood to see the code that powered those mscripts.
mscript-log
I've found that debugging scripts is best done with good logging. I have markedly sped up my scripting with logging used properly. It can be difficult and messy without it.
cx_Logging has been around for a long time. The source code is available as a .h / .c pair at GitHub in the src directory. The code is Python-centric and portable. I was able to take a machete to my copy and get it down to supporting Win32 with no provisions for anything else. Besides erasing a lot of code, I changed the global critical section into std::mutex / std::unique_lock, and I (had to) use strcpy_s and sprintf_s wherever possible, doing spot inspections of char* usage to validate security. Not for the faint of heart, but it's done, and now we can enjoy it as a simple mscript API:
mslog_start(filename, log_level, optional_settings_index)mslog_stop()mslog_setlevel(log_level)mslog_getlevel()mslog_error(message)mslog_info(message)mslog_debug(message)
Here is the source code for the logging DLL:
#include "pch.h"
#include "cx_Logging.h"
using namespace mscript;
wchar_t* __cdecl mscript_GetExports()
{
std::vector<std::wstring> exports
{
L"mslog_getlevel",
L"mslog_setlevel",
L"mslog_start",
L"mslog_stop",
L"mslog_error",
L"mslog_info",
L"mslog_debug",
};
return module_utils::getExports(exports);
}
void mscript_FreeString(wchar_t* str)
{
delete[] str;
}
static unsigned long getLogLevel(const std::wstring& level)
{
if (level == L"INFO")
return LOG_LEVEL_INFO;
else if (level == L"DEBUG")
return LOG_LEVEL_DEBUG;
else if (level == L"ERROR")
return LOG_LEVEL_ERROR;
else if (level == L"NONE")
return LOG_LEVEL_NONE;
else
raiseWError(L"Invalid log level: " + level);
}
static std::wstring getLogLevel(unsigned long level)
{
switch (level)
{
case LOG_LEVEL_INFO: return L"INFO";
case LOG_LEVEL_DEBUG: return L"DEBUG";
case LOG_LEVEL_ERROR: return L"ERROR";
case LOG_LEVEL_NONE: return L"NONE";
default: raiseError("Invalid log level: " + num2str(level));
}
}
wchar_t* mscript_ExecuteFunction(const wchar_t* functionName,
const wchar_t* parametersJson)
{
try
{
std::wstring funcName = functionName;
auto params = module_utils::getParams(parametersJson);
if (funcName == L"mslog_getlevel")
{
if (params.size() != 0)
raiseError("Takes no parameters");
return module_utils::jsonStr(getLogLevel(GetLoggingLevel()));
}
if (funcName == L"mslog_setlevel")
{
if (params.size() != 1 || params[0].type() != object::STRING)
raiseError("Pass in the log level: DEBUG, INFO, ERROR, or NONE");
unsigned result = SetLoggingLevel
(getLogLevel(toUpper(params[0].stringVal())));
return module_utils::jsonStr(bool(result == 0));
}
if (funcName == L"mslog_start")
{
if (params.size() < 2 || params.size() > 3)
raiseError("Takes three parameters: filename,
log level, and optional settings index");;
if (params[0].type() != object::STRING)
raiseError("First parameter must be filename string");
if (params[1].type() != object::STRING)
raiseError("Second parameter must be log level string:
DEBUG, INFO, ERROR, or NONE");
if (params.size() == 3 && params[2].type() != object::INDEX)
raiseError("Third parameter must be a settings index");
std::wstring filename = params[0].stringVal();
unsigned log_level = getLogLevel(toUpper(params[1].stringVal()));
object::index index =
params.size() == 3
? params[2].indexVal()
: object::index();
object max_files;
if
(
index.tryGet(toWideStr("maxFiles"), max_files)
&&
(
max_files.type() != object::NUMBER
||
max_files.numberVal() < 0
||
unsigned(max_files.numberVal()) != max_files.numberVal()
)
)
{
raiseError("Invalid maxFiles parameter");
}
if (max_files.type() == object::NOTHING)
max_files = double(1);
object max_file_size_bytes;
if
(
index.tryGet(toWideStr("maxFileSizeBytes"), max_file_size_bytes)
&&
(
max_file_size_bytes.type() != object::NUMBER
||
max_file_size_bytes.numberVal() < 0
||
unsigned(max_file_size_bytes.numberVal())
!= max_file_size_bytes.numberVal()
)
)
{
raiseError("Invalid maxFileSizeBytes parameter");
}
if (max_file_size_bytes.type() == object::NOTHING)
max_file_size_bytes = double(DEFAULT_MAX_FILE_SIZE);
object prefix;
if
(
index.tryGet(toWideStr("prefix"), prefix)
&&
prefix.type() != object::STRING
)
{
raiseError("Invalid prefix parameter");
}
if (prefix.type() == object::NOTHING)
prefix = toWideStr(DEFAULT_PREFIX);
unsigned result =
StartLogging
(
toNarrowStr(filename).c_str(),
log_level,
unsigned(max_files.numberVal()),
unsigned(max_file_size_bytes.numberVal()),
toNarrowStr(prefix.stringVal()).c_str()
);
return module_utils::jsonStr(bool(result == 0));
}
if (funcName == L"mslog_stop")
{
StopLogging();
return module_utils::jsonStr(true);
}
{
auto under_parts = split(funcName, L"_");
if (under_parts.size() != 2)
raiseWError(L"Invalid function name: " + funcName);
unsigned int log_level = getLogLevel(toUpper(under_parts[1]));
if (params.size() != 1 || params[0].type() != object::STRING)
raiseWError(funcName + L":
pass the log message as one string parameter");
unsigned result =
LogMessage(log_level, toNarrowStr(params[0].stringVal()).c_str());
return module_utils::jsonStr(bool(result == 0));
}
}
catch (const user_exception& exp)
{
return module_utils::errorStr(functionName, exp);
}
catch (const std::exception& exp)
{
return module_utils::errorStr(functionName, exp);
}
catch (...)
{
return nullptr;
}
}
mscript-db
I've written about 4db and various things that led up to it for ages now. If nothing better falls out of all of that than the db, dbreader and strnum classes for doing direct SQLite programming, then it's been worth it. I'm not going to force you to use 4db if you want to hack away at SQL directly. So there are two APIs in one DLL:
SQLite
msdb_sql_init(db_file_path)msdb_sql_close()msdb_sql_exec(sql_query)msdb_sql_rows_affected()msdb_sql_last_inserted_id()
4db
msdb_4db_init(db_file_path)msdb_4db_close()msdb_4db_define(table_name, primary_key_value, metadata_index)msdb_4db_query(sql_query, parameters_index)msdb_4db_delete(table_name, primary_key_value)msdb_4db_drop(table_name)
Note that I'm oversimplifying the interface. It looks like there would just be one database open globally. I didn't want it to be that restrictive, so I built in a mechanism for having multiple DBs open at once.
#include "pch.h"
std::mutex g_mutex;
std::unordered_map<std::wstring, std::shared_ptr<fourdb::ctxt>> g_contexts;
std::unordered_map<std::wstring, std::shared_ptr<fourdb::db>> g_db_conns;
static fourdb::strnum convert(const mscript::object& obj)
{
switch (obj.type())
{
case mscript::object::NOTHING:
return double(0.0);
case mscript::object::STRING:
return obj.stringVal();
case mscript::object::NUMBER:
return obj.numberVal();
case mscript::object::BOOL:
return obj.boolVal() ? 1.0 : 0.0;
default:
raiseError("Invalid object type for conversion to 4db: " +
mscript::num2str(int(obj.type())));
}
}
static mscript::object convert(const fourdb::strnum& obj)
{
if (obj.isStr())
return obj.str();
else
return obj.num();
}
static fourdb::paramap convert(const mscript::object::index& idx)
{
fourdb::paramap ret_val;
for (const auto& it : idx.vec())
ret_val.insert({ it.first.toString(), convert(it.second) });
return ret_val;
}
static std::shared_ptr<fourdb::ctxt> get4db(const std::wstring& name)
{
const auto& it = g_contexts.find(name);
if (it == g_contexts.end())
raiseWError(L"4db not found: " + name + L" - call msdb_4db_init first");
return it->second;
}
static std::shared_ptr<fourdb::db> getSqldb(const std::wstring& name)
{
const auto& it = g_db_conns.find(name);
if (it == g_db_conns.end())
raiseWError(L"SQL DB not found: " + name + L" - call msdb_sql_init first");
return it->second;
}
static mscript::object::list processDbReader(fourdb::dbreader& reader)
{
mscript::object::list ret_val;
ret_val.push_back(mscript::object::list());
unsigned col_count = reader.getColCount();
ret_val[0].listVal().reserve(col_count);
for (unsigned c = 0; c < col_count; ++c)
ret_val[0].listVal().push_back(reader.getColName(c));
while (reader.read())
{
ret_val.emplace_back(mscript::object::list());
mscript::object::list& row_list = ret_val.back().listVal();
row_list.reserve(col_count);
for (unsigned c = 0; c < col_count; ++c)
{
bool is_null = false;
fourdb::strnum val = reader.getStrNum(c, is_null);
row_list.push_back(is_null ? mscript::object() : convert(val));
}
}
return ret_val;
}
wchar_t* __cdecl mscript_GetExports()
{
std::vector<std::wstring> exports
{
L"msdb_sql_init",
L"msdb_sql_close",
L"msdb_sql_exec",
L"msdb_sql_rows_affected",
L"msdb_sql_last_inserted_id",
L"msdb_4db_init",
L"msdb_4db_close",
L"msdb_4db_define",
L"msdb_4db_undefine",
L"msdb_4db_query",
L"msdb_4db_delete",
L"msdb_4db_drop",
L"msdb_4db_reset",
L"msdb_4db_get_schema",
};
return mscript::module_utils::getExports(exports);
}
void mscript_FreeString(wchar_t* str)
{
delete[] str;
}
wchar_t* mscript_ExecuteFunction(const wchar_t* functionName,
const wchar_t* parametersJson)
{
try
{
std::wstring funcName = functionName;
auto params = mscript::module_utils::getParams(parametersJson);
std::unique_lock ctx_lock(g_mutex);
if (funcName == L"msdb_sql_init")
{
if
(
params.size() != 2
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
)
{
raiseError("Takes two parameters: a name for the database,
and the path to the DB file");
}
std::wstring db_name = params[0].stringVal();
std::wstring db_file_path = params[1].stringVal();
if (g_db_conns.find(db_name) != g_db_conns.end())
raiseWError(L"Database already initialized: " + db_name);
auto new_db = std::make_shared<fourdb::db>
(mscript::toNarrowStr(db_file_path));
g_db_conns.insert({ db_name, new_db });
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_sql_close")
{
if
(
params.size() != 1
||
params[0].type() != mscript::object::STRING
)
{
raiseError("Takes the name of the database");
}
auto db_it = g_db_conns.find(params[0].stringVal());
if (db_it != g_db_conns.end())
g_db_conns.erase(db_it);
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_sql_exec")
{
if
(
(params.size() != 2 && params.size() != 3)
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
||
(params.size() == 3 && params[2].type() != mscript::object::INDEX)
)
{
raiseError("Takes the name of the database, the SQL query,
and an optional index of query parameters");
}
auto sql_db = getSqldb(params[0].stringVal());
std::wstring sql_query = params[1].stringVal();
auto params_idx =
params.size() >= 3
? params[2].indexVal()
: mscript::object::index();
fourdb::paramap query_params = convert(params_idx);
auto reader = sql_db->execReader(sql_query, query_params);
auto results = processDbReader(*reader);
return mscript::module_utils::jsonStr(results);
}
else if (funcName == L"msdb_sql_rows_affected")
{
if
(
params.size() != 1
||
params[0].type() != mscript::object::STRING
)
{
raiseError("Takes the database name");
}
auto sql_db = getSqldb(params[0].stringVal());
int64_t rows_affected =
sql_db->execScalarInt64(L"SELECT changes()").value();
return mscript::module_utils::jsonStr(double(rows_affected));
}
else if (funcName == L"msdb_sql_last_inserted_id")
{
if
(
params.size() != 1
||
params[0].type() != mscript::object::STRING
)
{
raiseError("Takes the database name");
}
auto sql_db = getSqldb(params[0].stringVal());
int64_t last_inserted_id =
sql_db->execScalarInt64(L"SELECT last_insert_rowid()").value();
return mscript::module_utils::jsonStr(double(last_inserted_id));
}
else if (funcName == L"msdb_4db_init")
{
if
(
params.size() != 2
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
)
{
raiseError("Takes two parameters: a name for the context,
and the path to the DB file");
}
std::wstring db_name = params[0].stringVal();
std::wstring db_file_path = params[1].stringVal();
if (g_contexts.find(db_name) != g_contexts.end())
raiseWError(L"Context already initialized: " + db_name);
auto ctxt_ptr = std::make_shared<fourdb::ctxt>
(mscript::toNarrowStr(db_file_path));
g_contexts.insert({ db_name, ctxt_ptr });
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_4db_close")
{
if
(
params.size() != 1
||
params[0].type() != mscript::object::STRING
)
{
raiseError("Takes the name of the context");
}
auto ctxt_it = g_contexts.find(params[0].stringVal());
if (ctxt_it != g_contexts.end())
g_contexts.erase(ctxt_it);
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_4db_define")
{
if
(
params.size() != 4
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
||
params[3].type() != mscript::object::INDEX
)
{
raiseError("Takes four parameters: the name of the context,
the table name, the key value, and an index of name-value pairs");
}
auto ctxt = get4db(params[0].stringVal());
const std::wstring& table_name = params[1].stringVal();
const fourdb::strnum key_value = convert(params[2]);
const fourdb::paramap metadata = convert(params[3].indexVal());
ctxt->define(table_name, key_value, metadata);
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_4db_undefine")
{
if
(
params.size() != 4
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
||
params[3].type() != mscript::object::STRING
)
{
raiseError("Takes four parameters: the name of the context,
the table name, the key value, and the name of the metadata to remove");
}
auto ctxt = get4db(params[0].stringVal());
const std::wstring& table_name = params[1].stringVal();
const fourdb::strnum key_value = convert(params[2]);
const std::wstring& metadata_name = params[3].stringVal();
ctxt->undefine(table_name, key_value, metadata_name);
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_4db_query")
{
if
(
params.size() != 3
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
||
params[2].type() != mscript::object::INDEX
)
{
raiseError("Takes three parameters: the name of the context,
the SQL query, and an index of name-value parameters");
}
auto ctxt = get4db(params[0].stringVal());
fourdb::select sql_select = fourdb::sql::parse(params[1].stringVal());
const fourdb::paramap query_params = convert(params[2].indexVal());
for (const auto& param_it : query_params)
sql_select.addParam(param_it.first, param_it.second);
auto reader = ctxt->execQuery(sql_select);
auto results = processDbReader(*reader);
return mscript::module_utils::jsonStr(results);
}
else if (funcName == L"msdb_4db_delete")
{
if
(
params.size() != 3
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
)
{
raiseError("Takes three parameters: the name of the context,
the table name, and the key value");
}
auto ctxt = get4db(params[0].stringVal());
const std::wstring& table_name = params[1].stringVal();
const fourdb::strnum key_value = convert(params[2]);
ctxt->deleteRow(table_name, key_value);
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_4db_drop")
{
if
(
params.size() != 2
||
params[0].type() != mscript::object::STRING
||
params[1].type() != mscript::object::STRING
)
{
raiseError("Takes two parameters: the name of the context,
and the table name");
}
auto ctxt = get4db(params[0].stringVal());
const std::wstring& table_name = params[1].stringVal();
ctxt->drop(table_name);
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_4db_reset")
{
if
(
params.size() != 1
||
params[0].type() != mscript::object::STRING
)
{
raiseError("Takes the name of the context to reset");
}
auto ctxt = get4db(params[0].stringVal());
ctxt->reset();
return mscript::module_utils::jsonStr(mscript::object());
}
else if (funcName == L"msdb_4db_get_schema")
{
if
(
params.size() != 1
||
params[0].type() != mscript::object::STRING
)
{
raiseError("Takes the name of the context to get the schema of");
}
auto ctxt = get4db(params[0].stringVal());
const auto schema = ctxt->getSchema();
mscript::object::index table_schema;
for (const auto& tables_it : schema.vec())
{
const std::wstring& table_name = tables_it.first;
mscript::object::list table_columns;
table_columns.reserve(tables_it.second->size());
for (const std::wstring& col : *tables_it.second)
table_columns.push_back(col);
table_schema.set(table_name, table_columns);
}
return mscript::module_utils::jsonStr(table_schema);
}
else
raiseWError(L"Unknown function");
}
catch (const mscript::user_exception& exp)
{
return mscript::module_utils::errorStr(functionName, exp);
}
catch (const std::exception& exp)
{
return mscript::module_utils::errorStr(functionName, exp);
}
catch (...)
{
return nullptr;
}
}
Argument Parsing
A big feature request was powerful command-line argument parsing. You'd want to be able to say what flags the application handled, like -i, and a longer version of the flag, like --input, and a description for the flag, like "Specify the input file", and whether the flag requires a value, like an "--input" flag would, and whether the value has to be numeric, a default value, and whether the flag is required. Wow, that was one sentence! Powerful stuff.
Here's out the header:
#pragma once
#include "object.h"
namespace mscript
{
object
parseArgs
(
const object::list& arguments,
const object::list& argumentSpecs
);
}
It takes the list of argument strings mscripts can pass in the global "arguments" variable set by the mscript EXE, but this code and the mscript wrapper function do not assume that. It also takes a list of "argument specs". Each argument spec is an index that defines an argument that the program can handle, its flag, its long flag, etc. This function returns an index mapping a flag to the value for that flag, or true for flags that don't take values, or false for missing flags. The special pseudo-flag "" returns a list of values not associated with flags, e.g., your primary arguments. Your calling code would probe the returned index to control how it behaves.
Here's the implementation of the argument parser:
#include "pch.h"
#include "parse_args.h"
#include "exe_version.h"
#include "utils.h"
using namespace mscript;
struct arg_spec
{
std::wstring flag; std::wstring long_flag; std::wstring description;
bool takes = false;
bool numeric = false;
bool required = false;
object default_value;
};
static std::wstring getStrFlagValue(const object::index& argumentSpec,
const std::string& flagName)
{
object flag_value;
if (!argumentSpec.tryGet(toWideStr(flagName), flag_value) ||
flag_value.type() != object::STRING)
raiseError("Argument spec lacks " + flagName + " string setting");
std::wstring flag_str = trim(flag_value.stringVal());
if (flag_str.empty())
raiseError("Argument spec lacks " + flagName + " non-empty string setting");
return flag_str;
}
static bool getBoolFlagValue(const object::index& argumentSpec,
const std::string& flagName)
{
object flag_value;
if (!argumentSpec.tryGet(toWideStr(flagName), flag_value))
return false;
if (flag_value.type() != object::BOOL)
raiseError("Argument spec " + flagName + " setting is not bool");
return flag_value.boolVal();
}
object
mscript::parseArgs
(
const object::list& arguments,
const object::list& argumentSpecs
)
{
object ret_val = object::index();
object::index& ret_val_index = ret_val.indexVal();
object raw_arg_list = object::list();
ret_val_index.set(toWideStr(""), raw_arg_list);
std::vector<arg_spec> local_specs;
for (const auto& cur_input_spec : argumentSpecs)
{
if (cur_input_spec.type() != object::INDEX)
raiseError("Invalid argument spec type: " +
object::getTypeName(cur_input_spec.type()));
const object::index& cur_input_index = cur_input_spec.indexVal();
arg_spec cur_spec;
cur_spec.flag = getStrFlagValue(cur_input_index, "flag");
cur_spec.long_flag = getStrFlagValue(cur_input_index, "long-flag");
cur_spec.description = getStrFlagValue(cur_input_index, "description");
cur_spec.takes = getBoolFlagValue(cur_input_index, "takes");
cur_spec.numeric = getBoolFlagValue(cur_input_index, "numeric");
cur_spec.required = getBoolFlagValue(cur_input_index, "required");
object default_value;
if (cur_input_index.tryGet(toWideStr("default"), default_value))
cur_spec.default_value = default_value;
else if (!cur_spec.takes) cur_spec.default_value = false;
local_specs.push_back(cur_spec);
}
bool already_had_help = false;
for (const auto& cur_spec : local_specs)
{
if (cur_spec.flag == L"-?" || cur_spec.long_flag == L"--help")
{
already_had_help = true;
break;
}
}
if (!already_had_help)
{
arg_spec new_spec;
new_spec.flag = L"-?";
new_spec.long_flag = L"--help";
new_spec.description = L"Get usage of this script";
new_spec.default_value = false;
local_specs.insert(local_specs.begin(), new_spec);
}
for (const auto& arg_spec : local_specs)
ret_val_index.set(arg_spec.flag, arg_spec.default_value);
bool help_exit_suppressed = false;
for (size_t a = 0; a < arguments.size(); ++a)
{
if (arguments[a].type() != object::STRING)
{
raiseWError(L"Invalid command-line argument,
not a string: #" + num2wstr(double(a)) +
L" - " + arguments[a].toString());
}
if (arguments[a].stringVal() == L"--suppress-help-quit")
help_exit_suppressed = true;
}
bool help_was_output = false;
for (size_t a = 0; a < arguments.size(); ++a)
{
const std::wstring& cur_arg = arguments[a].stringVal();
if (cur_arg == L"--suppress-help-quit")
continue;
bool has_next_arg = false;
object next_arg;
if (a < arguments.size() - 1)
{
next_arg = arguments[a + 1];
if (!startsWith(next_arg.toString(), L"-"))
has_next_arg = true;
}
if (cur_arg.empty() || cur_arg[0] != '-')
{
raw_arg_list.listVal().push_back(cur_arg);
continue;
}
if (!already_had_help && (cur_arg == L"-?" || cur_arg == L"--help"))
{
std::wstring mscript_exe_path = getExeFilePath();
std::wcout
<< mscript_exe_path
<< L" - v" << toWideStr(getBinaryVersion(mscript_exe_path))
<< L"\n";
size_t max_flags_len = 0;
for (const auto& arg_spec : local_specs)
{
size_t cur_len = arg_spec.flag.size() + arg_spec.long_flag.size();
if (cur_len > max_flags_len)
max_flags_len = cur_len;
}
static std::wstring flag_separator = L", ";
static size_t flag_separator_len = flag_separator.length();
size_t max_flags_output_len = max_flags_len + flag_separator_len;
for (const auto& arg_spec : local_specs)
{
std::wcout
<< std::left
<< std::setfill(L' ')
<< std::setw(max_flags_output_len)
<< (arg_spec.flag + flag_separator + arg_spec.long_flag)
<< L": "
<< arg_spec.description;
if (arg_spec.takes)
{
std::wcout << " - type=" << (arg_spec.numeric ? "num" : "str");
if (arg_spec.default_value.type() != object::NOTHING)
std::wcout << " - default=" << arg_spec.default_value.toString();
if (arg_spec.required)
std::wcout << " - [REQUIRED]";
}
std::wcout << L"\n";
}
std::wcout << std::flush;
if (!help_exit_suppressed)
exit(0);
ret_val_index.set(toWideStr("-?"), true);
help_was_output = true;
continue;
}
bool found_flag = false;
for (const auto& cur_spec : local_specs)
{
if (!(cur_spec.flag == cur_arg || cur_spec.long_flag == cur_arg))
continue;
else
found_flag = true;
const std::wstring& cur_flag = cur_spec.flag;
if (cur_spec.takes)
{
if (!has_next_arg)
{
if (cur_spec.default_value != object())
next_arg = cur_spec.default_value;
else
raiseWError(L"No value for flag that takes
next argument: " + cur_arg);
}
if (cur_spec.numeric && next_arg.type() != object::NUMBER)
{
try
{
ret_val_index.set(cur_flag, std::stod(next_arg.toString()));
}
catch (...)
{
raiseWError(L"Converting argument to number failed: " +
cur_spec.flag + L" - " + next_arg.toString());
}
}
else
ret_val_index.set(cur_flag, next_arg);
a = a + 1;
}
else {
ret_val_index.set(cur_flag, true);
}
}
if (!found_flag)
raiseWError(L"Unknown command line flag: " + cur_arg);
}
if (!help_was_output)
{
for (const auto& cur_spec : local_specs)
{
if (cur_spec.required && ret_val_index.get
(cur_spec.flag).type() == object::NOTHING)
raiseWError(L"Required argument not provided: " + cur_spec.flag);
}
}
return ret_val;
}
Conclusion
I invite you to check out mscript.io for version history, including breaking changes in version 3.0, full documentation of functions new and old, and other language changes alluded to above, like the preprocessor, the syntax checker, and the fundamental change that * and & statement prefixes are no longer required.
I really think you'll enjoy writing mscripts. Give it a try, and let me know what you think.
History
- 26th May, 2022: Initial version
Michael Balloni is a manager of software development at a cybersecurity software and services provider.
Check out https://www.michaelballoni.com for all the programming fun he's done over the years.
He has been developing software since 1994, back when Mosaic was the web browser of choice. IE 4.0 changed the world, and Michael rode that wave for five years at a .com that was a cloud storage system before the term "cloud" meant anything. He moved on to a medical imaging gig for seven years, working up and down the architecture of a million-lines-code C++ system.
Michael has been at his current cybersecurity gig since then, making his way into management. He still loves to code, so he sneaks in as much as he can at work and at home.
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
