Guide line of integrated ShellScript with R Hybrid programming

Download ShellScript Hybrid R script example SourceCode

Introduction & Background

VisualBasic Shellscript(vbss) language is a scientific research tool that i've developed for debugging my virtual cell program engine, but now It just have a few of extension package. So I decided to extended the shellscript function using hybrid programming feature, and now a lot of R package are avaliable on CPAN that can be used for ShellScript! Awesome! 

NOTE: for more details of VisualBasic Shellscript programming syntax, you can review on this article:

Quote:

Powerful ShellScript for bioinformatics researchers

<span style="color: rgb(128, 128, 128); background-color: rgb(255, 255, 255);">The ShellScript is originally developed for the debugging of my "genome-in-code" virtual cell simulation engine, but now it becomes a script language for a .Net program.</span>

Currently most of the biological scientific research algorithm and research tools were implement in R language because of its open source and mathematics statics language features, and the R language is the most useful tools in the research job. The idea of hybrids the R script and my shellscript is awesome, and hybrid the shellscript with R can extends the function of my script and I also can write two type of the programming language both in one script file. Incredible thing! Right?

Using the code

ShellScript external script scripting language environment entry point interface

For the support of the hybridable dynamic scripting with other script language and makes the shellscript more extendable, then I added hrbrid programming feature into the shellscript interpreter

There are two custom attributes that you should implement on your shellscript API library, and you can treat these custom attributes as the interface in .NET language:

1. Namespace entry point (ScriptEntryPoint)
2. Method pointer handler interface (ScriptEntryInterface)

Language syntax of hybrid scripting in shellscript

1. Attach the scripting environment:

!<EntryName>

2. Hybrid scripting with external script language

Syntax for calling external script: 
[Shellscript_variable] << [hybrids_script]

Syntax for assign the variable in the external script environment: 
[ShellScript_variable] >> [hybrids_script_variable]

Hybrids programming using shellscript and the R language

The hybrid programming of the shellscript and R language is original develop on the RDotNET project which was developed by hmansell and jperraud. You can get the RDotNET source code in my attachment files in this article.

RDotNET Project on codeplex: [^] http://rdotnet.codeplex.com/

The RDotNET API for shellscript was defining in the Project RDotNET.Extensions.ShellScriptAPI and you can see all of the implementation details in the class source file API.vb. And then you can use this project as a template example to create a hybrid programming for other script language

Here is the developing steps of the hybrid scripting interface API for my shellscript:

1. Define the scripting environment entry point in the API project, this step just easy and put the Microsoft.VisualBasic.ShellScript.HybridsScripting.ScriptEntryPoint custom attribute on your API module class with a specific script type name. the example is show below:

<Microsoft.VisualBasic.ShellScript.HybridsScripting.ScriptEntryPoint("R.NET")>
<[Namespace]("r.net")>
Public Module API
……
End Module

2. Implements the interface of the script entry point, in this step you should implement three interface: init, setvalue and evaluate(init is for the initialization of the hybrid scripting environment and the setvalue and evaluate interface is for the data exchange between the shellscript and the hybrid scripting environment) through the custom attribute, which you can found in the namespace Microsoft.VisualBasic.ShellScript.HybridsScripting

And here is the three interface implementation example:

First, the init is needed for the initialize the scripting environment between the hydrides script language and the shellscript, and the method of this custom attribute belongs to is a non-parametric method. If the script environment does not required for the initialization, then you can leave this method blank, but the empty method is still needed for declared. Here is the example for initialize the R environment for scripting in the shellscript:

Dim REngine As RDotNET.REngine

<Microsoft.VisualBasic.ShellScript.HybridsScripting.ScriptEntryInterface(ShellScript.HybridsScripting.ScriptEntryInterface.InterfaceTypes.EntryPointInit)>
<Command(".init_r_dotnet()")> Public Function Init() As RDotNET.REngine
    REngine = RDotNET.REngine.CreateSession
    Return REngine
End Function

Second, then you can implement the evaluate interface for the scripting language, this interface should have one parameter of string type to accept the script text and a return type of object. Here is the example for the implement of the interface for the R scripting entry:

<Microsoft.VisualBasic.ShellScript.HybridsScripting.ScriptEntryInterface(ShellScript.HybridsScripting.ScriptEntryInterface.InterfaceTypes.Evaluate)>
Public Function Evaluate(scriptLine As String) As Object
    Return REngine.Evaluate(statement:=scriptLine)
End Function

At last, an interface for passing through the .NET object into the hybrid scripting environment is also required, in this method, two parameter was required: the first parameter is for the variable name in the hybrid scripting environment will be set and then the second parameter is the variable value, and you should always considered that does the scripting environment support the .NET object, if not support, a data type convention operation should be implement before the variable object passing through:

<Microsoft.VisualBasic.ShellScript.HybridsScripting.ScriptEntryInterface(ShellScript.HybridsScripting.ScriptEntryInterface.InterfaceTypes.SetValue)>
Public Function SetSymbol(Variable As String, value As Object) As Boolean
    Try
        Call Engine.SetSymbol(Variable, value)
    Catch ex As Exception

        Return False
    End Try

    Return True
End Function

So, just three simple steps, then you can implement an interface of hybrid programming for the R and the shellscript language.

Hybrid Scripting Test

The test hybrid scripting shellscript file can be found at shellscript debug directory, and here is the script text for the testing:

imports r.net

# telling the shellscript runtime mount the R scripting entrypoints

!r.net

# integerated the R script with shellscript, if successful interacting with R, then a new text file will be make in the save directory!

# test1:  r function invoke
r << data <- read.csv("./test_.csv")
r << write.csv(data, "./test___successfully.txt")

# get the return value of the r function
call $r -> cowsay

# test2: variable setting into the r enviroment
v <- array.new 1,2,3,4,5,6,7,8,9,10
v <- $v -> integervector
v2 <- array.new 1,3,4,5,6,7,8,9,4,3,2,2
v2 <- $v2 -> integervector

mat <- array.new \$v2,\$v2
mat <- $mat -> data.frame

#verify that the variable is passing into the R enviroment successfully!
$mat >> mat
r << write.csv(mat, "./matrix_test_successfully.txt")
$v >> v
v << write.csv(v, "./vector_test_successful.txt")

call "ALL TEST OK!!!" -> msgbox title "test ok!!!"

Script running details
 

1. Initialize the hybrid scripting mount point

# telling the shellscript runtime mount the R scripting entrypoints
!r.net

The script was parsing as the hybrid scripting environment initialize entry point:

The syntax of the initialize statement is the statement should start with a ! character prefix and then the following name string is the scripting environment register name. so that the parsing process can be describe like below:

<span style="color: black; font-family: Consolas, 'Courier New', Courier, mono; font-size: 9pt; white-space: pre; background-color: rgb(251, 237, 187);">If Command.First = "!" Then 'Mount the hybrid programming entry point.(挂载混合编程的脚本环境的执行载入点)</span>

      Dim Name As String = Mid(Command, 2).Trim

      Return New Runtime.Objects.ObjectModels.ShellScript.CodeLine With {
           .LineNumber = LineNumber, .LineFlag = LineNumber, .OrignialScriptLine = Command,
           .VariableAssigned = "$",
           .Method = Function() _ShellScriptHost.AttachesEntryPoint(Name)}
End If

2. Evaluate and set value test

You always should remember that the syntax for evaluate of the external hybrids script is using << operator, if the script have no return value, then the null value will be return.

# integerated the R script with shellscript, if successful interacting with R, then a new text file will be make in the save directory!

# test1:  r function invoke

r << data <- read.csv("./test_.csv")
r << write.csv(data, "./test___successfully.txt")

# get the return value of the r function
call $r -> cowsay

here is the syntax parsing code:

'invoke the external hybrid programming script statement and get the return value.(调用外部的脚本并返回计算结果)
If InStr(Command, " << ") > 0 Then
    Dim ExternalScript As String = Command.Replace(VariableName, "").Trim

    VariableName = VariableName.Replace(" << ", "").Trim
    GetValue = Function() _ShellScriptHost.ExternalScriptInteropEvaluate(ExternalScript)
    Return New Runtime.Objects.ObjectModels.ShellScript.CodeLine With {
         .LineNumber = LineNumber, .LineFlag = LineNumber,
         .VariableAssigned = VariableName,
         .OrignialScriptLine = OriginalCommand,
         .Method = GetValue}
End If

When you want to passing the shellscript variable into the hybrid scripting environment, then you can using >> operator to do this job.(if the scripting environment support the .NET type then you can directly passing the object value, if not then you should convert the.NET object type into the script support type)

Here is the example of passing a .NET integer vector into the R as a data frame object. Due to the R does not support the .NET object type, so before the variable is pushing into the R environment, a data type convertiion operation was implement using integervector function.

v2 <- $v2 -> integervector

mat <- array.new \$v2,\$v2
mat <- $mat -> data.frame

#verify that the variable is passing into the R enviroment successfully!

$mat >> mat
r << write.csv(mat, "./matrix_test_successfully.txt")

here is the code for parsing the pushing variable syntax:

' Pushing the .NET variable into the hybirds scripting environment
If InStr(Command, " >> ") > 0 Then
     Dim Test As String = "$" & VariableName

     If InStr(Command, Test) >= 1 Then
          VariableName = Test
     End If

     Dim ExternalScriptVariable As String = Command.Replace(VariableName, "").Trim
     VariableName = VariableName.Replace(" >> ", "").Trim
     GetValue = Function() _ShellScriptHost.ExternalScriptInteropSetValue(ExternalScriptVariable, _ShellScriptHost._HostMemory.TryGetValue(VariableName))

     Return New Runtime.Objects.ObjectModels.ShellScript.CodeLine With {
          .LineNumber = LineNumber,
          .LineFlag = LineNumber,
          .VariableAssigned = VariableName,
          .OrignialScriptLine = OriginalCommand,
          .Method = GetValue}
End If

Now let’s start the shellscript program and using the source command for calling the R_test.txt script file in the debug directory and test the hybrid programming script. The testing result has been show below (you should install the R program first before you running this script), dadada......