Click here to Skip to main content
15,867,985 members
Articles / Productivity Apps and Services / Biztalk

Transfer Large Files using BizTalk - Send Side

Rate me:
Please Sign up or sign in to vote.
4.29/5 (6 votes)
20 Apr 2011CPOL4 min read 42.4K   493   6   13
How to transfer large files using BizTalk - Send side

Introduction

Dealing with large files (200MB+) on BizTalk can be a major performance bottleneck when a file is parsed by a receive pipeline and stored into the MessageBox. The CPU usage of both the BizTalk Server host instance that receives the file and the SQL Server can go very high and slow down the system, especially when more large files are received at the same time.

One way to solve this problem is to create a custom pipeline component that receives the large file, stores it to disk and creates a small XML message that contains the information about where the large file is stored. The small XML message is stored into the MessageBox instead of the large file. It contains the same context information as the large file and it can be picked up by an orchestration or send pipeline. The sending component then gets the location to the large file and does the sending process. Deleting the large files from disk can be done in a scheduled task.

Transfer large files using BizTalk - Receive side describes the receive side on BizTalk. In this article, I describe how to get the information about where the large file is stored and send it in a custom send pipeline component. I also describe briefly how to delete the large files from disk.

The Custom Pipeline Component

Much of this code is basic when creating a custom pipeline component and can also be generated by wizards. However, I list all code below and give a short explanation. The essential code is found in the Execute method which is implemented from the IComponent interface.

The custom pipeline component is created as new Class Library in Visual Studio and signed with a strong name key file.

A reference to Microsoft.BizTalk.Pipeline.dll is added.

The code starts by adding the following namespaces to create a custom pipeline component and to read the files from disk.

C#
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component.Interop;
using System.IO;

The code at the beginning of the class is listed below. One attribute tells that this is a pipeline component and a second restricts the component to be used only in the encode stage in a pipeline. The encoder pipeline component needs to implement the interfaces IBaseComponent, IComponentUI and IComponent.

C#
namespace Stm.LargeFileEncoder
{
   [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
   [ComponentCategory(CategoryTypes.CATID_Encoder)]
   [System.Runtime.InteropServices.Guid("52dcc4e5-28e1-49a2-81fd-de496ac80fe8")]
   public class LargeFileEncoder: IBaseComponent, 
		IComponentUI, IComponent, IPersistPropertyBag
   {

The IBaseComponent interface provides properties that provide basic information about the components.

C#
#region IBaseComponent
private const string _description = 
	"Pipeline component used to read large files from disk";
private const string _name = "LargeFileEncoder";
private const string _version = "1.0.0.0";

public string Description
{
   get { return _description; }
}
public string Name
{
   get { return _name; }
}
public string Version
{
   get { return _version; }
}
#endregion

The interface IComponentUI defines a method and property that are used within the Pipeline Designer environment. To keep it simple, I have not provided any code here.

C#
#region IComponentUI
private IntPtr _icon = new IntPtr();
public IntPtr Icon
{
   get { return _icon; }
}
public System.Collections.IEnumerator Validate(object projectSystem)
{
   return null;
}
#endregion

The core interface is IComponent. In this case, it contains one method that executes the pipeline component and reads the large file from disk and returns it as a message.

C#
#region IComponent
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
   // Read filepath from the context properties            
   string largeFilePath = pInMsg.Context.Read("LargeFilePath", 
  "http://Stm.CustomPipelineComponents.Schemas.PropertySchema.PropertySchema").ToString();

   // Read file from disk
   int bufferSize = 1024; // 4096
   FileStream fs = new FileStream(largeFilePath, FileMode.Open, 
		FileAccess.Read, FileShare.Read, bufferSize);

   if (fs.CanSeek)           
      fs.Position = 0;
                                                                
   pInMsg.BodyPart.Data = fs;
            
   return pInMsg;
}
#endregion

As mentioned in the article Transfer large files using BizTalk - Receive side, a schema for the small XML message is deployed and in this case the field that contains the location to the large file is promoted. The first thing to do in this method is to read that location from the message. Next, a FileStream reads from the file using an internal buffer. The Data property of the BodyPart of the message is set to the FileStream and the message is returned. There is no code to delete the large file here. If an error occurs in a send adapter and BizTalk is set to resend the message after a period, the large file cannot be deleted.

The Send Pipeline

When the component above is built, it can be stored in the same folder as the other pipeline components on the BizTalk Server. In a BTS send pipeline project, this component DLL must be added to the toolbar and used in the encode stage and the pipeline must be deployed.

Testing on BizTalk

To test this project, the components on the receive side(a custom decoder component, a schema and a receive pipeline), the custom encoder component and the send pipeline need to be built and deployed. The large file will be stored as a .msg file on disk and the output from the send port will be a unique name plus the original name when using %MessageID%%SourceFileName% as macros in the send port. This works fine for all types of files.

Deleting the Large Files

Deleting the large files can be done in a scheduled task. In cases where there are more partners sending and receiving large files, there can also be more locations where the large files are stored and need to be deleted.

One way to do this is to create an XML file that contains one node for each of these locations. These nodes contain the path to the directory where the large files are stored. A .NET assembly can be set up as a scheduled task and read the directories from the XML file and delete each file in each directory. Check first if the file is not used by another process if the scheduled task is run often.

There are many ways to delete the large files and to store the path to the directories. This is just one simple suggestion.

Related Articles

The article Transfer large files using BizTalk - Receive side describes the receive side on BizTalk.

The article Transfer extremely large files using Windows Service and BizTalk Server describes how to transfer extremely large files (up to 2GB) using Windows Service and BizTalk.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Norway Norway
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
QuestionError : Object reference not set to an instance of an object. Pin
Member 1107234513-Mar-15 3:42
Member 1107234513-Mar-15 3:42 
GeneralGetting Object reference not set to an instance of an object. error Pin
sarvanvs1232-Aug-12 0:05
sarvanvs1232-Aug-12 0:05 
GeneralRe: Getting Object reference not set to an instance of an object. error Pin
Member 1107234513-Mar-15 3:41
Member 1107234513-Mar-15 3:41 
GeneralSend Host Adapter not closing filestream Pin
WernerDT12-May-11 23:48
professionalWernerDT12-May-11 23:48 
GeneralRe: Send Host Adapter not closing filestream Pin
Steinar Moen from Norway28-May-11 17:34
Steinar Moen from Norway28-May-11 17:34 
GeneralRe: Send Host Adapter not closing filestream Pin
WernerDT31-May-11 21:57
professionalWernerDT31-May-11 21:57 
GeneralRe: Send Host Adapter not closing filestream Pin
Steinar Moen from Norway6-Jun-11 21:01
Steinar Moen from Norway6-Jun-11 21:01 
GeneralRe: Send Host Adapter not closing filestream Pin
WernerDT8-Jun-11 1:52
professionalWernerDT8-Jun-11 1:52 
GeneralMy vote of 5 Pin
Member 432084426-Apr-11 2:45
Member 432084426-Apr-11 2:45 
GeneralRe: My vote of 5 Pin
Steinar Moen from Norway26-Apr-11 22:48
Steinar Moen from Norway26-Apr-11 22:48 
GeneralMy vote of 5 Pin
rue00120-Apr-11 23:58
rue00120-Apr-11 23:58 
GeneralRe: My vote of 5 Pin
Steinar Moen from Norway22-Apr-11 18:18
Steinar Moen from Norway22-Apr-11 18:18 
GeneralMy vote of 1 Pin
srikanthpp20-Apr-11 1:40
srikanthpp20-Apr-11 1:40 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.