Shashikant's Biztalk Streams

SSIS Package Error

2011-11-10T20:28:00.006-05:00

Biztalk is not always the right tool for Bulk data inserts. SQL Integration Services provide powerful tool set for inserting large bytes of data from source ( flat files / excel files/DB) into DataTables. In my scenario, I created SSIS package using SQL 2008R2 Business Intelligence Development Studio. It was a simple Data flow, read the excel file from a network location and insert to a remote DataTable. Image below:

On executing the package, OLE Destination 1 shape would threw the following error:
“[OLE DB Destination 1 [421]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "The statement has been terminated.".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Violation of PRIMARY KEY constraint 'PK_tbl_so'. Cannot insert duplicate key in object 'dbo.tbl_so'.".”

I tried various online forums but the solutions didn’t work for me. Then I tried data encryption and it worked. Check the image below.

Update

After I deployed the package to remote server, I tried executing it and received validation errror around login/username. I am using SQL authentication for the DB that will receive the data from excel sheet. Internally, SSIS encrypts the credentials information. If you open the package in a notepad, you can view the encrypted information. But, when the same package is run on a different machine the user context changes and decryption process throws the error. Setting the protection level as below can get you around that issue.

Microsoft BUILD

2011-08-17T09:28:00.003-04:00

Starting this year, Microsoft rebranded its Professional Developer Conference (PDC) as BUILD. I have no idea what message they are trying to convey with this name change. For some strange reasons, this change was not broadcasted strongly by the numerous Microsoft blogs and sites. The registration page was so hidden that even search engines could not find it using standard words like “PDC Microsoft conference” etc. Well, for people who still don’t know, BUILD is being held in Anaheim, California in middle of September this year. At the time of writing this article, no agenda have been published by Microsoft for the conference. This is rather strange for an elite conference like this. My guess is that there will be breaking news around direction of Biztalk Cloud technologies. Update to Windows mobile platform (“Mango” code name) will get good coverage. Microsoft is really pushing hard to break into the Smartphone market. Even though Mango is looking like a robust platform, it is hard to penetrate the barrier created by Android and iPhone. Recent buy out of MMI by Google, bodes well for Microsoft. It leaves windows mobile as the only true Open Source mobile platform in the market. It is rather ironic that a company traditionally linked with developing proprietary technologies is taking this mantle from a company like Google that is traditionally linked with developing and encouraging Open Source systems. This demonstrates the fast changing technology landscape and underlines the difficulties faced by CIOs in planning strategic IT initiatives.

Microsoft is consolidating its offering around Desktop, Web, Cloud and Mobile with the latest release of Visual Studio 2011. These are very exciting times for Microsoft Developers, it seems that finally Microsoft is streamlining its technologies and removing the redundancies. Biztalk-Appfabric consolidation is happening along with Silverlight-HTML5 consolidation. Sharepoint story is unchanged and it is one of the few non-redundant technologies in the Microsoft stack.

So, where do we stand amidst all this? Consolidation of technologies will lead to more clarity for customers. One reason, Biztalk didn’t maintain the steep growth curve was because of the confusion created by conflicting technologies on the Microsoft Connected System stack. Customers like clarity, especially while evaluating different technologies to implement. They tend to select products with more clarity around their future roadmap. Microsoft shops in general are prime candidate for Biztalk, but they saw technologies like AppFabric and WCF close substitute for Biztalk. For most of them, sole reason for buying Biztalk was because they were an EDI shop. On the other hand, Biztalk was a tough sell in non-microsoft shops as it works with Microsoft framework. A java shop will most likely go for a GIS or webMethods implementation.

Real value of Biztalk lies in developing it as an integration hub for both EDI and Non-EDI integrations and then extending it to link with Sharepoint. Next step will be to combine on-premise and cloud integration on same platform. This is where Azure Appfabric is leading us to. Keep an eye on PDC (BUILD) space for more on this.

Death of Biztalk -- Are you kidding me?

2011-06-22T20:44:00.003-04:00

Honestly, I am tired of statements like “BizTalk is Dead” or “BizTalk is Obsolete”. With every release of new technology around Connected System stack, the noise around speculated BizTalk demise gets louder. Right from the time Oslo was released (which never saw light of the day) to recently released Azure App Fabric –June CTP, blogosphere gets noisy with shouts of BizTalk is Dead.

In my opinion, if one thinks of BizTalk as a brand then it could be said that BizTalk may die as a brand in near future. Even that is a speculation for sake of argument. But if one considers BizTalk as Integration enabler and a set of tools, then it can’t be dead. There may be re-organization around the tools but they will not die down. I agree that redundancies in form of Windows App Fabric and Windows Azure App Fabric do take something away from BizTalk. They also create confusion in the mind of new and existing clients. Especially, when the marketing pitch is not in tune with technical pitch. Few years ago, I would just blindly recommend BizTalk in scenarios involving hosted workflows and service compositions. But now it is not so easy. Microsoft has labeled BizTalk as Integration Server, Windows App Fabric is termed as Application Server and Windows Azure App Fabric is a cloud integration platform. It can get confusing between these 3 technologies. If I need to mathematically establish a relationship between these 3, it will look something like:

BizTalk >= Windows Server Fabric + Windows Azure App Fabric.

It can be argued that in near future Windows Azure App Fabric may expand to include more of the existing BizTalk tool set and that may make the relationship more like

BizTalk = Windows Server Fabric + Windows Azure App Fabric.

This is where people jump in and say that BizTalk is dead as it can be replaced by combination of the two technologies! In my opinion:

1. Azure App Fabric is not a mature product yet and it is too early to say anything. Remember Oslo?

2. It is quite possible that BizTalk tools will be re-organized around cloud platform using Azure App fabric as core. Depending on how one sees it, it could be branded as a new version of BizTalk with Azure App Fabric as the cloud enabler. This will not mean end of BizTalk.

3. Other possibility is that BizTalk and Azure App Fabric will continue their parallel paths for foreseeable future. This will not mean end of BizTalk.

It could be said that Azure App Fabric is a new technology that BizTalk developers will have to learn. Just like BizTalk 2004 release, made BizTalk developers learn .net. BizTalk 2006R2 made developers look into WCF.

Windows Server App Fabric is more of an Application Server than an Integration server. However, BizTalk Appfabric connect allows .net developers leverage LOB adapters and Mapper to create workflow solutions. This compliments BizTalk just like Azure App fabric does.

As a BizTalk developer, I have mixed feelings. On one side I feel unhappy that the BizTalk tool set is being broken and leveraged around different Microsoft stacks. Now everyone else will be able to enjoy the excellent tool sets of BizTalk that were once sole proprietorship of BizTalk developers. It kind of takes power away from us. But at the same time, I feel happy that as a BizTalk developer we are placed uniquely to leave our footprints across the different technology stacks that use BizTalk tools .

Following are list of tools that are still exclusive to BizTalk. These are key touch points while making a choice between BizTalk and other technologies.

• Accelerators (EDI, SWIFT, Rosettanet, RFID, EDIFACT, HL7)
• Pipelines
• Flat File processing
• BAM
• BRE (it is exposed as a REST API now but still under BizTalk licensing)
• BizTalk Mapper (BizTalk AppFabric connect enables you to use the mapper with Server App Fabric, still under BizTalk licensing)
• Non-WCF adapters like FileAct/Interact
• Host Integration Server

Processing Empty Files in Biztalk

2011-04-19T11:29:00.008-04:00

In my current integration project, there is a need to process empty files (0 byte) in Biztalk. Default behavior of FILE adapter is to delete an empty file and raise an event. This empty file is then deleted even before it reaches pipeline on way to Biztalk MsgBox. So there is no way to do a routing based on filename either in a pure message scenario or using an orchestration.

One possible solution is to extend the File Adapter that is part of SDK (Image 1.1). Add the 3 projects in a Visual Studio Solution as shown in Image 1.2

Image1.1

Image 1.2

Key steps:

1. Build the solution, GAC Microsoft.Samples.BizTalk.Adapter.Common.dll.

2. Registry Key.

Use the StaticAdapterManagement.reg registry key file given in the sample and update the locations of the dlls mentioned. Microsoft recommends adding additional string for 64 bit machines. I didn’t do and it worked.

3. Go to Biztalk admin console and try to add the new EMPTY FILE adapter. It is possible that you may get an error similar too :
The system cannot find the file specified.

/C:\Program Files (x86)\Microsoft BizTalk Server 2009\SDK\Samples\CustomAdaptersDevelopment\File Adapter\Runtime\Microsoft.BizTalk.SDKSamples.Adapters.DotNetFile.Runtime.dll

This error threw me off as my registry key file did not contain this path at all and it left me wondering why is Biztalk even looking for this path. I did a search for this file in the entire registry and I found couple of references. I had no idea what were they used for but I knew that Biztalk was somehow reading them from registry key. I renamed the key for these entries without deleting . I was able to add the adapter to Biztalk successfully.

4. Once adapter was added, I assigned it to appropriate Biztalk host so I can use it in send/receive port. While setting the receive handler for this newly added adapter, I got the following error:
The Messaging Engine encountered an error when intializing the receive adapter "EmptyFile", HRESULT:"Property /Config/pollingInterval not found on adapter configuration XML.".

I decided to debug the code to find the point of error. I found the error in method : ExtractPollingInterval in file ConfigProperties.cs under project Microsoft.Samples.BizTalk.Adapter.Common.csproj

Update the following code

long pollingInterval = ExtractInt(document, "/Config/pollingInterval");
string pollingUnitOfMeasureStr = Extract(document,"/Config/pollingUnitOfMeasure", "Seconds");

long pollingInterval = Convert.ToInt32(IfExistsExtract(document, "/Config/pollingInterval","5"));
string pollingUnitOfMeasureStr = IfExistsExtract(document, "/Config/pollingUnitOfMeasure", "Seconds");

Or update the root name of ReceiveHandler schema to Config. I did the code change and not the schema change

I was in a position to configure this new custom FILE adapter and receive an empty file. In order to get a handle on the empty file, I need to go to method PickUpFilesAndSubmit which is part of DotNetFileReceiverEndpoint.cs under DotNetFile.csproj The line of code that got me the handle to the incoming file is if (item.Length == 0).
Once I got the handle, I was able to add my logic to deal with the incoming empty file. In my case, I decide to add a string “EmptyFile” if the incoming file was empty. I would then detect this file in the send adapter using the same custom adapter and assign 0 bytes to it. In between, I will have my orchestration reading the file name of this empty file (added string “EmptyFile”) and route it to appropriate directory folder.

This solution requires tampering with contents of an emptry file. It may be to acceptable in some cases. In my case, it was acceptable.

Send Side Changes:

To intercept the EmptyFile (file with “EmptyFile” string) on the send side, I used the method: ProcessMessage in file DotNetFileTransmitterEndpoint.cs under project DotNetFile.csproj.
I inserted the following snippet after cloning the stream. This writes an empty file.
StreamReader sr = new StreamReader(Stream);
string fileContents = sr.ReadLine();
if (fileContents.Contains("EmptyFile"))
{
fileStream.Write(new byte[0],0,0);
return null;
}

Dynamic Send Port:

In order to get this adapter to work with Dynamic Send port following changes will be needed:

1. Use DotNetFILE:// (same as the entry in registry) instead of FILE:// in orchestration for setting dynamic address

2. In the adapter use the following code to extract destination string from the context properties:

filePath = message.Context.Read("OutboundTransportLocation", "http://schemas.microsoft.com/BizTalk/2003/system-properties").ToString();

Use this destination string to parse the final destination of the file and add code to write the file to that location.

Configuring BAM portal on Biztalk 2010

2010-12-30T10:25:00.002-05:00

So, I installed Biztalk 2010 on Win2008 server with SQL server 2008 R2 on a 32 bit machine. Biztalk configuration worked smoothly as one would expect it to until the wizard tried to enable BAM portal. Biztalk configuration wizard threw an error saying "Default Web Site can't be validated". I did some research and found that almost everyone that had this error got it fixed by installing "IIS6 compatibility" role on the Server. But in my case, this role was already installed. But just to be sure, I uninstalled it--> restarted the server and installed it back. Only to find that it did not make any difference!

I tried hacking my way by physically creating BAM application on the IIS but did not work either. Since BAM is a pre-compiled version, I could not debug it as I would do in an asp.net web application. Then, I thought of creating a new website called "BAM" at the same level as the "Default Website" but on a different port (not 80). In my case, I used port 81. I ran the Biztalk Configuration wizard, pointed it to "BAM" site instead of "Default WebSite" and it worked!!

Later, I did some research and found out that this is a "bug" or a "known behavior" with Biztalk 2010 installation. It seems creating a new site for BAM is generally accepted solution.

Check out :http://social.msdn.microsoft.com/Forums/en/biztalkgeneral/thread/ef650460-c219-4b34-bc7a-41bb7e75ec81

I would like to hear from others who faced/or not faced same issue.

Biztalk and CEP (Complex Event Processing)

2010-12-27T12:26:00.004-05:00

For people interested in integrating CEP with Biztalk, check out the white paper by Jesus Rodriguez at :

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6c1da607-4785-4668-a274-6898c38ea603

StreamInsight is part of SQL 2008R2. Do some reading on StreamInsight before checking out this paper.

This paper is very conceptual and does not provide any code snippets. If you are looking for some code samples, then use the samples that come with StreamInsight download and extend them.

Appfabric Connect Feature on Biztalk 2010

2010-12-20T13:54:00.002-05:00

When Dublin (now Windows Server Appfaric) was close to being released, there was lot of talk around the speculative demise of Biztalk. Some of the key differentiators identified at that time were Mapper tool and ability to resume message flow using persistence data store. At that time, I wrote that it will be matter of time that these differentiators between Biztalk and Dublin will fade away (check the comment section of http://geekswithblogs.net/cyoung/archive/2008/10/15/125848.aspx). Current Avatar of Dublin aka WS Appfabric , provides the ability to persist data and supports resumption of messages. “AppFabric Connect” feature, that is part of Biztalk 2010 installation, exposes Biztalk mapper and LOB directly. So to be used with workflows, that can then be hosted on Windows Server Appfabric which in turn is part of IIS. Light weight applications does not need heavy weight Biztalk database anymore. Low latency scenarios can now easily be constructed using IIS hosting using Dublin.

Since this feature is part of Biztalk 2010 installation, Biztalk license is needed to use this feature. In future this can and will change. It will then be possible to create custom WF that can be persisted to a DB and exposed as WCF endpoints, while mapping .net types using Biztalk with same set of mapping functoids that are part of Biztalk world. And all of this will be free!

Little trick....that will help you long way

2010-12-16T14:07:00.003-05:00

During the process of creating cloud project using VStudio, don't forget to mark external "reference" assemblies as "Copy Local =True". If this is not done, your application will run fine on your local cloud emulator but throw an "asp.net application error" when executed on the cloud. Every dll that is referenced, other than the default dlls, that are part of cloud project, should have property "copy local = true". When Visual Studio creates deployment package for cloud, it looks for this property. It adds all the dlls with "copy local = true" in the package and when the solution is deployed on the cloud, these additional dlls are deployed on the cloud too. In the absence of these dlls, your cloud application will throw “asp.net application error”.

Biztalk and the Cloud

2010-12-14T15:35:00.015-05:00

I was in NY city on Dec 1 for the Azure Firestarter event at the Microsoft Office. It was a nice overview of Azure. For past year or so, I have been thinking of getting my hands dirty with this technology. Between work and family, never really got a chance to do so. But this time, after the event I promised myself to actually put what I learnt into practice. Moreover so, because the free registration to Azure portal would expire in few weeks time.

So, I came up with a simple scenario.

1. Expose a CreditScore service using Biztalk within a firewall.
2. Publish this Biztalk Orchestration as WCF to Cloud Endpoint using WCF publishing
wizard
3. Use a simple windows client application to send a message and receive a reply from this Biztalk service exposed within firewall.

Step 1:

Create a two way Biztalk Orchestration that accepts an xml message(First Name, Last Name, SSN) and returns a message (Credit Score and Comments).

(Image 1)

Step 2.

Publish this Orchestration as the cloud endpoint.

Pre-Requisite:

1. Biztalk 2010
2. Windows AppFabric SDK 1.0
3. Windows Azure AppFabric Account to create the URL of cloud Endpoint (Free, if you have MSDN subscription)
4. Biztalk 2010 adapter pack

For details checkout : support.microsoft.com/kb/2449478

Step 2. Create a windows client to test this Biztalk service using endpoint exposed in the cloud.

Pre-Requisite for the client machine:

1. Visual Studio(2008 sp1 or 2010)
2. Azure AppFabric SDK 1.0 (to get NetTcpRelayBinding and TransportClientEndpointBehavior).

a) Open a "Windows Form Application" Project

b) Add Reference to "Microsoft.ServiceBus.dll" from location

"C:\Program Files\Windows Azure AppFabric SDK\V1.0\Assemblies\NET3.5\Microsoft.ServiceBus.dll"

c)Add Service Reference point to the URL exposed in the cloud : Sample URL:

shashiraina-demo.servicebus.windows.net/Test.Biztalk.Cloud.Orch/. This namespace in URL is created using AppFabric Account. (Image below)

(Image 2)

d) Use the proxy to generate client side objects. I used code instead of config file because, I was not able to get "TransportClientEndpointBehavior" and "NetTcpRelayBinding" to show up in Config file. This may be because I was just using a Windows Application as test client instead of "cloud" supported client. With code, I could easily use the ServiceBus namespace and get it working.

Sample client code is below:

GetCreditScore.GetCreditScoreRequest req = new GetCreditScoreRequest();
GetCreditScore.GetCreditScoreResponse res = new GetCreditScoreResponse();
NetTcpRelayBinding tcpRelayBinding = new NetTcpRelayBinding();
TransportClientEndpointBehavior clientEndBeh = new TransportClientEndpointBehavior();
clientEndBeh.CredentialType = TransportClientCredentialType.SharedSecret;
clientEndBeh.Credentials.SharedSecret.IssuerName = "owner";
clientEndBeh.Credentials.SharedSecret.IssuerSecret = "Extract this from Appfabric Portal";// (it will look something like the crossed out section in the Image 2" )
EndpointAddress endpoint = new EndpointAddress("sb://shashiraina-demo.servicebus.windows.net/Test.Biztalk.Cloud.Orch/WcfService_Test_Biztalk_Cloud_Orch.svc");

WcfService_Test_Biztalk_Cloud_OrchClient client = new WcfService_Test_Biztalk_Cloud_OrchClient(tcpRelayBinding, endpoint);
client.Endpoint.Behaviors.Add(clientEndBeh);

req.Score = new Score(); ;
req.Score.FirstName = "Shashi";
req.Score.LastName = "Raina";
req.Score.SSN = "123-234-2323";

res.Root = new Root();
res.Root = client.GetCreditScore(req.Score);

Response from the service is below: (Go Ahead and tried it out from your machine)!

Image 3

So, without punching a hole in your DMZ, you could expose your Biztalk services to outside world. I have been involved in complex configurations in ISA reverse proxy to expose Biztalk services to outside world. But using "Appfabric Connect for Services" , all these complex configurations can be eliminated. Moreover, the ACS (Access Control Services) that is part of Azure Appfabric allows creating your own plug-in security framework.

Integrating Biztalk with Mobile CE device

2010-11-29T17:07:00.000-05:00

One of my recent projects involved developing Biztalk services to be consumed by Windows CE mobile devices. I got a chance to work with compact Microsoft framework. In an ideal world, one would expose Biztalk as WCF service and use advanced feature set to integrate the services with the mobile client using compact framework. There is a whole set of compact power tools with own svc generating utility that matches svcutil.exe of .net framework world. Compact framework svcutil may work well with pure WCF kind of services (not tested though) but it will not work with Biztalk services that are exposed as WCF using WCF publishing wizard. It seems there is a bug in the svc util tool for compact framework and the only other option is to consume exposed WCF as a webservice from the client side. This can be done by simply using Add Web Reference option in the mobile client project. Doing this will generate requisite client side proxy objects to talk with the exposed Biztalk service. Service side is to be exposed as WCF with basic-http binding to be consumed as a web service from mobile client. There is one caveat to this approach though. While using a SSL at transport level, it was observed that the client could not pass the request to the service hosted inside the corporate domain from a client situated on WAN (internet). Everything would work fine for calls made within the LAN though. The key difference between LAN and WAN scenarios was the use of reverse proxy ISA. Client requests would have to be routed through ISA for WAN scenarios whereas in LAN scenario ISA will be automatically by passed. This issue could be circumvented by exposing Biztalk as ASMX service (instead of WCF) and using SOAP adapter. Apparently, the security information that is carried over from client through ISA to Biztalk is carried in the envelope part of the SOAP message over the wire. It is possible that the way this security information is formatted in a service exposed as WCF v/s the one exposed as ASMX service is different. That could possibly explain why the client call will work in WAN scenario for ASMX type service and not work for WCF type service.

Batching Non-EDI received documents

2010-09-28T16:00:00.003-04:00

EDI Batching in Biztalk is done using Batching orchestration that is shipped with Biztalk. This batching orchestration is directly binded with MessageBox and picks up message with EDI.ToBeBatched=True. This flag is set when the message is received using edi receive pipeline. There is a component called "BatchMarker" in the receive pipeline that promotes and sets "EDI.ToBeBatched=True".

In scenario when the incoming message is a Non-EDI type (i.e a Positional Flat file), we have two options to promote the context properties that are required for Batching orchestration to work correctly. First option is to use Orchestration and second is to use a custom pipeline. In this example, I will demonstrate how to use an orchestration:

1. Create an orchestration that takes the Flat File message as input message
2. Create a Outbound Msg (Msgout) representing EDI schema
3. Add a map that does FF-->EDI conversion.
4. Create a correlation set containing 3 EDI context properites. Set these 3 propeties inside an expression shape within the context of a constuct block.

EDI834_Out(EDI.ToBeBatched)=true;
EDI834_Out(EDI.BatchId)=1; //get this ID from "Parties"
EDI834_Out(EDI.EncodingType)=0;// 0 for X12; 1 FOR EDIFACT

5. Initialize the correlation set in send port. This is the key step to promote the 3 context properties else you may end up "writing" the properties and not promoting them.
6. Bind the Orchestration Send Port directly to MessageBox
7. Create a Send Port that Filters "EDI.ToBeBatched=False". You may further differentiate by adding filter for BatchID.
8.Deploy the orchestation and start the send port.

Modifying Biztalk EDI Schemas

2010-09-04T23:54:00.006-04:00

There are times when standard Biztalk EDI schemas need to be expanded to include additional validations. It is a 2 step process.

1. Modify the relevant EDI schema and preferably modify the namespace to differentiate it from standard schema. Deploy it.

2. Go To-->Biztalk Admin Console-->Parties-->Select Partner--> Right Click-->EDI Properties-->Click on "Validation and ACK"-->Check "Extended Validation" (Image below)

Checking "Extended Validation" tells Biztalk Engine to validate against the custom schema validations. This is in addition to standard EDI schema validations that come out of box.

Extending VPC hard drive

2010-08-27T09:45:00.004-04:00

It has been a while since I last blogged. During this period, lot of things have happened in my life. We were blessed with twins and I took a new role with a new company. A role that lets me focus on integration more deeply and also puts me more closer to my clients. Recently, I completed a POC integrating Mobile Windows Devices with Biztalk. It was an exciting experience. No, this blog is not about Mobile integration. This is about something that we face a lot while using VPC images. Well, sooner or later we all will run out of C drive space on our VPC. Yes, it is straight forward to add a new hard drive and name it anything (say D). But that still does not do anything to existing C drive space. There are some programs that by default install on C drive and for these programs C drive should have enough space. There is a way t0 install programs that are looking for C drive to a D drive for example. This post is not talking about that. This post will help to extend existing C drive without using paid softwares like Partition Magic. I actually used a free software to achieve this. Below are the steps:

1. Add a new hard drive (.vhd file) to VPC of desired size but more than the existing hard drive (.vhd drive) that needs to be extended. Call this New partition D:
2. Use DriveImageXML a free software to create an image of existing hard drive (C:)
3. Restore the created image on to (D:)
4. shut down the VPC and remove the original drive C and keep only drive D:
5. Fire it up, you may get issues in booting up the new drive. Use http://www.runtime.org/driveimage_faq.htm. Go to section:
I restored my boot drive, why does it not boot?
and follow Fix Manually section.
6. This should boot up the machine with new hard drive D:. But you may end up having a blank desktop and no programs will work. This is because they all were installed in drive C and the new drive being used is D.
7. To fix this use: http://support.microsoft.com/kb/223188

Using Biztalk in transaction mode with a WCF client

2008-12-27T14:31:00.004-05:00

While trying to use a WCF client to send a message to Biztalk as a transaction, I got the following error:

"Cannot access a disposed object. Object name: 'TransactionScope' "

On investigation, I found that this can happen when:

Recieve port/location does not have a corresponding subscription (orchestration / send port) created.

Send Port is created but not enlisted. This is same as no subscription found in database.

When enable transactions check box is clicked while configuring receive location as shown below:

Behind the scenes, a transaction scope object is created that will attach itself to incoming message sent from a client and form a one continious transaction involving MSDTC. This enables WCF client to control the whole transaction involving Biztalk. Client can use "commit" and "rollback" commands to commit and rollback message in Biztalk messagebox database. Very powerful!

Reason:

At core, Biztalk works on pub-sub mechanism. In absence of any subscription, receive location can't really participate in transaction and create a transaction object. Hence the error is raised "cannot access a dispose object" (transaction object)

Solution is simple: Create a subscription for the receive port/location in form of a send port or an orchestration.

Using SAP Biztalk 2006 R2 WCF adapter

2008-10-05T12:43:00.012-04:00

Work involved in configuring SAP adapter can be divided into 2 steps. First step involves work done by Biztalk team and second step involves work done by SAP team.

Step 1: BIZTALK TEAM

Biztalk side needs to define a Program ID. SAP adapter then registers this Program ID to the SAP gateway defined in the URL. This Program ID then needs to be registered from SAP side. SAP admin should follow “Performing Tasks Using the SAP GUI for Specific SAP Adapter Scenarios” section of SAPAdapter.chm file.

Program ID will be unique per listener else SAP will send the document to random listeners. Each Receive location will be tied to a unique Program ID for an environment. Example: If same receive location is being used in 3 environments (DEV, QA and Prod) then 3 Program IDS are required to be set up for that single receive location. RFC checks the connection between SAP and biztalk using program id registered channel. SAP then sends IDOC to biztalk. Biztalk accepts this IDoc and processes it.
Configure recv location as shown:

Configure following URL at recv location

sap://CLIENT=[client ID] ;LANG=EN;@a/[servername]/[system ID]
?ListenerGwHost=[hostname]&ListenerGwServ=[gateway service name] &ListenerProgramId=[Program ID}

SAP team should supply with the above variables.
Do not forget to ask for user name and password to configure screen below:

Step 2: SAP TEAM

SAP team to supply:

1. client ID
2. servername
3. system ID
4. hostname
5. gateway service name
6. User ID
7. Password

Configuring BAM and WSS adapter on 64 bit machine

2008-04-24T13:23:00.005-04:00

BAM and WSS adapter have conflicting requirements for 64 bit machine. BAM needs 32 bit IIS and WSS 3.0 needs 64 bit IIS. To make matters further complex, it is not possible to host 32 bit IIS and 64 bit IIS on same 64 bit machine. So how can Biztalk communicate with a MOSS server running on 64 bit machine? Following configuration makes this possible:

1) Enable 32 bit IIS on the 64 bit machine by running following command:

cscript %SystemDrive%\inetpub\AdminScripts\adsutil.vbs set w3svc/AppPools/Enable32bitAppOnWin64 1

(1 means “true” and 0 means “false”)

2) Now, IIS is 32 bit and follow standard BAM configuration steps.
3) Install WSS 2.0 SP2 on the Biztalk machine to enable configuring WSS adapter. Also, configure WSS adapter on MOSS machine that hosts WSS 3.0
4) To install WSS 2.0, register ASP.Net 2.0 (32 bit version) from “Windows\Framework” folder by running aspnet_regiis.exe –i
5) Download and install WSS 2.0 SP2
6) Install WSS 2.0 in server farm mode and then create and extend the website. This website will host the WSS adapter.
7) When installing WSS 2.0 on clustered machine, use option “Join existing …”
8) Run Biztalk Configuration wizard and configure WSS adapter on Biztalk machine.
9) Run Biztalk Configuration Wizard on MOSS machine and configure adapter there.
10) Make the AppPool Account that is running WSS 2.0 website a member of SharePoint Group and SharePoint Site Collection on MOSS machine hosting WSS 3.0.

When WSS is configured, it creates BTSharePointAdapterWS. WSS adapter uses this web service to communicate with MOSS form libraries, lists and document libraries. This service need not be running on Biztalk machine but it should be running on MOSS machine for proper communication back and forth.

Cross Updating Windows Workflow Tasks in Parallel Activity

2008-01-08T12:01:00.000-05:00

Parallel activities in Windows Workflow are independent as the name suggests. How could we then update Task A from Task B when Task A and Task B are running in parallel? After spending 1 hour, I figured it out.

1) Use "Update Task" shape for updating Task A and drop it in the "parallel" arm of Task B. The position of this task could be above the "Complete Task" or as required. But it should be under the "create task" shape for Task B.

2) Here is the imporant part. Set the correlation token as the correlation token used for Task A even though this task is under Task B related tasks. This correlation token will direct the update task to the correct task (Task A in our case)

3) Bind the TaskID to existing Task ID for Task A.

3) Use syntax "TaskProperties.ExtendedProperties["NameOfInfopathField_inTaskA_thatis_to_be_updated"]="desiredvalue"

Important:

Parallel workflows in WF are actually processed in a sequence. Left branch is processed first and then control moves towards the right . While cross updating tasks, correlation ID is the key that ties the different tasks together. For tasks on the extreme right arm, all the correlation IDs in the workflow are available. But this selection reduces are we move towards left. In order to update a task on the right side of a given task, the correlation ID will not be available from the work flow design view. But going to deign code file(*.designer.cs / *.designer.vb) gives the flexibility to link the correlation ID with the task that is to be updated.

i.e.

this.UpdateFromTaskOnLeftSide =CorrelationToken (this will be token for the task on the right side that is being updated from left side).

Accessing SQL data from Infopath Form Control

2008-01-07T11:31:00.000-05:00

A control in infopath form can be configured to receive data from XML, WebService, SharePoint List and SQL database (only). In this article I will focus on using SQL database as secondary datasource. I used SQL database to hold values for login in one column and second column gave the user groups for that LogIn. I used these Groups to control the views that a user can access. For Example: You could add rule to restrict an IT user to view data for IT only and not for Finance or Marketing.

This is what you do:

1) Create a table and name it "RulesGroup". Add two columns : UserLogIn and UserGroup

2) In the infopath form create two text box control: UserName and UserGroup

3) Create a secondary datasource and configure it to receive data from SQL database. Use the wizard to point to the correct table.

4) In the infopath form: Use infopath function username() to populate UserName text box.
Map the UserGroup text box with following Xpath:

xdXDocument:GetDOM("RolesGroup")/dfs:myFields/dfs:dataFields/d:RolesGroup[@UserLogIn = "shashi"]/@UserGroup

PS: XML generated by the wizard marks the 2 user defined columns as "attributes". So there is need to user @ in Xpath expression.

In our case if UserName is "shashi" the UserGroup will be "Operations".

New users and Groups can be added in DB as required and accordingly Infopath Views can be programmed based on Groups.

Remember: While using form services to host infopath form, additional requirement will be creating a new Univeral Data Connection(UDC) file in Data Connection Library under the site collection. This UDC file maps the connection details between infopath form and SQL service. SQL server can't really authenticate the user name being passed directly from the infopath form. The user name comes as "" (blank). One way of getting around is to use proxy user name and password in case the user name is blank. But this will compromise security in Big Way. Use of UDC file helps in this regard. Also, it allows you set up custom User Groups based on User Names. These user groups can be then used to limit access to certain views as explained before in this article.

Integrating MOSS, WF and Infopath

2007-12-31T14:51:00.001-05:00

I ventured into relatively "unchartered" territory of SharePoint world. I have been working on setting up a a framework involving MOSS, Windows WorkFlow and Infopath 2007. Business requirements are pretty standard. Client wants to set up a business process for "New Hire". When a new hire joins in certain business processes need to be fired. Like creating domain accounts, allocating hardware and setting up the user in different databases. Some of the processes will be executed in series and others in parallel.

Goal is to set up a scalable architecture that can potentially hook up with Biztalk in near future. Essentially we are setting up "Business Process Management (BPM)" system. Recently my company established a partnership with a company (PNM Soft) that is one of the leaders in "BPM" software. I evaluated their software to be in a position to recommend it to my clients. It is a very powerful and easy to use software. Does not require much of coding and even a pure business user can design complex BPM processes using it.

My ongoing work in developing a scalable framework for my client using MOSS and other tools will also help me in determining the value added by the "BPM" software with respect to designing a BPM process from scratch using standard MSFT technologies.

Solution:

1) 2 Infopath forms ( First form is a brower enabled form that is published on "sharepoint", Second Form is infopath Task form that is published on "network"). I tried to get "Single" form to do both. But I ran into problems. Sharepoint won't let me publish the same form as "workflow" attachment and also as "sharepoint" enabled form. The second form is actually what you will call "Task Item" Form. It is one of the 4 forms that Workflow uses.

2) Workflow that is attached to a form library. Users publish the "first" form to the form library that triggers the "Workflow". This work flow is "sharepoint" sequential workflow.A good working demo can be found at:

http://weblog.vb-tech.com/nick/archive/2007/02/25/2207.aspx

Following are some helpful tips:

1) Cloning an infopath to create a second form is not a good practice. Infopath has a unique internal ID that will also copy over to new form and can create issues when both forms are used in a WF set up. I tried using one form as an Initiation form and second as Task Item form but could not get it working. I came across issue in passing parameters from work flow to the "Task Item" form. I am not aware of any workaround. But I think it is a good practice to create forms from scratch. I will not recommend the "schema first" approach for same reasons.

2) Make sure that the "network" published infopath forms (*.xsn) listed under Metadata section of "workflow.xml" (part of WF solution) exists and are under the correct folder location. Delete the references of forms not being used from "workflow.xml". Else you may get error like "Form is closed".

3) If you are using VSTA to code infopath forms. Make sure that the "dll" file is also part of the folder where *.xsn forms are located. Else you may be get an error " Type: InfoPathLocalizedException, Exception Message: The specified form cannot be found". Another thing that is to be kept in mind is use of "FormState" to define global variables. For browser enabled forms, there is need to preserve state across browser sessions. "FormState" helps you do that. The other option is to use workflow to pass the values into the "resource" file that is part of the infopath form. This "resource" file serves as "Secondary" data source and accepts values from Workflow and then updates the Infopath (Task form). I like this option better because using WorkFlow to maintain the state ensures that the "infopath" form is updated only after the workflow is executed. Thus making
sure that "state" the infopath work represents is actually the state that workflow
indicates. There is no need to use "FormState" property in latter case.

Example: Suppose user clicks on a check box on "Infopath Task form" indicating that the particular step is complete and workflow should flow to next step. If one preserves this state (of "check box" being clicked") using "FormState" then even if the workflow gives error the "checkbox" will still show as clicked and user will get an impression that the process is completed successfully as "check box" is clicked. But having workflow update this "checkbox" using "Extended Properties" ensures that this check box preserves the right state only after the workflow has done what it was suppose to do.

I noticed that order of populating "secondary data source" from workflow does matter. If the order is not correct the values will not propogate from workflow to secondary data source to Infopath form.

Check out the image below:

Note: The order of fields in "secondary" data source schema matches the order in which WorkFlow assign values.

Use "Trusted Security" while publising forms with embedded VSTA code.

PS: VSTA and VSTO carry the same weight when used for "broswer" only forms.

4) Windows Worfklow dll needs to be GACed but the dll for VSTA code is not required to be GACed. But it needs to be copied to the same folder on network where the forms are published.

PS: Suppose you create a infopath form with embedded VSTA code and then you remove the code before "publishing" it. You still need to "copy the dll" file in the correct folder containing .xsn form. Else you will get "Form have been closed" error. I found this hard way!

5) Everytime infopath form is changed it should be republished to the network. Also updated dll should be copied. Unistall/Install the feature. Workflows that are already in progress will be updated with the new changes. Keep this is mind before applying the changes. Accordingly wait for "In Progress" workflows to finish.

6) There is no need to RE GAC the workflow dll if only infopath forms are changed. Also it is important to restart the IIS after GACing any new version updates. Restart ensures that the new changes come in effect.

7) Do not try to change the Unique ID of form by going to "File----> Properties" in design view. It seems that there is more than 1 place where this unique ID is referenced in the form. This is the same ID that is referenced in "workflow.xml" and indicates to the workflow which form should be opened as Task Form, Initiation Form, Association form and Modification
Form.

8) "check box" control in Infopath form can be changed from "Boolean" data type to "Text" data type and can be used to accept updates from workflow and this also gives more flexibility to
program it using VSTA and VSTO.

To Do this:

Go to Design View -----> DataSource-----> Double Click on the "check box control" ----->Following type window will "pop-up"----->Change Date Type from "Boolean" to "String"---> This will enable the icon "Fx" and then click there. Now you can select the value that you want to pass from "Secondary Data Source" . This seconday data source will be configured to receive the XML from workFlow

Also, double click on "check" box in form view and make the properties for the "check box" look like.
Now when you pass values of "true" or "false" as string datatype from workflow, the check box will accept them.

9) Use of Initiation form gives easy access the XML data stream that is following into WorkFlow. XML stream can be accessed through workflowProperties.InitiationData. I did not want to use Inititiation form as I wanted to "automatically" start WorkFlow when something new is published or something existing changes in Form Library. To do so you need to use following code in "WorkFlow Activity activated shape"

SPWeb thisWeb = new SPSite(workflowProperties.SiteId).OpenWeb(workflowProperties.WebId);

SPListItem thisItem = thisWeb.Lists[workflowProperties.ListId].GetItemById(workflowProperties.ItemId);

byte[] fileBytes = thisItem.File.OpenBinary();
UTF8Encoding encoding = new UTF8Encoding();
String xmlString;
xmlString = encoding.GetString(fileBytes);
xmlString = xmlString.Trim();

// Desiralization. InitForm is class generated from schema of incoming message

XmlSerializer serializer = new XmlSerializer(typeof(InitForm));
XmlTextReader reader = new XmlTextReader(new System.IO.StringReader(xmlString));
InitForm initform = (InitForm)serializer.Deserialize(reader);

At the end, I concluded that the BPM software is very efficient when it comes to designing pure BPM solutions. MSFT tools are definitely powerful in complex B2B scenario and integration with other systems. But they are "overkill" for simple BPM scenarios.

Implementing WCF

2007-08-29T17:53:00.000-04:00

Why WCF?

WCF aka Windows Communication Foundation services is latest product from MSFT for creating Distributive Applications. Year 2007 has been buzzing with new product releases on a weekly basis. It is sometimes hard to keep track of latest releases. I can speak for myself that I end up missing some new releases now and then. It is tough to keep one eye on client work and another eye on MSFT press releases. Now it has become part of job. Working in consulting sector is a 24/7 job. When you are not working with client, you are scanning different websites looking for bits and bytes on latest product releases and technology trends. I do love my job but sometimes it gets really tough to keep up with the pace of technology changes. Sometimes you have to be like Tiger Woods and chose to leave. If only everyone was as good as him!

Distributive Technologies have become center of turf war for technology companies. Webservices is an important piece in this war. Turf is dotted with companies like Java and also open source products like Tungsten.

WCF is a further enhancement of ASMX webservice released with .Net framework 2.0. WCF is part of .net framework 3.0 version. Version 3.5 is out too in form of beta version. WCF lets users get benefit of ASMX , WCE and messaging services as one product. WCF supports MSMQ, HTTP and TCP protocol. ASMX webservice only supports HTTP. Additional transports can be configured to WCF. WCF also offers configuration editor with GUI for monitoring performance, tracking messages and errors. This tool helps in reducing operation cost with out of box monitoring and tracing of data. Details at the end.

It is highly flexible and configurable solution but does have some drawbacks as compared to .Net remoting. Especially when it comes to running multiple service host instances under single host. .Net remoting actually lets you add service instances. WCF can be run as self-hosting service, IIS or WAS. While running WCF as self hosting service, check the Task manager window. WCF service will show up as Services.exe. There will be only one instance of this service at one given time. Multiple instances are not supported.

Demo:

Used MSFT base sample, Organized it and added personal notes and updates
1) Create a Blank Solution call WCF
2) Add a Project and Call it “Host”
3) Add a C# library file and call it “Host.cs”
· Add reference to System.ServiceModel.dll
4) Add another Project and call it “Client”

I ) Create WCF Host

5) Define Service Contract
// Define a service contract.
[ServiceContract(Namespace = "http://Microsoft.Demo.WCF")]
public interface IMath
{
[OperationContract]
double Add(double n1, double n2);

}
Attribute “ServiceContract” marks the Interface to carry metadata for WCF.
Attribute “OperationContract” is equivalent to “Public” keyword. Client can access the service.
PS: Since the datatype used in simple we have not explicityly defined “DataContract”. For complex datatyype it is important to define DataContract explicitly. It is also a good practice to keep “ServiceContact” and “DataContract” as separate implementation.

6) Implement the service

Implementing service contract is a rather easy process. Just use a C# (VB) class and implement the service interface. This is the part where the logic behind the service is implemented.

//Implement the service
public class MathService : IMath
{
public double Add(double n1, double n2)
{
double result = n1 + n2;
Console.WriteLine("Return: {0}", result);
return result;
}

7) Create Host for the service

Till now the service is in form of library class. We will need a host to host the service. As mentioned earlier, there are 3 options:

a. Create Self Host
b. Use IIS
c. Use WAS

In this demo we will create our own host because it is easy to Debug. Before we create a host, we need to define the address where the service will reside.

This step will contain following five steps:

d. Create a base address for the service.
e. Create a service host for the service.
f. Add a service endpoint
g. Enable metadata exchange.
h. Open the service host

// Step d
Uri baseAddress = new Uri("http://localhost:8000/DemoWCF/Service");

// Step e
ServiceHost serviceHost = new ServiceHost(typeof(MathService), baseAddress);
try
{
// Step f
serviceHost.AddServiceEndpoint(
typeof(IMath),
new WSHttpBinding(),
"MathService");

// Step g
ServiceMetadataBehavior smb = new ServiceMetadataBehavior();
smb.HttpGetEnabled = true;
serviceHost.Description.Behaviors.Add(smb);

This step is required to run the svcutil command later on to download 2 client files (config and class). Otherwise user may get following error:

“ There was no endpoint listening at http://localhost:8000/DemoWCF/service that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
The remote server returned an error: (404) Not Found.”

This step is optional if user decides to manually create both client files.

// Step h
serviceHost.Open();
Console.WriteLine("The service is ready.");
Console.WriteLine("Press to terminate service.");
Console.WriteLine();
Console.ReadLine();
serviceHost.Close();

II) Create WCF client

8) Run svcutil command to generate client class and config file

C:\Program Files\Microsoft Visual Studio 8\Common7\IDE> svcutil /language:cs /out
:generatedClient.cs http://localhost:8000/DemoWCF/service

2 generated files will be : output.config and generatedClient.cs.

9) Add both of these clients to “Client” Project. The complete solution will look something like the imaage below. Note the added references. Rename them to app.config and MathClient.cs

10) Add the following code to Client.CS

namespace Microsoft.Demo.WCF
{
class Client
{
static void Main()

{

try
{
//Create an EndpointAddress instance for the base address
This Endpoint address should be concatenation of baseAddress variable in Host class and stringaddress paramter in AddEndPoint method of Host class.

EndpointAddress epAddress = new EndpointAddress("http://localhost:8000/DemoWCF/Service/MathService");
MathClient client = new MathClient(new WSHttpBinding(), epAddress);

//Call the contract implementation in Host class from Client class.
double value1 = 29;
double value2 = 16.00;
double result = client.Add(value1, value2);
Console.WriteLine("Add({0},{1}) = {2}", value1, value2, result);

//Close WCF client
client.Close();

Console.WriteLine();
Console.WriteLine("Press to terminate client.");
Console.ReadLine();
}

This is simple scenario where we have single service contract , single implementation and single endpoint.

Scenario:

A new client comes in and wants to a new service contract called “Subtract”.

Two ways of doing this:

1) Add the new service “Subtract” in existing interface IMath
2) Create a new Interface IMath1 for “Subtract”

Add the new service “Subtract” in existing interface IMath

Most simplest of updates. But it does not gives us seperation between both as it is part of same interface. Implementation is still with same class and same service host and endpoint. Only thing that change is on the client side where the client calls “client.subtract” just like “client.Add” in the demo above.

Create a new Interface IMath1 for “Subtract”

Create a new interface say IMath1 that contains “Subtract”. Create a second class say “MathService1”. We can’t use the same class to implement 2 interfaces under the same namespace, so we will need to create a new class to implement the new interface (service contract). Another problem that comes with this is the need to have 2 service shosts. Each service host is tied to unique class (service contract implementation). Since we are using “self created” host we can’t create 2 service hosts to implement 2 unique classes.
This is one of the drawback of WCF. Unlike .net remoting where we can add multiple instants of service host under single host, we can’t do same in WCF.
Using IIS as host can help solve this issue.

Key Points

1) Each WCF service class implements at least one service contract, which defines the operations this service exposes.
2) For distributing different set of service to multiple numbers of clients, class can implement multiple endpoints each with a unique service contract.
3) Minimum 1 end point for each service host.
4) Each service host is tied up to only one implementation of contract (class)
5) To expose multiple services, you can open multiple service Host instances within the same host process.
5)
6) Although you can have multiple endpoints with a single service but only service type.
7) You can also dynamically create service with different implementing types. This can be achieved by dynamically creating new Service Hosts.
8) Each endpoint should have unique relative address

WCF Configuration Editor

This tool can be accessed from Visual Studio interface. Go to Tools ---->WCF Configuration Editor.

Read the help file for details. In summary:

1) Open the config file that was generated after running svcutility. This will populate different placeholders in GUI with relevant data. Click on EndPoint folder to view existing configuration. This information can be changed for the GUI without even touching XML file

2) Click on Diagnostics folder and use Toggle switch to browse through various settings. Check out help file for details.

3) Closing the editor after changes will update the changes in the original app.config that was opened in first step. If we open config file, we see additional tags added to perform all the functions that we proposed in GUI.

4) Click on Host.exe file to start the service. Click on client.exe to start the client. All the events will be logged in file(s) whose location(s) can be set from "Diagnostic" folder in GUI.

Rules for Successful BAM implementation

2007-08-21T18:04:00.000-04:00

I learnt from my experience that by observing following rules BAM execution can become fairly smooth: Some of the steps are not "need to have" but rather "good to have".

1) Install SQL 2005 sp2
2) Ensure Analysis Services is installed (Else cubes/aggregations can't be created)
3) Do not forget to Configure BAM tools to "Enable Analysis Services for Aggregation" using Biztalk config tool. (Else cubes/aggregations can't be created)
4) Ensure that account being used is a member of the Group that is configured in Biztalk Config tool for BAM portal. (Else views in BAM portal may not be visible to the user )
5) Ensure that the user account under which BAM Application Pool is running is member of Biztalk Isolated Host and IIS_WG group.
6)Run following commands: (Else you may get 401 authentication error)

cscript adsutil.vbs SET w3svc/NtAuthenticationProviders "NTLM"
setspn -A HTTP/servername domain\account
setspn -A HTTP/servername.fullyqualifieddomainname domain\account

domain\account is account under which BAM App Pool is running on IIS.

cscript can be found under: C:\Inetpub\AdminScripts

setspn.exe can be found under the Windows 2003 CD-ROM at \support\tools\suptools.msi

If you view the BAM portal but can't view all the "VIEWS" you created. It is most likely that the account you are using does not have permissions on "BAMPrimaryImport" table. Add that account (i.e, biztalkuser) and then assign the role (view) you want that user to see on BAMPortal. Check the image below:

In the image above. The user (biztalkuser) only have permission for bam_EndToEnd. If you want this user to see other views that you can for example click on "bam_Test1View". User will now be able to see both views on BAM portal. This way you can selectively grant users access to different views(roles).

Please note: SQL 2005 creates Roles. You just have to assign them to the appropriate user. SQL 2005 does not supports concept of "Groups" anymore. Instead use Roles.

Strange behavior of Biztalk 2006 on SQL cluster environment

2007-08-17T14:05:00.001-04:00

While testing Biztalk application on server farm mode at one of my client, I came across something interesting. This biztalk application used SQL adapter to connect from orchestration to execute stored proc. All tests for server farm went OK. Biztalk server fail over sequence worked fine. Net scala worked fine for web farm. We ran into issue when we tried failing over SQL cluster from original (initial) node to another. On the original node, everything worked perfect. On failing over to second node, I found that Biztalk application would not connect to the custom database to execute stored procedure. We thought may be it is a connection pool caching issue. We tried restarting the physical biztalk servers and hosts. Nothing worked. Then I thought that may be it is something to do with 32 Bit SQL adapter not working properly on 64 bit cluster (not a very smart assumption on retrospection). We also played with MSDTC settings by reducing the authentication level. Nothing worked. The fact that threw me off was that everything was working fine on original node. We only had one physical copy of entire Biztalk database on cluster and not multiple database instances. So, it did not make sense to me as if why the same database was acting differently with different instances of SQL on each node.

I thought logically for 4 hours and tried different things. Nothing worked. I came back next day and starting thinking out of box or rather illogically. We gave explicit permissions to Biztalk service account on the "custom" database that contained stored procedures. We made Biztalk service account database owner and wolla! it worked! Yeah that simple! We could not understand the behavior but it worked. We thought of doing further post mortem. We took the permission away and failed over to original cluster node. It worked as it was working before. For some reasons Biztalk did not care for permissions while working on original node. We failed over again to second node without the permissions and it did not work again. We granted the permissions and it worked!

We left the solution in that working state! But could not explain the reason for this behavior. We have couple of theories but none of them convincing. Is it a Bug? May be. Is it a cluster configuration? May be.

Biztalk and SilverLight

2007-08-07T14:59:00.000-04:00

Gloves are off! MSFT is going head on against Adobe. With recent launch of Silverlight, a product that will compete against market leader Flash. MSFT dreams to snatch some serious business from Adobe. With Java version not lagging behind, market has just started heating up.

SilverLight will enable developers deliver rich browser based applications. Great new tool for web developers. Richer out of Box BAM portal is one area that Biztalk can gain from this new development. Rich BAM portal with interactive charts and graphical real time display of data can really grab attention of Business Users. How about adding this on Biztalk 2008 wishlist!

Implementing Business Rule Engine

2007-08-06T17:33:00.000-04:00

This weekend I got down and dirty with BRE (Business Rule Engine). It is a powerful tool for managing dynamic business rules. It is very efficient in complex logic scenarios. I thought of playing with it and worked out some scenarios.

Scenario:

Company wants to categorize its workers under different “Bonus” levels to decide on dollar value of annual bonuses. Each bonus level is based on variables like Number of Projects sold, Dollar value of each project sold. Is it a new client or existing one? Other variables like future potential etc can also play in. This business logic can change over period of time and needs to be controlled by Business Analysts/ Management team. That means no complex SQL queries. In real world scenario, this means having a policy that can interact with multiple databases/tables on client side. This policy can be invoked by a webservice . This webservice will pass on objects as facts in the BRE. BRE will then update the specified columns in the database with the correct “Bonus” categories.

Scenarios can range from simple scenarios that take single object as input fact to complex scenarios that take multiple objects.

Proof of Concept :

1) Create a simple database called “TestRules”.
2) Create a table called “Customers”
3) Define 3 columns: Author, ID and Status
4) Create a policy called “Testing”.
5) Under this policy, define a rule “Test1” (Image 1)
6) Define 2 Vocabularies TestBRE 1.0 and 1.2 (Image 2 and 3). Difference between them will be the nature of “Binding”. I will explain that later on.

Email me for complete solution @ shashi_1273@yahoo.com

Image 1:

Image 2 (Database Binding type as DataConnection )

Image 3 (Database Binding type as Data Table/ Data Row )

There are 3 facts in the rule:

1. Name of Author (Coming from Database)
2. Input Object on right hand side of “Contains”. This can be a simple string that gives the lookup value as name of the “Author”. In a complex scenario this can be an XML object that contains multiple Fields for “Author”. These name(s) will be then matched against the “Author” column of database “Customers”.
3. StatuD under the action pane. This will be updated based on above 2 facts.

This policy can then be invoked by a WebService . This webservice will use Policy.execute method after passing the required object parameters. For this Proof of Concept solution we will use following combinations:

1) Passing Single Object as Parameter:

In this case the object passed will be database connection only. The second fact can be a hard coded string i.e. “Shashi”. In 2 object scenario we will use an XML object instead of "string". This XML object will be array of values.

Implementation for 2 object scenario will cover this section too.

2) Passing Two Objects as Parameter:

In this case the object passed will be database connection and a XML document. This XML document can contain multiple fields and one of the fields can be “Author”. This “Author” field can have multiple names. In our sample we have 3 names: Shashi, Kent and Jeff.

a) Database Binding type as DataConnection
b) Database Binding type as Data Table/ Data Row

Both of the above types can be set up while using Wizard to creating Vocabulary. Use DataConnection type if the count of rows returned is more than 10 (http://blogs.msdn.com/biztalkbre/ ). Otherwise use Data Table type. Also for Long Term Facts that are based on Caching, use Data Table/ Data Row Type. Reason being that no additional benefit will be gained by caching Dataconnection object as Connection Pool already does that. Implement IFact Retriever for Long Term facts. For this particular example, we will not use Long Term facts.

Important thing to note is that in case of Database facts ( with binding type Data Table/ Data Row ) and XML facts, Typed objects need to be used. .Net facts do not need any “Typing”. You will see in the implementation how both XML and Database facts use TypedXMLDocument and TypedDataTable respectively.

It is a good practice to create a tracking file to understand how BRE works and this file can also be used as a Debugging tool.

Implementation for Case a)

SqlConnection conn = new SqlConnection("Integrated Security = SSPI;Database= TestRules; Server =(local)"); // establish connection to server

conn.Open(); //open the connection

DataConnection dc = new DataConnection("TestRules", "Customers", conn);

XmlDocument xd1 = new XmlDocument();
xd1.Load(@"C:\BiztalkProjects\BRE\TestSmall.xml");
TypedXmlDocument doc1 = new TypedXmlDocument("Books", xd1);
object[] shortTermFacts = new object[2];
shortTermFacts[0] = doc1;
shortTermFacts[1] = dc; // Note: No Typed dataset being used in this implementation

Policy pol = new Policy("Testing",1,0);
DebugTrackingInterceptor dtraking = new DebugTrackingInterceptor("ShashiOut.txt");

try
{ pol.Execute(shortTermFacts, dtraking);
dc.Update();
conn.Close();
}
catch (Exception ex)

{ System.Console.WriteLine(ex.ToString()); }

Implementation for Case b)

SqlConnection conn = new SqlConnection("Integrated Security = SSPI;Database= TestRules; Server =(local)"); // establish connection to server

conn.Open(); //open the connection

SqlDataAdapter da = new SqlDataAdapter("select * from Customers", conn); // create adapter to fill dataset .

//------explicitly define Update method for database adapter object. Directly using da.update will give error

SqlCommandBuilder commBldr= new SqlCommandBuilder(da);

commBldr.GetUpdateCommand();

da.UpdateCommand = commBldr.GetUpdateCommand();

//...

DataSet ds = new DataSet("TestRules"); // create a dataset

da.Fill(ds, "Customers"); //fill dataset

//....Not required for Single Object scenario
XmlDocument xd1 = new XmlDocument();

xd1.Load(@"C:\BiztalkProjects\BRE\TestSmall.xml"); //xml instance to be passed

TypedXmlDocument doc1 = new TypedXmlDocument("Books", xd1);

//............

TypedDataTable tdc = new TypedDataTable(ds.Tables["Customers"]);

// Only pass one fact (tdc) for single object scenario (First scenario)
object[] shortTermFacts = new object[2];
shortTermFacts[0] = doc1;
shortTermFacts[1] = tdc;

Policy pol = new Policy("Testing",1,0); //Major and Minor Versions

DebugTrackingInterceptor dtraking = new DebugTrackingInterceptor("ShashiOut.txt"); //writes the debug file under the "bin" folder.

try

{ pol.Execute(shortTermFacts, dtraking);
da.Update(ds,"Customers");

conn.Close();

} catch (Exception ex)

{ System.Console.WriteLine(ex.ToString()); }

}

}

}

Take Away :

1) While testing a Policy, BRE create Database connection/objects. While executing the policy from .net class, database objects need to be explicitly created.

2) It is a good practice to create tracking file while executing policy. This can help to troubleshoot issues while development. It can also be compared with the tracking log that is created by Business Rule composer when Testing a policy. This way we can observe the list of parameters that policy is expecting and make sure that these parameters are correcly passed while executing the same policy from outside Rule Composer. By comparing these 2 logs, I was able to figure out when policy is expecting TypedXMLDocument and TypedDataTable.

3) DataSet object and XML document have to be passed as "Typed".

4) The 2 Object scenario can be extended for multiple Objects.

5) I know it is very painful to copy and create new versions of policy whenever any modifications are required. Same applies to vocab. Following SQL commands can ease that pain by undeploying the policies/vocabs and allowing users to do modifications. It is highly recommended that these commands are used only in Development phase.

For Policies:

declare @RuleSetId Int
select @RuleSetID =nRuleSetID
FROM re_Ruleset
WHERE strName = 'Testing'
AND nMajor=1 and nMinor=0
UPDATE re_ruleset
SET nStatus =0
WHERE nRuleSetID = @RuleSetID
DELETE FROM re_deployment_config WHERE nRuleSetID=@RuleSetID
DELETE from re_tracking_id WHERE nRuleSetID=@RuleSetID

For Vocabs:

UPDATE re_VocabularySET nStatus = 0 /* to republish make it back to 1 */
WHERE strName = 'Vocabulary1' AND nMajor =1 AND nMinor=0

I think BRE is a very powerful tool if used properly. Key is to use it for complex scenarios that can leverage functions like Update, Assert, Retract. Using it for simple queries is an example of under utilization of tool. In terms of performance, BRE will work better than traditional SQL query in case of complex scenarios.

It is not truely a Business User tool. Concepts of version, vocabulary and facts are not easy for a business user to understand. Also, the fact that any policy that deployed can't be changed unless a new version is created is not helpful either. Developers are required for initial set up and creating new vocab etc. Business users can definitely change variable values that are already set up. They will need some training to do more than that.

I would imagine that MSFT understands these concerns and we can definitely see improvements in subsequent versions.

Biztalk Best Practices

2007-07-31T13:17:00.000-04:00

We recently finished a Biztalk Project with one of our client. It was a 4 week project with tight deadlines. Our team did a great job to deliver quality product on time. I joined the project midway after 2 weeks and finished error handling and fine tuning. The overall architecture of the project was straight out of “Biztalk Best Practices” book. This was one of the reasons it turned out to be successful. I would recommend everyone this approach as documented by Marty.

Scenario:

1. Main Orchestration A will be kicked off by a web service or a File Drop (as back up)
2. Orch A will send Email informing users using Orch B (Email Orch).
3. Orch A will then trigger 5 Different Orchestration (C,D,E,F and G) in a pre-defined sequence. 4. This sequence will be set up during design time. Some of these orchestrations will execute in
parallel.
5. Each of these orchestrations (C, D,E,F and G) will call Orch B after each call before control
goes to next orchestration in sequence.
6. Users will click on Email and that will trigger web application. This web application will send a "correlation" ID back to one of the orchestration (C,D,E,F and G) and that will complete the orchestration. Thus sending control to next orchestration.

Our approach:

1. Use Multipart messages
2. Use Direct Binding through Message Box to link multiple orchestrations.
3. Use separate project Internal and External Schemas
4. Never expose Internal Schemas directly to entities outside Biztalk.

Great deal of information can be found at:

http://msdn.microsoft.com/msdnmag/issues/07/05/BizTalk/default.aspx

We could have also gone with the (Partner Port) approach. This approach would have reduced some of the Orchestration code we had to put for changing the promoted values to eliminate the chances of "infinite loop". But then we would have lost on "flexibility". and client demanded flexibility.

Using Correlation in Loop

2007-07-25T20:26:00.000-04:00

Using Correlation set within a Loop is not a straightforward process in biztalk. One way around this is to define a Scope and within the scope define the correlation sets . This scope will be within the loop . With each iteration of Loop, correlation set will be instantiated again. This approach can lead to overheads due to addtional persistence points in form of transactional scope. Another approach that I recently used in a direct binding scenario is use of "Null Adapter". Have a dummy send shape outside the loop that initiate the correlation set. Use "Null Adapter" with this send port.

DebugViewer in Biztalk

2007-07-25T20:04:00.001-04:00

"DebugViewer" is another cool feature that comes with Biztalk 2006. Actually it is part of Studio 2005. Simple steps to use it:

1) Download and Install the tool from http://www.microsoft.com/technet/sysinternals/Miscellaneous/DebugView.mspx.

2) Use System.Diagnostics.Debug.WriteLine ( " Write the Message Here") inside Orchestration to display the message.
3) Check the capture menu on tool to ensure that all options are check marked except the last one (Log Boot). By default they are.
4) You can even filter the events on the viewer. Add a prefix before the message and use this Prefix to filter. Example: Use System.Diagnostics.Debug.WriteLine ( Prefix +" Write the Message Here") . Filter gives both "include" and "exclude" options.

PS: Ensure that you are using "Development/Debug" mode while deploying the assembly. Debug command does not work with"Release/Deployment" mode. This is added benefit of using Debug namespace. There is no need to remove these lines of code while moving from development to deployment version. These lines will be supressed in the deployment version by design. Not quiet the case if System.Diagnostics.EventLog.WriteEntry namespace is used. In that case, these lines will have to be removed when moving from development to deployment version. Else there will be performance hit.

SWIFTNet Adapter

2007-06-22T14:51:00.000-04:00

I started looking into SWIFT as an apparent area of push by Microsoft into financial sector. For people who have not worked in Financial services sector. SWIFT stands for “Society for Worldwide Interbank Financial Telecommunication”. SWIFT is both a standard body and a secure network like VAN. It also provides software solutions. This secure network connects various banks and corporations. Helping them to settle payments resulting from various buy and sells trades. It is also known as SWIFTNet.

Microsoft introduced first SWIFT accelerator with Biztalk 2004. Biztalk 2006 contains updated version of SWIFT accelerator (version 2.3). Partnership with SWIFT to develop SWIFT accelerator and out of box Rules via Rules Engine is an important strategic move from MSFT to capture Financial Services Market. It is aimed at targeting Back Office where all the trade settlements take place. Java technologies reign supreme in the front end of the trading spectrum using FIX protocol. MSFT aims to capture the back office market that is relatively less volume per day. Here, more emphasis is paid on Security and Reliability and less on speed.

Microsoft is leveraging its influence on Global Banking Industry to push SWIFT standards and SWIFT-BizTalk combination. Internally, MSFT is in process of implementing this solution for its Treasury/Accounting department. It is using in-house implementation as a case study for various corporations and banks to implement Biztalk-SWIFT combination.

Technically, it is a great solution. Users can use Infopath templates to change the Messages. Then use Biztalk as single point of contact to connect with SWIFT network. All the Relational validations in incoming/outgoing message are created using BRE and hence are easy to change by non-technical users. This provides greater degree of flexibility and control in the ever changing world of Banking/Financial Regulations.

Passed my Certification Exam!

2007-05-14T13:23:00.000-04:00

I passed by Biztalk 2006 Certification Exam Today. I would say that I am more relieved than happy. I have One regret though, I wish I was working on Biztalk in my Current project. My last Biztalk Project was 3 months ago and it was heavily tilted towards Biztalk 2004. Ongoing real world project experience in Biztalk 2006 would have definitely helped my preparation . My preparation was spread just over TWO weekends. I used 2 weekends (4 days) to go over Two Books: Biztalk Recipes - 2006, Biztalk - 2004 Unleashed. Tried my hands on SDK samples for BRE/BAM and read as much I could read from Microsoft website. I was sleep deprived on these weekends :)

I will not recommend the approach I took. I had to take this approach because of my situation. Ideally, I would recommended to spend at least 2 days on doing BAM, BRE and other examples from SDK. Working on ongoing Biztalk project will definitely give an edge while preparing for exam. There will be no need to specially allocate time to prepare for the exam like I had to do. I knew that Orchestration, Messaging and Administration were my strong areas. For this reason, I made an effort to focus more on areas like BAM, BRE and Role Link that are not my strengths.

Percentage breakdown of my preparation:

1. 50% of my time on BAM, BRE and Role Links
2. 20% on Orchestration, Messaging
3. 10% What is new in Biztalk 2006
3. 10% on Deployment and Administration
4. 10% on Random Readings on Biztalk Blogs.

Wild Wild World ....

2007-04-21T16:09:00.000-04:00

It has been a while since I sat down in front of my computer to do any Blogging. I joined a new company in February and then took a 2-week vacation. I came back and have been working like crazy. Current project has been very challenging and a great learning experience. It is not an Integration Project and I am diversifying into area of Business Intelligence. I am working on collecting Business Requirement for a Financial System and also doing Project Management. Also providing direction to the financial department in the face of changes they have been going through lately. Doing this I am facing the classic dilemma: to diversify or to specialize?

I love working on Business Integration Projects involving Biztalk and this is my core strength. At the same time I want to get more exposure to other aspect of technologies to give me a broader perspective while dealing with Business Problems and providing the solutions. Well, there are definitely pros and cons of each approach. One of the immediate challenges I am facing is that I have my Biztalk 2006 exam due in less than a month and I still have no plan in place to prepare for the exam. I have been struggling to take out some time but the present project is really demanding and does not leave me anytime on week or on the weekend. The only time I get is at night but then I am too tired to do anything. Fact that I am not working on Biztalk project also does not help me in putting my mind in the right Biztalk perspective at the end of the day when I sit with my Biztalk book to prepare for the exam.

I really need to find a way to put a plan in place and I have to do it soon! Wish we had a 40 hour day.

BAM Deployment Error

2007-02-22T16:11:00.000-05:00

I received following error while deploying BAM project in Biztalk 2006. Error read “BAM Deployment failed.
Encountered error while executing command on SQL server “server name”.
Incorrect Syntax near ‘@@Error’.
Must declare scalar variable “@@UniqueID”.”

Screenshot of the error is below:

I opened the tracer log at SQL server level. I extracted the stored procedure that was giving me an error and ran the stored procedure directly on the server. I got following error:

Msg 102, Level 15, State 1, Procedure bam_Test1_UpsertInstance, Line 15
Incorrect syntax near '@@Error'.
Msg 137, Level 15, State 2, Procedure bam_Test1_UpsertInstance, Line 68
Must declare the scalar variable "@@UniqueID".
Msg 137, Level 15, State 2, Procedure bam_Test1_UpsertInstance, Line 111
Must declare the scalar variable "@@UniqueID".
Msg 137, Level 15, State 2, Procedure bam_Test1_UpsertInstance, Line 140
Must declare the scalar variable "@@UniqueID".

I changed the two variables representing the name of BAM activity “Error” and “Unique ID” to “Exception” and “TransactionID” respectively. Ran the command line for deployment and Bingo it worked!!

It seems that Biztalk uses these 2 variables (Error and UniqueID) internally and will not accept them as user defined variables. There may well be more than two of these variables that will cause similar errors. Well, if we get similar error again then it is very likely that we stepped on some more of these variables.

MOM Architecture

2007-02-22T11:57:00.000-05:00

MOM is another great tool to monitor distributed Biztalk environment. It can easily configure alerts to trigger some activity. This activity can be like sending an email to a person or a group. It can be running some batch file or a script. This event/error based triggering is very helpful while administering a widespread distributive environment. It enables people to take necessary action at right times.

The basic components that constitute MOM are captured below.

Another Successful Implementation

2007-01-24T20:24:00.000-05:00

Months of hard work finally helped us execute another successful BizTalk Implementation this week. This was the reason that I was not able to Blog for past few months. I spent past 3 months managing and organizing the activities for the BIG implementation. It was a great team work and everything fell in place. Hours and Hours of work finally paid. Impact of implementation can be understood from the fact that entire system was off line for the users for almost a week! There were about 5 cross functional teams with about 100 people working on this Big Implementation. World wide offices were under the impact. Biztalk was a major part of the complete implementation if not the entire implementation. Every thing has to be timed to perfection between all the teams to get flawless implementation.

I also want to share a piece of code that I created as a workaround for existing process in place. With the recent implementation the transaction volume is projected to increase FOUR folds both in frequency and size. The present implementation is very database centric as all the business validations are run through queries in Oracle. This leads to instance of “connection pool time out” issues. Even though the code is optimized, this issue is something that we have not been able to completely eliminate. There are definitely some ideas architecture wise that can help. But we neither have budget nor time to implement them. So the workaround for that is to control or throttle the number of transactions passing through Biztalk. Since we can’t tell users not to send files in multiple batches, one thing we can do is to control the execution from our end.

The Plan:

1) Create a common folder that will receive the files
2) Create a way to copy the files from this folder to biztalk pick up locations and then delete the files. The following code is the core of the solution that will do the job.
---------------------------------------------------------------------------
Private Sub Button3_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button3.Click
Dim fileNameCollection() As String
Dim totalFiles As Collection

Dim ActualFilename As String
Dim LastIndex As Integer

Dim SetFolder As System.IO.Directory

fileNameCollection = SetFolder.GetFiles("C:\FileMover\") ' set source folder
Try
Dim i As Integer
For i = 0 To fileNameCollection.Length - 1
LastIndex = fileNameCollection(i).LastIndexOf("\")
ActualFilename = fileNameCollection(i).Substring(LastIndex + 1)
System.Threading.Thread.Sleep(10000) (May not be required when used as scheduled job)

System.IO.File.Copy(fileNameCollection(i), "C:\FileMover\Archive\" + ActualFilename, True)
System.IO.File.Delete(fileNameCollection(i))

Next

Catch
Finally

End Try

End Sub

------------------------------------------------------------

This code will be further made database configurable using SQL database. The variables being “Folder Locations”, “Number of files to be dropped” and “Time Interval” between each file.

Another possible use of this application can be during stress testing. Instead of manually dropping the file, this application can drop them automatically after all the parameters are configured. So switch on the performance indicator and go home! When you come back all the test statistics will be written in the file.

Optional Fields in FF Schema

2006-09-19T08:18:00.000-04:00

My tryst with FF Schema continues. In this edition it is the optional field settings that I will be tackling. I have a positional file with following structure.

A 123 456 789
B 735 323 456
C 824 359 531

A occurs 1 to unbounded
B occurs 0 to unbounded
C occurs 0 to unbounded

For optional group (B and C) I created a delimited level and made nillable = "true" and minoccurs ="0". I did same settings for positional group. I am not sure if it required at both levels so I did as a safety measure.

Now for some reasons, C started acting like an optional record but NOT B. I tried playing around with delimiter settings etc but nothing worked. Then I thought of change parser optimization property at root element. Default is "speed" and I changed it to "complexity" and bingo it worked. It seems that default parser setting did not recognize Min occurs ="0". But it is strange as it worked for C level.

I Googled the issue and I found out that people faced the same issue and did what I did. If only, I would have seen these sites before. I would have saved a DAY!

Flat File Schema and Early Termination

2006-08-29T22:55:00.000-04:00

For those who have worked with Flat File Schemas, must have used or at least heard the property called "Early Termination". This Property is exposed with Biztalk SP2 but otherwise it can only be edited using Text editor. There is some kind of confusion as if what happens if the "Early Termination" is set to True.

Let me illustrate it with an example:

Suppose FF schema have 5 fields (A,B,C,D and E) and each 10 in length. The file looks something like:

AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDEEEEEEEEEE.

Now if we set up "Early Termination" as True. The following scenarios are parsed properly and rest fail in Biztalk schema.

1)AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDEEEEEEE (Parsed)
2) AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDE(Parsed)
3) AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDD (Not Parsed)
4) AAAAAAAAAABBBBBBBBBBC CCCCCCCDDDDDDDDDDE (Parsed)

So this property only allows flexibility for the last field in the schema.

I faced a situation in a project where Source system sent a file that looked something like:

AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDD

It sent less than 5 fields (say 4) and the last field in the file (not in the schema) was partially or completely filled. In Either scenario it will be rejected. Problem was that Source System was very inflexible and would not change the file to suit Biztalk parsing needs.

There were different options that I thought of. I knew that we have to pad the file but question was how to do that. Since Biztalk schema does not pad the incoming file but only pads the outgoing file. And we were receiving the file so we could not use padding.

The other option was do create a pipeline component that would read the stream and pad it before it hits the Biztalk FF schema. Or create some kind of pre processor (windows service) that would do that padding while reading the stream.

It seemed as stream manipulation would be the only way to get around this issue but I wanted to avoid it as I am not a BIG fan of stream manipulation because of inherent dangers associated with it.

So, I thought hard and it clicked to me that I could use the padding property of Biztalk schema to achieve what I needed. All I need to do is to turn the padding property "inside out". In other words. I did the following:

1) Created a simple orchestration with one receive and send shape.
2) Defined a schema with one element only. This way one can take 5 fields together as single field and then use padding property to pad from the last position of incoming field to the maximum positional; length of the complete record in the schema. 50 in our case.
3) In the schema set Padding as Left justified and use Hexadecimal value of "Tab".
4) Define a single message with above schema
5) Create a send pipleline that refers to this single schema with padding.
6) Recv and Send Port will refer to same message.

Complete the orchestration and then drop the file. For a file like:

AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDD

The output will be padded to complete length of 50.

Then run this file through original FF schema with 5 fields. It will be parsed properly and will give you the right XML.

Validating an Instance in Biztalk

2006-07-15T11:27:00.000-04:00

How can one validate an instance of the input file and ensure that the required fields needed for business process are present, data type is correct and min/max lengths are conformed with. This is done to filter “junk” data going into Business Database and to maintain Database Integrity. Further if one needs to catch the errors and populate a database it becomes even more complicated. So what can be done?

Biztalk 2004 has no ability to validate instance on the fly and accumulate errors at once. But, Dot Net offers “SchemaValidation” class to validate the input instance and errors that are trapped can be accumulated and passed as a string through a stored procedure to a SQL/Oracle database. This is how it works.

1. Pass “Instance” and the “Schema” to the schema validation class.
a. Instance is the sample XML message that is to be validated.
b. Schema is the xsd schema that will contain all the information about the required fields, data types, lengths etc.

2. Creating the “tight” schema is a tedious process especially if the file is long. Each field has to be selected and the properties set.

a. For required fields.

i. String type: Set following properties in schema

1. Base Data Type : string
2. Derived by: Restriction
3. Nillable : False
4. Min Occurs: 1 (empty is 1 too)
5. Max length = as required
6. Min length = 1
ii. Date type:

1. Base Data Type : Date
2. Derived by: Restriction
3. Nillable : False
4. Min Occurs: 1 (empty is 1 too)

iii. Integer type

1. Base Data Type : Integer
2. Derived by: Restriction
3. Nillable : False
4. Min Occurs: 1 (empty is 1 too)
5. Max Facet value = as required
6. Min Facet value = as required

b. For optional fields

i. String:

1. Base Data Type : string
2. Derived by: Restriction
3. Nillable : True
4. Min Occurs: 0 (empty is 1 too)
5. Max length = as required
6. Min length = 0

ii. Date Type:

1. Base Data Type : date
2. Derived by: List
3. Nillable : True
4. Min Occurs: 1 (empty is 1 too)

iii. Integer type

1. Base Data Type : Integer
2. Derived by: List
3. Nillable : True
4. Min Occurs: 1 (empty is 1 too)

Note: Decimal Types follow same rules as Integer types. It is not possible to control the lengths for Decimal and Integer OPTIONAL fields from Biztalk schemas. To do this use .Net class that can read the length and validate it in next stage and reject or accept depending on the results.

3. After this “tight” schema is created put this in a folder. Do NOT forget to append a namespace to this schema before passing it in the .Net class. The .Net Class will not work without namespace.

Creating FF Schemas in Biztalk - A Different Approach

2006-05-25T00:18:00.000-04:00

One of my projects requires integrating a file from a Bank into our financial system. The Bank file is a positional flat file with 8 records. The structure is as follow:

1 – File header
......2 – Transaction header
................5 – Voucher Line
................5A – Distribution Line
................5B - Vchr Tax Line
.......7 – Transaction trailer
8 – File Trailer

All the records can occur multiple times in the file. In past, I always had problems creating FF schemas for positional files especially in area of looping. Surprisingly the approach I took this time made it very easy . I will try to break it down in small easy steps:

I will not go in details of "postional" record entry but focus only on "Looping"

1) I used the “Bottoms-Up” approach (this is what I call it). I broke the file into smaller segments and designed starting from the deepest child element and then added other elements as I moved towards the top. i.e Record 5 to Record 1. And then I designed for rest of the file. i.e Record 5 to Record 8.

2) I first designed the schema for only Record 5. I assumed that the file has only 3 records that can repeat multiple times within a root level. The structure looks something like:

5, 5A and 5B are the positional records that repeat multiple times. So they will have to be under a “delimited” record that has max occurs as “unbounded”. This delimited record will have name “Level5” and Child Delimiter type as "Default Child Delimiter" and Child Order as "Infix". I used the same properties for all the "Delimited" levels.

3) Having designed Level 5, I then designed Record 2 . I followed the same principle. Since Record 2 is parent of Record 5, I placed it one level higher than Record 5 but at the same level as the delimited record “Level5”. Record 2 is positional record like Record 5 and can occur multiple times. So it will again go under “delimited” record that has properties as “unbounded”. We will name it as “Level7_2”. As it shared by Record 2 and Record 7 as we will see later.

4) I followed the same logic with Record 1 that is parent of Record 2.
5) Having completed the “Middle to Top” path, I started symmetrically copying the same looping from “Middle to Bottom” to get the entire structure which will look like:

6) At end do not forget to add 1 root delimited record (Root) above Level1_8 that will occur as Single.

7) The last important thing is to set up the properties of the “Schema” node as below:

"Debugging is not supported under current trust level settings"

2006-04-28T09:38:00.000-04:00

If you get the following error while trying to open ASP.net page on the host server.

"Debugging is not supported under current trust level settings"

You will also get 500 error if you hit this page using HTTP POST method from any other machine. This is what you do:

Open the web.config file for the ASP solution and add the "BOLD" text below.

(opening tag)system.web(closing tag)

(opening tag)trust level="Full" originUrl="" (closing tag)

Biztalk Server 2006 Clinic- Summary

2006-04-07T17:28:00.000-04:00

I attended the Biztalk 2006 clinic on Wed 4 th April at MSFT Office in Iselin,NJ. It was very informative. I left wanting to know more about BAM though. For some reasons it feels as there is so much that can be done with this tool. But lack of documentation always gets me. I have tried my hand on it and honestly I do not feel it very flexible and intuitive. BAM portal addition in 2006 is certainly a positive step. Acceptance of BAM by Biztalk community can dramatically increase the value offered by Biztalk. As a Business user nothing is better than able to use BAM. I scribbled some notes from the clinic that I am posting. These notes came handy while making a presentation to my team.

---------------------------

The presentation started with videos of some companies that are using Biztalk 2004 at present and how they are benefiting from it:

1) INVENSYS: Uses for manufacturing plant
2) Balboa: Financial/Insurance Company

These companies’ uses “Rules Engine” that enables Managers/Operational people to change the variables based on scenarios. This provides them with flexibility

Showed Funny Video We-SYP (We share ur pain).

First Speaker: Sai Prakash from MSFT

Managing and Monitoring:

Tools:

1) Admin Console : General Managing of Applications (Deploying, Enlisting, Starting etc)
2) Microsoft Operation Manager (MOM): For IT monitoring i.e can be used to put alerts to perform specific tasks in event of error. Example being “Contact John Doe if certain error happens”. In short it can be used as knowledge database
3) BAM: This is highest level of monitoring that is intended for a business user.

Tasks that required used of Visual studio for Biztalk 2004 can be performed from Admin Console for 2006 in a very efficient manner. Concept of Application is THE thing in this new version. Each solution can be named as an Application and then all the related artificats like Orchestrations, Maps, Schemas, Pipelines, Non- Biztalk assemblies and Ports can be added under this. Once this is done a single right click on the Application to start will do all of the following:

1) GAC the Non-biztalk assemblies
2) Deploy the Biztalk assemblies
3) Enlist and Start the orchestrations.

Developer can create MSI packages from Admin console and send them to Admins. Admins just need to import these MSI packages and run and the whole application is installed. The only step left is a right click that starts the application.

So no Need for Deployment Wizard or Batch files for deploying

One can even include a batch file (pre processing script) to create folders in case they need to be created while deploying to a new machine.

The new console will give a snapshot of all errors on per application basis. Further information about the error can be then obtained by simply clicking on the link for errors.

There is a new Role added called IT-OP. This gives same privileges like Admins but READ only. This also shields any sensitive financial/hipaa information that need not be seen across the board.

Another feature is subscribing to errors in the pipelines. Subscription can be based both on Content and Context. There can be a orchestration that sends these errors as an email to the concerned people in a “InfoPath Form” like format. The concerned people can then rectify the error that is causing the error and then send the message back to biztalk and resume the processing

Sai showed a simple demon by setting up and recv and send port. He then stopped the send port and dropped the messages. He then opened the Admin console and we could see the 15 error messages (corresponding to 15 files her dropped) against the specific application name.

Briefly demonstrated FF wizard.

Second Speaker: Kent Brown (Virtual TS) from CitiHudson group.

He spoke about BAM and how this is the most underutilized aspect of Biztalk. He compared it a performance counter monitor. BAM gives an eagle eye view of the business flow through Biztalk

BAM can tap into the work flow at different points and record the pre selected variables. These variables can be of different types. Like Milestones, Dates, Durations, Content variables (Variables inside the message like PO#, Invoice #).

Three Components: Ascending Hierarchy

1. BAM Activity

2. BAM view (Consolidation of BAM Activity)

3. Observation Model (Consolidation of BAM view and BAM Activity)

In the background, BAM uses OLAP processing. It creates cubes to process data

BAM: Roles and Tools:

Role:

1) Business Analyst: Creates the BAM spreadsheet underlining the variables and parameters he needs to track.
2) IT Pro:
3) Developer: Map these variables to the actual work flow through Orchestra ions, Pipelines etc

Tools:

1) BAM spreadsheet: Open the excel sheet. Go to Add Ins and add BAM.
2) Create BAM Activity and give it a name.
3) Add the variables that will be part of this activity and need to be populated. These can be milestones, duration of even the content from the message.

Use Tracking Profile Editor to map these variables from the Work Flow. i.e. In orchestration. These mapping points can be at send, receive shapes. Some field in the message body. Below is a pic of Editior.

I asked if fields need to be promoted to be accessed while mapping. The answer was NOT NECCESARILY.

BAM can also tap across multiple orchestrations across multiple servers.

He also gave an example how a variable that can change from one application to another can be still represented in the same row. Ie. PO ID: 123 can become SO ID: 246 in marketing application and SHIP ID: 243 in Shipping system. BAM can track this variable through out.

What is new in BAM 2006?

1) Out of box BAM Portal. Runs ASP.Net 2.0
2) Integration with Message Only scenario. i.e if Orchestrations not present in the end to end file process.
3) The BAM portal also has search feature.

Migration from Biztalk 2004 to Biztalk 2006

2006-03-28T19:14:00.000-05:00

I recently converted a Biztalk 2004 project to Biztalk 2006. Believe me this time MSFT is right when they say that "Migration is very Simple". From my experience, all the orchestrations, pipeline, schemas and Maps (All Biztalk Artificats) were migrated without any changes. Some changes were required for some of the .Net functions that changed their signature from Studio 2003 to Studio 2005. Below are some screen shots of the wizard.

(This Final step gives the Migration Report)

BTW: Biztalk 2006 RTM is ready to roll!

Coming soon!

2006-03-23T09:21:00.000-05:00

I am in process of writing a research paper on Business Integration in general and how Biztalk positions itself in the grand schema of things.

Difference in HAT screens

2006-03-22T10:10:00.000-05:00

I noticed that the HAT screen in 2006 is different than 2004. Actually there are certain functionalities that are absent in 2006. i.e "Operations" and "Configuration".
see below:

Trying to figure out as if how to terminate the "instances" in 2006 . In 2004 it was done by going to "Operations" and terminating the instances. Do not see anything similair in 2006 as for now. I think I have the answer, it can be done from Biztalk Admin Console

Some New Features in Biztalk 2006

2006-03-21T10:00:00.000-05:00

Biztalk 2006 has some little changes here and there that can contribute to efficiency. Some of them are:

1) Having a "drop" down on the top pane of Solution Explorer window.

As in Biztalk 2004 version there was a horizontal scrolling to view all the artifacts contained in the solution. This was very frustrating at times because one could not see all the artifacts at once and needed to keep clicking to get to the right one. 2006 has a drop down that gives the list of all the artifacts at once and one can just click the required one.

2) File adapter (recv and send) contains an option for authentication.

As in 2004 version, biztalk service account had to have full permissions on the shared drive that contains the receive or send folder. In 2006 it need not be. Any user name and password that has permissions to the shared drive can be used to access the shared driver from Biztalk Explorer. It need not be Biztalk service account.

3) File Adapter Batching

The receive location has option of batching the incoming file. This option was not available in 2004.

Event Name: BizTalk Server 2006 First Look Clinic - Iselin, NJ

2006-03-16T00:35:00.000-05:00

I registered for this seminar.

For interested people :

Session Code: 1032288971
Session Name: BizTalk Server 2006 for DevelopersSession
Location: Microsoft NJ Office/Long Beach Conference Room
Session Room:
Session City: IselinSession
Start Date: 04/04/06

Secure FTP Adapter

2006-02-21T23:08:00.000-05:00

Biztalk 2004 comes with out of box adapter for FTP. But if you want to transmit USING secure FTP then you need to install custom adapter. One of the recommended SFTP adapters by MSFT is from their partner /n Software. It is simple installation. Once the installation is done go to Biztalk Admin and click on Adapters. Then add New. Name it SFTP and then from the drop down "Adapter" chose the installed adapter.

Now, when you go and create a new Send Port. Under the Transport Type you can see the option for new adapter SFTP. You can open the properties for URI and add the information as you will add for standard FTP.

As far I know one of the drawbacks to this adapter is that it does not exposes its properties in Orchestration. I will dig deeper into this. If I can't access the properties in Orchestration then I can't use SFTP in a dynamic Port like I can use standard FTP.

Custom Pipeline Woes

2006-01-26T00:10:00.000-05:00

In one of my project, the source system was sending 2 almost similar messages to Biztalk. The only difference was the root node name. I had 2 ways of dealing with it.

Solution 1:

1) Create separate document schemas
2) Create separate Receive Pipelines referring to each document schema
3) Use a map to change the root node name so to have exactly same message that can pass through the common orchestration thereafter.

Pros:

1) Less Development time
2) Easy message retrieving in case of failure

Cons:

1) Additional artifacts required in form of additional schemas and pipelines.
2) Low in performance and high in memory usage as we have to use extra map.

Solution 2:

Use a custom pipeline component that will replace the “root node” name to a common node name.

Pros:

1) Cleaner and Faster

Cons:

1) More development time
2) If error handling is not proper then the entire message is lost. This is because there is no persistence to messagebox in the pipelines.

Whereas the first solution is very easy, the second is relatively difficult. I tried the second one for simple reason that it was challenging and I thought I could do it within reasonable time that I had. I started by following the standard procedure for developing custom pipeline components. I used the “FixMsg” custom pipeline sample from “SDK” as my base code. It is located at “..\Program Files\Microsoft BizTalk Server 2004\SDK\Samples\Pipelines\CustomComponent”. This saved me some time in not “reinventing” the wheel. I worked on the “Execute” method and converted the Stream to Byte and then Byte to string. Then I tried replacing the required node in string and then tried to convert back the new string back to stream. Since pipeline only deals with streaming data. I ensured that the return is a “stream”. My approached looked quiet logical to me but the results were NADA. This string replacement did not actually work as I was not able to pass the “stream” back to the next stage of the pipeline. Then, I thought of using XPath. But since pipeline deal with streaming data whereas Xpath requires the whole message to be loaded into memory. I could not use the normal Xpath but I could “streaming Xpath”. This is called “forward only” Xpath. I tried “cloning” the stream so I can use XMLReader class on it. But could not go anywhere, then I gave up the solution as I was running out of time. I lost this challenge!! But, I will definitely revisit it once I get some “breathing space”.

Any inputs will be welcome.

Capturing ACKs and NACKs

2006-01-17T18:12:00.000-05:00

I am presently working on "File Mover" application. This application will replace existing VB application to move files from point A to point B based on the filename. Filename contains the send location address. I am populating a SQL database to track the file moved from point A to point B. One of the problems I am having is to capture the "Error in Transmission". This is tough to capture because File adapter transmission is separate from Orchestration process. Once orchestration is complete, it does not really care if message was sent or error out. For this we need to use concept of ACKs (Acknowledgements) and NACKs (Negative Acknowledgements). ACKs indicate successful message transmission and NACKs indicate failures in transmissions. Since I am using one way "asynchronous" FILE transmission these come handy. Some readers may ask why not use Request-Response. The reason is that they can only be used for HTTP adapter when there is a response that can be captured. For File adapter transmissions, we can only use ACKs and NACKs

Follow steps below and that is all you need to capture ACKs and NACKs.

1) Put Send shape in a Scope and make "Transaction Type" as "None".
2) Set "Synchronized" property to "True".
3) Mark Orchestration with Delivery Notification = Transmitted
4) Use "Catch Exception shape"
5) Select the following exception type in properties:
"Microsoft.XLANGs.BaseTypes.DeliveryFailureException"
6) Call the insert method for populating database in the "Catch" shape.
8) Put the "Send" shape to save the "bad" files to a folder.
7) Then PUT the "terminate" shape in the "CATCH shape".

Wonderful Cruise!

2006-01-11T22:23:00.000-05:00

The other day I was talking to one of my co worker (Abhi). He also maintains a Biztalk blog http://biztalkland.blogspot.com. We discussed about putting more non-technical stuff on the blob. We gave argument for and against it. How it will look out of place ...blah..blah. Finally, we decided that Blog is a medium to express. So there are no Rules for it as such. People might argue on this. Everyone is entitled to his/her opinion. After the conversation, I decided to trickle some personal stuff now and then.

I went on cruise with my wife. It was our "HoneyMoon" cruise. Even though it took place after 6 months of our marriage. I had just enough time to get married in June so we did not push for Honeymoon that time. Although we did went to India for our Reception as we got married in USA. After we came back she became busy with her residency and I went down with my project. Xmas and New Year stretch was good for the break we needed and we used that time to hop on a ship and hit the Caribbean. We went to San Juan, Saint Maartin and Saint Thomas. I personally loved Saint Thomas. The beaches are pristine there. We had tons of fun before coming back to Sub Zero temps. I am also putting some pics from our trip!

FileMove Application and "MessageBox" error

2006-01-03T22:31:00.000-05:00

Recently, I was asked to create an application that will replace an existing VB application. The application was to move the Files from a common location and drop them to a different destination based on the FileName. The File Name had the destination name in it. This is what I did:

1) Created a simple Orchestration with single receive port that will have filter for every possible file. i.e. pdf, dat, xml, txt etc.

2) Create a Dynamic port

3) Create a "Helper" class in VB.net. This will parse the incoming file name to extract the URI for dynamic port. This URI will determine where the file should be routed too.

I set up the whole project but found some interesting facts:

1) If the URI is blank for some reasons. You will get the following error in the eventvwr.

"Exception occured when persisting state to database".

2) The syntax for the URI for dynamic port will be.

file://servername/sharedfolder/Filename.

The start of the URI will tell the port that it is a "FILE" adapter. The rest will give the complete path to drop the file.

3) While using File.ReceiveFileName in the expression share. It will not only give the whole filename but also the path of the receive folder. In order to get the actual FileName, one will have to parse the whole path string.

Changing external assembly reference in Maps & Orchestrations

2005-12-21T22:47:00.000-05:00

What do you do if for some reasons you have to change the name of external assembly that is being referenced by Maps and Orchestrations? The change can be for reasons like:

1) New SNK was created and hence the Public Token key changed. (Very rare to happen)

2) For some reasons the name of external assembly have to be changed. (We had to do this in our project. This was done in order to adhere to naming standards of Biztalk artifacts)

Standard way of doing this can be really tedious. As one have to go to each map and click on each "Functoid" and go to properties to change to new "Assembly name". Same thing will need to be done for Orchestrations.

If like us you have 20 maps with average 30 functoids per maps and 20 orchestrations then you need to keep 2 days aside for this "boring" job!

The second option can make your life easy and get you up and running in 30 mts as compared to 2 days. This is what you do:

1) Open the Map/Orchestration in Text Editor (Ultra Edit in our case)

2) Do a Find and replace in the whole file. "Find" "old assembly.dll" replace with "new assembly.dll".

3) Replacement will be done for "Assembly", "Class" and "Assembly Path" in the file.

4) Save the file and you are all set!! That easy!

Care should be taken while doing this operation as it is large scale "find" and "replace" operation.

Aggregating the "Split-Gatherer" Pattern using SQL

2005-12-13T01:28:00.000-05:00

One of the limitations of BTS 2004 as compared to BTS2006 is its inability to “gather” the transactions it has split from the original file. “Splitting” can be achieved by combination of “Document” and “Envelope” schemas. In my present project, we needed to split the file and then add the “split” transactions on file -by- file basis. Presently, it is achieved by the “Convoy” design. This involves using “Correlation” dataset that waits for the field that is “correlated” (FileName in our case). It then adds all the “split” transactions for that file. Since we rely on “serial” processing as compared to “parallel” processing that orchestrations are intended for, this is not an optimum design. The major drawbacks of this approach is 1) It leads to “zombie” scenario in Biztalk. 2) Performance declines exponentially as the transaction volume increases.

These 2 scenarios were slowly taking menacing proportions. In order to counteract them our team came up with “SQL database” aggregator. It is simple in design and very fast. The files are split as before. But now each transaction is stored in SQL database as XML node. Parameters like Orchestration ID, FileName and Unique identifiers are passed along with the XML node. At the end the XML file is extracted by combining these nodes and using parameters passed to aggregate the correct transactions from the database. Stored procedures are used to Build the final XML .

Biztalk 2006 is cool!!

2005-12-07T21:52:00.000-05:00

Finally I got some time to review 2006. Here are some features that you might find very useful in Biztalk 2006.

1) Calling Pipelines from Orchestrations using expression box

This can help in creating single common orchestrations for messages with different schemas as they can be routed using dynamic calls of "Pipelines" from Orchestration. This will be real cool!!

2) Flat File wizard for creating Schemas

This will be huge! From my experience, creating FF schemas are real pain. Imagine having wizard for it now.:)

3) Standard processing and Recoverable interchange process in Pipelines

In my present project we are using substantial .Net code to log the errors in pipelines. This actually slows down the processing time. This new feature will definitely help!

4) Can generate errors that can be subscribed by send ports using pass through. Use filter on original receive ports that we created. Catch the error message, change it and re process it. This is called “Failed message Routing”

5) POP for email receive

Biztalk 2004 CAN only send emails. 2006 will give ability to even receive and process them. Imagine activating Orchestration by sending an email.

6) Better BAM services

7) Interfaces

SAP adapter is standard with 2006!

8) Shared point adapter

9) Ordered delivery:

To achieve first-in-first-out or "Ordered" delivery is BTS 2004 we need MSMQ/T . But in BTS 2006, with little configuration the same thing can be achieved by using standard MSMQ. This is what is required.

1) Set Ordered Processing Property on Send Port
2) Set the same Order Processing property on Receive Port
3) Set MSMQ to transactional.

Connection Pool Issue

2005-11-16T21:33:00.000-05:00

Been a long time since I wrote my last Blog. Been busy with present project and also setting up my house with my wife. Aaah! Finally I am back to where I belong:). Now the problem statement.

In my present project, we extract data from PSFT database using SQL. These queries are called by Functoid in the Maps. The number of these functoids is about 40 for each map. We are using OracleClient class to Fill the dataset with the return value. Database connection is closed and opened after every query. Lately the transaction volume has increased and is projected to increase further. This increase in volume is returning "connection pool" error. Because a file can have multiple transactions and each transaction can potentially open 40 connections at a time. We talking about thousands per day.

After some brain storming sessions, we came up with the following solution.

It will be a 3 tier solution.

First layer will be windows service which will create a dataset from all the possible queries. The second layer will be a webservice that will work like a wrapper and pass the resulting XML dataset from webservice to the Orchestration. Orchestration will be third layer that will call the webservice and grab the XML file with the values that will be then mapped using existing map. This way we will able to limit the connection pool from 1 per transaction to 1 per file. Since a file can have anywhere till 1000 transactions. This is a huge reduction.

Below is my "brain dump".

Windows Service:

1) Will do the processing and all the queries reside here.
2) Uses Config file for queries with use of array so that new queries can be added easily
3) Pool at specific intervals
4) Load the data at first initialization and after that just validates it if it is latest.
5) Validation will be done on bases of some flags
6) Once change detected the whole dataset is loaded.
7) Dataset is per site.
8) Can have single window service per site or one for all sites

Web Service:

1) Acts as wrapper to pass the dataset
2) Runs the validation on data set whenever called and based on changes will re-load the dataset using windows service.
3) Will be called from Aggregator using Expression shape.
4) Call will be made per file that coming in to orchestration
5) Every time web service is called it will invoke validation method that will in turn look for “flags”. That way we will avoid loading data all the time when webservice is called.
6) Each call of webservice will do data validation and not data upload. Unless it needs to.
7) This way we open one connection per file and that too 99% of time will load no data. Just do the validation

Orchestration

1) With the present design, use expression shape.
2) Can pass System. XML as parameter
3) If use call orchestration, will have to have 3 different schemas to pass as parameter that can be appended by the Mapping dataset then

"NameSpace" error

2005-09-14T18:58:00.000-04:00

I am facing a strange problem in my current project. When, I am build the orchestrations with default name space of project i.e A.B.C.D.E . I get an error "Schema not found, missing reference". After trying so many things I finally tried changing name space to A_B_C_D_E and it worked! I have no explanation for this !

"Call" Orchestration and Dynamic Ports

2005-08-24T21:38:00.000-04:00

In my present project there came a requirement to create a "common orchestration" that would take the "response" XML from PSFT and apply business rules. This "common orchestration" should work with all present Orchestrations that are already on production. This Orchestration has to be very flexible and accommodate the various possible parameters from Main orchestration.

This "Common Orchestration" is just like any another orchestration but will have NO "Receive" shapes. I used an "Request-Response" "dynamic" port that will read PSFT HTTP gateway address from REGISTRY KEY. The two-way dynamic port will receive status XML from PSFT and analyze the node for "success" and "failure". If it is a failure then the "Main Orchestration" will go in suspended mode but "resumable". Once the error is fixed, manual intervention will be required to send the message again by resuming it using HAT. An error message will be generated in the event viewer so the Production support people can be informed.

Main Orchestration will call this "Common Orchestration" using "Call Shape" and pass the parameters.

I used the following code for the dynamic port: PortName(Microsoft.XLANGs.BaseTypes.Address) = getmethodURL()

Interesting Observation regarding HTTP adapter receive codes

2005-08-19T15:32:00.000-04:00

I had an interesting discussion with my co worker. The ASP.Net page I deigned to pass XML through HTTP POST to Biztalk was returning code "200" for succesful submission. The discussion rotated around the fact that this code is being send by webserver or Biztalk. According to MSFT website, code 202 is returned by one-way receive port and code 200 is returned by Http "Request-Response" port. Since we are using one way port, we should be getting a 202 message from Biztalk if any. But we are getting 200. I also read somewhere that Biztalk looks for 200 message from webserver before it starts processing files. This lead me to believe that 200 is infact sent my webserver. But, I am not entirely convinced by it.

Biztalk Macros for FileNames

2005-08-12T17:39:00.000-04:00

I want to thank Jan Tielen for the following formats:

%datetime%:Coordinated Universal Time (UTC) date time in the format YYYY-MM-DDThhmmss (for example, 1997-07-12T103508).

%datetime_bts2000%: UTC date time in the format YYYYMMDDhhmmsss, where sss means seconds and milliseconds (for example, 199707121035234 means 1997/07/12, 10:35:23 and 400 milliseconds).

%datetime.tz%:Local date time plus time zone from GMT in the format YYYY-MM-DDThhmmssTZD, (for example, 1997-07-12T103508+800).

%DestinationParty%:Name of the destination party. The value comes from message the context property BTS.DestinationParty.

%DestinationPartyID%:Identifier of the destination party (GUID). The value comes from the message context property BTS.DestinationPartyID.

%DestinationPartyQualifier%:Qualifier of the destination party. The value comes from the message context property BTS.DestinationPartyQualifier.

%MessageID%:Globally unique identifier (GUID) of the message in BizTalk Server. The value comes directly from the message context property BTS.MessageID.

%SourceFileName%:Name of the file from where the File adapter read the message. The file name includes extension and excludes the file path, for example, foo.xml. When substituting this property, the File adapter extracts the file name from the absolute file path stored in the FILE.ReceivedFileName context property. If the context property does not have a value, for example, if message was received on an adapter other than File adapter, then the macro will not be substituted and will remain in the file name as is (for example, C:\Drop\%SourceFileName%).

%SourceParty%:Name of the source party from which the File adapter received the message.

%SourcePartyID%:Identifier of the source party (GUID). The value comes from the message context property BTS.SourcePartyID.

%SourcePartyQualifier%:Qualifier of the source party from which the File adapter received the message.

%time%:UTC time in the format hhmmss.

%time.tz;%Local time plus time zone from GMT in the format hhmmssTZD (for example, 124525+530).

ASP.Net issue

2005-08-07T22:13:00.000-04:00

In my present project, I had to create an ASP.Net page to receive XML documents and pass it do biztalk on basis of Query string: This is what I found out.

1) Ensure that the "Virtual Directory" in IIS where the ASPX is located is configured for Execute permissions for "Scripts and Executibles"

2) Unlike ASP page where we use "server" object, ASP.Net takes the incoming XML as "stream". Be very careful in converting the stream to array and then to string. Make sure that the stream is "encoded" while its being received by ASPX and "decoded" while it is being sent by ASPX page to Biztalk.

3) I have seen some sample code on various website using ASCII encoding. I had lots of problems with it and after wasting several hours I found out that "UTF-8" encoding is the one that does the trick.

4) Last and not the least: Do not forget to restart the IIS (Run Command "iisreset") when you get 500 errror.

"Zombie" scenario in Aggregator

2005-05-11T22:59:00.000-04:00

In my current project we used Aggregator to batch the files after splitting. We used combination of "Activate" subscription and "Correlation" Subscription. The second receive shape (for Correlation) has a listener that had delay of 60 seconds. We found that in some large files with more that 200 transactions there was some transaction loss. There was no trace on HAT or Event vwr. After thorough investigation we found out that the Orchestration was terminating without waiting for all the messgages to be be consumed from the Message Box. Basically, some messages were not being consumed and hence discarded. This is called "zombie" scenario. There is a nice article on this at:http://blogs.msdn.com/Biztalk%5FCore%5FEngine/

Solution: It was simple. We increased the timer delay from 60 seconds to 240 seconds. It did effect the performance but that was offset by the "no loss" of transactions.

Seperating Items from Order without Orchestration

2005-04-03T23:16:00.000-04:00

Recently, a co worker approached me with a problem. He had the following xml struture:

1
ABC

01123
35
300.50

45221
27
1430.50

45222
127
11430.50

He wanted the following structure as output:

1 ABC

01123 35 300.50

1 ABC

45221 27 1430.50

There is a solution where you have to promote property and use XPath in orchestration to do this. I saw this on Stepehen's website. I found another way to achieve same. This new way is even faster.

1) Generate the schema from the original file call it "IN"

-
-
-
-
-
-
-

-
-
-

2) Generate another schema call it OUT

-
-
-
-
-
-
-

3) Use Map1 to populate fields from IN to OUT.

4) The resutlant file will be:

-
-
01123
35
300.50
1
ABC

-
45221
27
1430.50
1
ABC

-
45222
127
11430.50
1
ABC

5) Now you can drop this file to a folder. This file format is easy to split now. data

6) Split it using XML disassembler. Envelope schema will be type

7) Create another map that takes input structure as the "split" structure. i,e

01123
35
300.50
1
ABC

8) The output of that map will be of type: (Original type in step 1)

-
-
-
-
-
-
-

-
-
-

9) Now this map will give yu the desired output after completing the mapping.

10) You can put this "second" map in either a "Port" or "Orchestration"

Biztalk and Integration Patterns

2005-03-10T14:41:00.000-05:00

I found this very nice white paper on Integration Patterns. It gives a very macro level view of Solution Architecture and then breaks it down to specific Biztalk level. I recommend this paper to everyone who is involved in system architecture and implementation.

This paper can be found at:

www.eaipatterns.com/docs/integrationpatterns_biztalk.pdf

Efficient Mapping Technique

2005-03-02T02:16:00.000-05:00

If you have sample input and output files with same header level node names and structures(some of the field names can be different) . Then instead of doing individual drag and drop (can't use mass copy as these files are not 100% similair). You can use "Link by Node Name". This is how you do that:

1) Click on the MAP grid.

2) Go to properties box

3) Select "Auto Link By" : NODE NAME

Faster way of Deploying assemblies to the servers.

2005-02-25T23:07:00.000-05:00

One of techniques of deplying assemblies directly to server without building it on your local machine is:

1) Right Click the Project in Solution Explorer.

2) Go to Properties----Configuration Properties-----Deployment.

3) Select the server name, database and make "Redploy"= true.

4) Click Deploy

The Redeploy="True" enables to deploy over the older assemblies on the server. This way we do not have to undeploy and then deploy.

Warning: Make sure to revert back the settings from your server to your local machine after completing the above steps. Else you might end up overwriting the server solution with your local machine solution (good or bad).

Synching the Method executions from Orchestrations

2005-02-13T16:58:00.000-05:00

One of the problem I faced in one of my project was, in the event of multiple files entering a same method. And if this method is reading information from database. This can result in that method being executed by multiple xml documents and hence exceeding the database connection pool. In order to avoid this, DOT NET offers "SyncLock" object.

Wrap all your methods in this object. This will ensure that second document does not enter this method till the previous one is completely processed.

Example:

SyncLock object name
Call the function here
End SyncLock

De-Batching and Re-Batching the Files (Scatter-Gather)

2005-02-08T11:16:00.000-05:00

In my current project we had requirement to split the incoming XML files into individual transactions. These individual transactions would then go through some rule validations in Orchestration. Depending on the validation rules set up in the external assemblies, these transactions will be either rejected or accepted. The business need for splitting the file came from the fact that the transaction has to be accepted or rejected as a whole.

Splitting part was easy. We used envelope and document schemas for that. But the difficult part was to batch the split transactions per incoming file they originally belonged too. I will give possible solutions for "Re-Batching" here and also limitations for each.

1) Use a loop within the Orchestration. Add each transactions on basis of node. Co-relation set will have to be set up . This is important as the correlation ID will help us to add only the transactions that belong to a particular file. This approach is useful if we have something like a "Orders" file that consists of various "Items". Order ID will be the correlation ID in this case. This will tie up all the "Items" for a particular "Order ID".

This approach fails in my situation as I have no correlation ID. My sample file consists of a root element immediately followed by child nodes. These child nodes contain all the information. The sample file looks something like:

(a)(b)contains all the data(/b)(/a)

Ideal structure to use the looping would be:

(a)correlation ID here(b)contains all the data(/b)(/a)

2) Loop it and add on basis of the count.

Problem with this approach is that in scenario of 10 input files being dropped at the same time and each having 10 each transactions. The count will not ensure that the transactions we are adding are actually coming from a single file (as we need in our case). It is not guaranteed that the Transactions in each file will be processed in a sequence.

3) Third approach works for me. If I set up a unique ID at each transaction level. This ID is same for all transactions in one particular file. Then I can equate this ID as "File Name" in the Send Port and then use "Append". Biztalk will append all the transactions with the same unique file name. This is what I need!

Work around "Node" name changes in Schemas

2005-01-08T23:03:00.000-05:00

Once the mapping is done and the assembly is deployed. Any change in the root node name of the schema means re-mapping the whole thing again. There is a quick work around to this.

Create a function using VB.net/C#. Pass the xml document (with the old root name) to this function. Suppose the incoming Doc is "DocIn". This function will take DocIn as a parameter. Replace the "oldname" with "newname" using replace command on "Outerxml". Pass the resulting document to the final destination. This is how the function will look like.

Public Function Convert (Byval DocIn as XMLDocument)

Dim DocOut as new XMLDocument

DocOut.Load(DocIn.Outerxml.replace("OldName","RootName"))

Return DocOut

End Function

Stored Procedures for Oracle through Dot Net

2004-12-11T19:12:00.000-05:00

One of the requirements in the current Project is to call queries from Oracle and perform validations. We are calling about 100 of the queries using "script funtoids" in the mapper. I wanted to optimize the database connections. Instead of sending 100 queries and hitting the database 100 times. I wanted to have single stored procedure to replace the 100 queries. Having worked with stored procedures for SQL database before, I did not find it an issue. My plan was to create an array that is input to the Stored Procedure and get an array as an output to the stored procedure. The idea was to send all possible variables in an array to the stored procedure. Process them and get possible values as an array. Use this array to populate the "Right Hand" side of the map.

On course of my effort, I realized that there are limitations as such of working with Oracle through arrays. I gave up after trying!

Pipeline tools

2004-11-28T16:55:00.000-05:00

For those that don't know, there are some really handy pipeline tools in the SDK under the folder:

\Program Files\Microsoft BizTalk Server 2004\SDK\Utilities\PipelineTools

These tools allow you to test your pipeline components and, even more handily, flat file schemas without having to set up the BizTalk Server environment. They are pretty much essential when developing flat file schemas (particularly when you have a header, trailer and body schema).
Here is a list of the pipeline tools available:

DSDump.exe - this tool outputs a textual description of the schema.

Pipeline.exe - this tool executes a pipeline and produces one or more output files. Very useful for testing custom pipelines.

XMLDasm.exe - this tool executes the XML Disassembler component by simulating a receive pipeline.

XMLAsm.exe - this tool executes the XML Assembler component by simulating a send pipeline.

FFDasm.exe - this tool executes the Flat File Disassembler component by simulating a receive pipeline. Essential for flat file development!

FFAsm.exe - this tool executes the Flat File Assembler component by simulating a send pipeline. Again, essential for flat file development!

Note: The pipeline.exe tool does not access the BizTalk Server databases, so executing a pipeline that contains the BizTalk Framework Assembler/Disassembler may not work as it accesses the BizTalk Management database during runtime to match up receipts to documents.

Remote Connection to HAT

2004-11-09T21:29:00.000-05:00

I read some articles on Remotely connecting to HAT. Not very straightword. My requirements were to just get a view of transaction process without using Remote Desktop to the server. Due to limited server connections it was always a problem to connect to it. This is how you connect to server:

1) Open the HAT.

2) Go to Tools----Preferences-----Select "Archived Data"....Enter the SQL server name and select the Database.

You are all set!

Converting Eurpean/American Dates to ISO-format

2004-10-09T22:31:00.000-04:00

We are getting European/American dates in the inbound files. We need to convert them to ISO format dates before sending it over to database. This is how to do it:

1) Drag a script functoid over to the map grid.

2) One input to that "functoid" will be the date field from the input schema and the second input will be a constact value of either "1" ( for European format) and "2" (American format).

2) Create a Public Function in external assembly with two paramaters.

Public Function dateformat( byval strDate as string, Byval flag as integer)
Dim newdate As Date
Dim strReturnDate As String

Select Case flag
Case 1
newdate = Convert.ToDateTime(strDate , New CultureInfo("en-gb"))
strReturnDate = Format(newdate , "yyyy-MM-dd")

Case 2
newdate = Convert.ToDateTime(strDate , New CultureInfo("en-us"))
strReturnDate = Format(newdate , "yyyy-MM-dd")
End Select

Return strReturnDate

End Function