Changing external assembly reference in Maps & Orchestrations

What do you do if for some reasons you have to change the name of external assembly that is being referenced by Maps and Orchestrations? The change can be for reasons like:

1) New SNK was created and hence the Public Token key changed. (Very rare to happen)

2) For some reasons the name of external assembly have to be changed. (We had to do this in our project. This was done in order to adhere to naming standards of Biztalk artifacts)

Standard way of doing this can be really tedious. As one have to go to each map and click on each "Functoid" and go to properties to change to new "Assembly name". Same thing will need to be done for Orchestrations.

If like us you have 20 maps with average 30 functoids per maps and 20 orchestrations then you need to keep 2 days aside for this "boring" job!

The second option can make your life easy and get you up and running in 30 mts as compared to 2 days. This is what you do:

1) Open the Map/Orchestration in Text Editor (Ultra Edit in our case)

2) Do a Find and replace in the whole file. "Find" "old assembly.dll" replace with "new assembly.dll".

3) Replacement will be done for "Assembly", "Class" and "Assembly Path" in the file.

4) Save the file and you are all set!! That easy!

Care should be taken while doing this operation as it is large scale "find" and "replace" operation.

Aggregating the "Split-Gatherer" Pattern using SQL

One of the limitations of BTS 2004 as compared to BTS2006 is its inability to “gather” the transactions it has split from the original file. “Splitting” can be achieved by combination of “Document” and “Envelope” schemas. In my present project, we needed to split the file and then add the “split” transactions on file -by- file basis. Presently, it is achieved by the “Convoy” design. This involves using “Correlation” dataset that waits for the field that is “correlated” (FileName in our case). It then adds all the “split” transactions for that file. Since we rely on “serial” processing as compared to “parallel” processing that orchestrations are intended for, this is not an optimum design. The major drawbacks of this approach is 1) It leads to “zombie” scenario in Biztalk. 2) Performance declines exponentially as the transaction volume increases.

These 2 scenarios were slowly taking menacing proportions. In order to counteract them our team came up with “SQL database” aggregator. It is simple in design and very fast. The files are split as before. But now each transaction is stored in SQL database as XML node. Parameters like Orchestration ID, FileName and Unique identifiers are passed along with the XML node. At the end the XML file is extracted by combining these nodes and using parameters passed to aggregate the correct transactions from the database. Stored procedures are used to Build the final XML .

Biztalk 2006 is cool!!

Finally I got some time to review 2006. Here are some features that you might find very useful in Biztalk 2006.

1) Calling Pipelines from Orchestrations using expression box

This can help in creating single common orchestrations for messages with different schemas as they can be routed using dynamic calls of "Pipelines" from Orchestration. This will be real cool!!

2) Flat File wizard for creating Schemas

This will be huge! From my experience, creating FF schemas are real pain. Imagine having wizard for it now.:)

3) Standard processing and Recoverable interchange process in Pipelines

In my present project we are using substantial .Net code to log the errors in pipelines. This actually slows down the processing time. This new feature will definitely help!

4) Can generate errors that can be subscribed by send ports using pass through. Use filter on original receive ports that we created. Catch the error message, change it and re process it. This is called “Failed message Routing”

5) POP for email receive

Biztalk 2004 CAN only send emails. 2006 will give ability to even receive and process them. Imagine activating Orchestration by sending an email.

6) Better BAM services

7) Interfaces

SAP adapter is standard with 2006!

8) Shared point adapter

9) Ordered delivery:

To achieve first-in-first-out or "Ordered" delivery is BTS 2004 we need MSMQ/T . But in BTS 2006, with little configuration the same thing can be achieved by using standard MSMQ. This is what is required.

1) Set Ordered Processing Property on Send Port
2) Set the same Order Processing property on Receive Port
3) Set MSMQ to transactional.

Connection Pool Issue

Been a long time since I wrote my last Blog. Been busy with present project and also setting up my house with my wife. Aaah! Finally I am back to where I belong:). Now the problem statement.

In my present project, we extract data from PSFT database using SQL. These queries are called by Functoid in the Maps. The number of these functoids is about 40 for each map. We are using OracleClient class to Fill the dataset with the return value. Database connection is closed and opened after every query. Lately the transaction volume has increased and is projected to increase further. This increase in volume is returning "connection pool" error. Because a file can have multiple transactions and each transaction can potentially open 40 connections at a time. We talking about thousands per day.

After some brain storming sessions, we came up with the following solution.

It will be a 3 tier solution.

First layer will be windows service which will create a dataset from all the possible queries. The second layer will be a webservice that will work like a wrapper and pass the resulting XML dataset from webservice to the Orchestration. Orchestration will be third layer that will call the webservice and grab the XML file with the values that will be then mapped using existing map. This way we will able to limit the connection pool from 1 per transaction to 1 per file. Since a file can have anywhere till 1000 transactions. This is a huge reduction.

Below is my "brain dump".

Windows Service:

1) Will do the processing and all the queries reside here.
2) Uses Config file for queries with use of array so that new queries can be added easily
3) Pool at specific intervals
4) Load the data at first initialization and after that just validates it if it is latest.
5) Validation will be done on bases of some flags
6) Once change detected the whole dataset is loaded.
7) Dataset is per site.
8) Can have single window service per site or one for all sites

Web Service:

1) Acts as wrapper to pass the dataset
2) Runs the validation on data set whenever called and based on changes will re-load the dataset using windows service.
3) Will be called from Aggregator using Expression shape.
4) Call will be made per file that coming in to orchestration
5) Every time web service is called it will invoke validation method that will in turn look for “flags”. That way we will avoid loading data all the time when webservice is called.
6) Each call of webservice will do data validation and not data upload. Unless it needs to.
7) This way we open one connection per file and that too 99% of time will load no data. Just do the validation

Orchestration

1) With the present design, use expression shape.
2) Can pass System. XML as parameter
3) If use call orchestration, will have to have 3 different schemas to pass as parameter that can be appended by the Mapping dataset then

"NameSpace" error

I am facing a strange problem in my current project. When, I am build the orchestrations with default name space of project i.e A.B.C.D.E . I get an error "Schema not found, missing reference". After trying so many things I finally tried changing name space to A_B_C_D_E and it worked! I have no explanation for this !

"Call" Orchestration and Dynamic Ports

In my present project there came a requirement to create a "common orchestration" that would take the "response" XML from PSFT and apply business rules. This "common orchestration" should work with all present Orchestrations that are already on production. This Orchestration has to be very flexible and accommodate the various possible parameters from Main orchestration.

This "Common Orchestration" is just like any another orchestration but will have NO "Receive" shapes. I used an "Request-Response" "dynamic" port that will read PSFT HTTP gateway address from REGISTRY KEY. The two-way dynamic port will receive status XML from PSFT and analyze the node for "success" and "failure". If it is a failure then the "Main Orchestration" will go in suspended mode but "resumable". Once the error is fixed, manual intervention will be required to send the message again by resuming it using HAT. An error message will be generated in the event viewer so the Production support people can be informed.

Main Orchestration will call this "Common Orchestration" using "Call Shape" and pass the parameters.

I used the following code for the dynamic port: PortName(Microsoft.XLANGs.BaseTypes.Address) = getmethodURL()

Interesting Observation regarding HTTP adapter receive codes

I had an interesting discussion with my co worker. The ASP.Net page I deigned to pass XML through HTTP POST to Biztalk was returning code "200" for succesful submission. The discussion rotated around the fact that this code is being send by webserver or Biztalk. According to MSFT website, code 202 is returned by one-way receive port and code 200 is returned by Http "Request-Response" port. Since we are using one way port, we should be getting a 202 message from Biztalk if any. But we are getting 200. I also read somewhere that Biztalk looks for 200 message from webserver before it starts processing files. This lead me to believe that 200 is infact sent my webserver. But, I am not entirely convinced by it.

Biztalk Macros for FileNames

I want to thank Jan Tielen for the following formats:

%datetime%:Coordinated Universal Time (UTC) date time in the format YYYY-MM-DDThhmmss (for example, 1997-07-12T103508).

%datetime_bts2000%: UTC date time in the format YYYYMMDDhhmmsss, where sss means seconds and milliseconds (for example, 199707121035234 means 1997/07/12, 10:35:23 and 400 milliseconds).

%datetime.tz%:Local date time plus time zone from GMT in the format YYYY-MM-DDThhmmssTZD, (for example, 1997-07-12T103508+800).

%DestinationParty%:Name of the destination party. The value comes from message the context property BTS.DestinationParty.

%DestinationPartyID%:Identifier of the destination party (GUID). The value comes from the message context property BTS.DestinationPartyID.

%DestinationPartyQualifier%:Qualifier of the destination party. The value comes from the message context property BTS.DestinationPartyQualifier.

%MessageID%:Globally unique identifier (GUID) of the message in BizTalk Server. The value comes directly from the message context property BTS.MessageID.

%SourceFileName%:Name of the file from where the File adapter read the message. The file name includes extension and excludes the file path, for example, foo.xml. When substituting this property, the File adapter extracts the file name from the absolute file path stored in the FILE.ReceivedFileName context property. If the context property does not have a value, for example, if message was received on an adapter other than File adapter, then the macro will not be substituted and will remain in the file name as is (for example, C:\Drop\%SourceFileName%).

%SourceParty%:Name of the source party from which the File adapter received the message.

%SourcePartyID%:Identifier of the source party (GUID). The value comes from the message context property BTS.SourcePartyID.

%SourcePartyQualifier%:Qualifier of the source party from which the File adapter received the message.

%time%:UTC time in the format hhmmss.

%time.tz;%Local time plus time zone from GMT in the format hhmmssTZD (for example, 124525+530).

ASP.Net issue

In my present project, I had to create an ASP.Net page to receive XML documents and pass it do biztalk on basis of Query string: This is what I found out.

1) Ensure that the "Virtual Directory" in IIS where the ASPX is located is configured for Execute permissions for "Scripts and Executibles"

2) Unlike ASP page where we use "server" object, ASP.Net takes the incoming XML as "stream". Be very careful in converting the stream to array and then to string. Make sure that the stream is "encoded" while its being received by ASPX and "decoded" while it is being sent by ASPX page to Biztalk.

3) I have seen some sample code on various website using ASCII encoding. I had lots of problems with it and after wasting several hours I found out that "UTF-8" encoding is the one that does the trick.

4) Last and not the least: Do not forget to restart the IIS (Run Command "iisreset") when you get 500 errror.

"Zombie" scenario in Aggregator

In my current project we used Aggregator to batch the files after splitting. We used combination of "Activate" subscription and "Correlation" Subscription. The second receive shape (for Correlation) has a listener that had delay of 60 seconds. We found that in some large files with more that 200 transactions there was some transaction loss. There was no trace on HAT or Event vwr. After thorough investigation we found out that the Orchestration was terminating without waiting for all the messgages to be be consumed from the Message Box. Basically, some messages were not being consumed and hence discarded. This is called "zombie" scenario. There is a nice article on this at:http://blogs.msdn.com/Biztalk%5FCore%5FEngine/

Solution: It was simple. We increased the timer delay from 60 seconds to 240 seconds. It did effect the performance but that was offset by the "no loss" of transactions.

Seperating Items from Order without Orchestration

Recently, a co worker approached me with a problem. He had the following xml struture:

1
ABC

01123
35
300.50


45221
27
1430.50

45222
127
11430.50



He wanted the following structure as output:

1 ABC

01123 35 300.50

1 ABC

45221 27 1430.50




There is a solution where you have to promote property and use XPath in orchestration to do this. I saw this on Stepehen's website. I found another way to achieve same. This new way is even faster.


1) Generate the schema from the original file call it "IN"



-
-
-
-
-
-
-





-
-
-










2) Generate another schema call it OUT


-
-
-
-
-
-
-













3) Use Map1 to populate fields from IN to OUT.

4) The resutlant file will be:

-
-
01123
35
300.50
1
ABC

-
45221
27
1430.50
1
ABC

-
45222
127
11430.50
1
ABC



5) Now you can drop this file to a folder. This file format is easy to split now. data

6) Split it using XML disassembler. Envelope schema will be type

7) Create another map that takes input structure as the "split" structure. i,e

01123
35
300.50
1
ABC


8) The output of that map will be of type: (Original type in step 1)


-
-
-
-
-
-
-





-
-
-










9) Now this map will give yu the desired output after completing the mapping.

10) You can put this "second" map in either a "Port" or "Orchestration"

Biztalk and Integration Patterns

I found this very nice white paper on Integration Patterns. It gives a very macro level view of Solution Architecture and then breaks it down to specific Biztalk level. I recommend this paper to everyone who is involved in system architecture and implementation.

This paper can be found at:

www.eaipatterns.com/docs/integrationpatterns_biztalk.pdf

Efficient Mapping Technique

If you have sample input and output files with same header level node names and structures(some of the field names can be different) . Then instead of doing individual drag and drop (can't use mass copy as these files are not 100% similair). You can use "Link by Node Name". This is how you do that:

1) Click on the MAP grid.

2) Go to properties box

3) Select "Auto Link By" : NODE NAME

Faster way of Deploying assemblies to the servers.

One of techniques of deplying assemblies directly to server without building it on your local machine is:

1) Right Click the Project in Solution Explorer.

2) Go to Properties----Configuration Properties-----Deployment.

3) Select the server name, database and make "Redploy"= true.

4) Click Deploy

The Redeploy="True" enables to deploy over the older assemblies on the server. This way we do not have to undeploy and then deploy.


Warning: Make sure to revert back the settings from your server to your local machine after completing the above steps. Else you might end up overwriting the server solution with your local machine solution (good or bad).

Synching the Method executions from Orchestrations

One of the problem I faced in one of my project was, in the event of multiple files entering a same method. And if this method is reading information from database. This can result in that method being executed by multiple xml documents and hence exceeding the database connection pool. In order to avoid this, DOT NET offers "SyncLock" object.

Wrap all your methods in this object. This will ensure that second document does not enter this method till the previous one is completely processed.

Example:


SyncLock object name
Call the function here
End SyncLock

De-Batching and Re-Batching the Files (Scatter-Gather)

In my current project we had requirement to split the incoming XML files into individual transactions. These individual transactions would then go through some rule validations in Orchestration. Depending on the validation rules set up in the external assemblies, these transactions will be either rejected or accepted. The business need for splitting the file came from the fact that the transaction has to be accepted or rejected as a whole.

Splitting part was easy. We used envelope and document schemas for that. But the difficult part was to batch the split transactions per incoming file they originally belonged too. I will give possible solutions for "Re-Batching" here and also limitations for each.

1) Use a loop within the Orchestration. Add each transactions on basis of node. Co-relation set will have to be set up . This is important as the correlation ID will help us to add only the transactions that belong to a particular file. This approach is useful if we have something like a "Orders" file that consists of various "Items". Order ID will be the correlation ID in this case. This will tie up all the "Items" for a particular "Order ID".

This approach fails in my situation as I have no correlation ID. My sample file consists of a root element immediately followed by child nodes. These child nodes contain all the information. The sample file looks something like:

(a)(b)contains all the data(/b)(/a)

Ideal structure to use the looping would be:

(a)correlation ID here(b)contains all the data(/b)(/a)

2) Loop it and add on basis of the count.

Problem with this approach is that in scenario of 10 input files being dropped at the same time and each having 10 each transactions. The count will not ensure that the transactions we are adding are actually coming from a single file (as we need in our case). It is not guaranteed that the Transactions in each file will be processed in a sequence.

3) Third approach works for me. If I set up a unique ID at each transaction level. This ID is same for all transactions in one particular file. Then I can equate this ID as "File Name" in the Send Port and then use "Append". Biztalk will append all the transactions with the same unique file name. This is what I need!

Work around "Node" name changes in Schemas

Once the mapping is done and the assembly is deployed. Any change in the root node name of the schema means re-mapping the whole thing again. There is a quick work around to this.

Create a function using VB.net/C#. Pass the xml document (with the old root name) to this function. Suppose the incoming Doc is "DocIn". This function will take DocIn as a parameter. Replace the "oldname" with "newname" using replace command on "Outerxml". Pass the resulting document to the final destination. This is how the function will look like.

Public Function Convert (Byval DocIn as XMLDocument)

Dim DocOut as new XMLDocument

DocOut.Load(DocIn.Outerxml.replace("OldName","RootName"))

Return DocOut

End Function