• SSIS #101 – Use expressions to pass parameters to ADO.NET Source

• SSIS #99 – Pass a date time parameter to the Execute SQL Task

• SSIS – Pass a variable to a OLE DB Source in a Data Flow
• SSIS – Pass a list of values to a data flow source

As you gain more confidence in SSIS, you start to design your ETL process with higher efficiency and higher portability. That means using more variables, and more package configuration files. This is when you will start to hit the limitations. One of the limitations that is relevant to this topic comes from this unfortunate message:

Parameter information cannot be derived from SQL statements with sub-select queries

This message is generated from an OLE DB Data Source in a data flow. It has a simple SQL command, but with a parameter inside a sub-query.

This is what you will see when you try to Preview or click the Parameters… button.

This is a very interesting message actually. It is telling us why SSIS refuses to accept the query because a parameter is inside a sub-query. It also gives us a very interesting recommendation, that is to use the “SQL command from variable” mode.

Another recommendation Microsoft might have given us here is to create a stored procedure with parameters. It didn’t, because creating stored procedures is essentially taking us away from Integration Services, and back to the SQL Server land. I guess the Integration Services designers wouldn’t want that to happen.

Integration Services’ inability to parse SQL statements with parameters inside sub-select queries comes from the OLE DB provider. In my case, we are using the native OLE DB native provider. You would think that this provider will be “smarter” than other providers, such as OLE DB provider for Oracle. Unfortunately, no privilege here.

Use “SQL command from variable” for OLE DB Data Source

I don’t want to go back to the SQL Server land, so following the above recommendation is my choice. The screenshots and steps I am sharing with you here are really backwards in order. If you want to implement this recommendation in your own package, you will need to follow the steps below in backward order. 

To follow the recommendation from Microsoft, you will need to change the Data access mode from SQL command to SQL command from variable.

Select the user variable you have created that stores the query as an expression. You should see that your parameter(s) are actually showing the values you have assigned to the variable(s).

Create a variable to store the parameterized query

This is the package-scope variable I’ve created to store the parameterized query.

To make this work, there are a few things you will need to pay attention to:

1) The data type of this variable should be String. A word of caution here. My query has sub-query, but it’s really a very simple query. If you have a query that is very complicated, and possibly exceeds the max length of a String data type, then the recommendation from Microsoft is not really a good choice for you. Going back to the SQL Server land to create a stored procedure will be a more manageable solution.

2) In the Properties window, set EvaluateAsExpression to True. This will give you a chance to use the Expression Builder.

3) Open the Expression Builder by clicking on the eclipse button in Expression

Use the Expression Builder to build the parameterized query

Expression Builder is a invaluable tool in SSIS, but it’s not a tool I look forward to in my daily job. The reason is that everything needs to be wrapped around the double quote. When our variable itself is a String data type, it’s actually very easy to construct the query.

String is one of the most used data types, but it’s not the only data type. In my case, my variable varDAYS_HRS is set to 60, and it’s an integer. 

In the above screenshot, the two red boxes should be 60, if I hard-code it. Since all needs to be string, I must use some data type casting function to cast it as string. (DT_WSTR, <<length>>) is a way to do just that.

A more generic syntax for explicitly converting an expression from one data type to a different data type is here:

(type_spec) expression

See Cast (SSIS Expression): Convert SSIS Data Types from MSDN for details.

Always use the Evaluate Expression button on the Expression Builder screen to make sure that your expression is acceptable by Integration Services.

Create another variable to hold the dynamic parameter value

Variable varDAYS_HRS is created to hold the dynamic parameter we need to pass to the query. This is a simple user variable:

“SQL command from variable” for OLE DB Data Source is a valuable recommendation

Next time, when Integration Services cannot parse your parameterized queries, give this recommendation a try if your query is reasonably simple and short.

According to MDX Function Reference, it returns the set of tuples that are not empty from a specified set, based on the cross product of the specified set with a second set. The syntax is simple enough:

NONEMPTY(set_expression1 [,set_expression2])

Look closely, you will see that the usage of the set_expression2 is optional. I’ve been using NonEmpty() without the set_expression2 for many of my SSRS reports. Going from “out-of-memory” on the report server to only a few seconds to render my reports, I was so happy that I’ve never thought about testing record counts or some measures, until I finally had time.

What I saw in my test results shocked me. After some research and reading, I realized that the problem is in the omission of the set_expression2. By including the appropriate set_expression2, I got the perfect results.

In this blog, I’ll try to share with you this little lesson I’ve learned.

Non-empty is a characteristic of the cells references by the tuples, not the tuples themselves

This is an important note from the above MDX Function Reference from Microsoft. Now that I am reading this note, I start to visualize:

1) set_expression1 will be typically the cross product of members of some hierarchies, for example,  

DEDESCENDANTS([Queue].[Queue].[Level 03].[Online Email], [Queue].[Queue].[Level 05], self) *

[Time].[Day].[Day].MEMBERS

2) Since the non-empty is really a characteristic of the cells, and we want to remove sets that have no certain measures, does that that mean we really need to specify the measures we are referring to? Now it makes sense to me that we do need to specify the measures we are referring to.

If a second set is not provided, the expression is evaluated in the context of the current coordinates of the members of the attribute hierarchies and the measures in the cube

This is another important note from the above MDX Function Reference from Microsoft. So if the second set is not provided, we are blindly relying on the Analysis Services to find these “current coordinates”. “Current” really means at the moment of evaluation. Unfortunately, these “current coordinates” can be pretty random at the moment of evaluation, depending on how we setup our default members, and especially if we are defining calculated measures in our query. The default initial value of any calculated measure is null. This can lead to a very messy, confusing, and unexpected query results.

set_expression2 is not provided – wrong result

This is part of my initial query, where the second set expression is not provided.

Although the NonEmpty() function is used, rows with all nulls are plainly showing. Further testing showed that the record counts and measures are incorrect.

set_expression2 is provided – All looks good!

This is part of my revised query, where the second set expression is now provided with a measure.

There is no longer any rows that are all null. Further testing showed that all numbers are correct. 

Conclusion – Always include a measure in the second set in NonEmpty() function

I will put this lesson in my MDX best practice list. Hope you will do the same.

In the next few blogs, I’ll share with you a few simple steps to create your own XML configuration files and then how to use them to make your SSIS packages dynamic and portable.

Benefit of using package XML configuration files

There are many resources on this topic. In a nutshell, using package configuration files is a powerful way to make your SSIS package extremely versatile.

Example 1 – Make a connection string for a connection manager dynamic

We typically have our server name, database name, user name (password as well sometimes) embedded in our Connection Manager.

When we are ready to move our packages from development to test, or to production, the least we would expect is that the server name will be different.

In this case, the goal of using a package configuration file is:

1) no modification to our packages themselves

2) the only change we will make is to the package configuration file. I’ll need to change the Data Source and the Initial Catalog accordingly.

Example 2 – Make a Value for a variable dynamic

A typical example is a path where we fetch our incoming files. This file path will be most likely different from what is on our local PC and what is on our test or production server, especially the root folder will be most likely different.

Again the goal of using a  a package configuration file in this case is:

1) no modification to our packages themselves

2) the only change we will make is to the package configuration file. I’ll need to change the ConfiguredValue accordingly when I move my package to the test or the production server.

So, it’s fair to summarize the benefits of using package configuration files as:

• no modification to our packages themselves

• the only change we will make is to the package configuration file

   

If it’s your first time to use a package configuration files, and you received it from a co-worker, you might wonder how the package configuration file is created.I’ll share two examples with you. One example is to create a package configuration file for a connection manager. Another example is for a variable.The steps of creating package configuration file is simple enough. There are two things that are not crystal clear though:1) The multiple choices of properties that are available to us. All the properties are exposed to us. Which one do we need? 

 

2) What are we going to do about the configuration file?

I’ll continue to share with you these two tasks when I get time in the future.

It’s actually pretty easy to accomplish that in both SSRS 2005 and 2008. I am going to show a few screen shots here.

Challenge

Suppose you have a very simple table with sales data for products sold from each store.

You want to rearrange the data so it shows the stores vertically, all products horizontally, and the Sales Amount at each intersection. Finally you want to show totals at the bottom and also to the right.

Matrix will be a perfect choice for this purpose with no extra coding on your part.

Solution – create a matrix and then add totals to row and column

The steps are very straightforward. I am going to show the screen shots from SSRS 2008 here.

Here is a matrix without any configuration yet.

Point the cursor to the Rows cell, an icon appear on the upper right corner. Click the icon, you will see the list of columns from your dataset. Select Store.

For the Columns cell, select Product.

For the Data cell, select SalesAmount.

Now, right-click the data cell, you will see choices under Tablix. Select Add Totals, and then Row.

Repeat the above step to create a total next the column Product.

But what if you already “infected” your report server” with unwanted default values for report parameters? It has happened to me many times. So I thought I should share with you this little annoying thing about default values for parameters on report server.

Here is a screenshot from the report server showing the unwanted default values.

Talking about unwanted or unwelcomed guests, this is one of those. It will stay on the report server, unless we take time to remove them.

Simply uncheck the box “Has Default”.

Complain itself will not solve any issue. But it drove me to focus on the frustration and research solutions or workarounds.

Issue – The MDX query designer in SSRS 2008 overwrites my parameter queries

When using Analysis Services as data source, the query designer is a very attractive tool for assembling MDX queries. When I decide to use parameters, it even creates the queries for me for the parameters. All is good, except when I found out that my customized parameter queries got overwritten over and over again.

There are many reasons why I write customized parameter queries. Here are a few:

1. The least reason is to format my MDX queries for better readability. If you have read some machine-generated MDX queries, such as queries from Excel, ProClarity, SSRS, you would not blame me for spending some extra time for better readability.

2. The auto-generated parameter queries most likely will retrieve the top level member ALL for the hierarchy along with children members. I personally do not like this feature. I often modify the parameter queries to exclude the ALL member.

3. To implement cascading parameters, the auto-generated parameter queries must be modified.     

Unfortunately, if I open the main data set again in the MDX query designer, all the customized parameter queries will be overwritten. 

A Workaround – Add a tag SuppressAutoUpdate = true to the RDL file

There are very few books on the integration between SSRS and SSAS, even fewer resources on solutions to the issues we face. Thanks to some dedicated bloggers, I found the following articles online regarding this annoying issue:

• Prevent SSRS 2008 from Overwriting Datasets
• Stop Reporting Services (SSRS) 2008 from overwriting custom Parameter Datasets

This link, MDX Query Designer Overwrites Parameter Queries, is actually a bug report on Microsoft Connect. It seems that this new issue in SSRS 2008 had never got fixed. I could be wrong on this. It will be appreciated if anybody can let me know if this has been fixed in later release of SSRS 2008.

It turned out that  the workaround to preserve my changes to the parameters queries is to modify the report’s RDL source code as follows by adding <rd:SuppressAutoUpdate>true</rd:SuppressAutoUpdate> between the tag <Query> and </Query>:

<Query>
   …
   <rd:SuppressAutoUpdate>true</rd:SuppressAutoUpdate>
</Query>

Here are some simple steps to accomplish this:

1. Right click the report in BIDS 2008, and select View Code.

2. Search string </DataSet> in the RDL/XML file.

Then add <rd:SuppressAutoUpdate>true</rd:SuppressAutoUpdate> to just before </Query>.

3. Repeat step 2 for every data set (or only the data sets you do not want to be updated).

Default values for parameters still need to be manually removed

The above workaround prevents SSRS 2008 from overwriting my parameters queries, but still leaves the Default Values for my parameters.

This is actually not a bad thing for testing purpose because it can save me many mouse-clicking to select parameters. When I am ready to send my report to the report server, I can just open the property window for my parameters, and change to No default value on the Default Values tab.

So I guess I will keep complaining about SSRS-SSAS-MDX till I can find either workarounds or solutions.

A co-worker helped me got a data source created with Microsoft SLQ Server Analysis Services as the Data Source Type.

For credentials for connection, we correctly flagged “Credentials stored securely in the report server” with the right User name and Password.

The initial try of using the newly created data source failed, however. After fixing two issues, the configuration for the data source finally worked. The final configuration is shown below.

Since I don’t do this kind of task every day, I thought it would be a good idea to write down how we fixed the issues.

1. The user name for the connection should be part of the server role

This is the error message from the report server before we made sure that the user name is part of the server instance role.

In SSMS, right click on the Analysis Services server instance, select Properties. On the Security tab, make sure that the user name is added. I have a little doubt about the user being part of the Server administrators. It seems to be too much privilege to me.

2. Check "Use as Windows credentials when connecting to the data source”

Before we check this option, here is the error we got:

I guess this has to do with the fact that we can only connect to Analysis Services (and also Integration Services) instance via Windows Authentication.

(Note: in the screen shot above, ignore the red warning about invalid password. I changed the User name to be a fake user name for this blog.)

In SSIS package design, however, I have used the Foreach Loop Container very often. I would not say that I have broken my vows though for the following reasons:

1.  The record set I loop through is usually relatively small.

2. Foreach Loop Container in Integration Services takes a data type of System.Object (not the physical SQL table), which is a in-memory object that is native to the Integration Services. So I trust that Microsoft engineers have done a wonderful job in terms of optimal performance.

In Integration Services, I have used two ways to generate this System.Object for the  Foreach Loop Container to consume.

One way is to use an Execute SQL Task. I have blogged about this way in the past.

• SSIS – “Full result set” to a set variable of Object.
• SSIS – Use “Foreach Loop Container” and pass a variable to a OLE DB Source in a Data Flow
• SSIS #99 – Pass a date time parameter to the Execute SQL Task

Here are two screen shots that shows how to configure the Execute SQL Task to receive the query result into a variable with data type of System.Object.

Although you can map the query result to more than one System.Object variables, this would not work for the problem I am facing now.

Challenge

1. Here is a pseudo query that I need to retrieve a collection of new QueueName and their parent IDs.

2. I want to use both the QueueName and the QueueParentId, and pass them through a Foreach Loop Container to another task and use them as query parameters.

For the 2nd challenge, where more than one columns must be split up from one single System.Object, I cannot use Execute SQL Task with a Full result set configuration.

Recordset Destination comes to the rescue

The ADO recordset fits perfectly for my problem.

A Design Pattern (Loop through rows with multiple columns) with Recordset Destination and Foreach Loop Container

1. First, I need three package level variables. vLoopSet is the System.Object that will store the ADO dataset. The other two non-System.Object variables are for each column in the ADO recordset.

2. Here is the Control Flow I will need. The first Data Flow will run some query against my data source and send the results to a Recordset Destination.

3. Here is the Data Flow.

Configuration the Recordset Destination is simple enough. All the interface on top is useless. The only thing you need is to set the  VariableName.

And also the Output columns.

4. For the Foreach Loop Container, select Foreach ADO Enumerator, User::vLoopSet, and Rows in the first table.

For the variable mapping, use the zero-based index to map the columns to the variable. Index zero will be the first column in the query, and 1 is the second column in the query.

5. Finally, we are ready to use the User::vQueryName and User::vQueueParentID as dynamic parameters for our task inside the Foreach Loop Container.

Here we are using the famous question mark again to represent our dynamic parameters. Note: the ? is position sensitive, meaning that the first ? will represent the first variable from the above mapping, and the second ? represents the 2nd variable in the above mapping in the Foreach Loop Container configuration.

I hope after reading this post, you will rush off to to open your own SSIS packages and find opportunities to use this Loop Through Rows with Multiple Columns pattern.

1. It counts rows as data passes through a data flow, and

2. it stores the final count in a variable. (Note that the variable is only populated when the data flow is finished, not while the data flow is still in the middle of transferring data.)

Here is a data flow design pattern I use very often.

At the end of the data flow, my variable @vNewQueueCount is populated with the row count from the data source.

Then I use this variable @vNewQueueCount in a Precedence Constraint expression, as shown below, to decide conditionally whether I need to process my Queue dimension or not.

Configuring the Row Count transformation is simply enough. I’ll show you that later in this blog.

What I want to say first is why the Row Count transformation is invaluable in this specific design pattern.

In this design, I’ve decided to use the Recordset Destination, which is a in-memory ADO dataset and it saves data in memory in a recordset that is stored in an Integration Services package variable of the Object data type. In my next blog, I’ll show you how to use a Foreach Loop container with the Foreach ADO enumerator to process one row of the recordset at a time.

The Recordset Destination is a handy component to use when you do not need to (nor want to) persist your data. One drawback of using the in-memory ADO dataset is that Integration Services does not provide any out-of-box methods to get the row count in the dataset. This is when Row Count transformation becomes invaluable to me. Without too much work on my side, I am able to accomplish the following:

1. Retrieve data from the source, and store the data in a, ADO dataset.

2. Also store the row count in a variable.

3. Finally, be able to use the row count variable to conditionally decide whether I need to process my dimension or not.

Here are a few simple steps to configure the Row Count transformation.

1. Define a variable vNewQueueCount.

2. Create a new data flow

3. Drag the Row Count transformation to the data flow design surface. On the Advance Editor, for VariableName select User::vNewQueueCount.

Finally, you can use the variable in a precedence constraint expression to do conditional processing:

@vNewQueueCount > 0

Have fun with the Row Count transformation!

The issue of integration between Reporting Services and Analysis Services is a never ending source of pain and frustration for many BI developers. Before I talk about the pain and frustration I’ve experienced, I want to give you some hope first by showing you two types of data providers we can use in SSRS to access your cubes.

Data Provider 1 – Microsoft SQL Server Analysis Services

Many blogs, such as my blog above, showed you the steps to create SSRS reports accessing cubes in Analysis Services database without telling you a very important piece of information. That is they are using the Microsoft SQL Server Analysis Services as the data provider for the integration between SSRS and SSAS.

This simple omission can be very beneficial to you, or can create frustration for you as well, depending on the nature of your project. If your cubes are designed well and your data navigation flows naturally as you would want, the drag-and-drop nature of this type of integration can be very beneficial to you. The well-liked SSAS query designer can be a life-saver for many developers who are so much experienced in creating manual MDX queries yet. On the other hand, when you start to experience serious performance issue with your SSRS reports, you would start to struggle with the un-optimized MDX queries that the integration has generated for you.

1) This is because this types of direct integration between SSRS and SSAS assumes that CROSSJOIN works for all data. Even without much experience in MDX, we all know that CROSSJOIN is absolutely not the best candidate for optimal performance in either TSQL or MDX.  

2) It also assumes that we need help to create MDX queries for every report parameter. Any customized MDX queries you managed to create for your report parameters will be overwritten without any notification or your consent. It also assumes that we do not have interest in seeing the datasets it has created for our parameters by hiding them from us. This can create serious performance problems for us when many unused datasets are hidden from us.    

Coupling the above two “helps” from this type of direct integration with the serious performance issue with your reports, there is nothing but pain and frustration.

(To be fair, this type of direct integration provides lots of benefits too. This post intends to focus on the pain and frustration, so I’ll ignore the benefits for now.)

Data Provider 2 – Microsoft OLE DB Provider for Analysis Services 9.0 (9.0 fro AS 2005 or 10.0 for AS 2008)

The OLE DB provider is not exactly better. What it interests me is that now I can put a little more customization/optimization to the MDX queries. With more customization, my hope for performance improvement (without any modifications to many shared/existing dimensions in the OLAP database) has increased a notch (if not soared).