Moved to Blogger [Reed]

Hi all, We've decided to move our blog to a new server. The new address is http://hccmsbi.blogspot.com/. (hccmsbi stands for Hitachi Consulting Company Microsoft BI.) Simon Sabin from sqlblogcasts.com was kind enough to move all our old posts over to the new site. Hope to see you there. (And I will try to be better about posting!) Cheers, Reed http://hccmsbi.blogspot.com/

MDX cell set into SSIS data stream [Reed + Pablo]


I've seen two techniques described for getting the results of an MDX query into an SSIS data stream. 1 - Use the OLE DB Source Adapter and ignore the whiny warning. This seems to work when you Preview, but crashes SSIS (even SP2) when you run the job. 2 - Use a linked server and an OpenQuery command. This works but has a lot of overhead. My colleague Pablo Guzman and I came up with a pretty slick way to do it. - Create a new ADO.Net Connection and from the .Net Providers\SQLClient Data Providers list choose the OLE DB Provider for AS9.0. - Create a DataReader Source adapter connected to the new AS connection manager. - Put the MDX into the SQLCommand property--be sure to map the external and output columns. This obviously flattens the cell set, but it doesn't limit what you put on the axes--probably a good idea to have the columns end up static. The DataReader Source adapter turns all the columns into DT_NTEXT, which you then have to convert to DT_WSTR to be able to convert to anything else, but that's standard SSIS data type manipulation. Seems a lot slicker than creating a linked server. Cheers, Reed

The 'Currency' number format style is Evil [Reed]


I always used to use Currency formatting for currency values, but it's totally wrong in a database environment. Currency displays using the current locale's currency setting. But the values in your database have a pre-known, specific currency. If the values in your database are in Euros, then you don't want somebody from California to see them showing up as dollars. The origin of the Currency style was to simplify life for people using tools like Excel to input numbers. But when used with a database, they can cause rediculous restatements. Once I thought about it, Currency format is so rediculous that I can't believe I used to use it all the time. Hence the blog entry. Even when doing currency conversions--you want to be in control of which currency format the values take. You NEVER want to be at the mercy of the current user's control panel whims. Cheers. Happy 4th of July to all us Americans. [Reed]

Sadness: RS and AS extended attributes bug [Reed]


There are two really cool chunks of functionality in the RS data extension for analysis services 1) the Aggregate function, which basically says, "just give me what the cube has for this aggregate level", and 2) the Extended Properties of a field, which allows you to grab the background color, formatted value, and other cool properties. The only problem is that you can't use both of them at the same time!!! In other words, you can get the formatted value (or background color, oetc) for the detail level, but not for higher levels. This is an acknowledged bug, but they won't fix it before Katmai unless a lot of people say it's important. If this is important to you, please go to Connect.Microsoft.Com and either file a new bug or vote for the one I posted. Here's the URL to mine: https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=245844 Cheers, Reed

More info on RS and AS [Reed]


A couple more ways the RS MDX Builder is smarter than I thought. I always wondered about some of the complexity. - When you use a parameter, there’s no way around using StrToSet to turn the parameter into an MDX set. That leaves an opening for inserting malicious code. The auto-generated MDX adds the ,CONSTRAINED flag to StrToSet, which says that the only acceptable contents of the string are literal MDX names, so it won’t run any MDX functions (such as .Children), and it won’t run any inserted code. - In AS2005, anytime you use a subcube subquery to filter down to a single date, there’s a problem with date-relative calculations because the “current member” in the report is still the “All” member of the date hierarchy. This is a serious problem right now with Excel 2007 PivotTables and MDX. The RS MDX builder adds a funny-looking WHERE clause that basically says, if you single-selected a date for the parameter, make it the current member so that date-relative calculations will work properly.” Reed

Office 2007 TR messes up Calculation Pane in BIDS [Reed]


Just spreading the word. If you get an error message when you activate the Calculations pane of a the Cube editor in BIDS for Analysis Services 2005 and get a message saying Unknown Error. Error in Application, did you just install Office 2007 Beta 2 TR? If so, that's what's causing the problem. Here's a link to the solution. http://sqlug.be/forums/339/ShowThread.aspx#339. Cost me a few days. Didn't make the link to the Office TR installation. It's possible that just copying the 2047 build files to the OLE DB folder might work, but I did the steps Frederick posted and it got me past the problem. - Reed

Combining Dimensions [Reed]


The way AS2005 manages attributes in a dimension changes the design dynamic a bit: Which attributes should be combined into a single dimension? I've run into this issue from two sides. On the one hand, we had a Project dimension with a lot of changing attributes. For example, the Project Manager would change frequently. To avoid dealing with creating lots of Type II instances of the Project, we decided to pull the Project Manager off as a separate key in the fact, making it a "Rapidly Changing Dimension". In 2000, this was often a very effective strategy. It's simple and easy to implement, but in 2005, we found that query performance was very bad. Even thougy the PM to Project relationship is very sparse, each crosstab query had to do a cartesian product and find the empty intersections. When we moved the PM back into the Project dimension, AutoExist suddenly made it really fast to find the projects for a given manager, or the managers for a given project. Conversely, I had a client with lots of customers and lots of products, but where the relationship between customers and products was very stable and very sparse. It seemed bizarre, but we explored putting both customers and products into a single dimension, with the root key as the Customer/SKU combination. Again, the query performance gain was substantial, especially when looking for which customers buy which products, etc. In 2000, I would never have considered putting Customer and Product into a single dimension, but given the high cardinality, stable relationship, and inherent sparsity, it seems to be a really good idea in 2005. - Reed

DSV joins vs. Dimension Usage [Reed]


I had a client who added a new column to a table in the DSV and then coverted existing joins from a different column to the new column. (e.g., OldDateKey --> NewDateKey). Because the DSV was changed to reflect the new join, they assumed that everything would properly. But the cube kept using the old join. The problem is that the DSV only shows potential pathways. The "real" joins between the fact table and the dimension table take place in the Dimension Usage tab. Without changing the join column in the Dimension Usage, changing the DSV doesn't do anything. The reason has to do with role-playing dimensions. In the DSV you can have multiple join possibilities--the classic example is from fact table OrderDateKey, ShipDateKey, DeliveryDateKey columns to a single DateKey in the time dimension. Any of these joins is "possible", but for a specific cube dimension, you need the Dimension Usage to specify the actualized relationship. So, get the DSV right, sure. But don't forget that the Dimension Usage is where the real "joins" take place. - Reed

ByAccount aggregation bug workaround [Joe Kasprzak]


When working with Account type dimensions and semi-additive Measures across a variety of account types with different aggregation functions, you can use the ByAccount aggregation function to set the proper aggregation behavior. There is a bug that sometimes occurs to the finance measure where the ByAccount option doesn’t properly set various aggregation rules for each account type. This can be manually fixed by the following process: * Right click the solution in the Solution Explorer window and click “Edit Database” to display the current settings for the finance measures by account type. The ByAccount aggregation function should have set for each account type to either sum for additive or lastnonempty for accounts which are semi-additive. * Click the Aggregation Function in the Account Type Mapping box and change the aggregation function from Sum to Lastnonempty for the Flow, Liability, Balance, Asset and Statistical account types. - Joe

Select All option missing [Kyle Finley]


Select All option missing for multi value parameters after SQL 2005 SP1 ==== Not sure if someone has spread the word on this or not but in case you haven’t seen it here’s a heads up. In short the RS team decided to remove the Select All option for multi value parameters. They have since realized that was a mistake. They will be adding it back in SP2 but if you need it for a client before then there is a patch you can get from MS Support Services. Check out Brian Welcker’s post [http://blogs.msdn.com/bwelcker/archive/2006/08/14/700189.aspx] for more info on this. Cheers, Kyle Finley Sr. Consultant MCSE Hitachi Consulting

Don't declare false Attribute Relationships [Reed]


It's really important to declare attribute relationships when they do exists. But what happens if you say A implies B when it isn't true. I somehow assumed that when you process the dimension, you'd get an error, but not so.

In the Adventure Works Date dimension, I created an attribute relationship from Day of Month to Day of Week. This is, of course, rediculous. Just because it's the 14th, that doesn't mean it's Wednesday. Just for fun, I then created a User Hierarchy of the same, with Day of Week drilling down to Day of Month. I got the nice, satisfying green triangle on the hierarchy. Then I deployed it.

As I said, I expected some kind of error or warning, but the days of the month just got a random weekday assigned. (I assume it has to do with the order the records were read from the dimension table.) So Sunday got associated with 12, 15, 18, and Monday with 1, 4, 21, 24, etc. This is actually the same behavior that AS2K had--if you created Day of Week as a member property of Day of Month, it would just pick one. I just thought 2005 would complain.

So, the moral is that you really really want to create attribute relationships when they are meaninful, but be careful not to create ones that are not, and don't trust silence as an indication of success.

-- Reed Jacobson


User and Attribute Hierarchy issues [Reed]


There are a couple interesting threads going on in the MS forums on attribute hierarchies vs user hierarchies. I added some long-winded comments to this one: http://forums.microsoft.com/MSDN/AddPost.aspx?PostID=560583&SiteID=1

-- Reed Jacobson


New Style and Old Style Calculated Member commands [Reed]


Chris Webb posted an interesting observation about a new style for calculated members in scripts, where instead of

CREATE MEMBER CURRENTCUBE.[MEASURES].[Old Style] AS 1, VISIBLE = 1

you can just say

CREATE [New Style] = 1;

He points out a "bug" with the old style syntax. I added some comments relating to some experiments he inspired me to do. My conclustion is that it's too predictable a behavior to be a real bug, but if MS wants to call it a bug, that's OK; and in any event, it might be worth being aware of, especially if you mix script assignments with calculated members. Here's the link to his July 2006 blog page:

http://cwebbbi.spaces.live.com/?partqs=amonth%3d7%26ayear%3d2006&beid=cns!7B84B0F2C239489A!823&d=1&wa=wsignin1.0

-- Reed Jacobson


Dynamic Dimension Security in Analysis Services 2005 [Reed]


Part I – The Measure Group Setup

 

The idea of dynamic dimension security is that with a single security role, each user gets a different list of members, based on an MDX expression that returns the permitted set. For example, User A may be allowed read access to different products than User B.

 

The MDX Username function returns the current user’s login information. If there is a many-to-many relationship between the User and the permitted products, you need a security map table, which is actually a “fact”, or bridging, table that contains one foreign key for the user and another for the product.

 

The SQL Server 2005 Adventure Works sample cube does not include a specific security map table, but it is possible to use the Internet Sales measure group as a surrogate: If a user (aka customer) purchased a product over the internet ([Internet Sales Amount]), then that user has the right to see the total sales for that product ([Sales Amount]). The structure is the same as if we used a regular security map table.

 

With a real security map table, you would create a measure group, and use a simple count of the rows as the measure, naming it something cool like [Security OK]. In this example, [Internet Sales Amount] is the surrogate for the [Security OK] measure.

 

Part II – The Username comparison

 

The format returned by the Username function is DOMAIN\User. In an ideal world, your dimension table would have an attribute in the exact same format. In the real world, you often have something a little different. For example, the name in the dimension may not include the domain. The closest thing to a user name in the Adventure Works Customer dimension is the Email Address attribute. Its format is User@adventureworks.com. This is not a bad approximation for a real-world user name attribute.

 

In the MDX security expression you compare the result of the Username function with the value of the [Email Address] attribute. Here’s an expression that will convert the Username to the right format for the Adventure Works [Email Address]:

 

Mid(Username,InStr(1,TestName,"\")+1) + "@adventure-works.com"

 

This expression will work even if users can be on multiple domains. If you know that all users will be on the same domain, you can make the expression evaluate much faster by replacing the (slow) InStr function with a hard-coded length. Assuming a constant domain name of DOMAIN, this formula is the simplified version of the previous one:

 

Mid(Username,7+1)  + "@adventure-works.com"

 

For testing purposes (in case your current domain user name doesn’t happen to be in the Adventure Works Customer dimension), you can create a calculated member as a surrogate for the Username function:

 

Member Measures.TestName as "DOMAIN\anne14"

 

Then, during early testing, you simply replace the Username function with the TestName calculated member. This makes it easy to try out different “users”. Then, during final testing, switch it back to use the Username function.

 

Part III: The MDX Set Function

 

The general concept of the set function is to find all the products where the intersection of the product, the current user, and the [Security OK] measure is not empty. (I’m going to show three methods for doing the set. If you just want the “best” answer, skip to the third one.)

 

** Double-Filter Method (2000 and 2005)

 

Here’s the way I usually did this in 2000. You use an “inner filter”, to find the User whose email alias matches the user logon (there should be only one), and use Item(0).Item(0) to turn it into a member. Then you combine that member with the test measure, and check for whether that tuple is empty in an outer filter against the list of products. This technique does work in 2005, but it’s a bit slow. On my test box it was about 1 minute.

 

Filter( [Product].[Model Name].[Model Name].Members, not IsEmpty(
            ( Filter ( [Customer].[Email Address].[Email Address].Members
            , [Customer].[Email Address].CurrentMember.Name = TestNameX
            ).Item(0).Item(0) , [Measures].[Internet Sales Amount] ) ) )

 

Note: To test the set, you can run a simple query:

 

With

            Member Measures.TestName as …

            Member Measures.TestNameX as …

            Set SecurityTest as …

Select [Measures].[ Sales Amount] on 0,

SecurityTest on 1

From [Adventure Works];

 

** NonEmpty Method (2005 only)

 

One way to simplify this is to use the NonEmpty function that is new in 2005. This basically puts the outer Filter and the IsEmpty test into a single function, but the logic is basically the same: Filter the list of products by testing it against the (single member) set of email members that match the current user. Note that you don’t need the .Item(0).Item(0) for this one, because NonEmpty is comparing two sets, so the result of the Filter function is fine as is. This is much faster—on my box, it’s about 3 seconds. Quite an improvement over 60 seconds.

 

NonEmpty ( [Product].[Model Name].[Model Name].Members
            , ( Filter ( [Customer].[Email Address].[Email Address].Members
            , [Customer].[Email Address].CurrentMember.Name = TestNameX

            ) , [Measures].[Internet Sales Amount] ) )

 

** NonEmptyCrossJoin Method (2000 and 2005)

 

Another way to do it—and this theoretically works in 2000 as well as 2005, but I don’t have a 2000 instance handy to test it on—is to use the NonEmptyCrossJoin function. Mosha says he hates the NECJ function (which is OK for him to say because he wrote it), and to use NonEmpty instead. But the NonEmpty function only takes two sets, and the NECJ function can take an indeterminate number. I have always had a great fondness for NECJ. The real goal is to find all the members of the Product dimension that actually exist in the Security measure group, when intersected with the specified member. This approach took < 1 second on my box, and I still like NECJ. Maybe there’s a better way in 2005 that can avoid NECJ, but I like this one. Plus, as an added bonus, it also exploits StrToMember which is another “no no” that I love to use.

 

NonEmptyCrossjoin(
            [Product].[Model Name].[Model Name].Members
            , StrToMember("[Customer].[Email Address].[Email Address].[" + TestNameX + "]")
            , [Measures].[Internet Sales Amount] , 1 )

 

-- Reed Jacobson


Designing legacy DTS packages from within SS2005 Management Studio [Michael B]


Problem solved. Thanks to all who wrote with suggestions.

Here is the full story. Sorry if it is a bit much, but there were several false starts.

Chapter 1:

I had been running SS2000 and had a bunch of DTS packages on my laptop. I installed SS2005 Developer’s Edition on the machine, which wiped out the SS2000 Enterprise Manager.

When I tried to open any of the legacy DTS packages from within Management Studio, I got an error message box opened telling me that I needed to install SQL Server 2000 DTS Designer Components.  Please note that I was not attempting to migrate the DTS packages to SSIS; I was merely trying to open the packages in the equivalence of the package designer used in the older Enterprise Manager.

I found the appropriate page on Microsoft’s site discussing the problem (http://support.microsoft.com/kb/905417) and the site to download the “fix” (http://www.microsoft.com/downloads/details.aspx?familyid=DF0BA5AA-B4BD-4705-AA0A-B477BA72A9CB&displaylang=en). I downloaded the recommended file (SQLServer2005_DTS.msi), installed it, and rebooted.

It was no help; symptoms were unchanged.

Chapter 2:

One of our colleagues suggested that I uninstall and reinstall SS2005. I was getting close to do this and was in the Add/Remove Programs utility when I noticed SS2005 Backward Compatibility Components item in the currently installed programs list for removal or repair. I ran the repair.

That didn’t fix it, but it changed the symptoms somewhat. Now when I tried to open a DTS package within SS2005, I got 3 separate error message boxes in succession (I believe the third was the same as the single error message I was originally getting (and oddly, sometimes I got just two messages -- the first and third. Also the package tried to open but only displayed some of the icons in the design window, usually just the connecting arrows.)

Chapter 3:

Two colleagues had sent a link to another Microsoft article  (http://support.microsoft.com/?kbid=917406), (and one of those also suggested I install the Backward Compatibility Components which I had already done, as mentioned above) and now, with the 3 error message boxes, my symptoms matched what was discussed on the Microsoft page.

The support page gave two workaround methods. The second one was in effect to reinstall the client components from SS2000. The first method promised to allow the correct opening of DTS packages from within Management Studio, which was my preferred route.

It said to remove the following lines of code from the Sqlwb.exe.manifest file in the \Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE folder.

The way the lines actually display on the Microsoft page, line 3 (red line numbering added by me for reference) is one