TBSM and eDayTrader sample contents - my comments.

Technically today.

The eDayTrader sample application for testing TBSM and JazzSM has been around, from even before JazzSM era, when we did want to demonstrate TBSM, TADDM and ITM/ITCAM capabilities all together. The eDayTrader application was deployed onto WAS and DB2, and thanks to TADDM you could discover its components, import to TBSM and then monitor with ITM. I remember such a TADDM class at least, long time ago, nevertheless, the eDayTrader J2EE application can give you quite a good sense of what BSM methodology might be about when it comes to service modeling.
Today you can download the eDayTrader sample code for TBSM and, without a need to have that application deployed onto your WAS, you can generate a service dependency tree in TBSM and run the predefined data fetchers to fetch the eDayTrader component service instances metric data from a sample DB2 table and present that data right on your JazzSM/DASH dashboards. This is very handful, first, what I find difficult for our users as such is - still, after years - it helps with understanding of how TBSM and BSM products (today it's NOI - Netcool Operations Insight, TADDM, TBSM, JazzSM and OMNIbus/Monitoring) can be really used, means not only for monitoring applications but also for monitoring some business metrics. It helps to some extent of course, as the eDayTrader templates and fetchers are taken out of context, not sure if even any real, although surely based on some real J2EE application examples. But it is still a good exercise to play around it in TBSM. Secondly, the eDayTrader sample code is handy now, because it familiarizes the users with JazzSM/DASH, the first really new, post-TBSM custom canvas era dashboards editor. So I truly encourage all of the readers who haven't had a chance to play around that exercise, to try, spend some time and see it. You can download the thing here, from developer works (you'll need IBM ID to do that):
https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=7d5ebce8-2dd8-449c-a58e-4676134e3eb8#fullpageWidgetId=Wea1cb2531f10_4ccd_99d7_6ab0334cb21f&file=b45d5b68-1dbd-4ecd-aa25-57c34f4d812f

The PDF describing what might be done with the contents in JazzSM after you deploy them to TBSM, is here:
https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=7d5ebce8-2dd8-449c-a58e-4676134e3eb8#fullpageWidgetId=Wea1cb2531f10_4ccd_99d7_6ab0334cb21f&file=293578da-e095-46be-8f35-594f476400d7

I found few minor issues that might be crossed on some readers' way to complete the exercise.

1. The database setup on Windows. I do have one of my TBSMs on Windows and I did use that particular one for installation of eDayTrader contents. I perhaps did misread the instructions and did something wrong, but I believe that the case of the administrative user used might have been bit better documented. Namely, you're supposed to install the eDayTrader database with an administrative user that will be used for accessing the database tables from withing the TBSM data fetchers. On Windows, the default administrative user is db2admin, and on Linux/Unix it is db2inst1. I passed db2admin everywhere but I executed the *.bat script in my administrative DB2 CLI windows, which is something that I open as my another user (I will not unveil its name here). So what happened - the tables were created fine, the schema was set fine, I could see the tables and data via SELECT statements fine, but I couldn't do same selects in TBSM / Impact queries. The JDBC using db2admin user couldn't see them. I did receive SQLCODE=-204, SQLSTATE=42704 in my data fetcher View Data dialog, means invalid schema in query. Almost went crazy about it, I found few useful commands to check what's up with the schema and here I share it:

>db2 "select substr(GRANTEE,1,16),GRANTEETYPE,substr(tabschema || '.' || tabname,1,64), CONTROLAUTH, ALTERAUTH, DELETEAUTH, INDEXAUTH, INSERTAUTH, REFAUTH, SELECTAUTH, UPDATEAUTH from syscat.TABAUTH where tabschema like 'DAYTRADE'"
should give you something like:
1                GRANTEETYPE 3
              CONTROLAUTH ALTERAUTH DELETEAUTH INDEXAUTH INSERTAUTH REFAUTH SELE
CTAUTH UPDATEAUTH
---------------- ----------- ---------------------------------------------------
------------- ----------- --------- ---------- --------- ---------- ------- ----
------ ----------
DB2ADMIN         U           DAYTRADE.EDAYTRADERDATA
              Y           G         G          G         G          G       G
       G
DB2ADMIN         U           DAYTRADE.LOST_REVENUE_ALL
              Y           G         G          G         G          G       G
       G
DB2ADMIN         U           DAYTRADE.LOST_REVENUE_VIP
              Y           G         G          G         G          G       G
       G

  3 record(s) selected.

so DB2ADMIN is the grantee.
If you've got another user there, use its name in your TBSM JDBC connection setup, or, rerun the DayTrade database creation with this change to create_daytrade_schema.sql:
CONNECT TO DAYTRADE USER db2admin USING <yourpasswordhere>;

So in this case simply drop daytrade database (use FORCE APPLICATION ALL to release all connections if DB2 doesn't allow you dropping it because of available connections) and rerun your installation script (Dashboards-Sample-Data_v1.1-Artifacts\Step2_TBSMConfiguration>daytrade_configuration.bat).

2. I had a nasty issue with locked contents of Impact: service list and data source list. As part of the eDayTrader package installation you've got to install TBSM/Impact project and if you have any SVN locks on any Impact data like lists, that step will fail. When you run the second step of the installation process, you may see friendly messages like:
     [exec] Please make sure no locks exist in "IMPACT_HOME\etc\<servername>_versioncontrol.locks"

which is ok, unless you see:

createImpactAccess:
     [echo] Importing using nci_import
     [exec] Please make sure no locks exist in "IMPACT_HOME\etc\<servername>_ver
sioncontrol.locks"
     [exec] Check impactserver.log for more details
     [exec] Final result - Failure : 1

Also, if you've got errors and exceptions like:
File etc/TBSM_datasourcelist is locked by user SYSTEM
you're in trouble.

How to deal with it - not sure what eventually worked, I went to TBSM/Impact UI and unlocked all data sources and services. It wasn't it. Then I found an SVN command to use:
C:\Program Files\IBM\tivoli\tbsm\platform\win\svn\bin>svn cleanup "C:\Program Files\IBM\tivoli\tbsm\etc"
but it didn't help. At the end I went to version control locks file and removed all entries like:
#This file was written by server.
#Sat Feb 28 23:26:45 CET 2015
impact.locks.numlocks=1
impact.locks.1.user=defaultWIMFileBasedRealm/tipadmin
impact.locks.1.file=etc\\TBSM_EDayTraderData.ds
to:
#This file was written by server.
#Sat Feb 28 23:26:45 CET 2015

and left nothing else inside that file. It didn't help either, so I restarted TBSM WAS profile and that did help. So probably the combination of the server restart and releasing the locks from the file or SVN command did work out.

So here we go. I did install the eDayTrader package from the DW site fine and can move on to playing around it more and show to my customers if necessary. Nice.

Netcool/Impact and ServiceNow! integration

Have you ever tried to integrate Netcool/Impact and ServiceNow! ?
ServiceNow!... It's an interesting piece of software I must admit and it's in a public cloud, in Internet. You can create a lot of customizations and your own project and applications like CMDB, incident management, problem management, probably marketing people from SN! would tell you more benefits.
I've had a need to integrate my Impact 6.1.1 with SN! via SOAP recently and just wanted to share few general tips.
1. SN! WSDLs seem to be not correctly generated for Axis2 and must be manually adjusted before you approach compiling them in Netcool/Impact, I basically go and open my WSDL, I visit every complextType entry and remove the name parameter and value, so I leave <complextType> only. Only with this your WSDL will compile in nci_compilewsdl. Thank you Yasser from Impact dev team to pointing me to that one! :-)
2. SN! CA-signed certificate must be imported to your ImpactProfile WAS. It's Entrust Inc. And here's the full instruction for Impact: http://www-01.ibm.com/support/docview.wss?uid=swg21592616
3. You've got a basic HTTP authentication in SN! and probably won't be allowed to switch it off, then compilation of WSDL is possible only locally with a downloaded WSDL file (both from CLI and GUI).
4. Policy generated by wizard is good and works well, usually a single parameter being selected helps it working better since the beginning (so you generate WSParams variable correctly).
The documentation of SN! in wiki is pretty well however not everything is documented and beware of your version of ServiceNow!
Everything else depends on your needs, if this is incident management integration (opening tickets) or CMDB (importing service trees to TBSM).
That would be it. I'll share more after I finish my integration with TBSM.

TBSM 6.1.x database installation gotchas - real disk space requirements.

Have you ever tried to install TBSM 6.1.x on non-single partition disk space? It’s easy, you usually go for it on your demo VMware image, and it can be a production case too.

It usually looks like this after all the installations (in this case we have logical volume to extend at any time by adding some more disk space):

Picture 1. Regular disk partitions layout on Unix-alike system, easy and boring.

Piece of cake. Let’s try something harder. Typically AIX systems storage is extensively divided on numerous disks in farm or Power System storage. Nevertheless a similar scenario can be easily emulated on your VMware box, assuming you create number of independent disks during your virtual OS installation, like in the example below:

Picture 2. Creating non-single-disk-based disk space in virtual Linux.

The goal is to achieve the following setup. I have home directory on one disk (sda5), opt directory (historically for “optional” packages, it's default directory for most of Tivoli packages on Linux platform) on another disk (sda2), tmp directory on sda6 and root directory on sda1. See this:

Picture 3. 6 successfully installed virtual disks on Linux

Now let's follow the available disk space degradation as we install next components of TBSM. We start from DB2 9.7 installation.

Picture 4. Disk space after installation of DB2 database manager code.

It looks quite obvious, DB2 default installation directory seats in /opt directory, which mounts sda2 disk. You can check for exact space value taken by DB2 database manager, by running this command:

Du –m /opt/ibm/db2/V9.7

871

If you compare to disk space consumed by DB2 instance itself, you'll see consumption in /home directory, this is because I installed default DB2 instance on Linux platform, called db2inst1 which uses db2inst1 user to run and this user's home directory to store the data files. Here's how fresh DB2 instance impacts the available disk space on my Linux vm:

Picture 5. Disk space after configuration one default DB2 database instance.

So the disk space taken by DB2 instance files (and don't forget you have to create fenced user for running stored procedures and at least one administrative user for managing all instances, so db2fenc1 and dasuser1) is about 80 MB.

I'm sorry for taking you so far to this place to tell you this. It's not going to work with your default TBSM installation. Why? It's too little space for DB2 instance data files. If you continue with TBSM database installation, you'll see these sizing options:

Picture 6. TBSM database installation size options.

According to the documentation for TBSM 6.1, available here, You need to secure the following disk space:

• Small - 3G of disk space
• Medium – 6G of disk space
• Large – 10G of disk space

Well, it's maximum limits, someone may say. If you plan for running just a simple demo, it's not going to break anything. This is true, however would you assume that you needed to monitor disk space consumption especially because of that? If yes, good for you, skip it. If not, consider this. Why? Let me show you what's going to happen and why you don't want to use your default settings during TBSM installation, related to data paths and log paths

Let's assume we simply continue the installation with defaults. Database configuration, especially transaction log files size for TBSM database won't accept your disk space offer. This is how your installation is going to finish, if you continue with it like presented above, it will simply fail:

Picture 7. Example of failure message during TBSM database installation if disk space is too low.

The same for TBSM Metric Marker tables and demo tables. If you go to

/opt/IBM/tivoli/tbsmdb/logs/db2_stdout.log log file, you'll read:

CREATE TABLESPACE TBSMSCR16KTS PAGESIZE 16K MANAGED BY AUTOMATIC STORAGE INITIALSIZE 100 M BUFFERPOOL TBSMSCR16KBP

DB21034E  The command was processed as an SQL statement because it was not a

valid Command Line Processor command.  During SQL processing it returned:

SQL0968C  The file system is full.  SQLSTATE=57011

Off the record, after that entry in the log you can see that the installer tries to continue executing the DDL script without validation of available disk space based on the first failure occurrence, what starts all series of unfortunate events in consequence and doesn't allow your installation succeeding at the end.

So it looks bad. Let's see the disk space now:

Picture 8. Disk space after failing installation.

Well, it doesn't look that bad now after it failed, we still have some space available, it means that creating the tablespace must have used up all the space first, then failed and then rolled-back and so it released some of taken disk space after all. But what happened? Well, after TBSM database migrated to DB2 from Postgres in version 6.1.0 (previous TBSM 4.2.1 - there is no TBSM 5.x - was using Postgres 8.x) it became exposed to all those challenges and gains coming from the fact. In order to understand it step by step we need to get closer to few DB2 design assumptions. Here's couple of settings which should draw your attention:

Picture 9. Default buffer pool settings for TBSM database.

So, this is buffer pool setting first. Each row in each table is being cached in memory in objects called pages (number of rows) every time when DB2 has to read that data from external memory, means disk. Also, if data is to be written to disk from buffer pool to table space, you can calculate the maximum amount of data in 16K buffer pool will be 48MB and 32MB in 32K buffer pool. This is not something dangerous to your installation yet. It tells you about potential RAM memory consumption in future when you launch TBSM into production mode. But the screen tells you also about table spaces being created for TBSM database, let's take a closer look on them.

Note, if you actively use TBSM Metric Marker and Metric History databases, they have their separate settings and disk and memory consumption rates.

From this command:

CREATE TABLESPACE TBSMSCR16KTS PAGESIZE 16K MANAGED BY AUTOMATIC STORAGE INITIALSIZE 100 M BUFFERPOOL TBSMSCR16KBP

You know that TBSM database has automated storage management which means that database manager creates new data containers if only needed. It is not System Managed Space (SMS) type of table space or database managed space (DMS) type. It means maximum capacity data can reach is defined by storage size while creating TBSM database, determined by the path. In the screenshot below you can see that TBSM database path is <default>. 

Picture 10. Sample configuration of TBSM database, it shows default database path.

What <default> is? Well, go to your DB2 command line, and run as DB2 instance owner the following command:

db2 get dbm cfg | grep DFTDBPATH

By default it is instance user's home directory, in my case: /home/db2inst1.

It means your database will grow as long as it hits the storage limits, it means endlessly until some free disk space remains. It's something good to remember, if didn't realize or didn't have clear answer. It means you don't want to select <default> during your TBSM database installation, you rather want to check on other disks space and allocate TBSM data files there.

Next, lets see the transaction logs, like below. These steps lets you define how many and how big logs can be created for you TBSM database. Again - for TBSM Metric Marker and history databases it's totally a separate story. So by default you define 10 primary logs and 2 secondary logs, 16000 4K pages big each, means up to 12*16000 = 192000 4K pages, means 768 MB of data max. Note, on Unix and Windows platform, the default log file size, both for primary and secondary is 1000*4K and range 4 - 1 048 572 (always *4K page size). This space gets allocated when only your database activates. It means you need to have this space available on your hard disk immediately when you start your database manager. Again, logs, similarly to data files, by default utilize the default log path name, see the screenshot below. By default it means it all goes to /home/db2inst1 directory again:

Picture 11. Transaction log for TBSM database configuration snippet.

What to do then? Again, if you miss space in your /home directory mount, select another value for Log path name. Be knowledgeable,  know what it all means to your installation. Take the installation hardware requirements seriously and monitor the usage.

Last but not least, TBSM DB directory which is used to store TBSM DB DDL files, executables to recreate TBSM DB and jars etc. takes it piece of cake too, it's 160 MB declared by the installer, and you can see it by Summary step:

Picture 12. Summary view

Don't forget about temporary disk space the installer takes and returns, but it must be available for installation time. The installer is flexible and will look for 200 MB in /tmp or /home directory of the user who runs it, in our case db2inst1 home directory (I have only 170 MB available in /tmp):

Picture 13. /tmp disk space is too low

Conclusions.
I'll need more than 1 GB disk space that I assigned to /home directory to install TBSM database successfully and the Install Guide is not specific about that.
This is because by default my data logs and transaction logs go to /home/db2inst1 directory and this is because I didn't simply create one single disk space for all directories, which can be a real case in production environments too. Additionally all the temporary files of the installer will be copied there for TBSM DB installation time.

So what is the real hard disk space requirement for TBSM DB installation?

/home
a) at least 1x768MB for transaction logs (10 primary, 2 secondary, 16000 4K pages each, all in one TBSM database - means you don't create separate TBSMHIST db)
b) 80 MB for fresh db2 instance installation
c) 200 MB for temporary files in case you don't have space in /tmp
d) suggested 3 GB for up to 5000 service instances
Total: 4048 MB minimum in /home

/opt
a) at least 160 MB for tbsmdb installation in /opt/IBM/tivoli/tbsmdb
b) 871 MB for DB2 database manager in /opt/ibm/db2/V9.7
Total: 1031 MB minimum in /opt

Keep in mind this is still before installing TBSM dataserver, dashboard server, JazzSM, XMLtoolkit and ITMAgent for TBSM or embedded Netcool OMNIbus with EIF probe.

So, if you're lucky and have one single / root partition only for all your files in your Linux or Unix box, prepare minimum 5,1 GB disk space for TBSM database preparation total. If you only get tempted to create TBSMHIST table, it will be next 768 MB for separate transaction logs. You can lower number of logs or decrease single log file size to accommodate it though. You'll be 200 MB ahead if you secure enough space in /tmp directory (I put it 500-1000 MB to be safe usually).

That's all for now, thanks and see you next time.

KPIs disappearing? No way!

Look at this article:
http://www.prelert.com/blog/623/Fair enough prelert tries to make their money but I spot this:
"KPI’s probably won’t disappear any time soon, but predictive analytics tools for IT operations, such as Prelert’s, will increasingly become indispensable for predicting and tracing the roots of performance issues."
KPI can be anything, I just invented a new one: time to find a needle in haystack, let's call it short: Time To Needle.
Hence, I think KPIs will stay with us forever. There always will be performance of something to measure and express.

What is the important thing in GUI?

I just read this article:
http://www.techrepublic.com/blog/european-technology/ok-so-stick-with-windows-xp-but-how-big-a-risk-do-you-run/1627?tag=nl.e101&s_cid=e101&ttag=e101

After reading the whole article out, I started wondering what is the best thing I like in Windows XP the most.

It's GUI.

There's something in common in Windows XP, Samsung TouchWiz or LG Smart TV GUI (its biggest sense can be seen in Home Dashboard) are EASY and SIMPLE or CLEAR, English dictionary suggests me to say legible. This, plus the fact those GUIs colorful in kind of elegant way (to me, some people may see they're kitch) makes me to stay and use it and enjoy.

I'll never forget those long hours spent on freshly new installed Windows XP systems, while working with HTML, JavaScript and PHP pages back in 2012/2013. So modern, user friendly and so cool Windows XP seemed to be so was the work with tools and apps. On top of that, music being played in WinAmp was adding a charm to those long evenings and nights over my computers. In fact, there's kind of sentiment in my generation and older too - Windows XP was the first one in Windows series which I didn't have to reinstall at least annually.

And now, well, XP is going to sunset in 12 months. Pity. Sentiment is there. And just a greatly simple GUI is something I cannot understand why MS gave up. 

XMLtoolkit stop issue

If this ever happened to you, that XMLtoolkit doesn’t want to stop normally or gives you other issues related to creating connection to itself, it must be registry error.

Here’s the symptom:

[netcool@tbsm61 bin]$ ./tbsmrdr_stop.sh GTMCL5478W: The request could not be delivered, the toolkit may be down. If the toolkit is busy processing data, allow it to complete and shutdown gracefully. If the toolkit is idle but will not stop, reissure the request with the -f flag. The exception was: Exception creating connection to: 172.16.1.103; nested exception is:         java.net.NoRouteToHostException: No route to host retCode: 4

So I try to stop my XMLtoolkit instance and the script fails and the toolkit itself keeps running.

This is mainly because of XMLtoolkit failover capability feature. Each instance registers itself by IP of the machine it runs on, and it’s usually IP configured for the first network interface. You can check on this at any time:

[netcool@tbsm61 bin]$ ./registryupdate.sh -U db2inst1 -P smartway -v GTMCL5457I: Toolkit registry table information. ID: 1         Name: 172.16.1.103         Primary: true         Action: 0         LastUpdate: Thu Jan 01 01:00:00 CET 1970 ID: 2         Name: null         Primary: false         Action: 0         LastUpdate: Thu Jan 01 01:00:00 CET 1970 GTMCL5358I: Processing completed successfully. retCode: 0

The corresponding value is written to xmltoolkitsvc.properties under DL_Toolkit_Instance_ID property.

It may happen especially on a virtual machine, which is being reconfigured as moved to new networks. There’s quick remedy for this: update DL_Toolkit_Instance_ID property with a new value, like static IP or a unique hostname in xmltoolkitsvc.properties file and register that value in the database:

[netcool@tbsm61 bin]$ ./registryupdate.sh -U db2inst1 -P smartway -s 1 GTMCL5458I: Setting the toolkit registry table GTMCL5358I: Processing completed successfully. retCode: 0 [netcool@tbsm61 bin]$ ./registryupdate.sh -U db2inst1 -P smartway -v GTMCL5457I: Toolkit registry table information. ID: 1         Name: 10.10.10.21         Primary: true         Action: 0         LastUpdate: Thu Jan 01 01:00:00 CET 1970 ID: 2         Name: null         Primary: false         Action: 0         LastUpdate: Thu Jan 01 01:00:00 CET 1970 GTMCL5358I: Processing completed successfully. retCode: 0

The second run of the script with associated –v flag will help you verify if value was set ok.

Now it’s time to stop the toolkit without issues:

[netcool@tbsm61 bin]$ ./tbsmrdr_stop.sh GTMCL5443I: The script toolkit_stop.xml has been submitted for processing. retCode: 0

And this is it.

TBSM 6.1.1 GA

Tivoli Business Service Manager 6.1.1 was released 14.03.2013.

Just look how many new features were added:
http://pic.dhe.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=%2Fcom.ibm.tivoli.itbsm.doc_4.2.1%2Fscenarios%2Fbsms_dlkc_field_valuepairs.html

I'll make bit more of overview soon. 

Here you can find the download instructions:
https://www.ibm.com/developerworks/mydeveloperworks/wikis/home?lang=en#/wiki/Tivoli%20Business%20Service%20Manager1/page/TBSM%206.1.x%20Milestone%206

Enjoy!