IBM and Watson

Interesting read about IBM and Watson

http://www.computerweekly.com/feature/For-IBM-the-answer-is-Watson-But-what-is-the-question?utm_content=recipe3&utm_medium=EM&asrc=EM_ERU_65309723&utm_campaign=20160929_ERU%20Transmission%20for%2009/29/2016%20(UserUniverse:%202195049)_myka-reports@techtarget.com&utm_source=ERU&src=5560425

What the Top 9 Data Visualization Skills Pay

http://www.information-management.com/gallery/what-the-top-9-data-visualization-skills-pay-10029793-1.html?utm_medium=email&ET=informationmgmt:e7628532:1822140a:&utm_source=newsletter&utm_campaign=daily-sep%2015%202016&st=email&eid=83b1ad123aa52bc4d96f3cc34765ab31

Items for your Data Governance team

In order to maintain a healthy data governance team, they need to tackle the tough questions pertaining to your data security, data quality and protocols.    Lets look at some of the questions to address.

1.  How much reporting if any will happen on your source system?  Remember once you give access to the source system, it will be difficult to pry people away from that data because its "real time" 

2.  Will you allow SQL queries on your source system or just canned reports?  What about your data warehouse or ODS?  SQL queries or canned reports?

3. That begs another question.   How often will you update the warehouse with source system data.  IE. How current is your warehouse data?  Noon and close of business?  Just close of business?  

4.  When will you allow people to export data from your data warehouse?  Not times of day but will you allow your data to be exported to another system?   Remember once your data leaves your warehouse, you have just opened up another can of worms.   Your data warehouse team has just become the middle man in the equation and now you are fielding questions from a third party.  Everything from "I didn't receive all of my data" to "customer 999 is not in the data feed."

5.  How will my organization communicate changes to source system data?  For example, if I add a field to the source system, when will the data warehouse team find out about that change?  If that happens, how will it affect other system downstream from the data warehouse/ODS.?

6.  Which tools will you allow to hit the data warehouse?    Will they have table access using Tableau or will a team build Cognos/Microstrategy reports and feed them to the business to consume.

I hope that these questions have stimulated you to think about your data and placing a framework around that data to ensure data quality and integrity.

Interesting stuff from Synchrony Financial.

https://www.synchronyfinancial.com/Synchrony_Financial_BigData_IT_and_Marketing_Analytics_white_paper.pdf

We have an interesting culture at my present engagement.  The business comes to IT and wants them to develop their KPI's.    I have seen this in the past but never to this extent.   Industry wise, I think we are in this culture now of Business involvement all the way through the development process, IE Agile/Scrum.   Lets involve the business through our sprints and they set the direction of the work from IT.  Not so in this case.

So let me take this time to rant.  The business and IT must be joined together.   Collaboration with the business is key and of the utmost importance.  When I started this blog, I talked about the importance of the Business Analyst.  That person who stands between the IT and the business in order to help both sides work in harmony.   The problem is that for whatever reason, the business sometimes doesn't want to pay for that role.  Then they end up with problems. Let me say that this role can be picked up by others but the BA acts like the SME of the group.  Someone who can direct the business down the proper path for executing the best results.  If the business understands the work load, then they can help with prioritization, necessary approvals, business goals and knowledge of where they need to go as a company/organization.  

How to become a Big Data Superhero - Capgemini

How to become a Big Data Superhero - Capgemini

Informatica Marketing Data lake

https://www.informatica.com/content/dam/informatica-com/global/amer/us/collateral/analyst-report/ema-revolutionary-approaches-to-marketing-informatica-marketing-data-lake_analyst-report_3192en.pdf

Where there is no vision, the IT organization will perish

Does your organization know where it is going?   Too often we in the IT world are so busy putting out fires that we forget that we have to plan.   The inevitable Proactive vs Reactive environment.   So in order to be proactive and make plans, you need to know where you are going.   What is the plan for upgrading your infrastructure?  What is the plan for your ERP?  What is the plan for reporting?   What is the plan for security?   Do you have business units on multiple technology platforms that might need to be consolidated.   These are the kind of questions that need to be answered when looking at your plan.   That plan is called a vision statement.  What is the vision of your IT organization.  I have attached a link for a vision statement below.   If you don't have a plan for where you are going, then you might wake up one day to realize that you haven't moved or that you have moved and you are now walking in a circle.  Your upgrade path has only succeeded in giving you a larger circle.   Start with the basics and then build from there.



What Is a Vision Statement?

Microstrategy Work Fencing

Data solutions that make cents

I'm sitting in a meeting with my current client and they are talking about reporting from Peoplesoft.   This is a story that repeats itself too often in the life of a company.   There was talk about tweaking the queries, hitting a snapshot, or limiting users access to the queries.  Lets let them report from the QA environment.   Is there a true solution in any of these "Solutions?"   I would call this client a mid sized company.  

Regardless of the size of your IT department, there could be a number of reasons why you don't make the right decisions concerning reporting.   You might have a "Yes Man" in the architect position of your company.   The architect might not feel, they have the "right" to question what they know the leadership wants to do.  One other reason is they might not be aware of the decisions being made in all areas of the company.  This architect might be pulled in too many directions and can not focus on the true problems.   I have worked for clients in the past that are supporting an application for every person in IT.   You may not have the resources to bring an architect in to your office.   Is your IT department "Silo'd" where every group is solely focused on their one area of concentration?   

Lets look at some true BI solutions to this issue.   Feed Peoplesoft into your ODS or Operational Data Store.   Setup a dashboard for general reporting and allow your power users to use a tool like Tableau to tap into advanced queries.  Your ODS is the correct place for this reporting.  Your DBA can monitor the performance of the database and indicate any bottlenecks or issues with queries running long, capping of CPU's or lack of space.   When you add infrastructure to this environment, it lifts up everyone.   More memory on your ODS benefits all reporting and not just HR reporting.  What a novel concept for the business to have a one stop shop for all reporting needs.    This is a solution with teeth.  This industry standard solution adds value to business and technology. 

Stop spending money on tweaking one off reports.  Stop telling the business to limit reporting.  Stop fixing the symptom and start fixing the problem with an Enterprise solution.

Analytics that makes sense

There is a huge difference between creating analytics for the sake of analytics and using analytics to not just improve but transform your customer service.   One of my clients in the past used a data warehouse to query the calls to their call center.   The calls to the call center were measured and used to find out where are my gaps in the call center.  Leaders could drill down to their teams to compare call center to call center, manager to manager or agent to agent.   Call center leadership would compare call centers to identify gaps.  Then managers would use the tool to measure call time and use those metrics to train agents and improve the customer experience.  We had a campaign at that client known as "The Customer Rules".

1. TAKE OWNERSHIP AND SHOW WE CARE

   We value our customers, and we let them know it by all we do.
   • We are friendly and courteous to all customers and treat them with respect.
   • We take personal responsibility for meeting each customer’s needs and demonstrate to our customers that we value them.

2. BE RESPONSIVE AND DELIVER

   We listen to customers and deliver with speed.
   • We are attentive to our customers and recognize that their time is precious.
   • We make every effort to resolve issues within the customer’s timeline. If we can’t, we explain why and set more reasonable deadlines.
   • Whenever appropriate, we provide our customers with updates on resolving their issues.
   • If a customer truly deserves consideration outside standard procedures, we evaluate the circumstance, exercise good judgment, and, if necessary, proactively escalate in order to respond and deliver.

3. DO IT RIGHT

   We deliver with quality the first time, every time.
   • We resolve customer issues the right way.
   • We make every effort to resolve issues the first time.
   • We let our managers know if there are obstacles that prevent prompt and correct issue resolution.
   • We make sure we have the right training and tools to do the job right. If we don’t, we let our managers know.

4. MAKE IT SEAMLESS

   AT&T has many parts, but to our customers we are one team.
   • We are committed to working together to be the only communications and entertainment company our customers will ever want.
   • We understand our policies and processes, and use them consistently.
   • We resolve our customers’ issues or work proactively across appropriate departments to get their issues addressed.
   • We are a team. So we share information with co-workers and other departments.

5. MEET OUR COMMITMENTS

   We communicate, follow through and work hard to keep our promises.
• We inform and educate our customers about our products, services and fees.
• We make every effort to exceed our customers’ expectations.
• When we make our commitments to our customers, we keep them.
• If a commitment is missed, we remain accountable and form a new shared commitment with our customer.
• We proactively inform a customer if a commitment cannot be met and provide both status and an explanation.

Now its one matter to start a campaign around improving customer satisfaction.   But its another matter entirely to put metrics around those commitments that you can truly measure.   This client truly created a program with teeth.   That's analytics that make sense.

Essential steps for the integrated Data Warehouse

Kimball Group White Paper

Source Systems - How to win friends and influence people

Before companies with new BI implementations put that data warehouse into production they should understand that this might change your IT culture.  For example, any changes in the upstream systems need to notify the BI team that those changes are on the radar.   That's really what data governance is all about.   Lets dig into this point a little bit.  What changes am I talking about?   Adding a field to the source system, Changing a field from numeric to alpha numeric., adding a flag.  The other thing that might happen is a fields meaning to change.   Lets say that you assume that the first character means A condition, and the change might be to add a field that changes that A condition.   You may not find out about this condition for days or weeks.

When you think about data governance and ensuring that data quality is high, you must be in lock step with your source systems. Ideally, I think it makes sense for your source systems and data warehouse team to be under the same leadership.   These teams should be communicating on what changes are being made and to collaborate on testing to ensure that the changes in the upstream source system are also being accounted for in the data warehouse tables.  (When I say data warehouse, you could have an ODS).  I worked at one client and the source system changes were made and the data warehouse team would only discover the change the next day.  

Lets take this one step further.  Your source system could nor only effect your data warehouse but your data warehouse could have a feed to another system.   Data warehouse groups have the information and the tools to move data from system A to system B.  This data might come from multiple sources.  So if the source system data changes, then the subsequent downstream data may change as well.  

A proper CCB or Change Control Board where every group in IT would attend and have an opportunity to understand all changes would have been necessary as a stop gap measure.  The CCB should have an opportunity to stop any change that might break anything downstream.   Again, your primary source system should have a test bed that might be shared by the data warehouse team.  Everything needs to be tested.   Lets add to this CCB an annual audit by the data warehouse team to understand what your ETL rules are and what they need to be changed to in order to meet the needs of the business.  Over time these rules might change.   This would be a good task for your data governance team.

The Presidential polls and what the analytics really say.

Be careful in your business that you don't fall into this trap. Let the numbers speak. Sales might want to skew numbers in their favor to make the next big sale but if you are trying to make business decision report what the numbers say. The business needs the opportunity to understand the market and make course corrections based on the data so just because you can report the positives doesn't mean that you should. 

 Look at the following article. Notice what the article says about Clinton leads in four swing states. Interesting how the article also throws in that Trump is leading in 4 battleground states. If you look at the electoral college, you would know that FL and PN have higher electoral college votes than states like NC. So Trump's four states are a bigger deal than Hillary's 4 states. However, this article focuses on Hillary's wins. Why? Because we can always see the glass as half empty or half full. The interpretation of your analytics can have more impact on the story than the numbers themselves. 

Hillary Clinton leads in four swing states including NC, according to poll


POSTED 3:52 PM, JULY 15, 2016, BY CNN WIRE





Presidential candidates Donald Trump and Hillary Clinton. (Nigel Parry for CNN)


Hillary Clinton leads Donald Trump in four crucial swing states, according to a new poll out Friday, good news for the Clinton campaign that has seen other surveys show the presidential race tightening in recent weeks.


The NBC News/Wall Street Journal/Marist poll finds Clinton with high single-digit leads in Colorado, Florida, North Carolina and Virginia. Clinton maintains these leads, though at slightly smaller margins, when third-party candidates Gary Johnson and Jill Stein are included.


Clinton leads Trump in Colorado 43% to 35%. Johnson performs best in this state — he garners 12% support when included in the poll, which shrinks Clinton and Trump’s support to 39% and 33%, respectively.


In Florida, Clinton also paces Trump, 44% to 37%. Clinton’s lead slips to 5 points with third party candidates factored in.


Clinton is ahead of Trump in North Carolina 44% to 38%, a state which President Barack Obama won for Democrats in 2008 for the first time since 1976. It reverted red in 2012, but polling has shown Clinton competitive there, drawing significant support from minority communities.


Finally, Clinton leads Trump comfortably in Virginia, 44% to 35%. Johnson performs well here too — Clinton leads Trump 41% to 34% when he’s included, and he draws 10%.


A Quinnipiac poll of other battleground states released earlier this week — Florida, Pennsylvania, Ohio, and Iowa — showed closer races, with Trump leading or tied with Clinton. In Florida, Quinnipiac found Trump up 3 percentage points to Clinton, 42%-39%. Both surveys found about one in five voters in Florida are undecided or don’t support either Trump or Clinton.


The WSJ/NBC/Marist findings in Virginia and Colorado findings are in line with other recent polls in those states.


In the WSJ/NBC Marist surveys, both candidates are slowed by steadfastly high unfavorable ratings, with significant majorities in all four states saying they have negative views of Clinton and Trump. And in each state, more than one in 10 say they won’t vote for either, or prefer a third party candidate.


The poll also suggests that Trump still faces difficulty unifying the Republican Party after the contentious primary campaign, as he gets no more than 79% of the Republican vote in all four swing states polled.


Additionally, the poll found that gender and educational divides continue to shape the 2016 race, with lower-educated male voters favoring Trump, and more highly educated female voters favoring Clinton.


The WSJ/NBC/Marist poll was conducted from July 5 through 10, and surveyed 871 registered voters in Florida; 907 in North Carolina; 876 in Virginia; and 794 in Colorado. The margin of error is plus or minus 3.3 points in Florida, North Carolina and Virginia, and 3.5 points for Colorado. TRADEMARK AND COPYRIGHT 2016 CABLE NEWS NETWORK, INC., A TIME WARNER COMPANY. ALL RIGHTS RESERVED.

The real reason behind Pokemon Go

This is truly brilliant.  A game that is a marketing tool.  Have you seen the story on gamification where the group is putting piano keys on steps to drive people to walk on the stairs instead of using the escalator.   From that we have  moved into gamification of actions that lead to sales.  So here's the steps:  Find the customer, Make the customer go to your business, then sell to the customer.  Brilliant marketting.   Love to see the analytics on this.

Check out the following article about Pokemon Go driving sales.

http://www.inc.com/walter-chen/pok-mon-go-is-driving-insane-amounts-of-sales-at-small-local-businesses-here-s-h.html

Joe

Oh how the mighty have fallen

I think that Analytics can be useful but we really have to look at what makes an impact to the business.  Analytics for the sake of analytics is mute.  But true business driven analytics that address the needs of not only an organization but an entire company can transform your company to be an industry leader.  I'm reminded of a movie with Robert Redford called "Sneakers".  My favorite line of all time is, "Its not about the money Marty, its who controls the information".  So true.  So So true.   We need to understand that we are in a data driven society.  When Coca Cola is spending millions on social media marketing, we know that social media is not about 14 year olds, yacking about Taylor Swift but its about a completely new market.  A completely new way to reach the next generation.  If you are not prepared for that market now, you are already behind.   I currently work for an auto parts company and some of the execs went to a seminar by Amazon where he said they would sell 4 Billion in Auto Parts.   Amazon people.   That's auto parts.  Not books, applications, or emoji's.  

Think about your data. Attain the data.   Read the data, Make decisions on your data.  Because if you don't, your competitors will and then your the next EDS.  The next IBM.   Did they have bad products or services?   No they had their blinders on and then the market passed them by and the realized their new name was Titanic.

To Purge or not to Purge

There comes a time in every data warehouse team tenure where you have to ascertain if you truly need to keep data from 1993.   Data Governance would dictate that you have a policy surrounding your data retention.  I worked for a client that had data "back to Noah" per the General Manager and the way the data warehouse code was written to calculate subscribers.  A database needs to be maintained or you experience a lot of problems.   But back to the data purging issue, let’s look at some reasons why you want and don't want to purge data.   


What is Purging
Purging is the process of freeing up space in the database or of deleting obsoletedata that is not required by the system. The purge process can be based on the age of the data or the type of data. Archiving is the process of backing up the obsolete data that will be deleted during the purge process

Reasons not to purge:
1. Disk space is cheap, why not keep the data
2. You don't know if you will ever "Need" the data
3. What if we are audited

Reason to purge the data
1. Improve performance - Why would you want to search through  1 Million records when you could search through 500k records - Query times, 
2. Legal reason -PII requires you to setup a purging policy.
3. Data beyond X number of years is irrelevant.
4. Costs of maintaining data - physical disks, space, bloated databases

Sample Purging policy

• Every day, purge data older than 45 days from the current data set, moving it to the history data set.
• The first day of every month, purge data older than 13 months from the history data set, moving it to the dormant data set.
• Every day, purge data older than eight years from the dormant data set


Even if your organization utilizes an ODS where you have current information for X years and then an EDW where you have historical data, the EDW would still need to have some sort of purging policy or you will experience problems down the road in database performance or lack thereof.   Working with your data governance team to review issues with data, setting data standards and retention policy can only be good for your organization.   Sometimes your organization has to champion the data governance for your organization in order to show the business that data is important to your organization but maintaining that data is important to both IT and the business.

The hidden costs of your warehouse solution

So you are still looking at building your first or next analytics solutions and you have 3-4 vendors arrive onsite to showcase their wares.   You narrow down that list and you really like the Pentaho demonstration because and I quote from the Pentaho page,

Any Analytics, Any Data, Simplified

Pentaho addresses the barriers that block your organization's ability to get value from all your data. Our platform simplifies preparing and blending any data and includes a spectrum of tools to easily analyze, visualize, explore, report and predict. Open, embeddable and extensible, Pentaho is architected to ensure that each member of your team — from developers to business users — can easily translate data into value.

That sounds great. Removing barriers.  The vendor can implement this data warhouse and it will be wonderful.   Then you have to look back at your situation.   You have 20 people in your IT shop from BI to Helpdesk, etc.. you get the picture...   And you have 7 reporting platforms including this new Pentaho platform.   No one in your organization knows anything about Pentaho so you need to send them to training.   Ohh and then you have to bring in a contractor to fill in the gap and help train your people, right?   Wait there are only 3 on the East Coast.   Ahh this raises issues.  All of a sudden the slick software doesn't seem so slick any more.  Suddenly you realize that although the software might remove barriers, the software company might raise a couple of barriers.  

Examine all of your hidden costs in a situation.  In no way, do I think that Pentaho is terrible, I just think that IT organizations need to think smarter not harder.   Small organizations need streamlined solutions that meet the business needs and sometimes, you need to tell your vendors to plug into your data warehouse instead of you plugging into theirs. 

Don't be so closed minded.

Industry changes.  Tools change and they become better.  Competitors add enhancements and this forces the rest of the industry to adapt.    One of the platforms that have worked to continuously improve is Microsoft SQL Server.   If you look over the years at the roadmap for this tool, you will see a robust platform that prior to 2008, you would have probably not appreciated.   Microsoft is now steering the industry.   Check out Microsoft BI, integration with SharePoint, etc.   But this is not a blog about Microsoft, it’s a blog about mindsets.  

My boss here at my current engagement is a strong advocate for the AS400.   The industry has walked around the AS400 and IBM.  We have moved to other platforms.  Right? I mean the AS400 is old school, right.   That's state of the art 1988 tech, right?  But it turns out that the AS400 is a stable platform that still exist in many businesses today.  They continue to refine and get better and now they have the I Series.  Great failover, great stack of redundancy, security.  I know what you're thinking.   It's not cool, it’s not new.   What is your goal?  Is it new tech or stable tech?  But this is not a blog about the AS400 either. 

Don't pass over the tools.   Tableau is for data mining, correct?   Qlik is for dashboards, correct?   Cognos is for reporting, correct?   I think if you look at Tableau, you will see that they have enhanced that tool to collaborate.  To share online.  Publish a report online with the click of a button.  

Back in 2005, MS SharePoint was a collaboration tool primarily used as a document repository.   But behind the scenes, you can create portals and web pages that envy others.   Think about this.   Stop looking at how you used the tool at your previous employer and start to think about how this tool can be used today by my current employer.   Look at price.  Use your analysis experience to look at tools in different ways.  Find out if your bias are warranted or not?  Be open to new ways of doing things.  Tunnel vision is limiting to you and your company.