We have an interesting culture at my present engagement.  The business comes to IT and wants them to develop their KPI's.    I have seen this in the past but never to this extent.   Industry wise, I think we are in this culture now of Business involvement all the way through the development process, IE Agile/Scrum.   Lets involve the business through our sprints and they set the direction of the work from IT.  Not so in this case.

So let me take this time to rant.  The business and IT must be joined together.   Collaboration with the business is key and of the utmost importance.  When I started this blog, I talked about the importance of the Business Analyst.  That person who stands between the IT and the business in order to help both sides work in harmony.   The problem is that for whatever reason, the business sometimes doesn't want to pay for that role.  Then they end up with problems. Let me say that this role can be picked up by others but the BA acts like the SME of the group.  Someone who can direct the business down the proper path for executing the best results.  If the business understands the work load, then they can help with prioritization, necessary approvals, business goals and knowledge of where they need to go as a company/organization.  

Source Systems - How to win friends and influence people

Before companies with new BI implementations put that data warehouse into production they should understand that this might change your IT culture.  For example, any changes in the upstream systems need to notify the BI team that those changes are on the radar.   That's really what data governance is all about.   Lets dig into this point a little bit.  What changes am I talking about?   Adding a field to the source system, Changing a field from numeric to alpha numeric., adding a flag.  The other thing that might happen is a fields meaning to change.   Lets say that you assume that the first character means A condition, and the change might be to add a field that changes that A condition.   You may not find out about this condition for days or weeks.

When you think about data governance and ensuring that data quality is high, you must be in lock step with your source systems. Ideally, I think it makes sense for your source systems and data warehouse team to be under the same leadership.   These teams should be communicating on what changes are being made and to collaborate on testing to ensure that the changes in the upstream source system are also being accounted for in the data warehouse tables.  (When I say data warehouse, you could have an ODS).  I worked at one client and the source system changes were made and the data warehouse team would only discover the change the next day.  

Lets take this one step further.  Your source system could nor only effect your data warehouse but your data warehouse could have a feed to another system.   Data warehouse groups have the information and the tools to move data from system A to system B.  This data might come from multiple sources.  So if the source system data changes, then the subsequent downstream data may change as well.  

A proper CCB or Change Control Board where every group in IT would attend and have an opportunity to understand all changes would have been necessary as a stop gap measure.  The CCB should have an opportunity to stop any change that might break anything downstream.   Again, your primary source system should have a test bed that might be shared by the data warehouse team.  Everything needs to be tested.   Lets add to this CCB an annual audit by the data warehouse team to understand what your ETL rules are and what they need to be changed to in order to meet the needs of the business.  Over time these rules might change.   This would be a good task for your data governance team.

The Presidential polls and what the analytics really say.

Be careful in your business that you don't fall into this trap. Let the numbers speak. Sales might want to skew numbers in their favor to make the next big sale but if you are trying to make business decision report what the numbers say. The business needs the opportunity to understand the market and make course corrections based on the data so just because you can report the positives doesn't mean that you should. 

 Look at the following article. Notice what the article says about Clinton leads in four swing states. Interesting how the article also throws in that Trump is leading in 4 battleground states. If you look at the electoral college, you would know that FL and PN have higher electoral college votes than states like NC. So Trump's four states are a bigger deal than Hillary's 4 states. However, this article focuses on Hillary's wins. Why? Because we can always see the glass as half empty or half full. The interpretation of your analytics can have more impact on the story than the numbers themselves. 

Hillary Clinton leads in four swing states including NC, according to poll


POSTED 3:52 PM, JULY 15, 2016, BY CNN WIRE





Presidential candidates Donald Trump and Hillary Clinton. (Nigel Parry for CNN)


Hillary Clinton leads Donald Trump in four crucial swing states, according to a new poll out Friday, good news for the Clinton campaign that has seen other surveys show the presidential race tightening in recent weeks.


The NBC News/Wall Street Journal/Marist poll finds Clinton with high single-digit leads in Colorado, Florida, North Carolina and Virginia. Clinton maintains these leads, though at slightly smaller margins, when third-party candidates Gary Johnson and Jill Stein are included.


Clinton leads Trump in Colorado 43% to 35%. Johnson performs best in this state — he garners 12% support when included in the poll, which shrinks Clinton and Trump’s support to 39% and 33%, respectively.


In Florida, Clinton also paces Trump, 44% to 37%. Clinton’s lead slips to 5 points with third party candidates factored in.


Clinton is ahead of Trump in North Carolina 44% to 38%, a state which President Barack Obama won for Democrats in 2008 for the first time since 1976. It reverted red in 2012, but polling has shown Clinton competitive there, drawing significant support from minority communities.


Finally, Clinton leads Trump comfortably in Virginia, 44% to 35%. Johnson performs well here too — Clinton leads Trump 41% to 34% when he’s included, and he draws 10%.


A Quinnipiac poll of other battleground states released earlier this week — Florida, Pennsylvania, Ohio, and Iowa — showed closer races, with Trump leading or tied with Clinton. In Florida, Quinnipiac found Trump up 3 percentage points to Clinton, 42%-39%. Both surveys found about one in five voters in Florida are undecided or don’t support either Trump or Clinton.


The WSJ/NBC/Marist findings in Virginia and Colorado findings are in line with other recent polls in those states.


In the WSJ/NBC Marist surveys, both candidates are slowed by steadfastly high unfavorable ratings, with significant majorities in all four states saying they have negative views of Clinton and Trump. And in each state, more than one in 10 say they won’t vote for either, or prefer a third party candidate.


The poll also suggests that Trump still faces difficulty unifying the Republican Party after the contentious primary campaign, as he gets no more than 79% of the Republican vote in all four swing states polled.


Additionally, the poll found that gender and educational divides continue to shape the 2016 race, with lower-educated male voters favoring Trump, and more highly educated female voters favoring Clinton.


The WSJ/NBC/Marist poll was conducted from July 5 through 10, and surveyed 871 registered voters in Florida; 907 in North Carolina; 876 in Virginia; and 794 in Colorado. The margin of error is plus or minus 3.3 points in Florida, North Carolina and Virginia, and 3.5 points for Colorado. TRADEMARK AND COPYRIGHT 2016 CABLE NEWS NETWORK, INC., A TIME WARNER COMPANY. ALL RIGHTS RESERVED.

Big Data terms

Analytics and Big Data Glossary

Last updated: 9/24/14

ACID test

A test applied to data for atomicity, consistency, isolation, and durability.

ad hoc reporting

Reports generated for a one-time need.

ad targeting

The attempt to reach a specific audience with a specific message, typically by either contacting them directly or placing contextual ads on the Web.

algorithm

A mathematical formula placed in software that performs an analysis on a set of data.

analytics

Using software-based algorithms and statistics to derive meaning from data.

analytics platform

Software or software and hardware that provides the tools and computational power needed to build and perform many different analytical queries.

anonymization

The severing of links between people in a database and their records to prevent the discovery of the source of the records.

application

Software that is designed to perform a specific task or suite of tasks.

automatic identification and capture (AIDC)

Any method of automatically identifying and collecting data on items, and then storing the data in a computer system. For example, a scanner might collect data about a product being shipped via an RFID chip.

behavioral analytics

Using data about people’s behavior to understand intent and predict future actions.

big data

This term has been defined in many ways, but along similar lines. Doug Laney, then an analyst at the META Group, first defined big data in a 2001 report called “3-D Data Management: Controlling Data Volume, Velocity and Variety.” Volume refers to the sheer size of the datasets. The McKinsey report, “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” expands on the volume aspect by saying that, “’Big data’ refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.”

Velocity refers to the speed at which the data is acquired and used. Not only are companies and organizations collecting more and more data at a faster rate, they want to derive meaning from that data as soon as possible, often in real time.

Variety refers to the different types of data that are available to collect and analyze in addition to the structured data found in a typical database. Barry Devlin of 9sight Consulting identifies four categories of information that constitute big data:

1.    Machine-generated data. This includes RFID data, geolocation data from mobile devices, and data from monitoring devices such as utility meters.

2.    Computer log data, such as clickstreams from websites.

3.    Textual social media information from sources such as Twitter and Facebook.

4.    Multimedia social and other information from Flickr, YouTube, and other similar sites.

IDC analyst Benjamin Woo has added a fourth V to the definition: value. He says that because big data is about supporting decisions, you need the ability to act on the data and derive value.

biometrics

The use of technology to identify people by one or more of their physical traits.

brand monitoring

The act of monitoring your brand’s reputation online, typically by using software to automate the process.

business intelligence (BI)

The general term used for the identification, extraction, and analysis of data.

call detail record (CDR) analysis

CDRs contain data that a telecommunications company collects about phone calls, such as time and length of call. This data can be used in any number of analytical applications.

Cassandra

A popular choice of columnar database for use in big data applications. It is an open source database managed by The Apache Software Foundation.

cell phone data

Cell phones generate a tremendous amount of data, and much of it is available for use with analytical applications.

clickstream analytics

The analysis of users’ Web activity through the items they click on a page.

Clojure

Clojure is a dynamic programming language based on LISP that uses the Java Virtual Machine (JVM). It is well suited for parallel data processing.

cloud

A broad term that refers to any Internet-based application or service that is hosted remotely.

columnar database or column-oriented database

A database that stores data by column rather than by row. In a row-based database, a row might contain a name, address, and phone number. In a column-oriented database,  all names are in one column, addresses in another, and so on. A key advantage of a columnar database is faster hard disk access.

competitive monitoring

Keeping tabs of competitors’ activities on the Web using software to automate the process.

complex event processing (CEP)

CEP is the process of monitoring and analyzing all events across an organization’s systems and acting on them when necessary in real time.

Comprehensive Large Array-data Stewardship System (CLASS)

A digital library of historical environmental data from satellites operated by the U.S. National Oceanic and Atmospheric Association (NOAA).

computer-generated data

Any data generated by a computer rather than a human–a log file for example.

concurrency

The ability to execute multiple processes at the same time.

confabulation

The act of making an intuition-based decision appear to be data-based.

content management system (CMS)

Software that facilitates the management and publication of content on the Web.

cross-channel analytics

Analysis that can attribute sales, show average order value, or the lifetime value.

crowdsourcing

The act of submitting a task or problem to the public for completion or solution.

customer relationship management (CRM)

Software that helps businesses manage sales and customer service processes.

dashboard

A graphical reporting of static or real-time data on a desktop or mobile device. The data represented is typically high-level to give managers a quick report on status or performance.

data

A quantitative or qualitative value. Common types of data include sales figures, marketing research results, readings from monitoring equipment, user actions on a website, market growth projections, demographic information, and customer lists.

data access

The act or method of viewing or retrieving stored data.

data aggregation

The act of collecting data from multiple sources for the purpose of reporting or analysis.

data analytics

The application of software to derive information or meaning from data. The end result might be a report, an indication of status, or an action taken automatically based on the information received.

data analyst

A person responsible for the tasks of modeling, preparing, and cleaning data for the purpose of deriving actionable information from it.

data architecture and design

How enterprise data is structured. The actual structure or design varies depending on the eventual end result required. Data architecture has three stages or processes: conceptual representation of business entities. the logical representation of the relationships among those entities, and the physical construction of the system to support the functionality.

database

A digital collection of data and the structure around which the data is organized. The data is typically entered into and accessed via a database management system (DBMS).

database administrator (DBA)

A person, often certified, who is responsible for supporting and maintaining the integrity of the structure and content of a database.

database as a service (DaaS)

A database hosted in the cloud and sold on a metered basis. Examples include Heroku Postgres and Amazon Relational Database Service.

database management system (DBMS)

Software that collects and provides access to data in a structured format.

data center

A physical facility that houses a large number of servers and data storage devices. Data centers might belong to a single organization or sell their services to many organizations.

data cleansing

The act of reviewing and revising data to remove duplicate entries, correct misspellings, add missing data, and provide more consistency.

data collection

Any process that captures any type of data.

data custodian

A person responsible for the database structure and the technical environment, including the storage of data.

data-directed decision making

Using data to support making crucial decisions.

data exhaust

The data that a person creates as a byproduct of a common activity–for example, a cell call log or web search history.

data feed

A means for a person to receive a stream of data. Examples of data feed mechanisms include RSS or Twitter.

data governance

A set of processes or rules that ensure the integrity of the data and that data management best practices are met.

data integration

The process of combining data from different sources and presenting it in a single view.

data integrity

The measure of trust an organization has in the accuracy, completeness, timeliness, and validity of the data.

data management

According to the Data Management Association, data management incorporates the following practices needed to manage the full data lifecycle in an enterprise:

• data governance
• data architecture, analysis, and design
• database management
• data security management
• data quality management
• reference and master data management
• data warehousing and business intelligence management
• document, record, and content management
• metadata management
• contact data management

Data Management Association (DAMA)

A non-profit international organization for technical and business professionals “dedicated to advancing the concepts and practices of information and data management.”

data marketplace

A place where people can buy and sell data online.

data mart

The access layer of a data warehouse used to provide data to users.

data migration

The process of moving data between different storage types or formats, or between different computer systems.

data mining

The process of deriving patterns or knowledge from large data sets.

data model, data modeling

A data model defines the structure of the data for the purpose of communicating between functional and technical people to show data needed for business processes, or for communicating a plan to develop how data is stored and accessed among application development team members.

data point

An individual item on a graph or a chart.

data profiling

The process of collecting statistics and information about data in an existing source.

data quality

The measure of data to determine its worthiness for decision making, planning, or operations.

data replication

The process of sharing information to ensure consistency between redundant sources.

data repository

The location of permanently stored data.

data science

A recent term that has multiple definitions, but generally accepted as a discipline that incorporates statistics, data visualization, computer programming, data mining, machine learning, and database engineering to solve complex problems.

data scientist

A practitioner of data science.

data security

The practice of protecting data from destruction or unauthorized access.

data set

A collection of data, typically in tabular form.

data source

Any provider of data–for example, a database or a data stream.

data steward

A person responsible for data stored in a data field.

data structure

A specific way of storing and organizing data.

data visualization

A visual abstraction of data designed for the purpose of deriving meaning or communicating information more effectively.

data virtualization

The process of abstracting different data sources through a single data access layer.

data warehouse

A place to store data for the purpose of reporting and analysis.

de-identification

The act of removing all data that links a person to a particular piece of information.

demographic data

Data relating to the characteristics of a human population.

Deep Thunder

IBM’s weather prediction service that provides weather data to organizations such as utilities, which use the data to optimize energy distribution.

distributed cache

A data cache that is spread across multiple systems but works as one. It is used to improve performance.

distributed object

A software module designed to work with other distributed objects stored on other computers.

distributed processing

The execution of a process across multiple computers connected by a computer network.

document management

The practice of tracking and storing electronic documents and scanned images of paper documents.

Drill

An open source distributed system for performing interactive analysis on large-scale datasets. It is similar to Google’s Dremel, and is managed by Apache.

elasticsearch

An open source search engine built on Apache Lucene.

electronic health records (EHR)

A digitized health record meant to be usable across different health care settings.

enterprise resource planning (ERP)

A software system that allows an organization to coordinate and manage all its resources, information, and business functions.

event analytics

Shows the series of steps that led to an action.

exabyte

One million terabytes, or 1 billion gigabytes of information.

external data

Data that exists outside of a system.

extract, transform, and load (ETL)

A process used in data warehousing to prepare data for use in reporting or analytics.

failover

The automatic switching to another computer or node should one fail.

Federal Information Security Management Act (FISMA)

A US federal law that requires all federal agencies to meet certain standards of information security across its systems.

grid computing

The performing of computing functions using resources from multiple distributed systems. Grid computing typically involves large files and are most often used for multiple applications. The systems that comprise a grid computing network do not have to be similar in design or in the same geographic location.

Hadoop

An open source software library project administered by the Apache Software Foundation. Apache defines Hadoop as “a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model.”

HANA

A software/hardware in-memory computing platform from SAP designed for high-volume transactions and real-time analytics.

HBase

A distributed columnar NoSQL database.

high-performance computing (HPC)

HPC systems, also called supercomputers, are often custom built from state-of-the-art technology to maximize compute performance, storage capacity and throughput, and data transfer speeds.

Hive

A SQL-like query and data warehouse engine.

in-database analytics

The integration of data analytics into the data warehouse.

information management

The practice of collecting, managing, and dsitributing information of all types–digital, paper-based, structured, unstructured.

in-memory database

Any database system that relies on memory for data storage.

in-memory data grid (IMDG)

The storage of data in memory across multiple servers for the purpose of greater scalability and faster access or analytics.

Kafka

LinkedIn’s open-source message system used to monitor activity events on the web.

latency

Any delay in a response or delivery of data from one point to another.

legacy system

Any computer system, application, or technology that is obsolete, but continues to be used because it performs a needed function adequately.

linked data

As described by World Wide Web inventor Time Berners-Lee, “Cherry-picking common attributes or languages to identify connections or relationships between disparate sources of data.”

load balancing

The process of distributing workload across a computer network or computer cluster to optimize performance.

location analytics

Location analytics brings mapping and map-driven analytics to enterprise business systems and data warehouses. It allows you to associate geospatial information with datasets.

location data

Data that describes a geographic location.

log file

A file that a computer, network, or application creates automatically to record events that occur during operation–for example, the time a file is accessed.

long data

A term coined by mathematician and network scientist Samuel Arbesman that refers to “datasets that have massive historical sweep.”

machine-generated data

Any data that is automatically created from a computer process, application, or other non-human source.

machine learning

The use of algorithms to allow a computer to analyze data for the purpose of “learning” what action to take when a specific pattern or event occurs.

map/reduce

A general term that refers to the process of breaking up a problem into pieces that are then distributed across multiple computers on the same network or cluster, or across a grid of disparate and possibly geographically separated systems (map), and then collecting all the results and combines them into a report (reduce). Google’s branded framework to perform this function is called MapReduce.

mashup

The process of combining different datasets within a single application to enhance output–for example, combining demographic data with real estate listings.

massively parallel processing (MPP)

The act of processing of a program by breaking it up into separate pieces, each of which is executed on its own processor, operating system, and memory.

master data management (MDM)

Master data is any non-transactional data that is critical to the operation of a business–for example, customer or supplier data, product information, or employee data. MDM is the process of managing that data to ensure consistency, quality, and availability.

metadata

Any data used to describe other data–for example, a data file’s size or date of creation.

MongoDB

An open-source NoSQL database managed by 10gen.

MPP database

A database optimized to work in a massively parallel processing environment.

multi-threading

The act of breaking up an operation within a single computer system into multiple threads for faster execution.

NoSQL

A class of database management system that does not use the relational model. NoSQL is designed to handle large data volumes that do not follow a fixed schema. It is ideally suited for use with very large data volumes that do not require the relational model.

online analytical processing (OLAP)

The process of analyzing multidimensional data using three operations: consolidation (the aggregation of available), drill-down (the ability for users to see the underlying details), and slice and dice (the ability for users to select subsets and view them from different perspectives).

online transactional processing (OLTP)

The process of providing users with access to large amounts of transactional data in a way that they can derive meaning from it.

OpenDremel

The open source version of Google’s Big Query java code. It is being integrated with Apache Drill.

Open Data Center Alliance (ODCA)

A consortium of global IT organizations whose goal is to speed the migration of cloud computing.

operational data store (ODS)

A location to gather and store data from multiple sources so that more operations can be performed on it before sending to the data warehouse for reporting.

parallel data analysis

Breaking up an analytical problem into smaller components and running algorithms on each of those components at the same time. Parallel data analysis can occur within the same system or across multiple systems.

parallel method invocation (PMI)

Allows programming code to call multiple functions in parallel.

parallel processing

The ability to execute multiple tasks at the same time.

parallel query

A query that is executed over multiple system threads for faster performance.

pattern recognition

The classification or labeling of an identified pattern in the machine learning process.

performance management

The process of monitoring system or business performance against predefined goals to identify areas that need attention.

petabyte

One million gigabytes or 1,024 terabytes.

Pig

A data flow language and execution framework for parallel computation.

predictive analytics

Using statistical functions on one or more datasets to predict trends or future events.

predictive modeling

The process of developing a model that will most likely predict a trend or outcome.

query analysis

The process of analyzing a search query for the purpose of optimizing it for the best possible result.

R

An open source software environment used for statistical computing.

radio-frequency identification (RFID)

A technology that uses wireless communications to send information about an object from one point to another.

real time

A descriptor for events, data streams, or processes that have an action performed on them as they occur.

recommendation engine

An algorithm that analyzes a customer’s purchases and actions on an e-commerce site and then uses that data to recommend complementary products.

records management

The process of managing an organization’s records throughout their entire lifecycle, from creation to disposal.

reference data

Data that describes an object and its properties. The object may be physical or virtual.

report

The presentation of information derived from a query against a dataset, usually in a predetermined format.

risk analysis

The application of statistical methods on one or more datasets to determine the likely risk of a project, action, or decision.

root-cause analysis

The process of determining the main cause of an event or problem.

Sawzall

Google’s procedural domain-specific programming language designed to process large volumes of log records.

scalability

The ability of a system or process to maintain acceptable performance levels as workload or scope increases.

schema

The structure that defines the organization of data in a database system.

search

The process of locating specific data or content using a search tool.

search data

Aggregated data about search terms used over time.

Semantic Web

A project of the World Wide Web Consortium (W3C) to encourage the use of a standard format to include semantic content on websites. The goal is to enable computers and other devices to better process data.

semi-structured data

Data that is not structured by a formal data model, but provides other means of describing the data and hierarchies.

sentiment analysis

The application of statistical functions on comments people make on the web and through social networks to determine how they feel about a product or company.

server

A physical or virtual computer that serves requests for a software application and delivers those requests over a network.

smart grid

The smart grid refers to the concept of adding intelligence to the world’s electrical transmission systems with the goal of optimizing energy efficiency. Enabling the smart grid will rely heavily on collecting, analyzing, and acting on large volumes of data.

smart meter

An electrical meter that monitor and report energy usage and are capable of two-way communication with the utility.

solid-state drive (SSD)

Also called a solid-state disk, a device that uses memory ICs to persistently store data.

software as a service (SaaS)

Application software that is used over the web by a thin client or web browser. Salesforce is a well-known example of SaaS.

storage

Any means of storing data persistently.

Storm

An open-source distributed computation system designed for processing multiple data streams in real time.

structured data

Data that is organized by a predetermined structure.

Structured Query Language (SQL)

A programming language designed specifically to manage and retrieve data from a relational database system.

terabyte

1,000 gigabytes.

text analytics

The application of statistical, linguistic, and machine learning techniques on text-based sources to derive meaning or insight.

transactional data

Data that changes unpredictably. Examples include accounts payable and receivable data, or data about product shipments.

transparency

As more data becomes openly available, the idea of proprietary data as a competitive advantage is diminished.

unstructured data

Data that has no identifiable structure–for example, the text of email messages.

variable pricing

The practice of changing price on the fly in response to supply and demand. It requires real-time monitoring of consumption and supply.

weather data

Real-time weather data is now widely available for organizations to use in a variety of ways. For example, a logistics company can monitor local weather conditions to optimize the transport of goods. A utility company can adjust energy distribution in real time.

Whole Earth Model

An integrated data management system that allows geophysicists, engineers, and financial managers in the oil and gas industry evaluate the potential of oil and gas fields.

- See more at: http://data-informed.com/glossary-of-big-data-terms/#sthash.4ha1lRlj.dpuf