Monday, June 13, 2022
HomeBusiness IntelligenceEnterprise Intelligence Elements and How They Relate to Energy BI

Enterprise Intelligence Elements and How They Relate to Energy BI

Business Intelligence Components and How They Relate to Power BI

Once I determined to jot down this weblog submit, I assumed it could be a good suggestion to be taught a bit concerning the historical past of Enterprise Intelligence. I searched on the web, and I discovered this web page on Wikipedia. The time period Enterprise Intelligence as we all know it right now was coined by an IBM pc science researcher, Hans Peter Luhn, in 1958, who wrote a paper within the IBM Methods journal titled A Enterprise Intelligence System as a selected course of in knowledge science. Within the Aims and rules part of his paper, Luhn defines the enterprise as “a set of actions carried on for no matter goal, be it science, know-how, commerce, business, legislation, authorities, protection, et cetera.” and an intelligence system as “the communication facility serving the conduct of a enterprise (within the broad sense)”. Then he refers to Webster’s dictionary’s definition of the phrase Intelligence as the flexibility to apprehend the interrelationships of offered info in such a method as to information motion in direction of a desired aim”.

It’s fascinating to see how a implausible concept previously units a concrete future that may assist us have a greater life. Isn’t it exactly what we do in our each day BI processes as Luhn described of a Enterprise Intelligence System for the primary time? How cool is that?

After we discuss concerning the time period BI right now, we consult with a selected and scientific set of processes of reworking the uncooked knowledge into helpful and comprehensible data for varied enterprise sectors (corresponding to gross sales, stock, legislation, and many others…). These processes will assist companies to make data-driven choices primarily based on the prevailing hidden info within the knowledge.

Like the whole lot else, the BI processes improved quite a bit throughout its life. I’ll attempt to make some wise hyperlinks between right now’s BI Elements and Energy BI on this submit.

Generic Elements of Enterprise Intelligence Options

Typically talking, a BI resolution incorporates varied parts and instruments which will range in several options relying on the enterprise necessities, knowledge tradition and the organisation’s maturity in analytics. However the processes are similar to the next:

  • We normally have a number of supply methods with totally different applied sciences containing the uncooked knowledge, corresponding to SQL Server, Excel, JSON, Parquet recordsdata and many others…
  • We combine the uncooked knowledge right into a central repository to cut back the chance of creating any interruptions to the supply methods by consistently connecting to them. We normally load the info from the info sources into the central repository.
  • We rework the info to optimise it for reporting and analytical functions, and we load it into one other storage. We purpose to maintain the historic knowledge on this storage.
  • We pre-aggregate the info into sure ranges primarily based on the enterprise necessities and cargo the info into one other storage. We normally don’t hold the entire historic knowledge on this storage; as a substitute, we solely hold the info required to be analysed or reported.
  • We create stories and dashboards to show the info into helpful data

With the above processes in thoughts, a BI resolution consists of the next parts:

  • Information Sources
  • Staging
  • Information Warehouse/Information Mart(s)
  • Extract, Remodel and Load (ETL)
  • Semantic Layer
  • Information Visualisation

Information Sources

One of many important objectives of operating a BI venture is to allow organisations to make data-driven choices. An organisation might need a number of departments utilizing varied instruments to gather the related knowledge every single day, corresponding to gross sales, stock, advertising and marketing, finance, well being and security and many others.

The info generated by the enterprise instruments are saved someplace utilizing totally different applied sciences. A gross sales system would possibly retailer the info in an Oracle database, whereas the finance system shops the info in a SQL Server database within the cloud. The finance workforce additionally generate some knowledge saved in Excel recordsdata.

The info generated by totally different methods are the supply for a BI resolution.


We normally have a number of knowledge sources contributing to the info evaluation in real-world eventualities. To have the ability to analyse all the info sources, we require a mechanism to load the info right into a central repository. The principle purpose for that’s the enterprise instruments required to consistently retailer knowledge within the underlying storage. Subsequently, frequent connections to the supply methods can put our manufacturing methods prone to being unresponsive or performing poorly. The central repository the place we retailer the info from varied knowledge sources is named Staging. We normally retailer the info within the staging with no or minor adjustments in comparison with the info within the knowledge sources. Subsequently, the standard of the info saved within the staging is normally low and requires cleaning within the subsequent phases of the info journey. In lots of BI options, we use Staging as a short lived setting, so we delete the Staging knowledge frequently after it’s efficiently transferred to the following stage, the info warehouse or knowledge marts.

If we need to point out the info high quality with colors, it’s truthful to say the info high quality in staging is Bronze.

Information Warehouse/Information Mart(s)

As talked about earlier than, the info within the staging isn’t in its greatest form and format. A number of knowledge sources disparately generate the info. So, analysing the info and creating stories on prime of the info in staging could be difficult, time-consuming and costly. So we require to seek out out the hyperlinks between the info sources, cleanse, reshape and rework the info and make it extra optimised for knowledge evaluation and reporting actions. We retailer the present and historic knowledge in a knowledge warehouse. So it’s fairly regular to have a whole bunch of tens of millions and even billions of rows of knowledge over an extended interval. Relying on the general structure, the info warehouse would possibly include encapsulated business-specific knowledge in a knowledge mart or a set of knowledge marts. In knowledge warehousing, we use totally different modelling approaches corresponding to Star Schema. As talked about earlier, one of many main functions of getting a knowledge warehouse is to maintain the historical past of the info. This can be a large profit of getting a knowledge warehouse, however this energy comes with a value. As the quantity of the info within the knowledge warehouse grows, it makes it costlier to analyse the info. The info high quality within the knowledge warehouse or knowledge marts is Silver.

Extract, Transfrom and Load (ETL)

Within the earlier sections, we talked about that we combine the info from the info sources within the staging space, then we cleanse, reshape and rework the info and cargo it into a knowledge warehouse. To take action, we observe a course of referred to as Extract, Remodel and Load or, in brief, ETL. As you possibly can think about, the ETL processes are normally fairly complicated and costly, however they’re a vital a part of each BI resolution.

Semantic Layer

As we now know, one of many strengths of getting a knowledge warehouse is to maintain the historical past of the info. However over time, preserving large quantities of historical past could make knowledge evaluation costlier. For example, we may have an issue if we need to get the sum of gross sales over 500 million rows of knowledge. So, we pre-aggregate the info into sure ranges primarily based on the enterprise necessities right into a Semantic layer to have an much more optimised and performant setting for knowledge evaluation and reporting functions. Information aggregation dramatically reduces the info quantity and improves the efficiency of the analytical resolution.

Let’s proceed with a easy instance to raised perceive how aggregating the info might help with the info quantity and knowledge processing efficiency. Think about a state of affairs the place we saved 20 years of knowledge of a sequence retail retailer with 200 shops throughout the nation, that are open 24 hours and seven days per week. We saved the info on the hour stage within the knowledge warehouse. Every retailer normally serves 500 prospects per hour a day. Every buyer normally buys 5 objects on common. So, listed below are some easy calculations to grasp the quantity of knowledge we’re coping with:

  • Common hourly data of knowledge per retailer: 5 (objects) x 500 (served cusomters per hour) = 2,500
  • Every day data per retailer: 2,500 x 24 (hours a day) = 60,000
  • Yearly data per retailer: 60,000 x 365 (days a 12 months) = 21,900,000
  • Yearly data for all shops: 21,900,000 x 200 = 4,380,000,000
  • Twenty years of knowledge: 4,380,000,000 x 20 = 87,600,000,000

A easy summation over greater than 80 billion rows of knowledge would take lengthy to be calculated. Now, think about that the enterprise requires to analyse the info on day stage. So within the semantic layer we mixture 80 billion rows into the day stage. In different phrases, 87,600,000,000 ÷ 24 = 3,650,000,000 which is a a lot smaller variety of rows to cope with.

The opposite profit of getting a semantic layer is that we normally don’t require to load the entire historical past of the info from the info warehouse into our semantic layer. Whereas we’d hold 20 years of knowledge within the knowledge warehouse, the enterprise won’t require to analyse 20 years of knowledge. Subsequently, we solely load the info for a interval required by the enterprise into the semantic layer, which reinforces the general efficiency of the analytical system.

Let’s proceed with our earlier instance. Let’s say the enterprise requires analysing the previous 5 years of knowledge. Here’s a simplistic calculation of the variety of rows after aggregating the info for the previous 5 years on the day stage: 3,650,000,000 ÷ 4 = 912,500,000.

The info high quality of the semantic layer is Gold.

Information Visualisation

Information visualisation refers to representing the info from the semantic layer with graphical diagrams and charts utilizing varied reporting or knowledge visualisation instruments. We might create analytical and interactive stories, dashboards, or low-level operational stories. However the stories run on prime of the semantic layer, which supplies us high-quality knowledge with distinctive efficiency.

How Completely different BI Elements Relate

The next diagram reveals how totally different Enterprise Intelligence parts are associated to one another:

Business Intelligence (BI) Components
Enterprise Intelligence (BI) Elements

Within the above diagram:

  • The blue arrows present the extra conventional processes and steps of a BI resolution
  • The dotted line gray(ish) arrows present extra trendy approaches the place we don’t require to create any knowledge warehouses or knowledge marts. As an alternative, we load the info instantly right into a Semantic layer, then visualise the info.
  • Relying on the enterprise, we’d must undergo the orange arrow with the dotted line when creating stories on prime of the info warehouse. Certainly, this strategy is respectable and nonetheless utilized by many organisations.
  • Whereas visualising the info on prime of the Staging setting (the dotted pink arrow) isn’t best; certainly, it isn’t unusual that we require to create some operational stories on prime of the info in staging. A superb instance is creating ad-hoc stories on prime of the present knowledge loaded into the staging setting.

How Enterprise Intelligence Elements Relate to Energy BI

To know how the BI parts relate to Energy BI, we’ve to have understanding of Energy BI itself. I already defined what Energy BI is in a earlier submit, so I counsel you test it out if you’re new to Energy BI. As a BI platform, we count on Energy BI to cowl all or most BI parts proven within the earlier diagram, which it does certainly. This part appears to be like on the totally different parts of Energy BI and the way they map to the generic BI parts.

Energy BI as a BI platform incorporates the next parts:

  • Energy Question
  • Information Mannequin
  • Information Visualisation

Now let’s see how the BI parts relate to Energy BI parts.

ETL: Energy Question

Energy Question is the ETL engine obtainable within the Energy BI platform. It’s obtainable in each desktop functions and from the cloud. With Energy Question, we are able to hook up with greater than 250 totally different knowledge sources, cleanse the info, rework the info and cargo the info. Relying on our structure, Energy Question can load the info into:

  • Energy BI knowledge mannequin when used inside Energy BI Desktop
  • The Energy BI Service inner storage, when utilized in Dataflows

With the combination of Dataflows and Azure Information Lake Gen 2, we are able to now retailer the Dataflows’ knowledge right into a Information Lake Retailer Gen 2.

Staging: Dataflows

The Staging part is accessible solely when utilizing Dataflows with the Energy BI Service. The Dataflows use the Energy Question On-line engine. We are able to use the Dataflows to combine the info coming from totally different knowledge sources and cargo it into the inner Energy BI Service storage or an Azure Information Lake Gen 2. As talked about earlier than, the info within the Staging setting shall be used within the knowledge warehouse or knowledge marts within the BI options, which interprets to referencing the Dataflows from different Dataflows downstream. Take into account that this functionality is a Premium function; subsequently, we should have one of many following Premium licenses:

Information Marts: Dataflows

As talked about earlier, the Dataflows use the Energy Question On-line engine, which implies we are able to hook up with the info sources, cleanse, rework the info, and cargo the outcomes into both the Energy BI Service storage or an Azure Information Kale Retailer Gen 2. So, we are able to create knowledge marts utilizing Dataflows. It’s possible you’ll ask why knowledge marts and never knowledge warehouses. The elemental purpose relies on the variations between knowledge marts and knowledge warehouses which is a broader subject to debate and is out of the scope of this blogpost. However in brief, the Dataflows don’t at present assist some elementary knowledge warehousing capabilities corresponding to Slowly Altering Dimensions (SCDs). The opposite level is that the info warehouses normally deal with huge volumes of knowledge, way more than the quantity of knowledge dealt with by the info marts. Keep in mind, the info marts include enterprise particular knowledge and don’t essentially include plenty of historic knowledge. So, let’s face it; the Dataflows will not be designed to deal with billions or hundred tens of millions of rows of knowledge {that a} knowledge warehouse can deal with. So we at present settle for the truth that we are able to design knowledge marts within the Energy BI Service utilizing Dataflows with out spending a whole bunch of hundreds of {dollars}.

Semantic Layer: Information Mannequin or Dataset

In Energy BI, relying on the placement we develop the answer, we load the info from the info sources into the info mannequin or a dataset.

Utilizing Energy BI Desktop (desktop utility)

It is suggested that we use Energy BI Desktop to develop a Energy BI resolution. When utilizing Energy BI Desktop, we instantly use Energy Question to hook up with the info sources and cleanse and rework the info. We then load the info into the info mannequin. We are able to additionally implement aggregations throughout the knowledge mannequin to enhance the efficiency.

Utilizing Energy BI Service (cloud)

Creating a report instantly in Energy BI Service is feasible, however it isn’t the beneficial methodology. After we create a report in Energy BI Service, we hook up with the info supply and create a report. Energy BI Service doesn’t at present assist knowledge modelling; subsequently, we can’t create measures or relationships and many others… After we save the report, all the info and the connection to the info supply are saved in a dataset, which is the semantic layer. Whereas knowledge modelling isn’t at present obtainable within the Energy BI Service, the info within the dataset wouldn’t be in its cleanest state. That is a superb purpose to keep away from utilizing this methodology to create stories. However it’s potential, and the choice is yours in any case.

Information Visualisation: Reviews

Now that we’ve the ready knowledge, we visualise the info utilizing both the default visuals or some customized visuals throughout the Energy BI Desktop (or within the service). The subsequent step after ending the event is publishing the report back to the Energy BI Service.

Information Mannequin vs. Dataset

At this level, it’s possible you’ll ask concerning the variations between a knowledge mannequin and a dataset. The brief reply is that the info mannequin is the modelling layer present within the Energy BI Desktop, whereas the dataset is an object within the Energy BI Service. Allow us to proceed the dialog with a easy state of affairs to grasp the variations higher. I develop a Energy BI report on Energy BI Desktop, after which I publish the report into Energy BI Service. Throughout my improvement, the next steps occur:

  • From the second I hook up with the info sources, I’m utilizing Energy Question. I cleanse and rework the info within the Energy Question Editor window. To date, I’m within the knowledge preparation layer. In different phrases, I solely ready the info, however no knowledge is being loaded but.
  • I shut the Energy Question Editor window and apply the adjustments. That is the place the info begins being loaded into the info mannequin. Then I create the relationships and create some measures and many others. So, the info mannequin layer incorporates the info and the mannequin itself.
  • I create some stories within the Energy BI Desktop
  • I publish the report back to the Energy BI Service

Right here is the purpose that magic occurs. Throughout publishing the report back to the Energy BI Service, the next adjustments apply to my report file:

  • Energy BI Service encapsulates the info preparation (Energy Question), and the info mannequin layers right into a single object referred to as a dataset. The dataset can be utilized in different stories as a shared dataset or different datasets with composite mannequin structure.
  • The report is saved as a separated object within the dataset. We are able to pin the stories or their visuals to the dashboards later.

There it’s. You’ve got it. I hope this weblog submit helps you higher perceive some elementary ideas of Enterprise Intelligence, its parts and the way they relate to Energy BI. I might like to have your suggestions or reply your questions within the feedback part beneath.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments