Download ebook: Managing big data – part 1

In Big Data, Data & Analytics, Featured on App, Technology by Simon Crompton-Reid

Big Data words landscape

This post is written in reference to the ebook "How to scale up for big data" by Derek Slater. You may want to download the full ebook here >

While a small amount of data might already be difficult to manage, what is to be expected when managing big data? Organisations must bear in mind that when managing big data, they have to think through the framework and construction requirements.

One important point to note when managing big data is "keeping it close". The key problem is storage and where the data will be stored. Most organisations engage in storage area network (SAN) for more control and retrieving data access. But the problem is that the process of shifting data from the SAN to the processing takes up a lot of time. Instead, big data's implementation has different conditions. Keeping all the data in one place is actually wise, as the large amount of data that has to be processed is being stored together. However, not many companies might be willing to accept a shift of data from the SAN structure to a network attached storage (NAS) structure, as it may cause many adjustments in the organisation. The organisation might also have to maintain and repair the disks when it is not functioning thus incurring extra costs. As such, many organisations might be resistant to the shift of data.

For organisations to discover what additional data they want to add in to their data storage, they should inspect the data they have on hand. Although it may be difficult to decide how big the storage space for the additional data needs to be, it is still doable. This is done by gaining data from different databases and keeping track and describing the different types of data, then the organisation may be able to dictate how much data they have and how much it will grow in the future. Another way is that organisations may calculate its "typical network utilisation" to anticipate present and future data. We would be able to find out how much data is being shifted in a year; this will then help the organisation to estimate how much additional data space would be needed in the next few years. However, as data is always shifting, the organisation would have to take time to re-calculate the data every year in order for it to be accurate.

There is still much to learn about big data. You might be interested to learn more through our 3 Day MBA in Big Data training course. Download the brochure to view the full course agenda.

This post is written in reference to the book "How to scale up for big data" by Derek Slater. You may want to download the full ebook here >

– Post by Hilda Yin

[Image: Marius B – Flickr]