10 successful big data sandbox strategies

Keep in mind these ten strategies when building and managing big data test environments. 

Being able to experiment with big data and queries in a safe and secure “sandbox” test environment is important to both IT and end business users as companies get going with big data. Nevertheless, setting up a big data sandbox test environment is different from establishing traditional test environments for transactional data and reports. Here are ten key strategies to keep in mind for building and managing big data sandboxes:

1. Data mart or master data repository?

The data base administrator needs to make a decision early-on as to whether to have test sandboxes use data directly from the master data repository that production uses, or whether the best solution is to replicate and splinter off sections of this data into separate data marts that are reserved for testing purposes only. The advantage of the full data repository is that testing actually uses data that is used in production, so test results will be more accurate. The disadvantage is that data contention can be created with production itself. With the data mart strategy, you don’t risk contention with production data—but the data will likely need to be periodically refreshed to stay in some degree of synchronization with data being used in production if it is going to closely approximate the production environment.

2. Work out scheduling

Scheduling is one of the most important big data sandbox activities. It ensures that all sandbox work is optimally being run. It usually achieves this by concurrently scheduling a group of smaller jobs that can be completed while a longer job is being run. In this way, resources are allocated to as many jobs as possible. The key to this process is for IT to sit down with the various user areas that are using sandboxes so everyone has an upfront understanding of the schedule, the rationale behind it, and when they can expect their jobs to run.  

3. Set limits

If months go by without a specific data mart or sandbox being used, business users and IT should have mutually acceptable policies in place for purging these resources so they can be put back into a resource pool that can be re-provisioned for other activities. The test environment should be managed as effectively as its production environment counterpart so that resources are called into play only when they are actively being used.

4. Use clean data

One of the preliminary big data pipeline jobs should be preparing and cleaning data so that it is of reasonable quality for testing, especially if you are using the “data mart” approach. It is a bad habit (dating back to testing for standard reports and transactions) to use data in test regions that is incomplete, inaccurate, or even broken—simply because it was never cleaned up before it was dumped into a test region. Resist this temptation with big data.

5. Monitor resources

Assuming big data resources are centralized in the data center, IT should set resource allowances and monitor sandbox utilization. One area often requiring close attention is the tendency to over-provision resources as more end user departments engage in sandbox activities.

6. Watch for project overlap

At some point, it makes sense to have a corporate “steering committee” for big data that tracks the various sandbox projects going on throughout the company to ensure that there is no overlap and/or duplicated effort.  

7. Consider centralizing compute resources and management in IT

Some companies start out with big data projects in specific departments but quickly learn that they can’t work on big data, do their daily work, and then manage compute resources, too. Ultimately, they move the equipment into the data center for IT to manage. This frees them to focus on the business and ways that big data can bring in value.

8. Use a data team

Even in sandbox experimentation, it’s important to have the requisite big data skills team on hand to assist with tasks. Typically, this team consists of a business analyst, a data scientist, and an IT support person who can fine-tune hardware and software resources and coordinate with database specialists.

9. Stay on task with business cases

It’s important to infuse creativity into sandbox activities, but not to where you totally forget the initial charge of the business case you’re trying to bring value to.

10. Define what a sandbox is!

Especially participants coming from the end business might not be familiar with the term “sandbox” or what it implies. Like the childhood sandbox, the purpose of a big data sandbox is to freely play and experiment with big data—but to do it with purpose. Part of this purposeful activity should be abiding by the ground rules of the sandbox, such as when, where and how to use it, as well as experimenting to derive meaningful results for the business.

Why everyone wants a private cloud

Concerns about security and control make the "private" cloud a more palatable model for many companies. How sound is this kind of thinking? 

“We’re going to the cloud for VDI (virtual desktop infrastructure), and we’re going to have our own cloud,” said an IT manager of a one-man shop (himself) at a manufacturing company with 20 employees.

The manager and the CEO of the company believed that they could implement their own private cloud by using a “cloud in a box” solution for office applications that would save the company money in the form of fewer license fees for office software. The way that they planned to implement the project was by relying on the cloud equipment vendor that had sold them the solution to provide both implementation and system-tuning expertise and support.

For these managers, there were also the benefits of “bragging rights”—because it’s popular today to have a cloud of your own, no matter how small you are.

The question is, why?

Inevitably, fears about the security of applications and data are the first things mentioned when the alternative of going to a public cloud comes up.

However, for many small companies with limited IT resources, data and application security have always been lax, even when they are running their own internal IT operations. Many of these companies routinely accept the downtime brought on by a denial of service attack (DNS) or the loss of data that is suffered when a system unexpectedly goes down.

So given this, why is it so important to have your own private cloud?

Some speculate that organizations have been developing their own IT infrastructures for years, and that these infrastructures have been used and continue to be used to host business critical applications for the organization. In addition, organizations, regardless of their size, like the idea of data sovereignty, where they can keep business critical data internally, without exposing it through widely available public interfaces that characterize the public cloud environment. Finally, businesses are aware that they must satisfy regulations and regulators, especially if they are in industries like finance or healthcare.

Still other companies are uncomfortable at relinquishing control of the information lifelines of their businesses to outside vendors, even if they are convinced that their data is absolutely secure. In back of this is a concern about control—and a fear that a breakup with a cloud vendor could lead to major risk and disruption for the business as it struggles to re-insource data that it should have never outsourced.

The truth is, we all understand that cloud is here to stay and that it will continue to make inroads into data centers and IT infrastructure. But what we don’t know is where the inevitable “pushbacks” are going to occur down the road.

“When you’ve been in IT for over thirty years, you see a lot of changes in thinking—and invariably, thought cycles reverse and “old thoughts” resurface in new ways,” said former and now retired CIO for Caterpillar, John Heller. Heller was talking about the days of centralized computing in the 1960s and 1970s which then gave way to decentralized, distributed computing in the 1980s—and then once again returned to centralized computing with the growth of virtualization in the 1990s and 21st century.

Consequently, it isn’t too far-fetched for organizations to hedge against the turns that technology thinking  takes—and to embark on their own cloud journeys with the desire to understand fully what cloud is all about and how it works, regardless of how small they are. For most companies, this means engagement with a private cloud.

Five future technologies I can't wait for

Takeaway: If you think the past two decades have been amazing for tech, wait until you see what’s next. See the five hottest technologies that are on the way.

“The future is already here — it’s just not evenly distributed.” William Gibson

One of the best things about my job covering the latest technologies is getting early information about some of the amazing things coming down the pipeline in the years ahead. But, the flip side of that is that I often learn about some really cool stuff that won’t be available to the general public at a reasonable cost for a long time, which often leads to a case of “Isn’t that here yet?” that can last for years.

With that mind, here are my current top five picks for “Isn’t that here yet?” These are the technologies that I’m seriously impatient to see show up in the real world.

Credit: iStockPhoto/audioundwerbung

1. Wireless docking of mobile devices

I’ve recently talked a lot about the utopian convergence of PC and mobile devices. I see this as the next big game-changer in the technology industry, and that’s why I’ve pinpointed it as Microsoft’s next big opportunity (and explained why they could miss it). However, the number one factor that’s needed to make this happen isn’t a super-fast CPU or a miniature SSD drive with lots of storage, it’s a common standard for wireless docking. That’s what will enable us to take a smartphone or tablet and set it on desk or on a charging station like the Palm Touchstone and then have it wirelessly connect to a keyboard, mouse, and large screen monitor. We need something easier and more robust than Bluetooth. A technology like Wireless USB could be the answer. The most important thing is that it will need to be a universal standard integrated into every phone and tablet so that we no longer need proprietary docking solutions like the ones for the Motorola Atrix. Timeframe: 3 years

2. Inexpensive mobile broadband everywhere

The arrival of true 4G wireless broadband is just beginning to hit critical mass in the U.S. in 2011 with the continuing rollout of Verizon’s LTE service (I don’t count the 3.5G of T-Mobile and AT&T as 4G). And, while LTE offers impressive speed and performance, it still has wrinkles that need to be ironed out (handoff between 3G and 4G often gets goofy). But, the biggest thing LTE needs — from a user standpoint — is a little more competition to drive the price down and force the telecom companies to fight tooth-and-nail for business by deploying 4G everywhere. It’s a shame WiMAX is floundering in the U.S. because it was a legitimate 4G competitor and was aimed at delivered low-cost, high-speed mobile broadband everywhere — and then just turning phone calls into VoIP calls (like Skype) since people are using their phones less and less for voice and much more for data. Still, 4G is going to happen because people want high-speed Internet everywhere and are willing to pay for it. There might even be creative companies that will give it away or offer a reduced rate for ad-supported access. Timeframe: 2 years

3. Three dimensional printing

One of the coolest and most futuristic things in the works has got to be 3D printing. No, I’m not talking about making a printout and using 3D glasses to create a silly illusion (that would be even dumber than 3D movies and 3D TV sets). The three dimensional printing that I’m talking about is where you make a three dimensional design on a computer and then send it to a special device to “print” a 3D model. There are already some expensive (over $15,000) models available in the real world and used by companies that need to make rapid prototypes of products. However, there will eventually be models available for average consumers and lots of templates of different things to create, which means it will someday be cheaper and easier to create certain things than to go out and buy them. In other words, this will likely be the first step toward the replicator in Star Trek. Timeframe: 5-10 years

4. HTML5 to make the web an app

There are a lot of things that HTML5 will bring to the web — and some of these elements are beginning to show up in a few sites today — but the biggest thing HTML5 is going to do is take the training wheels off the web and unleash it to compete with traditional software. The two things that I’m most excited about are that HTML5 is going to turn the web into app and it’s going to eliminate the need for most of the plugins that slow down browsers and introduce extra security risks. With HTML5, constantly refreshing pages will become a relic of the old web as pages become far more dynamic and interactive, automatically loading the latest content and changing the page based on user clicks, mouse-overs, and multi-touch gestures. And, of course, multimedia will be integrated into the experience and plugins for Flash, Shockwave, Silverlight, and other helper technologies will become unnecessary. Timeframe: 2 years

5. Flexible OLED displays

Another technology that has been promised for years but still needs several breakthroughs before it’s ready for the mainstream is OLED displays. We’ve seen some high-priced prototypes from Sony that feature ridiculously thin TV screens in small sizes (under 30 inches), but that only scratches the surface of what OLED will be able to do in the future — at least theoretically. These ultra-low-power displays will be able to be almost as thin as plastic wrap and will be completely flexible (even when in use). The result will be screens that can be integrated directly into walls and be virtually invisible when turned off. You’ll also see smartphones that can be folded in half and put into a pocket or a wallet, or even rolled into a bracelet. We could even see the re-emergence of the broadsheet as a way to read and consume news, but instead of unfolding a newspaper you’ll unfold an OLED display that is tied to a subscription to The New York Times, for example (here’s an example of how the Times is already envisioning this). Timeframe: 4-5 years

And you?

What future technologies are you dying to get your hands on? Post in the discussion below and include your prediction for how long it will take to hit the mainstream.