I wanted to share my experiences and highlight the things that I found most valuable while creating a Dedicated SQL Pool instance in Azure that is used by one of my clients to query and analyze over 5 billion rows of data. This is not a comprehensive list of the capabilities or best practices for Azure Dedicated SQL Pool. This is more of list of things that got me my best bang for my buck.
So you have a web app up and running in Azure and now you need to do some background work that’s related to your new web site but it definitely doesn’t belong in the web app itself. This could be anything from importing data, to sending out subscription emails, to updating cached data, or running database maintenance.
You may be wondering, with so many viable options at your fingertips in Azure, what is the best and most cost effective way to get work done? You want to strike the balance between having something that’s powerful enough to grow if it needs to grow but you don’t want to incur a bunch of unnecessary costs right out of the gate by choosing too much power!
The task at hand for me was that I needed to import and transform several different Excel formats into an existing SQL user table. The formats were not guaranteed to always be the same and I needed a way to quickly add new formats without writing a bunch of new code each time.
This past month I finally had a business case that called for Azure Event Hubs to be part of my overall solution to a real-world problem. Up until this point I was always able to get by mostly with plain old Table Storage Queues and sometimes Service Bus Topics, but the business case I started working on this past month involved sending many megabytes of information from potentially thousands of devices around the clock.
DocumentDB is a great NoSql database engine for working with big data. One thing that was difficult to implement though was an intelligent way to page through the results. In this article I will show how I used a combination of caching at the client side of my app along with utilizing continuation tokens from the SDK response to build a decent paging mechanism for big data queries using DocumentDB.
It seems to be a common story with almost every software dev I meet; they have a laptop full of side projects that were started many months or years ago but, sadly, never saw the light of day. Many of my projects have met the same fate. Why is this so common?
I think the answer is that it's easy to start something, it's very hard to finish it.
