NoSQL – Break The Shackles
Recent growth of social media networks and smartphone users have led to a sudden spike in internet user population. In today’s world a good application can become viral in a matter of few hours. Gone are the days when user-base growth was a slow, linear and predictable process. Having a million+ users is the new ‘Norm’ for any internet application. Growth estimates have become unpredictable and now companies need to implement/create applications such that they can effectively support dynamic increase in the number of users as and when required. Today’s web applications also generate a lot of unstructured data in form of comments, feedback and access patterns. This data contains lot of useful insights about the users and usage behavior. Applications should be able to preserve and analyze this unstructured or semi structured data. Let the facts speak for themselves, 2+ billion – Internet users 32 billion – Hours spent online daily 1+ billion – Smartphone users 566% – Growth in number of internet users in last 12 years Scalability and Performance have become key success factors for any web app company. Normally, web applications follow a three tier design approach – Front-end (UI) layer, Model layer and Back-end (Database) layer. There are a variety of solutions and design strategies available for UI and model layer implementations, however database layer has been dominated by relational databases. Relational databases are rigid and difficult to scale. Application developers find it extremely challenging (impossible in most of the cases), to get the dynamic scalability without compromising performance. Relational databases have also failed to provide effective storage and retrieval of unstructured data. So, in nut-shell Relational databases are biggest bottleneck for rapidly growing web applications? Not anymore; NoSQL databases have emerged to their rescue. NoSQL is a completely new approach of thinking about a database. NoSQL databases generally do not adhere to the traditional RDBMS principals and fundamentals. A NoSQL DB may not support SQL and may not provide ACID (atomicity, consistency, isolation, durability) guarantees, but compared to relational implementations, a NoSQL DB is more flexible, scalable and cost efficient. Let me clear the ambiguity around the term ‘NoSQL’. It is a misnomer. NoSQL should be understood as ‘Not Only SQL’.NoSQL databases are – non-relational, schema-free, distributed, open-source, eventually consistent and horizontally scalable. NoSQL DB solutions available today can be classified into four major design categories, namely: key-value stores, document store, Columnar, and graph databases. Irrespective of the design and implementation, NoSQL databases share following characteristics: a) Auto-sharding: A NoSQL Database automatically distributes stored data across servers, without any need of explicit applications logic to do that. Servers can be added or removed from the cluster without any data loss or major application downtime. b) Implicit cache: To increase throughput, advanced NoSQL database technologies seamlessly cache data in memory. This is implicit and transparent to the application development team. c) Schema free: The storage structure is schema free and thus more flexible to efficiently store and retrieve unstructured data. NoSQL may not be a fit for all applications, specially the ones which involve online transactions. However, most of the today’s web applications can break performance shackles through NoSQL. NoSQL is supported by many technology companies and is gaining momentum. It is here to stay! Stay tuned to dive deeper into the NoSQL world in my further posts.
Data Migration Challenges
According to a Gartner/Standish group study, 67% of data migration projects suffer from implementation delays. Given the importance of data in the modern enterprise, flawed data migration can have severe repercussions. A nuanced understanding of the various challenges in Data migration is required to mitigate many of the risks associated with this activity. Importance of Data Quality As with most data integration efforts, data quality is one of the biggest challenges in data migration and some of the challenges are: Given the non-recurring nature of most data migrations, handling data quality issues in a live post-migration environment can be very challenging. Poor data quality imposes significant costs post-migration, with issues ranging from poor business intelligence to delay/disruption in business processes. Data quality issues are amplified when migration happens from a legacy system (with poor data quality) to a newer application with a far richer feature set and a stricter data model. This necessitates a lot of planning before the migration process can commence. Solutions to manage Data Migration effectively To ensure good data quality coverage, complete profiling of the source data systems must be conducted. This process must be complemented with the reconciliation of business rules across the source and target systems. Both these activities, when completed with the involvement of all relevant stakeholders will provide the data migration team with a lot of insights into data/rule gaps between the source & target systems and help create data validation/transformation rules to be used in migration. These rules can be further fine-tuned by accounting for deduplication and consolidation if necessary (For instance, when multiple source systems are involved). If the target system is still evolving/being built, strong change management processes need to be put in place to ensure the data migration process keeps up with application changes. Choosing the right technology/tool stack is one of the biggest decisions that awaits the migration team. The choices range from a custom-built solution, data integration tools to a hybrid solution. While there are merits and drawbacks to each choice, the flexibility and comprehensive data integration capabilities offered by modern ETL tools make them a compelling choice. Many of these tools offer integrated development environments that speed up the development process and provide plenty of customization capabilities through scripting/reusability. The data migration solution offered by Tavant uses Talend, an open-source tool that provides a robust data integration toolkit and Java/Perl based scripting allowing for significant customization. Other technology challenges include differences between the underlying source/target database systems (mismatches in terms of data types supported, date/time format mismatches etc), encrypted data, and handling of character encoding and numeric precision. A good data quality management process and technology solution need to be backed up with a good operational process that can support an iterative data migration validation/ implementation process. Other operational challenges that need to be factored in are the migration of hot data (active business transactions) and the creation of failback provisions. A robust data testing strategy is required to not only ensure that all data within scope is migrated but also that the migrated data is functionally usable on the target system(s). Finally, an experienced data migration team can drastically cut down on the learning curve and hit the ground running. While each data migration project has its own dynamic, a good understanding of challenges and best practices can significantly reduce the chances of running into common data migration roadblocks. Recognition of these challenges will also ensure that the data migration process receives the support it requires.
Relational Models – Where Is The Bottleneck?
There are several reasons for the downfall of relational databases and before I start off with the various aspects of NoSQL designs, I would like to highlight a few factors which have made relational models a misfit for global applications today. Manual Intervention:Relational databases were designed in an era when human workforce was cheaper than technology. Thus, relational models have a lot of gaps. For example, data distribution, sharding, partitioning and similar functions are mostly managed through human intervention. Expensive Vertical Scalability:Relational model designs rely heavily on the underlying hardware’s capacity and quality for performance. They were never designed for clustered storage and operation. Thus, to upgrade a relational database’s capacity it has been necessary to upgrade server hardware which increases costs substantially. REDO and UNDO logs:Persistent REDO logs slow down the write performance as every write operation is recorded in REDO logs. UNDO logs are also updated for each transaction, slowing down the system considerably. Transaction Support Through Two-Phase Commit:Two-phase-commit transactions (used to ensure consistency) are known to degrade performance. Rigid Schema Design:RDBMS architecture requires fixed schema designs. All tables and columns need to be pre-defined along with the data type and length constraints. Most of today’s applications generate a lot of unstructured/semi structured data. What is required today is a flexible schema free architecture for storing such data. REDO and UNDO logs are used by relational databases to ensure ACID compliance of transactions. Getting rid of these persistent logs would be a huge performance booster but that would make them ACID non-compliant. Considerable design changes are needed to make the relational databases horizontally scalable and schema free. We can therefore conclude that a highly efficient and scalable database must have the following characteristics: • It must be designed to work in distributed architecture,• It must be horizontally scalable,• It must have memory based caching mechanism, and,• It must not make use of REDO and UNDO logs. (However, one must remember the fact that getting rid of REDO and UNDO logs will improve performance but such databases may not be ACID compliant and may not support transactions). NoSQL databases offer all the advantages mentioned above and they offer the right fit for most applications. Obviously they are here to stay! Do you agree or have you come across some hitches with NoSQL which makes you believe that there are other, better databases available today?
Tavant to Attend the Casual Connect 2013, San Francisco.
Santa Clara, CA, July 16, 2013 : Tavant, a leading Software solutions provider to the Interactive Entertainment Industry, today announced that it will be present at the 2013 Casual Connect to be held during July 30 – August 1, at the Hilton San Francisco Union Sq. in San Francisco, California. At the event, Tavant subject matter experts will be sharing best practices and the technology know-how on developing advanced online, social, and mobile games. The team will also be having in-depth discussions with the delegates and industry peers on technology areas such as developing gaming platforms, community websites, online stores, and analytics systems for casual game development. The experts will be sharing success stories on how Tavant is helping top gaming companies develop solutions for monetizing content, providing detailed analytics and simplifying transactions. As a development partner for companies building social and mobile games using leading-edge technologies, Tavant handles both cross-platform UI and backend services. Tavant manages all technical aspects of game development while the studio retains full control over the game design and art style. To schedule a meeting with us, send an email us at [email protected] About Casual Connect Casual Connect USA is produced by the Casual Games Association and is the premiere event for the casual games industry with over 5,500 professionals attending each year. Casual Connect brings the most talented and knowledgeable experts in the casual gaming field together to further the casual games industry with the best networking and learning opportunities. About Tavant Technologies Tavant Technologies is a specialized software solutions & services provider that leverages its expertise to provide impactful results to its customers. We have leveraged our unrivaled capabilities and domain insights to create game changing results for leading businesses across chosen industry verticals. We are known for our long-lasting customer relationships, engineering excellence and passionate employees. Founded in 2000, we are headquartered in Santa Clara, California and service customers across North America, Europe, and Asia-Pacific.
What’s so Big about Big Data?
About 90% of the data in today’s World has been created only in the last two years. As per Google, every 2 days we create as much information as we did till 2003. There are around 200 million Tweets every day. Facebook gets around 6 billion messages per day. When it comes to handling this kind of a data explosion, conventional RDBMS has its own limitations; that’s where Big Data comes into the picture. Big Data is about handling the 3Vs – Volume, Velocity, and Variety. Volume: Big Data can handle data in Petabytes or more; this is a difficult task in RDBMS. Velocity: Velocity describes the frequency at which data is generated, captured and shared. Big Data is capable of handling dynamic data from diverse sources—online systems, sensors, social media, web clickstream, and other channels. Variety: Big Data comprises all types of data—structured, semi-structured and unstructured data (such as text, sensor data, audio, video, click streams, log files and more). According to O’Reilly Media,., “Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the structures of your database architectures. To gain value from this data, you must choose an alternative way to process it.” The comparison chart shown below throws more light on the differences between RDBMS and Big Data. RDBMS Big Data Variety Places data inside well-defined structures or tables using meta data. But it can’t handle semi-structured and unstructured data—like photos, videos and posting messages on Social Media. Has the capability to handle a variety of data (structured, semi-structured and unstructured data) through different NoSQL databases like graph, document, key-value and column family databases. Volume Can handle data in MBs and GBs better than any Big Data system, but its performance goes down as the data size increases to TBs or PBs. The RDBMS system can be scaled up and not scaled out. Also, the cost of scaling up a system is high. Good in handling a large size of data. So it is very efficiently used by sites like Facebook, LinkedIn and Twitter, where the data size is huge. Big Data handles this task through scaling out on commodity hardware. Velocity Can handle small sets of data, but can’t manage the speed at which data arrives on sites like Facebook, Twitter, etc. So, the performance will be poor when the velocity is high. Can easily handle high velocity data like the millions/billions of messages arriving on social networking sites. It can handle the data through parallel processing, which is not possible in RDBMS. Apart from data storage and retrieval, Big Data has capabilities to process the data efficiently, e.g., we can divide the data to be processed into hundreds or thousands of commodity hardware, and the data can be processed independently on each machine. So the bottom line is that the omnipresent and ongoing buzz around Big Data is definitely not a passing fad. This claim can be substantiated with the fact that today’s technology-driven, net-enabled businesses are continuing to count on Big Data – big time!
Tavant Invited to Speak at the Warranty Chain Management Conference in San Diego.
Santa Clara, CA, March 11, 2013: Tavant, a leading software solutions and services provider, today announced that it has been invited to speak at the Warranty Chain Management Conference March 12-14, San Diego, California. The 9th Annual Warranty Chain Management Conference is the largest international warranty and service contract conference and provides a forum for warranty professionals and executives to meet and discuss warranty related issues and develop warranty management as a recognized discipline. Tavant experts will be presenting papers jointly with customers on important areas of service management such as: Identifying a Value Driven Warranty Solution, Supplier Cost Recovery and Integrating Service Contracts, Fleet Management & Warranty. Tavant will also be showcasing its web-based, real-time, full-lifecycle and flexible pricing model-based warranty solution that helps reduce spends, eliminate fraudulent claims and reduce claim processing time substantially. About Tavant Technologies Tavant Technologies is a specialized IT solutions & services provider that leverages its expertise to provide impactful results to its customers. We have leveraged our unrivaled capabilities and domain insights to create game changing results for leading businesses across chosen industry micro-verticals. We are known for our long-lasting customer relationships, engineering excellence and passionate employees. Founded in 2000, we are headquartered in Santa Clara, California and service customers across North America, Europe, and Asia-Pacific.
Tavant receives the Great Place to Work® award.
Bangalore, July 13, 2012: – Tavant has been ranked as one of the best employers in the ‘Best Companies to Work For’ study conducted by the Great Place to Work® Institute. With this recognition, Tavant has earned the top 25 best workplace award for the third time since it has started participating in this study – the first two times in the years, 2007 and 2009. Being rated as one of the best employers is a recognition of Tavant’s focus on its core values of People, Passion and Excellence. Great Place to Work study is in its 9th year and received overwhelming response with more than 580 organizations registering to participate making it the largest such study in India. Only 50 made it to the Best Companies list, making this list the gold standard of best workplaces. “We are proud to be part of the select group of best companies to work for,” said Sarvesh Mahesh, CEO, Tavant. He continued, “I believe that great people make a great workplace and I take this opportunity to thank all Tavantians – current and past, who have made this a truly Great Place to Work.” About Tavant Tavant is a specialized IT solutions & services provider that leverages its expertise to provide impactful results to its customers. We are known for our long-lasting customer relationships, engineering excellence and passionate employees. We have many years of real-world implementation experience and deep domain expertise along the entire trading and securities value chain. Tavant offers solutions that combine leading edge technology with business insights to power next generation, scalable applications that allow you to take customer experiences to new levels, while minimizing costs. Founded in 2000, we are headquartered in Santa Clara, California, and service customers across North America, Europe, and Asia-Pacific. Tavant also provides full end-to-end custom trading system development (front, mid and back-office integrations) for the top Barron’s-rated brokerage firms in the US and worldwide.