Copy
We love hearing about technologies from our sponsors doing good in the worldView in browser »
The New Stack Update

ISSUE 255: Tracking a Viral Infection Requires a Distributed Database

Talk Talk Talk

“If such biases go unchallenged in a future where life-changing decisions are largely automated, it could potentially pave the way to an ‘administrative and computational reordering of society,’ weakening civil society to the point that it becomes fertile ground for the growth of authoritarianism.”

Add It Up
Tool/Platform/Service Used for Self-service Data/Analytics Needs a Major Upgrade

Tools for data catalogs, metadata and the semantic layer are not meeting requirements for self-service business intelligence, analytics, artificial intelligence and related data preparation and integration. According to a 2020 survey included in TDWI’s “Evolving from Traditional Business Intelligence to Modern Business Analytics,” 49% believe their tools, platforms and cloud services need a major upgrade in order to meet their requirements in this area. On a similar note, 43% believe a major upgrade is needed to meet governance and data lineage tracking requirements. Access to real-time data is an area that continues to miss the mark as expectations are much higher than what today’s data warehouses are delivering.

Overall, few respondents were very satisfied that requirements are being met, but some areas are at least not doing too poorly. Only 18% believe “ease of use” needs a lot of work, while scalability, query performance and security also did better relatively well. However, while vendors have been meeting these table-stakes requirements, that has not helped them gain quick adoption of a wide range of self-service capabilities. Still, only 50% of respondents say people can access reports with little or no IT intervention, and that’s the most common type of self-service activity.

What's Happening

This episode of The New Stack Makers series with Okta, on all topics related to development and security at scale, features guest Anant Jhingran, CEO of StepZen, a GraphQL API provider. Jhingran’s deep well of experience, including long stints at IBM, Apigee and Google, prior to becoming CEO of StepZen, certainly qualifies him as a leading expert on APIs and their role in today’s DevOps environments. Co-hosted by Alex Williams, founder and publisher of The New Stack, and Randall Degges, head of developer advocacy at API security firm Okta, Jhingran offers his take on how APIs have evolved, their potential for the developer community and how their success accounts, in part, for their exposure to vulnerabilities.

Okta Series - APIs’ Evolution, Future and Vulnerabilities

Case Study: Tracking a Viral Infection Requires a Distributed Database

We love hearing about technologies from our sponsors doing good in the world. This week we got word about how Cockroach Labs’ open source distributed database is helping Germany get a handle on the COVID-19 pandemic. 

Last year, the German government sponsored a hackathon to deal with the emerging COVID-19 crisis. One German startup responded with an application designed to track people’s COVID-19 symptoms. Central to the success of the application has been a set of open source technologies — as well as a strong distributed database to ensure scalability and security.

The project, called Quarano, went live just 10 weeks after the Hackathon. This system allows patients to log a diary of symptoms which is then monitored by the Health Authorities.

For the Hackathon challenge, the Quarano team wanted to focus their project on making the COVID-19 documentation process more effective. The German government operates on a Health Authority basis, meaning that each region in Germany has its own authority that makes its own rules for tracing the virus. Health Authorities used (and in some cases are still using) pen and paper to record people’s symptoms and who they have been in contact with.

Not only is this manual documentation method inefficient and unreliable, but it also significantly slows down the process of tracking the virus.

After the hackathon, the team had an open source "contact tracing" prototype that would allow authorities to understand where and how the virus was spreading. It could also be used to organize quarantines, and even offer health services for citizens.

With a working prototype ready, the team was confronted with a number of challenges to make it scale into production use, as well as to make it secure.

The application itself would need to easily scale to new regions should other authorities want to use it. For starters, all the technologies that make up Quarano needed to be open source, including the infrastructure, and the database.

This requirement led to a design built on Spring Boot, which exposed a REST API to track cases of COVID-19 infections. The application backend uses a standard Maven project based on a series of open-source technologies. Deutsche Telekom donated the hardware. Currently, the project uses three nodes (one per region) in Germany and each region has four vCPUs.

The team needed a database that could be hosted as a cluster on this private cloud. As per requirements, it would need to be open source and it would need to scale.

The Quarano team started building their application on another distributed database, but found the performance lacking. So they tested CockroachDB, and were pleased by its ability to scale and were intrigued by its geo-partitioning capabilities, should their product grow beyond Germany.

CockroachDB has a node-based architecture where each node functions on its own but is part of a cluster of nodes that presents as a single logical database. Every node is a gateway to the entirety of the database, and if a node fails, it automatically comes back online. This ensures Quarano will survive any failure and will always be on and available.

Security, of course, was also a factor for the team. The application collects a lot of data about users (age, gender, symptoms, location, etc). Because CockroachDB has built-in encryption, Quarano did not have to implement a supplemental encryption solution.

If Quarano adds an additional Health Authority located in a different region, they can simply add more nodes. When a node is pointed at the cluster, it instantly rebalances the data across all of the nodes to incorporate this new resource. With the addition of a new node, you also scale out the volume of transactions the single logical database can handle. This means that as you add nodes, you scale both reads and writes which would ensure that Quarano is performant.

Should the application’s reach expand beyond Germany, that’s when Quarano could leverage CockroachDB’s geo-partitioning features. Geo-partitioning ties data to a location at the row level. This helps Quarano meet latency requirements by keeping data close to its users, and it would also help with regulatory compliance that may require data to reside in a particular geographic jurisdiction.

Got a case study of how your open source cloud native technology is being used? Drop us a line! And be sure to check out Cockroach Labs’ page of other customer success stories. 

The Ultimate Guide to Machine Learning Frameworks

Want to get your organization started in machine learning? You need a platform, one that offers a full range of features and handles the underlying infrastructure. We asked our AI Analyst Janakiram MSV to handpick the best frameworks to help you get started. Here you’ll read all about Scikit-Learn, TensorFlow, PyTorch, MXNet, DL4J, XGBoost, PaddlePaddle and ONNX.

Design a Better Kubernetes Experience for Developers

Here’s a fascinating contributed post from Spotify Senior Infrastructure Engineer Matthew Clarke about how the music streaming giant adopted Kubernetes. The company loved the scalability of Kubernetes, naturally, but found the K8s experience out of the box not well suited for its developers. So it created its own portal Backstage to adopt Kubernetes to service owners, not just cluster administrators. 

Solo.io Launches Gloo Mesh Enterprise to General Availability

After a couple of years in development and just three months in beta, Solo.io has released Gloo Mesh Enterprise service mesh to general availability this month, marking API stability and a slate of new features, built in response to customer feedback during the beta period. Gloo Mesh Enterprise is the company’s enterprise-grade, Kubernetes-native solution to help organizations install and manage Istio service mesh deployments.

On The Road
JavaScript Trends Revisited Wednesday, March 3 at 10 a.m. PT.

March 3 // LIVE STARTING AT 10 AM PST
JavaScript Trends: Revisited


It’s time to go back to the future with a retrospective look at Node.js. Join us at OpenJS as we look back to 2019 when we first gathered on the stage to discuss the year in JavaScript. Now it’s 2021 so what has changed? What’s hot? What’s not? Watch here live or on-demand!
The New Stack Makers podcast is available on:
SoundCloudFireside.fm — Pocket CastsStitcher — Apple PodcastsOvercastSpotifyTuneIn

Technologists building and managing new stack architectures join us for short conversations at conferences out on the tech conference circuit. These are the people defining how applications are developed and managed at scale.
Best of DevSecOps: Trends in Cloud Native Security Practices

This is the first in a new series of anthologies that assemble some of our best articles on a trending subject, paired with our editors’ insightful analysis to frame the bigger picture. These exclusive ebooks help developers, architects, operators and management go in-depth, quickly, on hot topics in at-scale development and management.

In this ebook, we explore how security practices are now being integrated into the development process, as well as the build pipeline and runtime operations of cloud native applications. You’ll learn more about:

  • How DevSecOps enables faster deployment cycles.
  • Why DevSecOps is necessary for cloud native architectures.
  • The challenges and benefits of DevSecOps practices.
  • The new role of developers and operators in security.
  • How to measure DevSecOps success.
  • Tools and best practices for adoption.
  • Emerging trends to pay attention to.
Download Ebook
We are grateful for the support of our ebook sponsor:


 
Copyright © 2021 The New Stack, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by Mailchimp