--- title: "Exploring several ways to process 25TB of data" publication: "DB Weekly" issue: 260 published_at: "2019-06-28" canonical_url: "https://dbweekly.com/issues/260" --- This week's database news #260 — June 28, 2019 [Read on the Web](https://dbweekly.com/issues/260) Database Weekly [![\](https://res.cloudinary.com/cpress/image/upload/w_1280,e_sharpen:60/oznmvemhqzly31jmrymk.jpg)](https://livefreeordichotomize.com/2019/06/04/using_awk_and_r_to_parse_25tb/) [Using AWK and R to Parse 25TB of Data](https://livefreeordichotomize.com/2019/06/04/using_awk_and_r_to_parse_25tb/) — This is a fun, practical look at several approaches taken to process a large data set, including the dead ends and lessons learned before settling on a reasonably ‘rustic’ solution. Nick Strayer [MongoDB 4.2 Previewed at MongoDB World](https://www.mongodb.com/blog/post/mongodb-42-previewed-at-mongodb-world) — The 4.2 release of the popular document-oriented database will be packed with new features - here’s a look at four of them, including distributed transactions and client-side field-level encryption. _(Via our [MongoDB-focused newsletter](https://mongodb.email/).)_ Dj Walker-Morgan (MongoDB) [![\](https://copm.s3.amazonaws.com/7d77f473.png)](https://studio3t.com/knowledge-base/articles/import-sql-database-to-mongodb/?utm_source=cooper&utm_medium=newsletter&utm_campaign=june21) [Studio 3T Makes SQL Migration to MongoDB, Powerfully Simple](https://studio3t.com/knowledge-base/articles/import-sql-database-to-mongodb/?utm_source=cooper&utm_medium=newsletter&utm_campaign=june21) — Now you can import an entire SQL database to MongoDB using Studio 3T and its new SQL Migration feature. Studio 3T sponsor [The Major Features Coming in PostgreSQL 12](https://momjian.us/main/writings/pgsql/features.pdf) — The much esteemed Postgres expert Bruce Momijan has released a simple slidedeck highlighting the most significant improvements and features coming to PostgreSQL 12, including JIT compilation and `REINDEX CONCURRENTLY`. It’ll only take you a minute to scan. Bruce Momijan [The Cloud is Now the Default Platform for Databases, Gartner Says](https://www.datanami.com/2019/06/26/cloud-now-default-platform-for-databases-gartner-says/) — The overall database market is growing, says Gartner, and cloud deployments are responsible for 68% of the growth, principally on AWS and Azure. While cloud deployments are rapidly becoming ‘default’, however, the cloud only accounts for 23% of the market’s revenue, so there’s still a long way to go. Datanami [A New Redis Benchmark Hits 200 Million Ops/Sec](https://redislabs.com/blog/redis-enterprise-extends-linear-scalability-200m-ops-sec/) — Redis Labs have pushed the enterprise version of Redis, the data structure store, up to 200 million operations per second with under 1 ms latency on 40 EC2 instances - a 4x improvement on a similar test last year. Redis Labs [Analyze BigQuery Data with Kaggle Kernels Notebooks](https://cloud.google.com/blog/products/data-analytics/analyze-bigquery-data-with-kaggle-kernels-notebooks) — Kaggle is a sort of social network/sharing platform for data scientists, Google acquired it in 2017, and it’s now [integrated into BigQuery](https://cloud.google.com/blog/products/data-analytics/analyze-bigquery-data-with-kaggle-kernels-notebooks), enabling BigQuery users to use Kaggle’s neat analysis tools. Google Cloud **IN BRIEF:** - Redis 6 is under development and [should be released this December.](https://twitter.com/RedisLabs/status/1144242384189788160) - MongoDB has unveiled [Atlas Data Lake](https://www.mongodb.com/blog/post/mongodb-atlas-data-lake-debuts-at-mongodb-world), a service for querying data stored in Amazon S3 buckets using MongoDB's query language. - EnterpriseDB has been [acquired by a private equity firm.](https://www.prnewswire.com/news-releases/enterprisedb-acquired-by-great-hill-partners-300875303.html) - MongoDB's CEO was on CNBC explaining to Jim Cramer [why MongoDB has the edge over Oracle.](https://www.cnbc.com/video/2019/06/20/mongodb-ceo-explains-why-the-company-has-an-edge-over-oracle-in-database.html) - Oracle has [launched Autonomous Database Dedicated](https://venturebeat.com/2019/06/26/oracle-launches-autonomous-database-dedicated-for-enterprise-customers/), an enterprise-focused service that provides autonomous database infrastructure management. 💻 Jobs [Senior Software Engineer (Santa Barbara or Remote)](https://grnh.se/2cce24422) — Join a team where everyone is striving to constantly improve their knowledge of software development tools, practices, and processes. Invoca [Find a DB Job on Vettery](https://www.vettery.com/tech?utm_source=newsletter&utm_medium=cooper-dbweekly&utm_term=tech&utm_content=grouped&utm_campaign=ad-88878) — Vettery specializes in tech roles and is completely free for job seekers. Vettery 📒 Tutorials and Stories ▶[Advanced NoSQL Data Modeling with Amazon DynamoDB](https://www.youtube.com/watch?v=nhUtZ7suZWI) — The latest in a series of YouTube videos that dig deep into using DynamoDB, Amazon’s scalable NoSQL database. DynamoDB is quite unique in how it works so content like this is valuable if you plan to use it. Amazon Web Services [An Introduction to Hypothetical Indexes in PostgreSQL](https://www.percona.com/blog/2019/06/21/hypothetical-indexes-in-postgresql/) — Why _would_ you want to create imaginary indexes for Postgres’s optimizer to chew over? It’s a way to find out if an index would be useful before you endure the expense of creating a real one. SQL Server and Oracle can [do this too](https://www.mssqltips.com/sqlservertip/3246/sql-server-performance-tuning-with-hypothetical-indexes/). Avinash Vallarapu [Building a Data Stream for IoT with NiFi & InfluxDB](https://www.influxdata.com/blog/building-a-data-stream-for-iot-with-nifi-and-influxdb/?utm_campaign=iot&utm_medium=newsletter&utm_source=cooperpress) — Combining NiFi & InfluxDB results in secure, accessible, and usable IoT data streams. This solution enables a single data view across all facilities providing proactive maintenance, failure detection, and more. InfluxData sponsor [Analyzing the Performance and Cost of Large-Scale Data Processing with AWS Lambda](https://aws.amazon.com/blogs/apn/analyzing-performance-and-cost-of-large-scale-data-processing-with-aws-lambda/) — A serverless approach isn’t applicable for every data analytics use case but the low TCO and flexibility of AWS Lambda has a lot going for it. Amazon Web Services [SQLsmith: Randomized SQL Testing in CockroachDB](https://www.cockroachlabs.com/blog/sqlsmith-randomized-sql-testing/) — Randomized testing lets you automate the discovery of interesting test cases that would be difficult to come up with on your own and CockroachDB has adopted the idea for its ultra-resilient SQL database. Matt Jibson (Cockroach Labs) [MongoDB's Plan to Stop Breaches With Dead Simple Database Encryption](https://www.wired.com/story/field-level-encryption-databases-mongobd/) — MongoDB has been working on a new encryption scheme that should help keep customers’ data more secure. Lily Hay Newman (Wired) [Spring Cleaning at OverOps: How (and Why) We Changed Our DB Cleaning Strategy](https://blog.overops.com/spring-cleaning-at-overops-how-and-why-we-changed-our-db-cleaning-strategy/) — _“after years of writing and executing code, our DB’s free disk space started to run out..”_ Here’s how they addressed the issue. Aviv Danziger [RedisTimeSeries: A Redis Module for Working with Time Series Data](https://redislabs.com/blog/redistimeseries-ga-making-4th-dimension-truly-immersive/) — Redis is already a great fit for time series work but this introduces some cool new features like timed retention policies on streams, downsampling, and integration with other tools. Redis Labs MongoDB user? 🍃 Check out [our MongoDB newsletter](https://mongodb.email/). A **[Cooperpress](https://cooperpress.com/)** publication. [Change your email address](https://dbweekly.com/edit_subscription/*|UID|*) or[Stop getting this newsletter](https://dbweekly.com/leave/*|UID|*) © Cooper Press Ltd · Fairfield Enterprise Centre, Louth, LN11 0LS, United Kingdom