--- title: "A bumper collection of our favorite data tools" publication: "DB Weekly" issue: 358 published_at: "2021-06-11" canonical_url: "https://dbweekly.com/issues/358" --- #​358 — June 11, 2021 [Read on the Web](https://dbweekly.com/issues/358) ✍️ As part of our winding down towards our hiatus in July, we're looking back at some of the best things we've covered over the past few years. This week it's a bumper roundup of data manipulation, processing, or transformation tools — all of which are still available or actively maintained :-) __ _Peter Cooper, editor_ Database Weekly The Data Processing Tools Special Edition The focus in this issue is on small or single use tools rather than large enterprise-y systems. Things you can run on the Web or on your own machine for slicing and dicing various forms of data or for connecting to and managing the databases and queries you already use. Enjoy! [![\](https://res.cloudinary.com/cpress/image/upload/w_1280,e_sharpen:60/tywvu8ueyclyi1p4anqp.jpg)](https://superintendent.app/) [Superintendent.app: Load CSV Files and Perform SQL Queries on Them](https://superintendent.app/) — The only 'news' item in this issue, but it fits so well.. Superintendent is a new desktop app (for Windows, macOS and Linux) for pulling in CSV files to then query using SQL. The results can then be saved back to CSV, if you wish. Superintendent Team [Datasette: An Open Source 'Multi-Tool' for Exploring and Publishing Data](https://datasette.io/) — A tool aimed at anyone who has data they wish to share with the world (e.g. data journalists, curators, archivists..) This has become a mature and popular tool over the years and is well worth exploring. Simon's [Personal Data Warehouses talk](https://simonwillison.net/2020/Nov/14/personal-data-warehouses/) is also worth a watch. Simon Willison [![\](https://copm.s3.amazonaws.com/5ce03523.jpg)](https://fauna.com/go/serverless?utm_source=DBWeekly&utm_medium=Newsletter&utm_campaign=CooperPress_Newsletters_2021) [Fauna - A Flexible, Developer-Friendly, Serverless Database](https://fauna.com/go/serverless?utm_source=DBWeekly&utm_medium=Newsletter&utm_campaign=CooperPress_Newsletters_2021) — Fauna is the “Data API” for modern applications. Whether you’re building new microservices or augmenting existing services applications, Fauna lets you simplify code, reduce costs and ship faster. [Learn more...](https://fauna.com/go/serverless?utm_source=DBWeekly&utm_medium=Newsletter&utm_campaign=CooperPress_Newsletters_2021) Fauna sponsor [SQL Fiddle: A Tool for Quick Testing of Database Queries](http://sqlfiddle.com/) — Set up your schema (which can be empty) on the left then issue queries on the right. Supports specific (slightly old) versions of MySQL, Oracle, Postgres, and SQL Server. [DB Fiddle](https://www.db-fiddle.com/) is a similar tool covering newer versions of MySQL and Postgres. ZZZ Projects [jOOQ SQL Translation: Translate SQL From One Dialect to Another](https://www.jooq.org/translate/) — While SQL has been standardized numerous times, its implementation does tend to vary, so this tool continues to provide an interesting way to see the differences on your own queries. I love it. Lukas Eder [SQLPad: A Web-Based SQL Editor to Run in Your Private Cloud](https://sqlpad.github.io/sqlpad/#/) — SQLPad supports a vast array of systems from standard RDBMS to things like BigQuery, Cassandra, and Apache Pinot. Rick Bergfalk [![\](https://res.cloudinary.com/cpress/image/upload/w_1280,e_sharpen:60/jztlv2no5zrxugrdgwpp.jpg)](https://github.com/saulpw/visidata) [Visidata 2: A Terminal Spreadsheet Multitool for Discovering and Arranging Data](https://github.com/saulpw/visidata) — Here’s [an introductory tutorial](https://jsvine.github.io/intro-to-visidata/index.html) to what you can use this for. Saul Pwanson [Fx: A Command-Line JSON Processing Tool](https://github.com/antonmedv/fx) — If you’ve got some files full of JSON that you want to process, Fx will slice and dice it however you want, including using JavaScript one-liners to add a bit of logic to the process. I’ve used this a lot. Anton Medvedev [sqlfmt: An Opinionated Online SQL Formatter](https://sqlfum.pt/) — Provide sqlfmt with some SQL, specify a target line width, and get back some better formatted SQL that takes SQL’s syntax into account. Ideal for blog posts or documentation. Matt Jibson [Avoiding Integer Overflows with Zero Downtime at Buildkite](https://buildkite.com/blog/avoiding-integer-overflows-with-zero-downtime?utm_campaign=CPress&utm_source=ppc&utm_medium=cp-database&utm_term=continuous-integration&utm_content=integer-overflow-blog) — Learn how Buildkite migrated over 2 billion rows in one of our largest tables. Buildkite sponsor [Unix Command Line Tricks For Data Scientists](http://kadekillary.work/post/cli-4-ds/) — OK, we’re kinda cheating here, but Unix favorites like sort, split, head, grep, sed and awk make up a serious part of my own data processing arsenal at least.. Kade Killary [A List of CLI Tools for Manipulating Structured Text Data](https://github.com/dbohdan/structured-text-tools) — A list of around 80 tools for working with DSV/CSV, XML, HTML, JSON, YAML, INI and other formats. Danyil Bohdan [usql: A Universal CLI Tool for Databases](https://github.com/xo/usql) — A CLI tool (written in Go) for working with Postgres, SQL Server, MySQL, SQLite3, Oracle Database, CockroachDB, and [many more.](https://github.com/xo/usql#database-support) A database Swiss Army knife, if you will. Kenneth Shaw [Mockaroo: A Random Data Generator](https://www.mockaroo.com/) — Specify your schema and get back ‘realistic’ test data in CSV, JSON, SQL or Excel format. You get up to 1000 rows of generation for free. [GenerateData.com](https://www.generatedata.com/) is a similar tool in this space. Mockaroo, LLC [SQLSmith: A Random SQL Query Generator](https://github.com/anse1/sqlsmith) — It hasn’t been updated in a few years, but this tool went beyond merely being interesting to actually discovering bugs in real database systems. [SQLancer](https://github.com/sqlancer/sqlancer) is another, newer tool in this space. Andreas Seltenreich Jobs [Find Data Engineering Jobs with Hired](https://hired.com/join/?utm_source=newsletter&utm_medium=sponsor&utm_campaign=%28b2c%29%28l-all%29%28r-all%29%28cooper-dbweekly%29&utm_content=5-minutes) — Take 5 minutes to build your free profile & start getting interviews for your next job. Companies on Hired are actively hiring right now. Hired [![\](https://res.cloudinary.com/cpress/image/upload/w_1280,e_sharpen:60/dqzgjvogtbzayvicgbbw.jpg)](https://dbdiagram.io/home) [DBDiagram: A Database Designer for Developers and Analysts](https://dbdiagram.io/home) — A Web-based tool to help you draw database relationship diagrams and flows quickly using a simple markup language/DSL. Holistics Software [dadbod.vim: A Modern Database Interface for Vim](https://github.com/tpope/vim-dadbod) — A Vim plugin for interacting with numerous databases, including Postgres. Tim Pope [q: Run SQL on CSV or TSV Files](https://harelba.github.io/q/) — A perennially useful utility. Harel Ben-Attia [xsv: A Fast CSV Command Line Toolkit Written in Rust](https://github.com/BurntSushi/xsv) — Another ‘Swiss Army knife’ for your slightly structured data. Andrew Gallant [Dbmate: A Lightweight, Framework-Agnostic Database Migration Tool](https://github.com/amacneil/dbmate) — Written in Go but can be used alongside database-using apps written in any language. Supports MySQL, Postgres, SQLite and ClickHouse. Adrian Macneil [dbcrossbar: Move Large Datasets Between Different Databases and Formats](http://www.dbcrossbar.org/) — Copy tabular data between databases, CSV files and cloud storage. Written in Rust. Faraday, Inc. [PostgREST: Serve a RESTful API from a Postgres Database](https://postgrest.org/en/stable/) — We’ve covered this popular tool numerous times in [Postgres Weekly](https://postgresweekly.com/) but never here.. It’s a standalone app that turns a PostgreSQL database into a RESTful API. Prefer GraphQL? [PostGraphile](https://www.graphile.org/) does similar things in that area. Joe Nelson and Steve Chavez [gron: Make JSON Greppable](https://github.com/tomnomnom/gron) — A tool written in Go that transforms JSON into more easily `grep`pable assignments, so you can use `grep` and see the context/path of the result. Useful. Tom Hudson A **[Cooperpress](https://cooperpress.com/)** publication. [Change your email address](https://dbweekly.com/edit_subscription/*|UID|*) or[Stop getting this newsletter](https://dbweekly.com/leave/*|UID|*) © Cooper Press Ltd · Fairfield Enterprise Centre, Louth, LN11 0LS, United Kingdom