• Stars
    star
    150
  • Rank 247,323 (Top 5 %)
  • Language
    Scala
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Add efficient pipelining, batching, and caching to any data source

ZIO Query

ZIO Query is a library for writing optimized queries to data sources in a high-level compositional style. It can add efficient pipelining, batching, and caching to any data source. ZIO Query helps us dramatically reduce load on data sources and improve performance.

Development CI Badge Sonatype Releases Sonatype Snapshots javadoc ZIO Query

Introduction

Some key features of ZIO Query:

  • Batching β€” ZIO Query detects parts of composite queries that can be executed in parallel without changing the semantics of the query.
  • Pipelining β€” ZIO Query detects parts of composite queries that can be combined together for fewer individual requests to the data source.
  • Caching β€” ZIO Query can transparently cache read queries to minimize the cost of fetching the same item repeatedly in the scope of a query.

Compared with Fetch, ZIO Query supports response types that depend on request types, does not require higher-kinded types and implicits, supports ZIO environment and statically typed errors, and has no dependencies except for ZIO.

A ZQuery[R, E, A] is a purely functional description of an effectual query that may contain requests from one or more data sources, requires an environment R, and may fail with an E or succeed with an A.

Requests that can be performed in parallel, as expressed by zipWithPar and combinators derived from it, will automatically be batched. Requests that must be performed sequentially, as expressed by zipWith and combinators derived from it, will automatically be pipelined. This allows for aggressive data source specific optimizations. Requests can also be deduplicated and cached.

This allows for writing queries in a high level, compositional style, with confidence that they will automatically be optimized. For example, consider the following query from a user service.

Assume we have the following database access layer APIs:

def getAllUserIds: ZIO[Any, Nothing, List[Int]] = {
  // Get all user IDs e.g. SELECT id FROM users
  ZIO.succeed(???)
}

def getUserNameById(id: Int): ZIO[Any, Nothing, String] = {
  // Get user by ID e.g. SELECT name FROM users WHERE id = $id
  ZIO.succeed(???)
}

We can get their corresponding usernames from the database by the following code snippet:

val userNames = for {
  ids   <- getAllUserIds
  names <- ZIO.foreachPar(ids)(getUserNameById)
} yield names

It works, but this is not performant. It is going to query the underlying database N + 1 times, one for getAllUserIds and one for each call to getUserNameById.

In contrast, ZQuery will automatically optimize this to two queries, one for userIds and one for userNames:

lazy val getAllUserIds: ZQuery[Any, Nothing, List[Int]]    = ???
def getUserNameById(id: Int): ZQuery[Any, Nothing, String] = ???

lazy val userQuery: ZQuery[Any, Nothing, List[String]] = for {
  userIds   <- getAllUserIds
  userNames <- ZQuery.foreachPar(userIds)(getUserNameById)
} yield userNames

Installation

In order to use this library, we need to add the following line in our build.sbt file:

libraryDependencies += "dev.zio" %% "zio-query" % "0.4.0"

Example

Here is an example of using ZIO Query, which optimizes multiple database queries by batching all of them in one query:

import zio._
import zio.query._

object ZQueryExample extends ZIOAppDefault {
  case class GetUserName(id: Int) extends Request[Nothing, String]

  lazy val UserDataSource: DataSource.Batched[Any, GetUserName] =
    new DataSource.Batched[Any, GetUserName] {
      val identifier: String = "UserDataSource"

      def run(requests: Chunk[GetUserName])(implicit trace: Trace): ZIO[Any, Nothing, CompletedRequestMap] = {
        val resultMap = CompletedRequestMap.empty
        requests.toList match {
          case request :: Nil =>
            val result: Task[String] = {
              // get user by ID e.g. SELECT name FROM users WHERE id = $id
              ZIO.succeed(???)
            }

            result.either.map(resultMap.insert(request))

          case batch: Seq[GetUserName] =>
            val result: Task[List[(Int, String)]] = {
              // get multiple users at once e.g. SELECT id, name FROM users WHERE id IN ($ids)
              ZIO.succeed(???)
            }

            result.fold(
              err =>
                requests.foldLeft(resultMap) { case (map, req) =>
                  map.insert(req)(Left(err))
                },
              _.foldLeft(resultMap) { case (map, (id, name)) =>
                map.insert(GetUserName(id))(Right(name))
              }
            )
        }
      }

    }

  def getUserNameById(id: Int): ZQuery[Any, Nothing, String] =
    ZQuery.fromRequest(GetUserName(id))(UserDataSource)

  val query: ZQuery[Any, Nothing, List[String]] =
    for {
      ids <- ZQuery.succeed(1 to 10)
      names <- ZQuery.foreachPar(ids)(id => getUserNameById(id)).map(_.toList)
    } yield (names)

  def run = query.run.tap(usernames => Console.printLine(s"Usernames: $usernames"))
}

Resources

Documentation

Learn more on the ZIO Query homepage!

Contributing

For the general guidelines, see ZIO contributor's guide.

Code of Conduct

See the Code of Conduct

Support

Come chat with us on Badge-Discord.

License

License

More Repositories

1

zio

ZIO β€” A type-safe, composable library for async and concurrent programming in Scala
Scala
4,077
star
2

zio-quill

Compile-time Language Integrated Queries for Scala
Scala
2,153
star
3

zio-http

A next-generation Scala framework for building scalable, correct, and efficient HTTP clients and servers
Scala
785
star
4

zio-prelude

A lightweight, distinctly Scala take on functional abstractions, with tight ZIO integration
Scala
451
star
5

zio-json

Fast, secure JSON library with tight ZIO integration.
Scala
406
star
6

zio-kafka

A Kafka client for ZIO and ZIO Streams
Scala
336
star
7

zio-actors

A high-performance, purely-functional library for building, composing, and supervising typed actors based on ZIO
Scala
268
star
8

zio-sql

Type-safe, composable SQL for ZIO applications
Scala
236
star
9

zio-config

Easily use and document any config from anywhere in ZIO apps
Scala
231
star
10

zio-intellij

A companion IntelliJ IDEA plugin for the ZIO library ecosystem.
Scala
212
star
11

zio-protoquill

Quill for Scala 3
Scala
205
star
12

zio-keeper

A ZIO library for building distributed systems
Scala
199
star
13

zio-nio

A small, unopinionated ZIO interface to NIO.
Scala
187
star
14

zio-logging

Powerful logging for ZIO 2.0 applications, with compatibility with many logging backends out-of-the-box.
Scala
174
star
15

zio-akka-cluster

ZIO wrapper for Akka Cluster
Scala
164
star
16

interop-cats

ZIO instances for cats-effect type classes
Scala
157
star
17

zio-direct

Direct-Style Programming for ZIO
Scala
154
star
18

zio-flow

Resilient, distributed applications powered by ZIO
Scala
142
star
19

zio-schema

Compositional, type-safe schema definitions, which enable auto-derivation of codecs and migrations.
Scala
141
star
20

izumi-reflect

TypeTag without scala-reflect. Supports Scala 2 and Scala 3.
Scala
140
star
21

zio-aws

Low level ZIO interface for the full AWS
Scala
139
star
22

zio-cli

Rapidly build powerful command-line applications powered by ZIO
Scala
130
star
23

zio-redis

A ZIO-based redis client
Scala
123
star
24

zio-telemetry

ZIO-powered OpenTelemetry library
Scala
112
star
25

zio-petclinic

An idiomatic pet clinic application written with ZIO.
Scala
103
star
26

zio-cache

A ZIO native cache with a simple and compositional interface
Scala
85
star
27

zio-zmx

Monitoring, Metrics and Diagnostics for ZIO
Scala
85
star
28

zio-shield

Enforce best coding practices with ZIO
Scala
82
star
29

zio-jdbc

A small, idiomatic ZIO interface to JDBC.
Scala
82
star
30

zio-sqs

ZIO-powered client for AWS SQS
Scala
82
star
31

zio-openai

Scala
66
star
32

zio-process

A simple ZIO library for interacting with external processes and command-line programs
Scala
64
star
33

zio-quickstarts

A minimal quickstart ZIO application for writing a RESTful Web Service
Scala
62
star
34

zio-analytics

Distributed stream processing using ZIO
Scala
61
star
35

zio-optics

Easily modify parts of larger data structures
Scala
57
star
36

zio-metrics-legacy

⛔️ DEPRECATED
Scala
56
star
37

interop-reactive-streams

Interoperability Layer Between ZIO and Reactive Streams
Scala
47
star
38

zio-s3

An S3 client for ZIO
Scala
44
star
39

zio-lambda

AWS Lambda Runtime built with ZIO
Scala
38
star
40

zio-dynamodb

Simple, type-safe, and efficient access to DynamoDB
Scala
37
star
41

zio-amqp

ZIO-based AMQP client for Scala
Scala
33
star
42

zio-macros

Macros to scrap ZIO boilerplate
Scala
32
star
43

zio-rocksdb

A ZIO-based interface to RocksDB.
Scala
32
star
44

zio-connect

Sources, Sinks and Pipelines for channeling data
Scala
32
star
45

zio-crypto

Fast, secure cryptographic primitives in a ZIO & ZIO Streams friendly package.
Scala
30
star
46

zio-metrics-connectors

Monitoring, Metrics and Diagnostics for ZIO
Scala
30
star
47

zio-ftp

A simple, idiomatic (S)FTP client for ZIO
Scala
29
star
48

zio-parser

Scala
28
star
49

zio-mock

Scala
27
star
50

zio-constraintless

An advanced library for building DSLs that allows defering the existence of type class instances until interpretation.
Scala
26
star
51

zio-codec

High-performance codecs for ZIO applications
Scala
23
star
52

zio-project-seed.g8

giter8 template used to start new ZIO projects for the ZIO organization
Scala
21
star
53

zio-webhooks

A microlibrary for reliable and persistent webhook delivery
Scala
20
star
54

zio-profiling

Scala
19
star
55

zio-gcp

A ZIO-based interface to Google Cloud API
Scala
18
star
56

zio-insight-ui

TypeScript
17
star
57

interop-java

Scala
17
star
58

caliban-deriving

Full-featured, robust deriving for Caliban.
Scala
16
star
59

zio-test-intellij

An optional ZIO Test runner support module for the ZIO IntelliJ plugin
Shell
16
star
60

zio-insight

Toolset for ZIO developers
Scala
15
star
61

zio-delegate

Scala
14
star
62

zio-wasm

WASM AST and syntax based on zio-parser.
Scala
14
star
63

interop-twitter

Scala
13
star
64

zio-sbt

SBT Plugins For ZIO Projects
Scala
12
star
65

zio-deriving

Scala
11
star
66

zio-memberlist

Cluster membership and failure detection
Scala
11
star
67

interop-monix

Scala
8
star
68

zio-spark

A simple, type-safe ZIO interface to Spark
Scala
8
star
69

interop-scalaz

Scala
7
star
70

interop-guava

Scala
6
star
71

zio-concurrent

Concurrency utilities for ZIO.
JavaScript
6
star
72

zio-morphir

Scala
5
star
73

zio-uring

Scala
5
star
74

interop-future

Scala
5
star
75

zio-quickstart-graphql-webservice

Quickstart for Writing GraphQL Servers
Scala
4
star
76

zio-bson

BSON library with tight ZIO integration
Scala
4
star
77

zio-cron

ZIO Cron
3
star
78

zio-simple-seed.g8

Scala
3
star
79

zio-meta

Scala
2
star
80

zio-direct-intellij

Scala
2
star
81

zio-insight-server

Scala
2
star
82

zio-quickstart-hello-world

A minimal working example of ZIO Application
Scala
2
star
83

zio-distributed

The future home of ZIO Distributed!
Scala
1
star
84

zio-docs

Experimenting with a new way of maintaining dev.zio
JavaScript
1
star