• Stars
    star
    20
  • Rank 1,121,974 (Top 23 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 8 years ago
  • Updated almost 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Datastore Connectivity for BigQuery in go

Datastore Connectivity for BigQuery (bgc)

Datastore Connectivity library for BigQuery in Go. GoDoc

This library is compatible with Go 1.5+

Please refer to CHANGELOG.md if you encounter breaking changes.

This library uses SQL mode and streaming API to insert data as default. To use legacy SQL please use the following /* USE LEGACY SQL */ hint, in this case you will not be able to fetch repeated and nested fields.

Configuration parameters

insertMethod

To control insert method just provide config.parameters with the following value:

_table_name_.insertMethod = "load"

Note that if streaming is used, currently UPDATE and DELETE statements are not supported.

insertIdColumn

For streaming you can specify which column to use as insertId with the following config.params

_table_name_.insertMethod = "stream"
_table_name_.insertIdColumn = "sessionId"
streamBatchCount

streamBatchCount controls row count in batch (default 9999)

insertWaitTimeoutInMs

When inserting data data this library checks upto 60 sec if data has been added. To control this behaviour you can set insertWaitTimeoutInMs (default 60 sec)

To disable this mechanism set: insertWaitTimeoutInMs: -1

insertMaxRetires

Retries insert when 503 internal error

datasetId

Default dataset

pageSize

Default 500

The maximum number of rows of data to return per page of results. In addition to this limit, responses are also limited to 10 MB.

Credentials

  1. Google secrets for service account

a) set GOOGLE_APPLICATION_CREDENTIALS environment variable

b) credential can be a name with extension of the JSON secret file placed into ~/.secret/ folder

config.yaml

driverName: bigquery
credentials: bq # place your big query secret json to ~/.secret/bg.json
parameters:
  datasetId: myDataset

c) full URL to secret file

config.yaml

driverName: bigquery
credentials: file://tmp/secret/mySecret.json
parameters:
  datasetId: myDataset

Secret file has to specify the following attributes:

type Config struct {
	//google cloud credential
	ClientEmail  string `json:"client_email,omitempty"`
	TokenURL     string `json:"token_uri,omitempty"`
	PrivateKey   string `json:"private_key,omitempty"`
	PrivateKeyID string `json:"private_key_id,omitempty"`
	ProjectID  string `json:"project_id,omitempty"`
}
  1. Private key (pem)

config.yaml

driverName: bigquery
credentials: bq # place your big query secret json to ~/.secret/bg.json
parameters:
  serviceAccountId: "***@developer.gserviceaccount.com"
  datasetId: MyDataset
  projectId: spheric-arcadia-98015
  privateKeyPath: /tmp/secret/bq.pem

Usage:

The following is a very simple example of Reading and Inserting data

package main

import (
    "github.com/viant/bgc"
    "github.com/viant/dsc"
    "time"
    "fmt"
    "log"
)


type MostLikedCity struct {
	City      string
	Visits    int
	Souvenirs []string
}

type  Traveler struct {
	Id            int
	Name          string
	LastVisitTime time.Time
	Achievements  []string
	MostLikedCity MostLikedCity
	VisitedCities []struct {
		City   string
		Visits int
	}
}


func main() {

    config, err := dsc.NewConfigWithParameters("bigquery", "",
    	    "bq", // google cloud secret placed in ~/.secret/bg.json
            map[string]string{
                "datasetId":"MyDataset",
            })

    if err != nil {
        log.Fatal(err)
    }

		
    factory := dsc.NewManagerFactory()
    manager, err := factory.Create(config)
    if err != nil {
        log.Fatalf("Failed to create manager %v", err)
    }
   

    traveler := Traveler{}
    success, err := manager.ReadSingle(&traveler, " SELECT id, name, lastVisitTime, visitedCities, achievements, mostLikedCity FROM travelers WHERE id = ?", []interface{}{4}, nil)
    if err != nil {
        panic(err.Error())
    }

    travelers :=  make([]Traveler, 0)
    err:= manager.ReadAll(&interest, "SELECT iid, name, lastVisitTime, visitedCities, achievements, mostLikedCity",nil, nil)
    if err != nil {
        panic(err.Error())
    }

   // ...

    inserted, updated, err := manager.PersistAll(&travelers, "travelers", nil)
    if err != nil {
           panic(err.Error())
    }
    // ...
    




   //Custom reading handler with reading query info type to get CacheHit, TotalRows, TotalBytesProcessed

   var resultInfo = &bgc.QueryResultInfo{}
   var perf = make(map[string]int)  
   	err = manager.ReadAllWithHandler(`SELECT DATE(date), COUNT(*) FROM performance_agg WHERE DATE(date) = ?  GROUP BY 1`, []interface{}{
   		"2018-05-03",
   		resultInfo,
   	}, func(scanner dsc.Scanner) (toContinue bool, err error) {
   	        var date string
   	        var count int
   	        err = scanner.Scan(&date, &count)
   	        if err != nil {
   	        	return false, err
   	        }
   	        perf[date] = count
   		return true, nil
   	})
   	log.Printf("cache: %v,  rows: %v, bytes: %v", resultInfo.CacheHit, resultInfo.TotalRows, resultInfo.TotalBytesProcessed)

   
    dialect := dsc.GetDatastoreDialect(config.DriverName)
    DDL, err := dialect.ShowCreateTable(manager, "performance_agg")
    fmt.Printf("%v %v\n", DDL, err)
   
}

GoCover

GoCover

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Credits and Acknowledgements

Library Author: Adrian Witas

Contributors: Mikhail Berlyant

More Repositories

1

afs

Abstract File Storage
Go
301
star
2

endly

End to end functional test and automation framework
Go
262
star
3

toolbox

Toolbox - go utility library
Go
196
star
4

bqtail

BigQuery Google Storage Based Data Loader
Go
49
star
5

dsunit

Datastore Testibility
Go
43
star
6

ptrie

A prefix tree implementation in go
Go
39
star
7

xunsafe

Faster golang reflection
Go
29
star
8

dsc

Datastore Connectivity in go
Go
27
star
9

dbsync

RDBMS synchronizer
Go
22
star
10

gohessian

Hessian serializer written for Go
Go
15
star
11

vec

Vectorization toolkit for golang
Assembly
12
star
12

afsc

Abstract File Storage Connectors
Go
11
star
13

assertly

Arbitraty datastructure validation
Go
10
star
14

asc

Datastore Connectivity for Aerospike for go
Go
9
star
15

bintly

super fast binary serialization for go
Go
9
star
16

smirror

Serverless cloud storage mirror
Go
9
star
17

datly

Go
8
star
18

bigquery

BigQuery database/sql golang driver
Go
8
star
19

CacheStore

A key-value hybrid storage system with a powerful API that utilizes both memory cache and disk to optimize performance.
Java
7
star
20

neatly

Neatly - a neat format for representing nested structured data.
Go
5
star
21

gmetric

Operation metric for go
Go
5
star
22

scache

Cache in go
Go
5
star
23

parsly

Parsing utility (lexer/tokenizer)
Go
5
star
24

etly

ETL framework
Go
4
star
25

mmcb

Memory mapped compactable buffer in go
Go
4
star
26

jdsunit

Java Client for Dsunit server
Java
4
star
27

velty

Template language for go
Go
4
star
28

bqwt

BigQuery Windowed Tables
Go
4
star
29

bitsy

Bitset data indexer
Go
4
star
30

igo

Go evaluator in go
Go
3
star
31

cloudless

Serverless toolbox
Go
3
star
32

mly

Machine Learning executor on steroid for golang
Go
2
star
33

voldemort-storage

HTML
2
star
34

gtly

Generic data stracture for go
Go
2
star
35

tapper

High performant transaction logger
Go
2
star
36

hessian-sm

HTML
2
star
37

rta

Real Time Aggregation for golang
Go
2
star
38

structql

Go struct query
Go
1
star
39

drone-gcloud

Drone plugin to execute gcloud and gsutil commands
Shell
1
star
40

drone-json

Drone plugin to validate JSON files
Python
1
star
41

CacheStore-deploy-cluster

A ready to use deployment of CacheStore Cluster. Includes CacheStore, cluster start up scripts, and cluster configurations.
Shell
1
star
42

scy

Scy - secure store api for golang
Go
1
star
43

sqlx

Go lang sql extensions
Go
1
star
44

CacheStore-deploy

A ready to use deployment of CacheStore remote (default). Includes CacheStore, start up scripts for remote, and remote configurations.
Shell
1
star