Throttler
Throttle the throughput of function calls and core.async channels.
Uses the token bucket algorithm to control both the overall rate as well as the burst rate.
Some of its key features:
- Control both average rate and burstiness.
- Accurate over a large range of rates.
- Lightweight. Based on core.async, does not rely on
Thread/sleepso each throttler does not require a dedicated thread. Use as many as you want. - Throttle a single function/channel or a group under the same rate using statistical multiplexing.
Get it
Leiningen
[throttler "1.0.1"]Gradle
compile "throttler:throttler:1.0.1"Throttling functions
Import throttler.core:
(require '[throttler.core :refer [throttle-chan throttle-fn]])Create a throttled function. Here, we create a slower + that runs at 100
calls/second:
(def +# (throttle-fn + 100 :second))
(+# 1 1) ; => 2
(time (dotimes [_ 300] (+# 1 1))) ; "Elapsed time: 3399.865 msecs"You can also create a bursty function by optionally supplying a burst size. Let's create a bursty multiply with an average rate of 100 calls/s and a burst size of 999 calls:
; goal: 100 calls/s on avg, bursts of up to 1000 calls/s
(def *# (throttle-fn * 100 :second 999))
(*# 2 3) ; => 6
; First 1000 calls go through unthrottled, the 1001th needs to wait for about a second
(time (dotimes [_ 1001] (*# 2 3))) ; "Elapsed time: 1125.117 msecs" (889/second)
; Over a large-ish number of invocations, the average rate is close to the goal
(time (dotimes [_ 10000] (*# 2 3))) ; "Elapsed time: 105041.395 msecs" (95/second)Throttling channels
For channels, simply swap throttle-fn with throttle-chan. It takes an
input channel that should be written to, and returns a throttled output channel
that should be read from:
(def in (chan 1))
(def slow-chan (throttle-chan in 1 :millisecond)) ; 1 msg/ms
(>!! in :token) ; => true
(<!! slow-chan) ; :tokenThis is functionally equivalent to clojure.core.async/pipe, except with a
controlled rate.
Here, we spin up a go thread that writes 5,000 messages to the in channel and
then closes it. On the main thread, we read from the throttled channel until we
get nil, which means the input channel was closed.
(go
(dotimes [_ 5000]
(>! in "hi!"))
(close! in))
(time (while (not (nil? (<!! slow-chan))))) ; "Elapsed time: 5686.198 msecs" (0.9 msg/millisecond)So it took 5.6 seconds to read 5000 messages, resulting in a rate of ~ 0.9 messages/millisecond.
Throttle many functions or channels under the same shared rate
With chan-throttler or fn-throttler you can limit the combined rate
of a group of channels or functions, respectively.
Say for instance you want to use a web API, but you want to limit the number of calls you make to avoid going beyond your currently paid plan. The API has 3 methods, and you don't want the sum of all calls to go over 1000 on a single day.
A naive approach would be to assign 1/3rd of the total rate to each method (or some other pre-fixed proportion). But what if you don't know in advance what the proportion will be, or if it's likely to change over time? In that case you'd be overthrottling some methods and not taking full advantage of your calling capacity.
We can do better. To throttle all 3 methods under the same combined rate, we create a function throttler:
(def api-throttler (fn-throttler 1000 :day))Then we wrap all three API methods with the same api-throttler:
(require [some.api :as api])
(def f1-slow (api-throttler api/f1))
(def f2-slow (api-throttler api/f2))
(def f3-slow (api-throttler api/f3))Now all {f1,f2,f3}-slow will honor the global rate of 1000 calls/day
whatever the calling ratio among the three methods is.
The same can be done for channels, using chan-throttler.
Throughput accuracy over a wide range of rates
While motivated by throttling to about 1-1000 messages/second, Throttler is reasonably accurate over a wide range of rates. High rates are accurate with an error margin of ~10% until we reach core.async's maximum possible pipe throughput. On my laptop, this happens at about 50,000 messages/second.
Here's the result of running some Criterium benchmarks on channels throttled at different rates.
Goal rate Observed rate (mean) Lower quantile (2.5%) Upper quantile (95.5%)
--------------------- -------------------- --------------------- ---------------------
0.1 msg/s 0.1010 msgs/s 0.1008 msgs/s 0.1016 msgs/s
1 msg/s 1.071 msgs/s 1.056 msgs/s 1.126 msgs/s
10 msgs/s 10.77 msgs/s 10.67 msgs/s 11.33 msgs/s
100 msgs/s 91.6 msgs/s 90.49 msgs/s 93.93 msgs/s
1,000 msgs/s 892.2 msgs/s 886.0 msgs/s 907.5 msgs/s
10,000 msgs/s 8,939 msgs/s 8,819 msgs/s 9,062 msgs/s
30,000 msgs/s 26,571 msgs/s 25,970 msgs/s 27,184 msgs/s
100,000 msgs/s 49,958 msgs/s 47,602 msgs/s 50,981 msgs/s
β msgs/s (raw pipe) 48,657 msgs/s 47,663 msgs/s 49,550 msgs/s
The error stays below and at about 10% until we get close to the theoretical maximum, which is the speed at which we can pipe messages through channels.
Same numbers apply to throttle-fn, as the implementation uses throttle-chan
under the hood.
More
Browse the API Docs or check out the blog post for more.
License
Copyright Β© 2014 Bruno Vecchi
Distributed under the Eclipse Public License, the same as Clojure.