• Stars
    star
    317
  • Rank 127,154 (Top 3 %)
  • Language
    HTML
  • Created over 10 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A simple, functional HTTP client library for F#

Http.fs logo Http.fs

A gloriously functional HTTP client library for F#! NuGet name: Http.fs.

.Net build (AppVeyor): AppVeyor Build status Mono build (Travis CI): Travis Build Status NuGet package: NuGet

How do I use it?

In it's simplest form, this will get you a web page:

open Hopac
open HttpFs.Client

let body =
  Request.createUrl Get "http://somesite.com"
  |> Request.responseAsString
  |> run

printfn "Here's the body: %s" body

To get into the details a bit more, there are two or three steps to getting what you want from a web page/HTTP response.

1 - A Request (an immutable record type) is built up in a Fluent Builder style as follows:

open System.IO
open System.Text
open Hopac
open HttpFs.Client

let pathOf relativePath =
  let here = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location)
  Path.Combine(here, relativePath)

let firstCt, secondCt, thirdCt, fourthCt =
    ContentType.parse "text/plain" |> Option.get,
    ContentType.parse "text/plain" |> Option.get,
    ContentType.create("application", "octet-stream"),
    ContentType.create("image", "gif")

let httpClientWithNoRedirects () =
    let handler = new HttpClientHandler(UseCookies = false)
    handler.AllowAutoRedirect <- false
    let client = new HttpClient(handler)
    client.DefaultRequestHeaders.Clear()
    client

// we can trivially extend request to add convenience functions for common operations
module Request =
    let autoFollowRedirectsDisabled h = 
        { h with httpClient = httpClientWithNoRedirects () }

let request =
    Request.createUrl Post "https://example.com"
    |> Request.queryStringItem "search" "jeebus"
    |> Request.basicAuthentication "myUsername" "myPassword" // UTF8-encoded
    |> Request.setHeader (UserAgent "Chrome or summat")
    |> Request.setHeader (Custom ("X-My-Header", "hi mum"))
    |> Request.autoFollowRedirectsDisabled
    |> Request.cookie (Cookie.create("session", "123", path="/"))
    |> Request.bodyString "This body will make heads turn"
    |> Request.bodyStringEncoded "Check out my sexy foreign body" (Encoding.UTF8)
    |> Request.body (BodyRaw [| 1uy; 2uy; 3uy |])
    |> Request.body (BodyString "this is a greeting from Santa")

    // if you submit a BodyForm, then Http.fs will also set the correct Content-Type, so you don't have to
    |> Request.body (BodyForm 
        [
            // if you only have this in your form, it will be submitted as application/x-www-form-urlencoded
            NameValue ("submit", "Hit Me!")

            // a single file form control, selecting two files from browser
            FormFile ("file", ("file1.txt", ContentType.create("text", "plain"), Plain "Hello World"))
            FormFile ("file", ("file2.txt", ContentType.create("text", "plain"), Binary [|1uy; 2uy; 3uy|]))

            // you can also use MultipartMixed for servers supporting it (this is not the browser-default)
            MultipartMixed ("files",
              [ "file1.txt", firstCt, Plain "Hello World" // => plain
                "file2.gif", secondCt, Plain "Loopy" // => plain
                "file3.gif", thirdCt, Plain "Thus" // => base64
                "cute-cat.gif", fourthCt, Binary (File.ReadAllBytes (pathOf "cat-stare.gif")) // => binary
          ])
    ])
    |> Request.responseCharacterEncoding Encoding.UTF8    
    |> Request.proxy {
          Address = "proxy.com";
          Port = 8080;
          Credentials = Credentials.Custom { username = "Tim"; password = "Password1" } }

(with everything after createRequest being optional)

2 - The Http response (or just the response code/body) is retrieved using one of the following:

job {
  use! response = getResponse request // disposed at the end of async, don't
                                      // fetch outside async body
  // the above doesn't download the response, so you'll have to do that:
  let! bodyStr = Response.readBodyAsString response
  // OR:
  //let! bodyBs = Response.readBodyAsBytes

  // remember HttpFs doesn't buffer the stream (how would we know if we're
  // downloading 3GiB?), so once you use one of the above methods, you can't do it
  // again, but have to buffer/stash it yourself somewhere.
  return bodyStr
}

3 - If you get the full response (another record), you can get things from it like so:

response.StatusCode
response.Body // but prefer the above helper functions
response.ContentLength
response.Cookies.["cookie1"]
response.Headers.[ContentEncoding]
response.Headers.[NonStandard("X-New-Fangled-Header")]

So you can do the old download-multiple-sites-in-parallel thing:

[ "http://news.bbc.co.uk"
  "http://www.wikipedia.com"
  "http://www.stackoverflow.com"]
|> List.map (createRequestSimple Get)
| > List.map (Request.responseAsString) // this takes care to dispose (req, body)
|> Job.conCollect
|> Job.map (printfn "%s")
|> start

If you need direct access to the response stream for some reason (for example to download a large file), you need to write yourself a function and pass it to getResponseStream like so:

open Hopac
open System.IO
open HttpFs.Client

job {
  use! resp = Request.createUrl Get "http://fsharp.org/img/logo.png" |> getResponse
  use fileStream = new FileStream("c:\\bigImage.png", FileMode.Create)
  do! resp.Body.CopyToAsync fileStream
}

Note because some of the request and response headers have the same names, to prevent name clashes, the response versions have 'Response' stuck on the end, e.g.

response.Headers.[ContentTypeResponse]

Building

  1. Download the source code
  2. Execute the build.sh (linux & macos) or build.cmd (windows)

Examples

Check out HttpClient.SampleApplication, which contains a program demonstrating the various functions of the library being used and (to a limited extent) unit tested.

SamplePostApplication shows how you can create a post with a body containing forms.

Version History

Http.fs attempts to follow Semantic Versioning, which defines what the different parts of the version number mean and how they relate to backwards compatability of the API. In a nutshell, as long as the major version doesn't change, everything should still work.

  • 0.X.X - Various. Thanks for code and suggestions from Sergeeeek, rodrigodival, ovatsus and more
  • 1.0.0 - First stable API release. Changed how 'duplicated' DUs were named between request/response.
  • 1.1.0 - Added withProxy, thanks to vasily-kirichenko
  • 1.1.1 - Handles response encoding secified as 'utf8' (.net encoder only likes 'utf-8')
  • 1.1.2 - Added utf16 to response encoding map
  • 1.1.3 - Added XML comments to public functions, made a couple of things private which should always have been (technically a breaking change, but I doubt anybody was using them)
  • 1.2.0 - Added withKeepAlive
  • 1.3.0 - Added getResponseBytes, thanks to Sergeeeek
  • 1.3.1 - Added project logo, thanks to sergey-tihon
  • 1.4.0 - Added getResponseStream, with thanks to xkrt
  • 1.5.0 - Added support for Patch method with help from haf, and xkrt fixed an issue with an empty response.CharacterSet
  • 1.5.1 - Corrected the assembly version
  • 2.0.0 - Production hardened, major release, major improvements
  • 3.0.3 - Async -> Job, withXX -> Request.withXX

FAQ

  • How does it work?

Http.fs currently uses HttpClient under the hood.

  • Does it support proxies?

Yes. By default it uses the proxy settings defined in IE, and as of 1.1.0 you can specify basic proxy settings separately using withProxy.

  • Can I set KeepAlive?

Yes, as of version 1.2.0. This actually sets the Connection header (to 'Keep-Alive' or 'Close'). Note that if this is set to true (which is the default), the Connection header will only be set on the first request, not subsequent ones.

Why?

Simplicity for F# programmers. An abstract, immutable API that you can build better abstractions beneath (if needed).

What other kick-ass open source libraries are involved?

The only thing that's used in the HttpClient module itself is AsyncStreamReader.fs, a source file taken directly from the Fsharpx library.

However, for testing a couple of other things are used:

  • Suave to create a web server for integration testing
  • FsUnit for unit testing
  • NancyFX to create a web server for integration testing

That's about it. Happy requesting!

Henrik Feldt – @haf

Originally built by Grant Crofton.

Post Scriptum

More Repositories

1

expecto

A smooth testing lib for F#. APIs made for humans! Strong testing methodologies for everyone!
F#
640
star
2

DotNetZip.Semverd

A fork of the DotNetZip project without signing with a solution that compiles cleanly. This project aims to follow semver to avoid versioning conflicts. DotNetZip is a FAST, FREE class library and toolset for manipulating zip files. Use VB, C# or any .NET language to easily create, extract, or update zip files.
C#
544
star
3

linux-intro-course

A gentle introduction to programming networked services on linux
F#
111
star
4

EventStore.Client.FSharp

EventStore Client F# API for https://github.com/EventStore/ (Contact hi at qvitoo dot com for an up to date version)
F#
28
star
5

Topshelf.FSharp

F# APIs for Topshelf
F#
26
star
6

Flotten

F# implementation of the Raft consensus algorithm from the paper 'In Search of an Understandable Consensus Algorithm'.
F#
25
star
7

YoLo

Effing utilities.
F#
24
star
8

macos

A complete macOS dev machine bootstrap repository.
Shell
20
star
9

Newtonsoft.Json.FSharp

Nice F# support for Newtonsoft.JSON - tuples as arrays, maps as objects, lists as arrays, unions as _name-metadata annotated arrays, decimals as strings to save precision, options as null/value, string-GUIDs and finally BigInt support. **I recommend using Chiron https://github.com/xyncro/chiron instead of Newtonsoft**
F#
19
star
10

NodaTime

A better date-time library for .Net by Mr Skeet and Mr Colebourne
C#
18
star
11

Castle.Facilities.NHibernate

The NHibernate Facility supercedes NHibernateIntegration and integrates with the new Transaction support. Please read http://stackoverflow.com/questions/4010265/how-to-let-nhibernate-retry-deadlocked-transactions-when-using-session-per-reque before using, it will help you.
C#
16
star
12

AleaGPUTutorial

Tutorial and samples for Alea GPU compiler.
F#
14
star
13

kanban-trello.js

A Greasemonkey plugin that colours the columns of a trello board according to the number of cards and the numbers in square brackets. [4-5] or [3] would both work.
JavaScript
13
star
14

FSharp.EventSource

A F# library of Server-Sent Events, with sample, MIT license and nuget.
JavaScript
12
star
15

System.Data.SQLite

A mirror to the official https://system.data.sqlite.org site which uses Fossil for their SCM, and configured to be buildable on mono.
C
12
star
16

vagrant-vault

A 3-node Vault PoC w/ the Consul backend
HCL
10
star
17

suave-presentation.2015-09-03

Builds a RxJS chat on Suave.IO
JavaScript
8
star
18

puppet-teamcity

Puppet TeamCity server and agent module
Puppet
8
star
19

grok-patterns

A repository where you can develop grok patterns for logstash and other services
Ruby
8
star
20

log4net.Azure

A log4net appender for use with Windows Azure
C#
8
star
21

miTLS

(mirroring) miTLS is a verified reference implementation of the TLS protocol.
F#
7
star
22

QuartzTopshelfSample

Trying out Quartz 2 with Topshelf
Ruby
7
star
23

kubernetes-hello-fsharp

F#
7
star
24

semantic-logging

For 'Semantic Logging' blog entry - the references might have changed since blog entry was published
C#
6
star
25

fpm-recipes

My collection of CentOS recipes.
Ruby
5
star
26

dockerfiles

This is the collection of dockerfiles needed to successfully and in a stable way deploy on docker (with F#, Mono, 12-factor app style)
Shell
5
star
27

MassTransit-AzureServiceBus

Azure Service Bus Transport for MassTransit
C#
4
star
28

Stateless

Mirror Github Repo of Stateless - with added bugfixes
C#
4
star
29

ReceiverSpike

Reorder, persist and integrate event message handlers with a service bus for DDD/CQRS endpoints.
C#
4
star
30

puppet-eventstore

A puppet module for deploying (get) EventStore.
Puppet
4
star
31

functional-programming-patterns

TypeScript
4
star
32

logirel

Release Management ruby gem for .Net
Ruby
3
star
33

mailgun

Mailgun F# wrapper (Not, not failgun :))
F#
3
star
34

openlung-website

The new OpenLung website and progressive web app for spreading knowledge. Parent org https://gitlab.com/open-source-ventilator
TypeScript
2
star
35

optimistic

An optimistic concurrency control/CQRS/React/Redux app
JavaScript
2
star
36

rmq-publisher-confirms-hopac

Showcasing the invariants you'll have to deal with when using RabbitMQ with an HTTP API
F#
2
star
37

puppet-mono

A puppet module that installs mono.
Puppet
2
star
38

puppet-dak

Getting Debian Archive Kit up and running on a debian (Ubuntu 12.04 LTS in this case)
Python
2
star
39

jws-masstransit-scala-java-interop

A bit of hacking, interop between Scala, C# and Java with MassTransit
C#
2
star
40

puppet-mono-src

A puppet module for installing mono from source on CentOS
Puppet
2
star
41

MassTransit.TransportSpecs

A repository for testing all of the MassTransit transport in unison in order to avoid regression bugs and implementations that are not substitutable for one another.
C#
2
star
42

puppet-on-windows

A talk / repo that sets up a Windows dev machine
Ruby
2
star
43

puppet-keepalived

Keepalived module for puppet
Puppet
2
star
44

dotnet-skaffold-spike

Just an example of how to configure Skaffold with .Net Core and SQL Server
C#
2
star
45

sh64

Forwards STDERR, STDOUT and STDIN to your local execution context. This is very useful if you, for example, are calling powershell from a rake script and want to see its output. Then you just call 'sh64 powershell -command &{ write-host "Hello World" }'
C#
1
star
46

EventStore-build

A docker container that builds EventStore, packages it
Shell
1
star
47

Code4Fun.WebShotter

An MassTransit Saga impl for screenshotting web sites as a Service!
JavaScript
1
star
48

EventStore.Rebuilder

A project for re-publishing domain events to a specified service endpoint.
1
star
49

zero-to-hero-talk

https://www.meetup.com/FuncProgSTHLM/events/239753911/
F#
1
star
50

vagrant-eventstore

A vagrant showcase for eventstore running on linux
Shell
1
star
51

puppet-logstash

A puppet LogStash 1.3.x (soon 1.4) module w/ kibana support, running in supervisord. Fork from mediawiki.
Puppet
1
star
52

puppet-fsharp

A puppet module for installing F# on a linux machine.
Puppet
1
star
53

haskell-nextjs-webhooks

Sample PoC for HMAC auth; deriving the action key from the master key
TypeScript
1
star
54

vagrant-fsharp

A quickie to get F# up and spinning
Ruby
1
star
55

fsharp-concurrency

A presentation on concurrency
F#
1
star
56

presentations

C#
1
star
57

vercel-monorepo-with-workspaces

TypeScript
1
star
58

puppet-zookeeper

A puppet module for deploying ZooKeeper on CentOS. Aims to provide: installation, configuration through parametized classes, Ongoing Data Directory Cleanup tasks, monitoring integration for ZK.
Puppet
1
star
59

SmbPingPong

Small utility to test whether SMB is using session semantics on writes in distributed systems
F#
1
star
60

DVar

Close functions over mutable variables for transitive system-wide reconfiguration/load balancing.
F#
1
star
61

Requester

Send a HTTP request from a Windows service at a specified interval. Good if you have a web site that musn't get lazy, or your IIS recycles your app pool every nth minutes. This project is good for that and that only, but the code had to be stored somewhere...
C#
1
star