diff --git a/blog.cabal b/blog.cabal
index e1bf90c310b32f0086600536804d8d23c9e67b79..b5b685aca5e52c6380c7a0f284d0b23aef0ed78e 100644
--- a/blog.cabal
+++ b/blog.cabal
@@ -17,7 +17,9 @@ executable blog
default-language: Haskell2010
ghc-options: -threaded
build-depends: base == 4.*,
+ lens,
hakyll == 4.7.*,
+ hakyll-serve,
filepath == 1.4.*,
split == 0.2.*,
random,
@@ -33,10 +35,12 @@ executable server
build-depends: base == 4.*,
safe == 0.3.*,
text,
+ lens,
bytestring == 0.10.*,
+ hakyll-serve,
warp == 3.*,
warp-tls == 3.*,
wai == 3.*,
- wai-extra <= 3.0.14,
+ wai-extra >= 3.0.14,
wai-app-static == 3.*,
streaming-commons == 0.1.*
diff --git a/images/posts/haskell-containers.png b/images/posts/haskell-containers.png
new file mode 100644
index 0000000000000000000000000000000000000000..04fef08ce3e7b1adaf7dc38ec8e52495451cff4f
Binary files /dev/null and b/images/posts/haskell-containers.png differ
diff --git a/posts/2016-04-20-haskell-travis-docker.md b/posts/2016-04-20-haskell-travis-docker.md
new file mode 100644
index 0000000000000000000000000000000000000000..a4b50c5f3af2db139612b50f5746bfb52ee91a8c
--- /dev/null
+++ b/posts/2016-04-20-haskell-travis-docker.md
@@ -0,0 +1,161 @@
+---
+title: Continuous integration for Haskell projects with Docker
+date: Wed April 20 14:21:00 EDT 2016
+author: Eduardo Trujillo
+uuid: ccdac18a-4592-4560-9505-8940bb69df46
+---
+
+At my current job, we are starting to adopt Haskell to write some of our
+backend APIs and, like other projects at the company, we are using Docker for
+deploying them to staging and production environments.
+
+Working with Haskell required some re-thinking on how we do certain parts of
+our workflow, but it also led to a much improved image size, with some extra
+room for improvement.
+
+
+
+Unlike PHP or JavaScript, Haskell is a compiled language, meaning that some of
+previous approaches we were using for building Docker images did not port that
+well.
+
+For dynamic languages, we followed roughly these steps to build our images:
+
+- Use a base image with packages for the software we need (Debian, Alpine).
+- Install server software, such as Nginx, Node.js, and PHP.
+- Install language tools and package managers.
+- Copy application sources.
+- Download dependencies.
+- Final optimizations and cleanup.
+
+This is a hybrid approach where we have both build and runtime dependencies on
+the same Docker image, which is useful because image builders such as Quay.io
+can automatically build the docker image for you on every commit without the
+need of an additional step in the CI pipeline, and this also has the slight
+advantage of having enough tooling inside a container for light debugging.
+
+As a result, the image size is not the smallest it could be since there is a
+lot of things that are not commonly used during runtime. However, the added
+convenience sort of out weighs the issues of having a larger image for these
+cases.
+
+For Haskell projects, though, things are bit different, and not in a bad way:
+
+The community has made an excellent build tool called [Stack][3]. `stack` takes
+care of mostly everything related to setting up your project: Installing GHC,
+pulling dependencies, building, testing, coverage reports, documentation. When
+paired with Nix, it can even pull non-Haskell dependencies for reproducible
+builds.
+
+<div class="callout-quote">
+Stack takes care of mostly everything related to setting up your project.
+</div>
+
+If we try to do a hybrid image approach like above using Stack, we mainly have
+to do the following on a Dockerfile:
+
+- Download and install Stack.
+- Copy application sources.
+- Install GHC (`stack setup`).
+- Compile project (`stack build`).
+
+This works, but it is extremely slow and the resulting images are huge
+(+800MB!).
+
+On every Docker build, `stack` would have to download and install GHC, and then
+proceed to download and compile every dependency of the project, which tended
+to a good 30 minutes on a [Quay.io][2] worker node.
+
+When developing locally, you only have to go through this process every now and
+then because most of it is cached in directories such as `~/.stack` and
+`.stack-work`.
+
+Looking for faster build times and smaller images, I decided to experiment with
+splitting the build and runtime aspects of the project.
+
+The build part was already setup since we were using Travis CI for running
+unit and some integration tests. When compared to basic Docker image builders,
+Travis CI has the clear benefit of having a cache that can be reused across
+builds without too much work. This cache allowed us to keep our built
+dependencies across builds, which reduced the build time to under 5 minutes.
+
+This enable caching of Stack builds, you just need to add the work directories
+to the cache section of `.travis.yml`:
+
+```yaml
+cache:
+ directories:
+ - "$HOME/.stack"
+ - .stack-work
+```
+
+So, getting the runtime half working meant taking the resulting build from the
+previous steps and building a Docker container with enough dependencies and
+data files for running the application.
+
+FPComplete has [a great article][1] out there on how to create minimal Docker
+images for Haskell projects.
+
+The main difficulty with this process is that Haskell programs are not built
+statically by default, meaning that you have to identify all the libraries the
+binary is linked against and include them in the final runtime image.
+
+<div class="callout-quote">
+The main difficulty with this process is that Haskell programs are not built
+statically...
+</div>
+
+In order to keep things simple, I decided to stick to using a base image, which
+we could use to pull in any dependencies we don't have, like `libcurl-dev`.
+
+I initially tried to use Alpine, since its known for being one of the smallest
+images out there. However, getting a Haskell program running in it was not
+trivial since it requires cross-compiling GHC.
+
+So I settled with `debian`, which is a larger image, but has almost everything
+we need out of the box.
+
+Building a Docker image on Travis CI is a fairly simple process. Pushing it to
+a registry and correctly tagging it was the annoying part. After a couple of
+hours of trial and error, I made a small shell script for authenticating with
+the registry and pushing a tagged image matching the current git tag and
+branch.
+
+This script is called on the `after-success` step of the Travis CI build:
+
+```bash
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+docker build -t myapp .
+
+# If this is not a pull request, update the branch's docker tag.
+if [ $TRAVIS_PULL_REQUEST = 'false' ]; then
+ docker tag myapp quay.io/myorg/myapp:${TRAVIS_BRANCH/\//-} \
+ && docker push quay.io/myorg/myapp:${TRAVIS_BRANCH/\//-};
+
+ # If this commit has a tag, use on the registry too.
+ if ! test -z $TRAVIS_TAG; then
+ docker tag myapp quay.io/myorg/myapp:${TRAVIS_TAG} \
+ && docker push quay.io/myorg/myapp:${TRAVIS_TAG};
+ fi
+fi
+```
+
+As a result, we now have Docker images for our Haskell projects that are about
+80 MB, which is not terrible, but can definitely be improved on.
+
+The next steps for me are investigating how to make our images even smaller by
+using a smaller base image, and automate the deployment of development
+and staging environments by having Travis CI notify a scheduler that a new
+image has been built.
+
+I'm including some of my scripts and an example Dockerfile on a
+[GitHub Gist][4] for reference. You will most likely have to modify them to
+meet your needs.
+
+[1]: https://www.fpcomplete.com/blog/2015/05/haskell-web-server-in-5mb
+[2]: https://quay.io
+[3]: http://haskellstack.org/
+[4]: https://gist.github.com/etcinit/d484b72de336836a956eb51b3da231ad
diff --git a/server.hs b/server.hs
index 3c57426e272c94672fa41a1112ce0e1551cc98d5..ee457166c7f94f66c1767175a99bae2d5089d3ff 100644
--- a/server.hs
+++ b/server.hs
@@ -1,193 +1,93 @@
{-# LANGUAGE OverloadedStrings #-}
-import Control.Concurrent (forkIO)
-import qualified Data.ByteString as BS (ByteString, pack)
-import Data.Maybe (fromMaybe, mapMaybe)
-import Data.Monoid ((<>))
-import Data.Streaming.Network (HostPreference)
-import Data.String (fromString)
-import qualified Data.Text as T (Text, concat, pack)
-import qualified Data.Text.Encoding as TE (encodeUtf8)
-import Network.Wai (Application, Middleware,
- pathInfo)
-import Network.Wai.Application.Static (defaultWebAppSettings,
- ss404Handler,
- ssAddTrailingSlash,
- ssIndices, ssMaxAge,
- ssRedirectToIndex,
- staticApp)
-import Network.Wai.Handler.Warp (defaultSettings, runSettings,
- setHost, setPort)
-import Network.Wai.Handler.WarpTLS (runTLS,
- tlsSettingsChain)
-import Network.Wai.Middleware.AddHeaders (addHeaders)
-import Network.Wai.Middleware.ForceDomain (forceDomain)
-import Network.Wai.Middleware.ForceSSL (forceSSL)
-import Network.Wai.Middleware.Gzip (def, gzip)
-import Network.Wai.Middleware.RequestLogger (logStdout)
-import Network.Wai.Middleware.Vhost (redirectTo)
-import Safe (lastMay)
-import System.Environment (lookupEnv)
-import WaiAppStatic.Types (MaxAge (MaxAgeSeconds),
- toPiece)
-
--- | The core application.
--- It serves files from `_site` whic is where Hakyll will place the generated
--- site.
-staticSite :: Maybe String -> Application
-staticSite path = staticApp
- (defaultWebAppSettings $ fromString $ fromMaybe "_site" path)
- { ssIndices = mapMaybe (toPiece . T.pack) ["index.html"]
- , ssRedirectToIndex = False
- , ssAddTrailingSlash = True
- , ss404Handler = Just redirectApp
- , ssMaxAge = MaxAgeSeconds 3600
- }
-
--- | 404 handler.
--- We will redirect users to a 404 page if we can't locate the resource they
--- are looking for.
-redirectApp :: Application
-redirectApp req sendResponse = sendResponse $ redirectTo "/"
-
--- | Gzip compression middleware.
-gzipMiddleware :: Middleware
-gzipMiddleware = gzip def
-
--- | Domain redirection middleware.
--- When the site is live, we want to redirect users to the right domain name
--- regarles of whether they arrive from a www. domain, the server's IP address
--- or a spoof domain which is pointing to this server.
-domainMiddleware :: Middleware
-domainMiddleware = forceDomain
- (\domain -> case domain of
- "localhost" -> Nothing
- "chromabits.com" -> Nothing
- _ -> Just "chromabits.com")
-
--- | Common headers middleware.
-headersMiddleware :: Middleware
-headersMiddleware = addHeaders
- [ ("X-Frame-Options", "SAMEORIGIN")
- , ("X-XSS-Protection", "1; mode=block")
- , ("X-Content-Type-Options", "nosniff")
- ]
-
--- | Strict Transport Security middleware.
-stsHeadersMiddleware :: Middleware
-stsHeadersMiddleware = addHeaders
- [("Strict-Transport-Security", "max-age=31536000; includeSubdomains")]
-
--- | Content Security Policy middleware.
--- Here we add the CSP header which includes the policies for this blog.
-cspHeadersMiddleware :: Middleware
-cspHeadersMiddleware = addHeaders
- [("Content-Security-Policy", TE.encodeUtf8 $ glue policies)]
- where
- glue :: [T.Text] -> T.Text
- glue [] = "default-src 'none'"
- glue [x] = x
- glue xs = T.concat $ map (\x -> T.concat [x, "; "]) (init xs) ++ [last xs]
-
- policies :: [T.Text]
- policies = [ "default-src 'self'"
- , "script-src 'self' 'unsafe-inline' https://use.typekit.net"
- <> " https://cdn.mathjax.org https://connect.facebook.net"
- <> " https://*.twitter.com https://cdn.syndication.twimg.com"
- <> " https://gist.github.com"
- <> " https://*.google-analytics.com"
- , "img-src 'self' https: data: platform.twitter.com"
- , "font-src 'self' data: https://use.typekit.net"
- <> " https://cdn.mathjax.org"
- , "style-src 'self' 'unsafe-inline' https://use.typekit.net"
- <> " platform.twitter.com https://assets-cdn.github.com"
- , "frame-src https://www.youtube.com https://www.slideshare.net"
- <> " staticxx.facebook.com www.facebook.com"
- ]
-
--- | De-indefify middleware.
--- Redirects any path ending in `/index.html` to just `/`.
-deindexifyMiddleware :: Middleware
-deindexifyMiddleware app req sendResponse =
- if lastMay (pathInfo req) == Just "index.html"
- then sendResponse $ redirectTo newPath
- else app req sendResponse
- where
- newPath :: BS.ByteString
- newPath = TE.encodeUtf8 $ processPath oldPath
-
- processPath :: [T.Text] -> T.Text
- processPath xs = case xs of
- [] -> "/"
- _ -> T.concat $ map prefixSlash xs
-
- oldPath :: [T.Text]
- oldPath = init $ pathInfo req
-
- prefixSlash :: T.Text -> T.Text
- prefixSlash x = T.concat ["/", x]
-
--- | Serves a WAI Application on the specified port.
--- The target port is printed to stdout before hand, which can be useful for
--- debugging purposes.
-listen :: Int -> Application -> IO ()
-listen port app = do
- let settings = setHost "*6" (setPort port defaultSettings)
-
- -- Inform which port we will be listening on.
- putStrLn $ "Listening on port " ++ show port ++ "..."
- -- Serve the WAI app using Warp
- runSettings settings app
-
--- | Serves a WAI Application on the specified port.
--- The target port is printed to stdout before hand, which can be useful for
--- debugging purposes.
-listenTLS :: Int -> Application -> IO ()
-listenTLS port app = do
+import Control.Lens
+import Data.Monoid ((<>))
+import Data.Maybe (fromMaybe)
+import System.Environment (lookupEnv)
+import Hakyll.Serve.Main (TLSConfiguration(..), Stage(..),
+ defaultServeConfiguration, port, stage, middleware, stagingTransform,
+ tlsConfiguration, tlsPort, prodTransform, path, serve)
+import Hakyll.Serve.Middleware (Directive(..), (<#>), gzipMiddleware,
+ domainMiddleware, securityHeadersMiddleware, stsHeadersMiddleware,
+ cspHeadersMiddleware, deindexifyMiddleware, forceSSLMiddleware,
+ loggerMiddleware)
+import Hakyll.Serve.Listeners (TLSSettings, tlsSettingsChain)
+
+directives :: [Directive]
+directives
+ = [ DefaultSrc ["'self'"]
+ , ScriptSrc [
+ "'self'", "'unsafe-inline'", "https://use.typekit.net",
+ "https://cdn.mathkax.org", "https://connect.facebook.net",
+ "https://*.twitter.com", "https://cdn.syndication.twimg.com",
+ "https://gist.github.com"
+ ]
+ , ImgSrc ["'self'", "https:", "data:", "platform.twitter.com"]
+ , FontSrc [
+ "'self'", "data:", "https://use.typekit.net", "https://cdn.mathjax.org"
+ ]
+ , StyleSrc [
+ "'self'", "'unsafe-inline'", "https://use.typekit.net",
+ "platform.twitter.com", "https://assets-cdn.github.com"
+ ]
+ , FrameSrc [
+ "https://www.youtube.com", "https://www.slideshare.net",
+ "staticxx.facebook.com", "www.facebook.com"
+ ]
+ ]
+
+getTLSSettings :: IO TLSSettings
+getTLSSettings = do
certPath <- lookupEnv "BLOG_TLS_CERT"
chainPath <- lookupEnv "BLOG_TLS_CHAIN"
keyPath <- lookupEnv "BLOG_TLS_KEY"
- let tlsSettings = tlsSettingsChain
- (fromMaybe "cert.pem" certPath)
- [fromMaybe "fullchain.pem" chainPath]
- (fromMaybe "privkey.pem" keyPath)
- let settings = setHost "*6" (setPort port defaultSettings)
-
- -- Inform which port we will be listening on.
- putStrLn $ "Listening on port " ++ show port ++ " (TLS)..."
- -- Serve the WAI app using Warp
- runTLS tlsSettings settings app
+ return $ tlsSettingsChain
+ (fromMaybe "cert.pem" certPath)
+ [fromMaybe "fullchain.pem" chainPath]
+ (fromMaybe "privkey.pem" keyPath)
-- | The entry point of the server application.
main :: IO ()
main = do
- stage <- lookupEnv "BLOG_STAGE"
- path <- lookupEnv "BLOG_PATH"
-
- let liveMiddleware = logStdout
- $ cspHeadersMiddleware
- $ headersMiddleware
- $ domainMiddleware
- $ forceSSL
- $ deindexifyMiddleware
- $ gzipMiddleware
- $ staticSite path
-
- -- Depending on the stage we will choose a different set of middleware to
- -- apply to the application.
- case fromMaybe "dev" stage of
- -- "Production"
- "live" -> do
- forkIO $ listenTLS 443 $ stsHeadersMiddleware liveMiddleware
- listen 80 liveMiddleware
- "staging" -> do
- forkIO $ listenTLS 8443 liveMiddleware
- listen 8080 liveMiddleware
- -- "Development"
- _ -> listen 9090 (logStdout
- $ headersMiddleware
- $ deindexifyMiddleware
- $ gzipMiddleware
- $ staticSite path
- )
+ rawStage <- lookupEnv "BLOG_STAGE"
+ rawPath <- lookupEnv "BLOG_PATH"
+
+ tlsSettings <- getTLSSettings
+
+ let liveMiddleware
+ = mempty
+ <#> loggerMiddleware
+ <#> cspHeadersMiddleware directives
+ <#> securityHeadersMiddleware
+ <#> domainMiddleware "chromabits"
+ <#> forceSSLMiddleware
+ <#> deindexifyMiddleware
+ <#> gzipMiddleware
+ let prodMiddlware = (mempty <#> stsHeadersMiddleware) <> liveMiddleware
+
+ let tlsConf = TLSConfiguration (const liveMiddleware) tlsSettings 8443
+
+ let serveConf
+ = defaultServeConfiguration
+ & stage .~ case rawStage of
+ Just "live" -> Production
+ Just "staging" -> Staging
+ _ -> Development
+ & port .~ 9090
+ & middleware .~ mempty
+ <#> loggerMiddleware
+ <#> securityHeadersMiddleware
+ <#> deindexifyMiddleware
+ <#> gzipMiddleware
+ & path .~ rawPath
+ & stagingTransform .~
+ ((set tlsConfiguration $ Just tlsConf)
+ . (set middleware liveMiddleware)
+ . (set port 8080))
+ & prodTransform .~
+ ((set tlsConfiguration $ Just (tlsConf & tlsPort .~ 443))
+ . (set middleware prodMiddlware)
+ . (set port 80))
+
+ serve serveConf
diff --git a/site.hs b/site.hs
index 31ad74a6ebca7b2aec523a7232851abd4930155e..b3003eca6c1fab0a43958324057f218b41344bc4 100644
--- a/site.hs
+++ b/site.hs
@@ -3,16 +3,17 @@
import Control.Applicative
import Control.Monad (liftM)
-import Control.Monad.IO.Class
+import Control.Lens ((&), (.~))
import qualified Data.Map as M
import Data.Maybe (fromMaybe)
-import Data.Monoid (mappend)
import Data.List (intersperse, isSuffixOf)
import Data.List.Split (splitOn)
import Hakyll
+import Hakyll.Serve (ServeConfiguration, defaultServeConfiguration,
+ hakyllConfiguration, hakyllServeWith)
import Text.Highlighting.Kate.Styles (haddock)
import Text.Pandoc.Options
-import System.FilePath (combine, splitExtension, takeFileName)
+import System.FilePath (splitExtension)
import System.Random (randomRIO)
--------------------------------------------------------------------------------
@@ -21,6 +22,9 @@ data SiteConfiguration = SiteConfiguration
, siteGaId :: String
}
+serveConf :: ServeConfiguration
+serveConf = defaultServeConfiguration & hakyllConfiguration .~ hakyllConf
+
--------------------------------------------------------------------------------
hakyllConf :: Configuration
hakyllConf = defaultConfiguration
@@ -37,18 +41,18 @@ siteConf = SiteConfiguration
, siteGaId = "UA-47694260-1"
}
-feedConf :: String -> FeedConfiguration
-feedConf title = FeedConfiguration
- { feedTitle = "Chromabits: " ++ title
- , feedDescription = "Personal blog"
- , feedAuthorName = "Eduardo Trujillo"
- , feedAuthorEmail = "ed@chromabits.com"
- , feedRoot = "https://chromabits.com"
- }
+-- feedConf :: String -> FeedConfiguration
+-- feedConf title = FeedConfiguration
+-- { feedTitle = "Chromabits: " ++ title
+-- , feedDescription = "Personal blog"
+-- , feedAuthorName = "Eduardo Trujillo"
+-- , feedAuthorEmail = "ed@chromabits.com"
+-- , feedRoot = "https://chromabits.com"
+-- }
--------------------------------------------------------------------------------
main :: IO ()
-main = hakyllWith hakyllConf $ do
+main = hakyllServeWith serveConf $ do
let writerOptions = defaultHakyllWriterOptions
{ writerHtml5 = True
, writerHighlightStyle = haddock
@@ -159,7 +163,7 @@ main = hakyllWith hakyllConf $ do
teaser <- loadAndApplyTemplate "templates/project-teaser.html"
siteCtx $ dropMore compiled
- saveSnapshot "teaser" teaser
+ _ <- saveSnapshot "teaser" teaser
saveSnapshot "content" full
>>= loadAndApplyTemplate "templates/default.html" siteCtx
diff --git a/stack.yaml b/stack.yaml
index e64e9f5cc31ec37a380632a5be582c900a63b0ba..35b9bae3d1bac560f47722d5ada10cc95915deee 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -1,11 +1,12 @@
# For more information, see: https://github.com/commercialhaskell/stack/blob/release/doc/yaml_configuration.md
# Specifies the GHC version and set of packages available (e.g., lts-3.5, nightly-2015-09-21, ghc-7.10.2)
-resolver: lts-5.2
+resolver: lts-5.11
# Local packages, usually specified by relative directory name
packages:
- '.'
+- '/home/etcinit/src/github.com/etcinit/hakyll-serve'
# Packages to be pulled from upstream that are not in the resolver (e.g., acme-missiles-0.3)
extra-deps: []