Friday, July 18, 2008

HTML Templating in HAppS using HSP

HAppS (the Haskell Application Server framework) recently got a whole lot more awesome when it added support for HTML templating using HSP (Haskell Server Pages).


HSP is an extension to Haskell which allows you to use XML syntax in your Haskell source. It is implemented as a preprocessor, trhxs, which transforms the source into regular Haskell code. Unlike some templating systems, you are not restricted to simple string substitution, or a limited templating language. Using HSP we can use the full-power of Haskell in our templates.


In this quickstart guide I will give a brief overview of how I use HSP with HAppS. I assume you already know the basics of using HAppS and so I do not cover those aspects, unless they differ when using HSP. This quickstart is very heavy on the "what to do", but a bit light on the "why" portion. The biggest contribution is the working example directory, which gets you started on the right foot.


Extracting and Running this QuickStart


This quickstart is written in Literal Haskell, and can be executed directly. But, more usefully, I have also written scripts which will automatically extract the code from this quickstart and create a nice template directory to start your projects with.


Among other things, you must be using a version of trhsx which was built against haskell-src-exts >= 0.3.5. And you should be using a version of HAppS-Server that has my support for on-the-fly output validation. However, it is trivial to comment out the validation support if you need to.


You can get a copy of this quickstart via


darcs get http://src.seereason.com/examples/happs-hsp-quickstart/


To recreate this quickstart as a .html file, just run:


make


To run the quickstart in-place do:


make test


and then point your browser at http://localhost:8000/


To load the quickstart Main.lhs into GHCi, set the option -ipages


To extract the code into a usable template run:


make template


This will create a subdirectory named template which you can copy, modify, and use for your own projects.


A Very Simple HSP Example


The following example highlights the basics of mixing XML and Haskell together. This example uses HSP, but not HAppS.


The first thing to notice is the extra options we pass to GHC, namely -F -pgmF trhsx. This tells GHC (and GHCi) to call trhsx to pre-process the source code before trying to compile it. trhsx will automatically translate the XML syntax into normal Haskell code which the compiler can understand.


> {-# OPTIONS_GHC -F -pgmF trhsx #-}
> module Main where
>
> import Data.Char (toUpper)
> import HSP
> import HSP.HTML

Next we see how to write a simple function which generates XML using the special XML syntax as well as dynamically creating some of the XML using ordinary Haskell expressions.


>
> -- |hello creates a simple hello <noun> page
> hello :: String -> HSP XML
> hello noun =
> <html>
> <head>
> <title>Hello, <% noun %></title>
> </head>
> <body>
> <p>Hello, <% map toUpper noun %></p>
> </body>
> </html>
>

To use the XML syntax, we just use it -- no special escaping is required to insert XML in the middle of a Haskell expression. On the other hand, if we want to evaluate a Haskell expression and insert it into the XML, then we need the special <% %> escape clause. If we had just written, <title> Hello, noun</title> then the title would have been Hello, noun. Likewise, if we did not have the <% %> around map toUpper noun, the page would say, "Hello map toUpper noun" instead of evaluating map toUpper noun and substituting the result in.


We can evaluate any arbitrary expression inside the <% %>, provided HSP knows how to turn the result into an XML value. By default, HSP knows how to handle String, Char, numbers, and some other common Haskell data-types. You can add additional class instances if you want to convert other datatypes (including your own) to XML automatically.


Next we have a simple main. The first line evaluates hello and gets back the generated XML. The second line uses renderAsHtml to turn the XML into HTML. renderAsHtml expects you to pass in valid XHTML which it will transform into valid HTML. However, it does not check the validity of the input or output. We will do that using a more general mechanism in the HAppS code.


>
> main :: IO ()
> main =
> do (_, xml) <- evalHSP (hello "World") Nothing
> putStrLn (renderAsHTML xml)
>

If we run this example, we get the following output:



<html
><head
><title
>Hello, World</title
></head
><body
><p
>Hello, WORLD</p
></body
></html
>

The formatting probably looks a bit funny to you -- but the web browser will read it fine. In HTML, the whitespace between the open and close tags is sometimes significant. However, the whitespace inside the open or close tag itself is never significant. The rendering algorithm is designed to exploit those properties to ensure it never adds significant whitespace where you didn't explicit have it in the input file.


Integrating HSP with HAppS


The integration of HSP with HAppS brings a few things to the table:



  • Dynamic HSP recompilation and loading - this means you can modify your HSP templates with out having to completely recompile and restart your HAppS application. Recompiling an HSP page usually happens faster than I can switch from emacs to the web browser and hit reload.
  • JSON data serialization - in order to pass data from HAppS to the HSP template, the data is serialized as a JSON object. This can be especially useful if you want to extend your application to support AJAX or if you want to expose your API to third parties using JSON, because you can use a single unified JSON API for all three instead of having the JSON API be some extra thing that is tact on.

If you decide you do not like using JSON and the dynamic page loading provided happs-hsp-template, but you do like HSP, it should be fairly straight forward to use HSP directly in your HAppS application, similar to how you would use Text.XHtml. However, this quickstart will focus on using happs-hsp-template.


Basic framework

This is the basic framework for creating a HAppS application using:



  • HAppS-Server for HTTP, cookies, etc
  • HAppS-State for persistent storage (aka, our database)
  • HSP for HTML templating
  • RJson for JSON marshalling

> module Main where
>
> import Control.Concurrent
> import Control.Monad
> import HAppS.Server
> import HAppS.State
> import HSP
> import HAppS.Template.HSP
> import HAppS.Template.HSP.Handle
> import Interface
> import State
> import System.Environment
> import Text.RJson

This main function is mostly boilerplate.


>
> main :: IO ()
> main =

newStore just creates an IORef to a Map which will map template source files to compiled object files.


>    do store <- newStore

startSystemState starts the state transaction system.


>       control <- startSystemState entryPoint

Next we parse the command-line arguments to extract a default config file. We also enable validation using wdg-html-validator. In theory, validation should be enabled and disable via a command-line argument, but I have not quite figured out how to add the desired functionality to parseConfig. If you do not have a version of HAppS-Server which support validation, then just change the last line to Right c -> c.


>       eConf <- liftM parseConfig getArgs
> let conf = case eConf of
> Left e -> error (unlines e)
> Right c -> c { validator = Just wdgHTMLValidator }

We then fork off the HTTP process in a new thread. This will take care of handling all incoming requests for us in the background.


>       tid <- forkIO $ simpleHTTP conf $ impl store

Now we are just waiting around for someone to terminate us.


>       putStrLn "running..."
> waitForTermination

Finally we cleanly shutdown the system. The system will not get corrupted if it is shutdown unexpected or improperly, however it may be possible that some pending transactions are lost. (That is true of most (or all?) database systems. The Consistency part of ACID means that the database will not get corrupt. That neccesarily means that sometimes data will be lost during an abrupt shutdown or intentionally thrown out during recovery.)


>       killThread tid
> shutdownSystem control
> destroyStore store

entryPoint is just there to provide type information to startSystemState. The type should be the one you want stored in your State. In our case that is HitCounter.


>
> entryPoint :: Proxy HitCounter
> entryPoint = Proxy

The impl function is your typical HAppS-Server impl function with a few new constructs:


>
> impl :: Store -> [ServerPart Response]
> impl store =
> [ runHSPHandle "pages" "objfiles" store $ multi
> [ dir "json"
> [ dir "times"
> [ path $ \c ->
> [ method GET $ ok (toResponse (toJson (if c == (1 :: Integer) then "time" else "times")))
> ]
> ]
> ]
> , method GET $ do hits <- webUpdate IncHits
> addParam "hits" hits
> ok =<< execTemplate Nothing "Index.hs"
> ]
> ]

runHSPHandle takes four arguments



  1. the directory which holds the templates
  2. the directory to store the compiled template object files in
  3. a handle to the store which manages the objects
  4. a ServerPart a

dir "json" provides our JSON API, which can be accessed via HSP, client-side javascript, or as a 3rd party API. We currently only provide one API call, times. It takes an integer and returns "time" if it is 1, otherwise "times". The toJson function converts the value into JSON data. Our JSON API must be inside runHSPHandle.


The addParam function adds the hits value to the environment used by the page template later. This allows us to pass information to the page template without having to add a JSON API call. We can not always use this method because we may not know what information the page template will need until we run it. More on this later.


The execTemplate function is responsible for actually invoking a specific template. It takes two arguments:



  1. default XMLMetaData (mime-type, doctype, etc), to use if the template does not provide any values.
  2. the name of the template file relative to the directory passed to runHSPHandle

JSON Interface


JSON, short for Javascript Object Notation, is a lightweight data-interchange format. It's primary appeal is that it is natively supported by Javascript. This means you can create javascript objects using JSON notation in the javascript program text, and you can easily convert between JSON and javascript objects at runtime.


happs-hsp-template uses RJson for converting Haskell values to and from JSON. See the README in the RJson tarball for more information on using RJson.


We define our datatypes which will be turned into JSON objects in pages/Interface.lhs. It is in the pages subdirectory because it needs to be imported by both our HAppS backend as well the the HSP templates. This ensures that they are both talking the same specification.


> {-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}

>
> module Interface where
> import HAppS.Data
> import HAppS.State

Next we define a type which will be used as JSON data:


>
> $(deriveAll [''Eq,''Ord,''Read,''Show,''Num,''Enum,''Default]
> [d|
> newtype HitCounter = HitCounter { hits :: Integer }
> |])

One key thing to note is that we use the record syntax in our type declaration. This is so that RJson can automatically convert our data-type into a JSON object. If we did not do this, we would have to manually declare instances of ToJson and FromJson for HitCounter.


It is critical that you read the RJson README to get an understanding of what it supports, or you will be mystified as to why your JSON data is getting silently corrupted.


Because we might want to store HitCounter in the HAppS State we also deriveSerialize for it, and create a Version instance. In theory this could also be useful if clients attempt to pass JSON data back to the server using an older format. However, I have no idea how to handle that in practice.


>
> $(deriveSerialize ''HitCounter)
> instance Version HitCounter

State Definition


In this quickstart our State will just be HitCounter. There is nothing HSP-specific about this module, aside from the fact that we have already defined HitCounter in Interface.lhs.


> {-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}
> {-# OPTIONS_GHC -F -pgmF trhsx #-}
> module State where
>
> import HAppS.Data
> import HAppS.State
> import Control.Concurrent
> import Control.Monad.State
> import Interface

>
> -- |increment hit counter, and return new value
> incHits :: Update HitCounter HitCounter
> incHits =
> do hc <- fmap succ get
> put hc
> return hc

>
> $(mkMethods ''HitCounter
> [ 'incHits
> ])

>
> instance Component HitCounter where
> type Dependencies HitCounter = End
> initialValue = HitCounter 0

Index Page


Now that we have our JSON interface and our State defined, we can implement a page template.


We will use the OPTIONS_GHC pragma to automatically pass this file through trhsx. happs-hsp-template will automatically add this command-line option when compiling pages, but adding it explicitly means we can easily load the file into GHCi.


> {-# OPTIONS_GHC -F -pgmFtrhsx #-}

> module Index where
>
> import Control.Monad.Trans
> import HSP
> import HAppS.Template.HSP
> import UACCT
> import HSP.Google.Analytics
> import HAppS.State
> import Interface

In our templates, the entry point is always page and it always has the type Web XML. (If you use HSP without HAppS then you will want HSP XML instead of Web XML). This is similar to how normal programs always start at main and main always has the type IO ().


>
> page :: Web XML
> page =
> withMetaData html4Strict $
> do (HitCounter hits) <- localRead "hits"
> times <- globalRead $ simpleRequest ("/json/times/" ++ show hits)
> page' hits times

Our page function does fours things.



  1. Sets the XML meta-data to html4Strict. This will:

    • render the page as HTML (instead of XML)
    • set the DOCTYPE to HTML 4.01 Strict.
    • set the content-type to "text/html"

  2. reads the "hits" value from the local environment.
  3. Uses the JSON API to determine if it should say, "time" or "times"
  4. Calls the page' function

The environment read by localRead is created way back in the impl function when we did:


>        addParam "hits" hits

globalRead simulates an HTTP request to the HAppS server. In this example, we just do a simple GET request to /json/times, but any type of HTTP request can be simulated.


If globalRead returns "Missing content-type", it is probably because your JSON API is not inside the runHSPHandle call.


The page' function is straight-forward HSP code. We could have put the contents in the page function itself, but I prefer to split it out, because it looks neater, and makes gives you the option of calling the page' function.


>
> page' :: Integer -> String -> Web XML
> page' hits times =
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
> <title>HitCounter 3000</title>
> <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
> <link type="text/css" rel="stylesheet" href="/css/hg.css" media="screen" title="default" />
> </head>
> <body>
> <h1>HitCounter 3000</h1>
> <p>This page has been viewed <% (show hits) ++ " " ++ times %>.</p>
> <% analytics uacct %>
> </body>
> </html>

Google Analytics Code


The UACCT module is just a place to keep our Google Analytics account code so that we can easily import it into all the pages on the site. Since you are likely to want to use a different Google Analytics code for each site, it makes more sense to keep this code in the pages directory on a per-project basis, rather than store it in a system-wide library.


> module UACCT where
>
> import HSP.Google.Analytics
>
> uacct :: UACCT
> uacct = UACCT "UA-4353757-1"

Conclusion


I hope you now have a basic idea of what is going on. The next step is to extract the template directory and attempt to make your own site.


In this quickstart, our State and our JSON interface are very closely related. In fact, they are just the HitCounter type. For a simple application such as this one, Storing your JSON objects directly in the State seems like a sensible approach. For other sites, you will find that you want to have one set of data types for your State, and a separate set of data types for your Interface, though some data types may still be shared between the two.

Thursday, July 3, 2008

Extending Asterisk with HAppS

I just glued two of my favorite technologies together, Asterisk (the opensource PBX/VoIP/etc system) and HAppS (the Haskell Application Server framework).



If you have heard of HAppS, but never used it, you may have the impression that HAppS is a web development platform -- but that is not quite correct. HAppS is actually a collection of several different server components which can be combined together or used separately.



In this post, I will show how to build as simple FastAGI server on top of the HAppS-State component. We will not be using the web component (HAppS-Server), which is the part that provides HTTP, templating, cookies, etc. This post assumes no prior knowledge of HAppS or AGI.



What You Will Need


If you want to build this demo you will need:



  • the latest version of HAppS-State and it's dependencies.
  • Asterisk (I think any version later than 1.0 should work. I use 1.4.17 from Ubuntu Hardy)
  • The haskell AGI library
  • darcs get http://src.seereason.com/fastagi-hitcounter


Run make HitCounter.hs to produce a nice, clean
.hs file from the .lhs file.



FastAGI



The asterisk server can be extended by using the Asterisk Gateway Interface (AGI). AGI provides the functionality you need to do stuff like "Please enter your 16-digit account number."



An AGI script is a standalone program you write (in a language of your choice). Asterisk communications with your AGI script by running it directly, and writing to its stdin and reading from its stdout. The AGI protocol consists of simple commands and
responses
which are human readable text.



Asterisk also has the option of communicating with your AGI script remotely via TCP instead of directly running a local program. This feature is known as FastAGI. The commands and responses are identical to normal AGI, the only differences are:



  • The communication channel is setup via TCP instead of forking off a local process
  • Some extra AGI variables are passed in by FastAGI


There is one additional importing difference, which is more of a side-effect. When using plain-old AGI, a new process will be spawned for each call. When using FastAGI, a new TCP connection will be opened -- typically to a single, long running server process. So, with AGI you will need to worry about how to provide communication and synchronization between multiple processes, but with FastAGI, you can just use threads.



HAppS-State



The HAppS State component provides in-memory state with ACID guarantees. It uses write ahead logging and checkpointing to ensure the state can be restored from disk in the event of a power outage, and also provides multimaster replication. Unlike a traditional relational database, HAppS-State works directly with native, arbitrary Haskell data types. This means you don't have to figure out how to get your beautiful data structures wedged into a relational database just to get ACID guarantees and replication.



Example Application


The remainder of the post is a simple example which implements a hit-counter. When you call the phone number, it tells you what caller number you are. I won't go into too much detail about the HAppS State portion, since this post is supposed to show how to integrate AGI, not how to use HAppS State.



> {-# LANGUAGE TemplateHaskell, UndecidableInstances, TypeFamilies, 
> TypeSynonymInstances, FlexibleInstances, DeriveDataTypeable,
> MultiParamTypeClasses, TypeOperators, GeneralizedNewtypeDeriving #-}


> module Main where
>
> import Control.Concurrent
> import Control.Monad
> import Control.Monad.Reader
> import Control.Monad.State
> import Control.Monad.Trans
> import HAppS.Data
> import HAppS.State
> import Network
> import Network.AGI
> import System.Random
> import System.Posix.Unistd

The first thing we do is define the type we will use to store our
persistent state (aka, our "database schema"). The deriveAll is
similar to deriving (Eq, Ord, Read, Show, Num, Enum, Default,
Data, Typeable)
. Since there is no way to extend
deriving, we have to use Template Haskell to add support
for deriving Default.



>
> $(deriveAll [''Eq,''Ord,''Read,''Show,''Num,''Enum,''Default]
> [d|
> newtype HitCounter = HitCounter { hits :: Integer }
> |])


deriveSerialize is part of the magic that allows HitCounter
to be serialized to disk or replicated between servers.



>
> $(deriveSerialize ''HitCounter)

The Version instance is used to migrate the
old data, if we modify the HitCounter data structure. That
is a subject for a different tutorial.



> instance Version HitCounter

Next we define a function which modifies the global state
(HitCounter). This function while be run
atomically. This means that there is no race condition between the
get and the put. The get and
put functions come from
Control.Monad.State.



>
> -- |increment hit counter, and return new value
> incHits :: Update HitCounter Integer
> incHits =
> do hc <- fmap succ get
> put hc
> return (hits hc)

This is the magic which converts the incHits function
into an atomic action for updating the global state.



>
> $(mkMethods ''HitCounter
> [ 'incHits
> ])

Next we define our top-level component which uses the global state. A more complex application might use a bunch of independent components similar to HitCounter. This allows us to easily build things like session support, user accounts, etc, in third party reusable libraries. I believe it also makes atomic actions finer grained and makes it possible to support shards.



>
> instance Component HitCounter where
> type Dependencies HitCounter = End
> initialValue = HitCounter 0

> entryPoint :: Proxy HitCounter
> entryPoint = Proxy

The main function starts up the state engine, forks off the fastAGI server, waits for a shutdown signal (for example, ^C), and then cleanly shuts down the state engine. The fastAGI function comes from the Haskell AGI library, and is in no way HAppS specific.



>
> main :: IO ()
> main =
> do control <- startSystemState entryPoint
> tid <- forkIO $ fastAGI Nothing agiMain
> putStrLn "running..."
> waitForTermination
> killThread tid
> shutdownSystem control

Here is our simple AGI application. It



  1. answers the call
  2. waits a second to give the caller time to finish setting up their end of the call
  3. increments the hit counter
  4. plays a pre-recorded file which says, "You are currently caller number"
  5. says the caller number
  6. plays a pre-recorded file which says "Goodbye."
  7. hangs up


The functions answer, streamFile,
sayNumber, and hangUp come from the AGI
library.



The update IncHits call is our database query. Note
that we don't call the incHits function
directly. Instead we call update and pass it the value
IncHits. The IncHits type was created for
us automatically by the call to mkMethods we made
earlier.



> 
> agiMain :: HostName -> PortNumber -> AGI ()
> agiMain hostname portNum =
> do answer
> liftIO $ sleep 1 -- give the caller time to get their end of the call setup
> h <- update IncHits
> streamFile "queue-thereare" [] Nothing
> sayNumber h []
> streamFile "vm-goodbye" [] Nothing
> hangUp Nothing
> return ()

Hooking it up


To test the application, first we need to update the asterisk dialplan to call our AGI application. Something like this should do the trick (be sure to reload the dialplan after modifying extensions.conf):



[default]
exten => 31415,1,AGI(agi://127.0.0.1)

Next we start our AGI application server:



$ runhaskell HitCounter.lhs

And finally, we dial 31415 and hope the magic happens.


Summary


The above code is a good starting template for a more interesting AGI application. Note that caller number is a bit fuzzy. The caller number is determined by who gets to the update function first -- which could be different from who actually connected to the asterisk server first.


Also, when calling a FastAGI application, it is possible to pass in a PATH and query string. The Haskell AGI library makes this information available, but does not provide any special mechanisms for doing something useful with it. This is likely to change in the future.