tag:blogger.com,1999:blog-63817774247134489432024-02-20T14:10:12.471-08:00n-heptane labJeremy Shawhttp://www.blogger.com/profile/18373967098081701148noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-6381777424713448943.post-36640034959735170202008-12-14T14:08:00.000-08:002008-12-14T14:20:39.779-08:00Data Migration with HApps-Data<p>HAppS applications, like any application with persistent data storage, are faced with the issue of migrating existing data when the format of the persistent data is changed. This tutorial will explore the binary serialization and migration facilities provided by HAppS-Data. If you think I am doing it all wrong, please let me know. Writing this tutorial is the extent of my experience using the <kbd>HApps-Data</kbd> migration facilities.</p><br /><h3>Requirements</h3><br /><p>This tutorial only uses the <kbd>HAppS-Data</kbd> (and dependencies) portion of <kbd>HAppS</kbd>. It has been tested with <kbd>HAppS-Data 0.9.3</kbd>. The first three lines of the module look like this:</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}</span><br /><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Main</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Data</span><br /></pre><br /><h3>Serialization</h3><br /><p>The most obvious way to serialize data in Haskell is to use the familiar <code>Read</code> and <code>Show</code> classes. Simply use <code>show</code> to turn a value into a <code>String</code>, and <code>read</code> to turn a <code>String</code> back into a value. This method has three serious flaws however:</p><br /><ol><br /> <li>The law <code>read . show == id</code> does not hold for all Show/Read instances. <br /> <li>The serialized representation is rather verbose<br /> <li>No migration path when types change, leaving your old data inaccessible<br /></ol><br /><p><kbd>HAppS-Data</kbd> provides a <code>Serialize</code> class which addresses these three issues. From an end user point of view the <code>Serialize</code> functionality provides three items of interest:</p><ol><li>The <code>Serialize</code> class<br /> <li>the <code>serialize</code> and <code>deserialize</code> functions<br /> <li>The <code>deriveSerialize</code> function<br /> </ol><br /><pre><span class='varop'>></span> <span class='keyword'>class</span> <span class='layout'>(</span><span class='conid'>Typeable</span> <span class='varid'>a</span><span class='layout'>,</span> <span class='conid'>Version</span> <span class='varid'>a</span><span class='layout'>)</span> <span class='keyglyph'>=></span> <span class='conid'>Serialize</span> <span class='varid'>a</span> <span class='keyword'>where</span> <br /><span class='varop'>></span> <span class='varop'>...</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='varid'>serialize</span> <span class='keyglyph'>::</span> <span class='conid'>Serialize</span> <span class='varid'>a</span> <span class='keyglyph'>=></span> <span class='varid'>a</span> <span class='keyglyph'>-></span> <span class='conid'>Lazy</span><span class='varop'>.</span><span class='conid'>ByteString</span><br /><span class='varop'>></span> <span class='varid'>deserialize</span> <span class='keyglyph'>::</span> <span class='conid'>Serialize</span> <span class='varid'>a</span> <span class='keyglyph'>=></span> <span class='conid'>Lazy</span><span class='varop'>.</span><span class='conid'>ByteString</span> <span class='keyglyph'>-></span> <span class='layout'>(</span><span class='varid'>a</span><span class='layout'>,</span> <span class='conid'>Lazy</span><span class='varop'>.</span><span class='conid'>ByteString</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='varid'>deriveSerialize</span> <span class='keyglyph'>::</span> <span class='conid'>Language</span><span class='varop'>.</span><span class='conid'>Haskell</span><span class='varop'>.</span><span class='conid'>TH</span><span class='varop'>.</span><span class='conid'>Syntax</span><span class='varop'>.</span><span class='conid'>Name</span><br /><span class='varop'>></span> <span class='keyglyph'>-></span> <span class='conid'>Language</span><span class='varop'>.</span><span class='conid'>Haskell</span><span class='varop'>.</span><span class='conid'>TH</span><span class='varop'>.</span><span class='conid'>Syntax</span><span class='varop'>.</span><span class='conid'>Q</span> <span class='keyglyph'>[</span><span class='conid'>Language</span><span class='varop'>.</span><span class='conid'>Haskell</span><span class='varop'>.</span><span class='conid'>TH</span><span class='varop'>.</span><span class='conid'>Syntax</span><span class='varop'>.</span><span class='conid'>Dec</span><span class='keyglyph'>]</span><br /></pre><br /><p>The <code>Version</code> superclass is used during data migration. The <code>serialize</code> and <code>deserialize</code> functions are the counterparts to <code>show</code> and <code>read</code>. <code>deriveSerialize</code> is a Template Haskell function which provides functionality similar to <code>deriving (Read, Show)</code>.</p><br /><h3>The <code>Version</code> class</h3><br /><p>The <code>Version</code> class is very straight-forward. It consists of a single function which returns the <code>Mode</code> (aka, the version) of a datatype.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>class</span> <span class='conid'>Version</span> <span class='varid'>a</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>::</span> <span class='conid'>Mode</span> <span class='varid'>a</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>=</span> <span class='conid'>Versioned</span> <span class='num'>0</span> <span class='conid'>Nothing</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Mode</span> <span class='varid'>a</span> <span class='keyglyph'>=</span> <span class='conid'>Primitive</span> <span class='comment'>-- ^ Data layout won't change. Used for types like Int and Char.</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span> <span class='conid'>Versioned</span> <span class='layout'>(</span><span class='conid'>VersionId</span> <span class='varid'>a</span><span class='layout'>)</span> <span class='layout'>(</span><span class='conid'>Maybe</span> <span class='layout'>(</span><span class='conid'>Previous</span> <span class='varid'>a</span><span class='layout'>)</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>newtype</span> <span class='conid'>VersionId</span> <span class='varid'>a</span> <span class='keyglyph'>=</span> <span class='conid'>VersionId</span> <span class='layout'>{</span><span class='varid'>unVersion</span> <span class='keyglyph'>::</span> <span class='conid'>Int</span><span class='layout'>}</span> <span class='keyword'>deriving</span> <span class='layout'>(</span><span class='conid'>Num</span><span class='layout'>,</span><span class='conid'>Read</span><span class='layout'>,</span><span class='conid'>Show</span><span class='layout'>,</span><span class='conid'>Eq</span><span class='layout'>)</span><br /></pre><br /><p>There are two categories of datatypes:</p><br /> <ul><br /> <li>primitives whose layout will never change, and, hence, will never need to be migrated<br /> <li>everything else<br /> </ul><br /><p>The <code>Versioned</code> constructor takes two arguments. The first argument is a version number which you increment when you make an change to the data-type. The second argument is an indicator of the previous version of the data-type. The exact details are covered in the next section.<br /><h3>Putting it all together</h3><br /><p>Let's say we have the following types:</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Foo</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>Bar</span> <span class='conid'>String</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span> <span class='conid'>Baz</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Beep</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /></pre><br /><p>The <code>deriveAll</code> template haskell function is similar to the normal haskell deriving clause, except it also has the ability to derive <code>Default</code> instances. Additionally, it always derives <code>Typeable</code> and <code>Data</code> instances even though they are not explicitly listed.</p><br /><p>To make the types serializeable we first need to create <code>Version</code> instances.</p><br /><pre><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Beep</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>=</span> <span class='conid'>Versioned</span> <span class='num'>0</span> <span class='conid'>Nothing</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Foo</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>=</span> <span class='conid'>Versioned</span> <span class='num'>0</span> <span class='conid'>Nothing</span><br /></pre><br /><p>We want to indicate that <code>Beep</code> and <code>Foo</code> are non-primative types, so we use the <code>Versioned</code> constructor. Next we specify a version number for the type. It could be anything, but <code>0</code> is the most sensible choice. Since there are now previous versions of these types we mark the previous type as <code>Nothing</code>.<br /><p>For all non-primitive types the initial version of <code>Versioned 0 Nothing</code> is sensible. So the <code>Version</code> class provides it as a default value for <code>mode</code>:</p><br /><pre><span class='varop'>></span> <span class='keyword'>class</span> <span class='conid'>Version</span> <span class='varid'>a</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>::</span> <span class='conid'>Mode</span> <span class='varid'>a</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>=</span> <span class='conid'>Versioned</span> <span class='num'>0</span> <span class='conid'>Nothing</span><br /></pre><br /><p>Hence, we could shorten our <code>Version</code> instances from above to:</p><br /><pre><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Foo</span><br /></pre><br /><p>Next we derive <code>Serialize</code> instances for our types:</p><br /><pre><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Beep</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Foo</span><span class='layout'>)</span><br /></pre><br /><p>Now we can use <code>serialize</code> to serialize values. Let's look at the output of <code>serialize Beep</code></p><br /><pre><br />*Main> Data.ByteString.Lazy.unpack $ serialize Beep<br />[0,0,0,0,0,0,0,0,0]<br />*Main><br /></pre><br /><p>We see that <code>Beep</code> serializes to 9 bytes. The first 8 bytes represent the <code>VersionId</code>. <code>VersionId</code> is basically an <code>Int</code>, and the serialization code always treats <code>Int</code>s as a 64-bit values to avoid cross-platform issues. The final byte indicates which constructor of <code>Beep</code> was used. In this case the zeroth constructor was used.</p><br /><p>At first it may seem like we don't have enough information here to deserialize the data, after all there are no type names, constructors, etc. But deserializing these bytes is no different than doing <code>read "1" :: Int</code>. Because we know the type of the value we want to be reading at compile time, we do not need to record that information in the stored data. We just do:</p><br /><pre><br />*Main> deserialize (serialize Beep) :: (Beep,ByteString)<br />(Beep,Empty)<br />*Main> <br /></pre><br /><p>As a side note, <code>String</code>s are serialized to a very compact representation. In fact, they are stored as compactly as <code>ByteString</code>s because they are first converted to a <code>ByteString</code>. </p><br /><pre><br />*Main> Data.ByteString.Lazy.unpack $ serialize "hello"<br />[0,0,0,0,0,0,0,5,104,101,108,108,111]<br />*Main> Data.ByteString.Lazy.unpack $ serialize (Data.ByteString.Lazy.Char8.pack "hello")<br />[0,0,0,0,0,0,0,5,104,101,108,108,111]<br />*Main> <br /></pre><br /><p>The first 8 bytes are the length of the <code>String</code>, and the remaining bytes are the utf-8 encoded characters of the <code>String</code>.</p><br /><p>So, if you application is best served by using <code>String</code>s instead of <code>ByteString</code>s, you do not have to take an extra steps to ensure that the serialized data is compactly represented.</p><br /><h3>Simple Migration</h3><br /><p>Let's say we want to add another constructor to the <code>Beep</code> type. As a first pass, we will actually create a whole new type named <code>Beep'</code>, which is similar to the old type, but has an additional constructor <code>BeepBeep</code>.</p><br /><pre><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Beep'</span> <span class='keyglyph'>=</span> <span class='conid'>BeepBeep'</span> <span class='keyglyph'>|</span> <span class='conid'>Beep'</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Beep'</span><span class='layout'>)</span><br /></pre><br /><p>Because we are extending a previous type, our <code>Version</code> instance will look a bit different:</p><br /><pre><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Beep'</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>=</span> <span class='varid'>extension</span> <span class='num'>1</span> <span class='layout'>(</span><span class='conid'>Proxy</span> <span class='keyglyph'>::</span> <span class='conid'>Proxy</span> <span class='conid'>Beep</span><span class='layout'>)</span><br /></pre><br /><p>This indicates that we are extending the old type <code>Beep</code>. The new version number must be higher than the old version, but does not have to be strictly sequential.</p><br /><p>Because we specified that this type is a newer version of an older type, we also need to tell HAppS how to migrate the old data to the new type. To do this, we simply create an instance of the <code>Migrate</code> class.</p><br /><pre><span class='varop'>></span> <span class='keyword'>class</span> <span class='conid'>Migrate</span> <span class='varid'>a</span> <span class='varid'>b</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>migrate</span> <span class='keyglyph'>::</span> <span class='varid'>a</span> <span class='keyglyph'>-></span> <span class='varid'>b</span><br /></pre><br /><p>The <code>Migrate</code> class is quite simple, it contains a single function, <code>migrate</code> which migrates something of type <code>a</code> to type <code>b</code>. In our current example, all we need is:</p><br /><pre><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Migrate</span> <span class='conid'>Beep</span> <span class='conid'>Beep'</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>migrate</span> <span class='conid'>Beep</span> <span class='keyglyph'>=</span> <span class='conid'>Beep'</span><br /></pre><br /><p>We can demonstrate migration by serializing a value of type <code>Beep</code> and deserializing it as type <code>Beep'</code>. The migration happens automatically in the <code>deserialize</code> function.</p><br /><pre><br />*Main> fst $ deserialize (serialize Beep) :: Beep'<br />Beep'<br />*Main><br /></pre><br /><p>When <code>deserialize</code> tries to deserialize the data produced by <code>serialize Beep</code>, it will first check the version number. When it sees that the version number in the stored data is lower than the version number of the current type it will instead try to decode it as the type you specified as the previous version. If the version associated with the previous type is still higher than the value in the serialized data, the migration code will recurse until it finds a matching version number. Once it finds a matching version number, it will call the corresponding deserialization "instance" to decode the old data. Then as the recursion unwinds, it will apply the <code>migrate</code> function to migrate the data to newer and newer formats until it reaches the newest format.</p><br /><h3>Managing History</h3><br /><p>A big issue in the above example is that when we added the new constructor we also changed the name of the type and its existing constructors. That is not very convenient in a real application where you have a multitude of references to the old names.</p><br /><p>Fortunately, we do not have to change the name of the type to add a new constructor. As we saw in the beginning, the name of the type and the names of the constructors are not actually stored in the serialized data. So, instead we can change the name of the old type from <code>Beep</code> to <code>OldBeep</code> and update its constructor as well.</p><br /><pre><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>OldBeep</span> <span class='keyglyph'>=</span> <span class='conid'>OldBeep</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>OldBeep</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>OldBeep</span><br /></pre><br /><p>Because <code>OldBeep</code> and <code>Beep</code> have the same shape, they will serialize to the same byte sequence:<br /><pre><br />*Main> Data.ByteString.Lazy.unpack $ serialize OldBeep<br />[0,0,0,0,0,0,0,0,0]<br />*Main> Data.ByteString.Lazy.unpack $ serialize Beep<br />[0,0,0,0,0,0,0,0,0]<br />*Main> <br /></pre><br /><p>that means we can serialize an <code>OldBeep</code> value and then deserialize it as a <code>Beep</code> value, like this:</p><br /><pre><br />*Main> fst $ deserialize (serialize OldBeep) :: Beep<br />Beep<br />*Main> <br /></pre><br /><p>Note that this is not the same as migration. Here we are just exploiting the fact that because the type name and constructor names are not encoded in the serialized data we can change those names and still be able to deserialize the data.</p><br /><h4>Full Migration Example #1</h4><br /><p>Here is the full example which shows:</p><br /> <ol><br /> <li><code>Beep</code> renamed to <code>OldBeep</code><br /> <li>the new <code>Beep</code> with the extra constructor<br /> <li>the migration code from <code>OldBeep</code> to <code>Beep</code><br /> </ol><br /><pre><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>OldBeep</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>OldBeep</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>OldBeep</span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>OldBeep</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>OldBeep</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>OldBeep</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Beep</span> <span class='keyglyph'>=</span> <span class='conid'>BeepBeep</span> <span class='keyglyph'>|</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Beep</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>=</span> <span class='varid'>extension</span> <span class='num'>1</span> <span class='layout'>(</span><span class='conid'>Proxy</span> <span class='keyglyph'>::</span> <span class='conid'>Proxy</span> <span class='conid'>OldBeep</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Beep</span><span class='layout'>)</span><br /></pre><br /><h4>Using separate files to manage type history</h4><br /><p>Keeping all the revisions of your type in one file, and changing the name of the type and its constructors every revision is tedious and hard to manage. Instead, we can use a system where we rename the files that contain our types. To start, we will put the types we want to serialize in a separate file (or files), such as <kbd>Types.lhs</kbd>.</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}</span><br /><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Types</span> <br /><span class='varop'>></span> <span class='layout'>(</span> <span class='conid'>Bar</span><span class='layout'>(</span><span class='keyglyph'>..</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='layout'>,</span> <span class='conid'>Foo</span><span class='layout'>(</span><span class='keyglyph'>..</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='layout'>)</span> <span class='keyword'>where</span><br /></pre><br /><pre><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Data</span><br /></pre><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Foo</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>Bar</span> <span class='conid'>String</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span> <span class='conid'>Baz</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Beep</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Beep</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Foo</span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Foo</span><span class='layout'>)</span><br /></pre><br /><p>Now let's say we want to add a constructor <code>Bop</code> to the type <code>Foo</code>. First we rename <kbd>Types.lhs</kbd> to <kbd>Types_000.lhs</kbd> and change the module name to reflect the changed file name:</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}</span><br /><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Types_000</span> <br /><span class='varop'>></span> <span class='layout'>(</span> <span class='conid'>Beep</span><span class='layout'>(</span><span class='keyglyph'>..</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='layout'>,</span> <span class='conid'>Foo</span><span class='layout'>(</span><span class='keyglyph'>..</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='layout'>)</span> <span class='keyword'>where</span><br /></pre><br /><pre><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Data</span><br /></pre><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Foo</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>Bar</span> <span class='conid'>String</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span> <span class='conid'>Baz</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Beep</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Beep</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Foo</span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Foo</span><span class='layout'>)</span><br /></pre><br /><p>Next we create a new <kbd>Types.lhs</kbd>:</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}</span><br /><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Types</span> <br /><span class='varop'>></span> <span class='layout'>(</span> <span class='conid'>Beep</span><span class='layout'>(</span><span class='keyglyph'>..</span><span class='layout'>)</span> <span class='comment'>-- ^ re-exported from Types_000</span><br /><span class='varop'>></span> <span class='layout'>,</span> <span class='conid'>Foo</span><span class='layout'>(</span><span class='keyglyph'>..</span><span class='layout'>)</span> <span class='comment'>-- ^ extended here</span><br /><span class='varop'>></span> <span class='layout'>)</span> <span class='keyword'>where</span><br /></pre><br /><pre><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Data</span><br /></pre><br /><p>We import the old <kbd>Types_000</kbd> qualified as <code>T0</code> to avoid name clashes.</p><br /><pre><span class='varop'>></span> <span class='keyword'>import</span> <span class='keyword'>qualified</span> <span class='conid'>Types_000</span> <span class='keyword'>as</span> <span class='conid'>T0</span><br /></pre><br /><p>Since we are only modifying Foo, we can import and re-export the old version of Beep unmodified (also see the module export list above):</p><br /><pre><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Types_000</span> <span class='layout'>(</span><span class='conid'>Beep</span><span class='layout'>)</span><br /></pre><br /><p> Then we extend <code>Foo</code> with the new constructor <code>Bop Int</code>:</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>data</span> <span class='conid'>Foo</span> <br /><span class='varop'>></span> <span class='keyglyph'>=</span> <span class='conid'>Bar</span> <span class='conid'>String</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span> <span class='conid'>Baz</span> <span class='conid'>Beep</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span> <span class='conid'>Bop</span> <span class='conid'>Int</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /></pre><br /><p>Next we create a <code>Version</code> instance which indicates that the previous version of <code>Foo</code> is <code>T0.Foo</code>.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>Foo</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>mode</span> <span class='keyglyph'>=</span> <span class='varid'>extension</span> <span class='num'>1</span> <span class='layout'>(</span><span class='conid'>Proxy</span> <span class='keyglyph'>::</span> <span class='conid'>Proxy</span> <span class='conid'>T0</span><span class='varop'>.</span><span class='conid'>Foo</span><span class='layout'>)</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Foo</span><span class='layout'>)</span><br /><span class='varop'>></span><br /></pre><br /><p>And, finally, we specify how to migrate the old data:</p><br /><pre><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Migrate</span> <span class='conid'>T0</span><span class='varop'>.</span><span class='conid'>Foo</span> <span class='conid'>Foo</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='varid'>migrate</span> <span class='layout'>(</span><span class='conid'>T0</span><span class='varop'>.</span><span class='conid'>Bar</span> <span class='varid'>str</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='conid'>Bar</span> <span class='varid'>str</span><br /><span class='varop'>></span> <span class='varid'>migrate</span> <span class='layout'>(</span><span class='conid'>T0</span><span class='varop'>.</span><span class='conid'>Baz</span> <span class='varid'>beep</span><span class='layout'>)</span> <span class='keyglyph'>=</span> <span class='conid'>Baz</span> <span class='varid'>beep</span><br /></pre><br /><p>Note that <code>Foo</code> in <kbd>Types.lhs</kbd> and <code>Foo</code> in <kbd>Types_000.lhs</kbd> are different types, namely <code>Types.Foo</code> and <code>Types_000.Foo</code>. So this works for the same reason that renaming <code>Beep</code> to <code>OldBeep</code> works.</p><br /><h3>Serializing Datatypes from 3rd Party Libraries</h3><br /><p>It is also possible to serialize datatypes from 3rd party libraries, provided those types have <code>Data</code> and <code>Typeable</code> instances. There is a caveat with this however. If the third party library changes the type, then you will not be able to read your data. This is not a fatal flaw however. You can simply copy the old type definition into a local module, and then migrate the old data to the new format.</p><br /><h3>Suggested Policy</h3><br /><ol><br /> <li>Put the types you will serialize in one or more files which only contain types<br /> <li>Deploy your web 2.718 killer app<br /> <li>Before you do any more development, copy the current type files to sequential versions and create new current type files which re-export all the types. You can skip this step if the current type file only contains re-exports. i.e., if no type changes were made to that type file during the previous iteration.<br /> <li>Make changes for current development cycle, and then go to step 2.<br /></ol>Jeremy Shawhttp://www.blogger.com/profile/18373967098081701148noreply@blogger.com4tag:blogger.com,1999:blog-6381777424713448943.post-34183317663936744492008-07-18T14:16:00.000-07:002008-07-18T14:29:14.881-07:00HTML Templating in HAppS using HSP<p><a href="http://happs.org/">HAppS</a> (the Haskell Application Server framework) recently got a whole lot more awesome when it added support for HTML templating using <a href="http://www.cs.chalmers.se/~d00nibro/hsp/">HSP</a> (Haskell Server Pages).</p><br /> <p>HSP is an extension to Haskell which allows you to use XML syntax in your Haskell source. It is implemented as a preprocessor, <kbd>trhxs</kbd>, which transforms the source into regular Haskell code. Unlike some templating systems, you are not restricted to simple string substitution, or a limited templating language. Using HSP we can use the full-power of Haskell in our templates.</p><br /> <p>In this quickstart guide I will give a brief overview of how I use HSP with HAppS. I assume you already know the basics of using HAppS and so I do not cover those aspects, unless they differ when using HSP. This quickstart is very heavy on the "what to do", but a bit light on the "why" portion. The biggest contribution is the working example directory, which gets you started on the right foot.</p><br /> <h4>Extracting and Running this QuickStart</h4><br /> <p>This quickstart is written in Literal Haskell, and can be executed directly. But, more usefully, I have also written scripts which will automatically extract the code from this quickstart and create a nice template directory to start your projects with.</p><br /> <p>Among other things, you must be using a version of <kbd>trhsx</kbd> which was built against <kbd>haskell-src-exts >= 0.3.5</kbd>. And you should be using a version of HAppS-Server that has my support for on-the-fly output validation. However, it is trivial to comment out the validation support if you need to.</p><br /> <p>You can get a copy of this quickstart via</p><br /> <p><kbd>darcs get http://src.seereason.com/examples/happs-hsp-quickstart/</kbd></p><br /> <p>To recreate this quickstart as a <kbd>.html</kbd> file, just run:</p><br /> <p><kbd>make</kbd></p><br /> <p>To run the quickstart in-place do:</p><br /> <p><kbd>make test</kbd></p><br /> <p>and then point your browser at <kbd>http://localhost:8000/</kbd></p><br /> <p>To load the quickstart <kbd>Main.lhs</kbd> into GHCi, set the option <kbd>-ipages</kbd></p><br /> <p>To extract the code into a usable template run:</p><br /> <p><kbd>make template</kbd></p><br /> <p>This will create a subdirectory named <kbd>template</kbd> which you can copy, modify, and use for your own projects.</p><br /><h4>A Very Simple HSP Example</h4><br /><p>The following example highlights the basics of mixing XML and Haskell together. This example uses HSP, but not HAppS.</p><br /><p>The first thing to notice is the extra options we pass to GHC, namely <code>-F -pgmF trhsx</code>. This tells GHC (and GHCi) to call <code>trhsx</code> to pre-process the source code before trying to compile it. <code>trhsx</code> will automatically translate the XML syntax into normal Haskell code which the compiler can understand.</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# OPTIONS_GHC -F -pgmF trhsx #-}</span><br /><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Main</span> <span class='keyword'>where</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Data</span><span class='varop'>.</span><span class='conid'>Char</span> <span class='layout'>(</span><span class='varid'>toUpper</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HSP</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HSP</span><span class='varop'>.</span><span class='conid'>HTML</span><br /></pre><br /> <p>Next we see how to write a simple function which generates XML using the special XML syntax as well as dynamically creating some of the XML using ordinary Haskell expressions.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='comment'>-- |hello creates a simple hello <noun> page </span><br /><span class='varop'>></span> <span class='definition'>hello</span> <span class='keyglyph'>::</span> <span class='conid'>String</span> <span class='keyglyph'>-></span> <span class='conid'>HSP</span> <span class='conid'>XML</span><br /><span class='varop'>></span> <span class='definition'>hello</span> <span class='varid'>noun</span> <span class='keyglyph'>=</span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>html</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>head</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>title</span><span class='varop'>></span><span class='conid'>Hello</span><span class='layout'>,</span> <span class='varop'><%</span> <span class='varid'>noun</span> <span class='varop'>%></</span><span class='varid'>title</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'></</span><span class='varid'>head</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>body</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>p</span><span class='varop'>></span><span class='conid'>Hello</span><span class='layout'>,</span> <span class='varop'><%</span> <span class='varid'>map</span> <span class='varid'>toUpper</span> <span class='varid'>noun</span> <span class='varop'>%></</span><span class='varid'>p</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'></</span><span class='varid'>body</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'></</span><span class='varid'>html</span><span class='varop'>></span><br /><span class='varop'>></span><br /></pre><br /><p>To use the XML syntax, we just use it -- no special escaping is required to insert XML in the middle of a Haskell expression. On the other hand, if we want to evaluate a Haskell expression and insert it into the XML, then we need the special <code><% %></code> escape clause. If we had just written, <title> Hello, noun</title> then the <code>title</code> would have been <kbd>Hello, noun</kbd>. Likewise, if we did not have the <code><% %></code> around <code>map toUpper noun</code>, the page would say, "Hello map toUpper noun" instead of evaluating <code>map toUpper noun</code> and substituting the result in.</p><br /><p>We can evaluate any arbitrary expression inside the <code><% %></code>, provided HSP knows how to turn the result into an XML value. By default, HSP knows how to handle <code>String</code>, <code>Char</code>, numbers, and some other common Haskell data-types. You can add additional class instances if you want to convert other datatypes (including your own) to XML automatically.</p><br /> <p>Next we have a simple <code>main</code>. The first line evaluates <code>hello</code> and gets back the generated XML. The second line uses <code>renderAsHtml</code> to turn the XML into HTML. <code>renderAsHtml</code> expects you to pass in valid XHTML which it will transform into valid HTML. However, it does not check the validity of the input or output. We will do that using a more general mechanism in the HAppS code.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='definition'>main</span> <span class='keyglyph'>::</span> <span class='conid'>IO</span> <span class='conid'>()</span><br /><span class='varop'>></span> <span class='definition'>main</span> <span class='keyglyph'>=</span> <br /><span class='varop'>></span> <span class='keyword'>do</span> <span class='layout'>(</span><span class='keyword'>_</span><span class='layout'>,</span> <span class='varid'>xml</span><span class='layout'>)</span> <span class='keyglyph'><-</span> <span class='varid'>evalHSP</span> <span class='layout'>(</span><span class='varid'>hello</span> <span class='str'>"World"</span><span class='layout'>)</span> <span class='conid'>Nothing</span><br /><span class='varop'>></span> <span class='varid'>putStrLn</span> <span class='layout'>(</span><span class='varid'>renderAsHTML</span> <span class='varid'>xml</span><span class='layout'>)</span><br /><span class='varop'>></span><br /></pre><br /><p>If we run this example, we get the following output:</p><br /><pre><br /><html<br />><head<br /> ><title<br /> >Hello, World</title<br /> ></head<br /> ><body<br /> ><p<br /> >Hello, WORLD</p<br /> ></body<br /> ></html<br />><br /></pre><br /><p>The formatting probably looks a bit funny to you -- but the web browser will read it fine. In HTML, the whitespace between the open and close tags is sometimes significant. However, the whitespace inside the open or close tag itself is never significant. The rendering algorithm is designed to exploit those properties to ensure it never adds significant whitespace where you didn't explicit have it in the input file.</p><br /><h4>Integrating HSP with HAppS</h4><br /><p>The integration of HSP with HAppS brings a few things to the table:</p><br /><ul><br /> <li>Dynamic HSP recompilation and loading - this means you can modify your HSP templates with out having to completely recompile and restart your HAppS application. Recompiling an HSP page usually happens faster than I can switch from emacs to the web browser and hit reload.<br /> <li>JSON data serialization - in order to pass data from HAppS to the HSP template, the data is serialized as a JSON object. This can be especially useful if you want to extend your application to support AJAX or if you want to expose your API to third parties using JSON, because you can use a single unified JSON API for all three instead of having the JSON API be some extra thing that is tact on.<br /></ul><br /> <p>If you decide you do not like using JSON and the dynamic page loading provided happs-hsp-template, but you do like HSP, it should be fairly straight forward to use HSP directly in your HAppS application, similar to how you would use Text.XHtml. However, this quickstart will focus on using happs-hsp-template.</p><br /> <h5>Basic framework</h5><br /> <p>This is the basic framework for creating a HAppS application using:</p><br /> <ul><br /> <li>HAppS-Server for HTTP, cookies, etc<br /> <li>HAppS-State for persistent storage (aka, our database)<br /> <li>HSP for HTML templating<br /> <li>RJson for JSON marshalling<br /> </ul><br /><pre><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Main</span> <span class='keyword'>where</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Concurrent</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Monad</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Server</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>State</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HSP</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Template</span><span class='varop'>.</span><span class='conid'>HSP</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Template</span><span class='varop'>.</span><span class='conid'>HSP</span><span class='varop'>.</span><span class='conid'>Handle</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Interface</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>State</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>System</span><span class='varop'>.</span><span class='conid'>Environment</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Text</span><span class='varop'>.</span><span class='conid'>RJson</span><br /></pre><br /> <p>This <code>main</code> function is mostly boilerplate.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='definition'>main</span> <span class='keyglyph'>::</span> <span class='conid'>IO</span> <span class='conid'>()</span><br /><span class='varop'>></span> <span class='definition'>main</span> <span class='keyglyph'>=</span> <br /></pre><br /> <p><code>newStore</code> just creates an <code>IORef</code> to a <code>Map</code> which will map template source files to compiled object files.</p><br /><pre><span class='varop'>></span> <span class='keyword'>do</span> <span class='varid'>store</span> <span class='keyglyph'><-</span> <span class='varid'>newStore</span><br /></pre><br /> <p><code>startSystemState</code> starts the state transaction system.</p><br /><pre><span class='varop'>></span> <span class='varid'>control</span> <span class='keyglyph'><-</span> <span class='varid'>startSystemState</span> <span class='varid'>entryPoint</span><br /></pre><br /> <p>Next we parse the command-line arguments to extract a default config file. We also enable validation using wdg-html-validator. In theory, validation should be enabled and disable via a command-line argument, but I have not quite figured out how to add the desired functionality to <code>parseConfig</code>. If you do not have a version of HAppS-Server which support validation, then just change the last line to <code>Right c -> c</code>.</p><br /><pre><span class='varop'>></span> <span class='varid'>eConf</span> <span class='keyglyph'><-</span> <span class='varid'>liftM</span> <span class='varid'>parseConfig</span> <span class='varid'>getArgs</span><br /><span class='varop'>></span> <span class='keyword'>let</span> <span class='varid'>conf</span> <span class='keyglyph'>=</span> <span class='keyword'>case</span> <span class='varid'>eConf</span> <span class='keyword'>of</span><br /><span class='varop'>></span> <span class='conid'>Left</span> <span class='varid'>e</span> <span class='keyglyph'>-></span> <span class='varid'>error</span> <span class='layout'>(</span><span class='varid'>unlines</span> <span class='varid'>e</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='conid'>Right</span> <span class='varid'>c</span> <span class='keyglyph'>-></span> <span class='varid'>c</span> <span class='layout'>{</span> <span class='varid'>validator</span> <span class='keyglyph'>=</span> <span class='conid'>Just</span> <span class='varid'>wdgHTMLValidator</span> <span class='layout'>}</span><br /></pre><br /> <p>We then fork off the HTTP process in a new thread. This will take care of handling all incoming requests for us in the background.</p><br /><pre><span class='varop'>></span> <span class='varid'>tid</span> <span class='keyglyph'><-</span> <span class='varid'>forkIO</span> <span class='varop'>$</span> <span class='varid'>simpleHTTP</span> <span class='varid'>conf</span> <span class='varop'>$</span> <span class='varid'>impl</span> <span class='varid'>store</span><br /></pre><br /> <p>Now we are just waiting around for someone to terminate us.</p><br /><pre><span class='varop'>></span> <span class='varid'>putStrLn</span> <span class='str'>"running..."</span><br /><span class='varop'>></span> <span class='varid'>waitForTermination</span><br /></pre><br /> <p>Finally we cleanly shutdown the system. The system will not get corrupted if it is shutdown unexpected or improperly, however it may be possible that some pending transactions are lost. (That is true of most (or all?) database systems. The Consistency part of ACID means that the database will not get corrupt. That neccesarily means that sometimes data will be lost during an abrupt shutdown or intentionally thrown out during recovery.)</p><br /><pre><span class='varop'>></span> <span class='varid'>killThread</span> <span class='varid'>tid</span><br /><span class='varop'>></span> <span class='varid'>shutdownSystem</span> <span class='varid'>control</span><br /><span class='varop'>></span> <span class='varid'>destroyStore</span> <span class='varid'>store</span><br /></pre><br /><p><code>entryPoint</code> is just there to provide type information to <code>startSystemState</code>. The type should be the one you want stored in your State. In our case that is <code>HitCounter</code>.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='definition'>entryPoint</span> <span class='keyglyph'>::</span> <span class='conid'>Proxy</span> <span class='conid'>HitCounter</span><br /><span class='varop'>></span> <span class='definition'>entryPoint</span> <span class='keyglyph'>=</span> <span class='conid'>Proxy</span><br /></pre><br /><p>The <code>impl</code> function is your typical HAppS-Server impl function with a few new constructs:</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='definition'>impl</span> <span class='keyglyph'>::</span> <span class='conid'>Store</span> <span class='keyglyph'>-></span> <span class='keyglyph'>[</span><span class='conid'>ServerPart</span> <span class='conid'>Response</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='definition'>impl</span> <span class='varid'>store</span> <span class='keyglyph'>=</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span> <span class='varid'>runHSPHandle</span> <span class='str'>"pages"</span> <span class='str'>"objfiles"</span> <span class='varid'>store</span> <span class='varop'>$</span> <span class='varid'>multi</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span> <span class='varid'>dir</span> <span class='str'>"json"</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span> <span class='varid'>dir</span> <span class='str'>"times"</span> <br /><span class='varop'>></span> <span class='keyglyph'>[</span> <span class='varid'>path</span> <span class='varop'>$</span> <span class='keyglyph'>\</span><span class='varid'>c</span> <span class='keyglyph'>-></span> <br /><span class='varop'>></span> <span class='keyglyph'>[</span> <span class='varid'>method</span> <span class='conid'>GET</span> <span class='varop'>$</span> <span class='varid'>ok</span> <span class='layout'>(</span><span class='varid'>toResponse</span> <span class='layout'>(</span><span class='varid'>toJson</span> <span class='layout'>(</span><span class='keyword'>if</span> <span class='varid'>c</span> <span class='varop'>==</span> <span class='layout'>(</span><span class='num'>1</span> <span class='keyglyph'>::</span> <span class='conid'>Integer</span><span class='layout'>)</span> <span class='keyword'>then</span> <span class='str'>"time"</span> <span class='keyword'>else</span> <span class='str'>"times"</span><span class='layout'>)</span><span class='layout'>)</span><span class='layout'>)</span> <br /><span class='varop'>></span> <span class='keyglyph'>]</span> <br /><span class='varop'>></span> <span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='layout'>,</span> <span class='varid'>method</span> <span class='conid'>GET</span> <span class='varop'>$</span> <span class='keyword'>do</span> <span class='varid'>hits</span> <span class='keyglyph'><-</span> <span class='varid'>webUpdate</span> <span class='conid'>IncHits</span><br /><span class='varop'>></span> <span class='varid'>addParam</span> <span class='str'>"hits"</span> <span class='varid'>hits</span><br /><span class='varop'>></span> <span class='varid'>ok</span> <span class='varop'>=<<</span> <span class='varid'>execTemplate</span> <span class='conid'>Nothing</span> <span class='str'>"Index.hs"</span><br /><span class='varop'>></span> <span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>]</span><br /></pre><br /><p><code>runHSPHandle</code> takes four arguments</p><br /><ol><br /> <li>the directory which holds the templates<br /> <li>the directory to store the compiled template object files in<br /> <li>a handle to the <code>store</code> which manages the objects<br /> <li>a <code>ServerPart a</code><br /></ol><br /> <p><code>dir "json"</code> provides our JSON API, which can be accessed via HSP, client-side javascript, or as a 3rd party API. We currently only provide one API call, <code>times</code>. It takes an integer and returns <code>"time"</code> if it is 1, otherwise <code>"times"</code>. The <code>toJson</code> function converts the value into JSON data. Our JSON API must be inside <code>runHSPHandle</code>.</p><br /><p>The <code>addParam</code> function adds the <code>hits</code> value to the environment used by the page template later. This allows us to pass information to the page template without having to add a JSON API call. We can not always use this method because we may not know what information the page template will need until we run it. More on this later.</p><br /><p>The <code>execTemplate</code> function is responsible for actually invoking a specific template. It takes two arguments:</p><br /><ol><br /> <li>default XMLMetaData (mime-type, doctype, etc), to use if the template does not provide any values.<br /> <li>the name of the template file relative to the directory passed to <code>runHSPHandle</code><br /></ol><br /> <h4>JSON Interface</h4><br /> <p><a href="http://json.org"><abbr title="Javascript Object Notation">JSON</abbr></a>, short for Javascript Object Notation, is a lightweight data-interchange format. It's primary appeal is that it is natively supported by Javascript. This means you can create javascript objects using JSON notation in the javascript program text, and you can easily convert between JSON and javascript objects at runtime.</p><br /> <p><kbd>happs-hsp-template</kbd> uses <a href="http://hackage.haskell.org/cgi-bin/hackage-scripts/package/RJson-0.3.4">RJson</a> for converting Haskell values to and from JSON. See the <kbd>README</kbd> in the RJson tarball for more information on using RJson.</p><br /> <p>We define our datatypes which will be turned into JSON objects in <kbd>pages/Interface.lhs</kbd>. It is in the <kbd>pages</kbd> subdirectory because it needs to be imported by both our HAppS backend as well the the HSP templates. This ensures that they are both talking the same specification.</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}</span><br /></pre><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Interface</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Data</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>State</span><br /></pre><br /><p>Next we define a type which will be used as JSON data:</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Num</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Enum</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>newtype</span> <span class='conid'>HitCounter</span> <span class='keyglyph'>=</span> <span class='conid'>HitCounter</span> <span class='layout'>{</span> <span class='varid'>hits</span> <span class='keyglyph'>::</span> <span class='conid'>Integer</span> <span class='layout'>}</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /></pre><br /> <p>One key thing to note is that we use the record syntax in our type declaration. This is so that RJson can automatically convert our data-type into a JSON object. If we did not do this, we would have to manually declare instances of <code>ToJson</code> and <code>FromJson</code> for HitCounter.</p><br /> <p>It is critical that you read the RJson <kbd>README</kbd> to get an understanding of what it supports, or you will be mystified as to why your JSON data is getting silently corrupted.</p><br /> <p>Because we might want to store HitCounter in the HAppS State we also <code>deriveSerialize</code> for it, and create a <code>Version</code> instance. In theory this could also be useful if clients attempt to pass JSON data back to the server using an older format. However, I have no idea how to handle that in practice.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>HitCounter</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>HitCounter</span><br /></pre><br /> <h4>State Definition</h4><br /> <p>In this quickstart our State will just be <code>HitCounter</code>. There is nothing HSP-specific about this module, aside from the fact that we have already defined <code>HitCounter</code> in <kbd>Interface.lhs</kbd>.</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# LANGUAGE TemplateHaskell, UndecidableInstances, FlexibleInstances, GeneralizedNewtypeDeriving, MultiParamTypeClasses, DeriveDataTypeable, TypeFamilies #-}</span><br /><span class='varop'>></span> <span class='comment'>{-# OPTIONS_GHC -F -pgmF trhsx #-}</span><br /><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>State</span> <span class='keyword'>where</span> <br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Data</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>State</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Concurrent</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Monad</span><span class='varop'>.</span><span class='conid'>State</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Interface</span><br /></pre><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='comment'>-- |increment hit counter, and return new value</span><br /><span class='varop'>></span> <span class='definition'>incHits</span> <span class='keyglyph'>::</span> <span class='conid'>Update</span> <span class='conid'>HitCounter</span> <span class='conid'>HitCounter</span><br /><span class='varop'>></span> <span class='definition'>incHits</span> <span class='keyglyph'>=</span> <br /><span class='varop'>></span> <span class='keyword'>do</span> <span class='varid'>hc</span> <span class='keyglyph'><-</span> <span class='varid'>fmap</span> <span class='varid'>succ</span> <span class='varid'>get</span><br /><span class='varop'>></span> <span class='varid'>put</span> <span class='varid'>hc</span><br /><span class='varop'>></span> <span class='varid'>return</span> <span class='varid'>hc</span><br /></pre><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>mkMethods</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>HitCounter</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span> <span class='chr'>'</span><span class='varid'>incHits</span><br /><span class='varop'>></span> <span class='keyglyph'>]</span><span class='layout'>)</span><br /></pre><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Component</span> <span class='conid'>HitCounter</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='keyword'>type</span> <span class='conid'>Dependencies</span> <span class='conid'>HitCounter</span> <span class='keyglyph'>=</span> <span class='conid'>End</span><br /><span class='varop'>></span> <span class='varid'>initialValue</span> <span class='keyglyph'>=</span> <span class='conid'>HitCounter</span> <span class='num'>0</span><br /></pre><br /> <h4>Index Page</h4><br /> <p>Now that we have our JSON interface and our State defined, we can implement a page template.</p><br /> <p>We will use the OPTIONS_GHC pragma to automatically pass this file through <kbd>trhsx</kbd>. <kbd>happs-hsp-template</kbd> will automatically add this command-line option when compiling pages, but adding it explicitly means we can easily load the file into GHCi.</p><br /><pre><span class='varop'>></span> <span class='comment'>{-# OPTIONS_GHC -F -pgmFtrhsx #-}</span><br /></pre><br /><pre><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Index</span> <span class='keyword'>where</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Monad</span><span class='varop'>.</span><span class='conid'>Trans</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HSP</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Template</span><span class='varop'>.</span><span class='conid'>HSP</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>UACCT</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HSP</span><span class='varop'>.</span><span class='conid'>Google</span><span class='varop'>.</span><span class='conid'>Analytics</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>State</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Interface</span><br /></pre><br /> <p>In our templates, the entry point is always <code>page</code> and it always has the type <code>Web XML</code>. (<i>If you use HSP without HAppS then you will want HSP XML instead of Web XML</i>). This is similar to how normal programs always start at <code>main</code> and <code>main</code> always has the type <code>IO ()</code>.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='definition'>page</span> <span class='keyglyph'>::</span> <span class='conid'>Web</span> <span class='conid'>XML</span><br /><span class='varop'>></span> <span class='definition'>page</span> <span class='keyglyph'>=</span> <br /><span class='varop'>></span> <span class='varid'>withMetaData</span> <span class='varid'>html4Strict</span> <span class='varop'>$</span><br /><span class='varop'>></span> <span class='keyword'>do</span> <span class='layout'>(</span><span class='conid'>HitCounter</span> <span class='varid'>hits</span><span class='layout'>)</span> <span class='keyglyph'><-</span> <span class='varid'>localRead</span> <span class='str'>"hits"</span><br /><span class='varop'>></span> <span class='varid'>times</span> <span class='keyglyph'><-</span> <span class='varid'>globalRead</span> <span class='varop'>$</span> <span class='varid'>simpleRequest</span> <span class='layout'>(</span><span class='str'>"/json/times/"</span> <span class='varop'>++</span> <span class='varid'>show</span> <span class='varid'>hits</span><span class='layout'>)</span><br /><span class='varop'>></span> <span class='varid'>page'</span> <span class='varid'>hits</span> <span class='varid'>times</span><br /></pre><br /> <p>Our <code>page</code> function does fours things.</p><br /> <ol><br /> <li>Sets the XML meta-data to html4Strict. This will:<br /> <ul><br /> <li>render the page as HTML (instead of XML)<br /> <li>set the DOCTYPE to HTML 4.01 Strict.<br /> <li>set the content-type to "text/html"<br /> </ul><br /> <li>reads the <code>"hits"</code> value from the local environment.<br /> <li>Uses the JSON API to determine if it should say, "time" or "times"<br /> <li>Calls the <code>page'</code> function<br /> </ol><br /> <p>The environment read by <code>localRead</code> is created way back in the <code>impl</code> function when we did:</p><br /><pre><span class='varop'>></span> <span class='varid'>addParam</span> <span class='str'>"hits"</span> <span class='varid'>hits</span><br /></pre><br /> <p><code>globalRead</code> simulates an HTTP request to the HAppS server. In this example, we just do a simple <kbd>GET</kbd> request to <kbd>/json/times</kbd>, but any type of HTTP request can be simulated.</p><br /> <p>If <code>globalRead</code> returns <kbd>"Missing content-type"</kbd>, it is probably because your JSON API is not inside the runHSPHandle call.</p><br /> <p>The <code>page'</code> function is straight-forward HSP code. We could have put the contents in the <code>page</code> function itself, but I prefer to split it out, because it looks neater, and makes gives you the option of calling the <code>page'</code> function.</p><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='definition'>page'</span> <span class='keyglyph'>::</span> <span class='conid'>Integer</span> <span class='keyglyph'>-></span> <span class='conid'>String</span> <span class='keyglyph'>-></span> <span class='conid'>Web</span> <span class='conid'>XML</span><br /><span class='varop'>></span> <span class='definition'>page'</span> <span class='varid'>hits</span> <span class='varid'>times</span> <span class='keyglyph'>=</span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>html</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>head</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>meta</span> <span class='varid'>http</span><span class='comment'>-</span><span class='varid'>equiv</span><span class='keyglyph'>=</span><span class='str'>"Content-Type"</span> <span class='varid'>content</span><span class='keyglyph'>=</span><span class='str'>"text/html; charset=utf-8"</span> <span class='varop'>/></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>title</span><span class='varop'>></span><span class='conid'>HitCounter</span> <span class='num'>3000</span><span class='varop'></</span><span class='varid'>title</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>link</span> <span class='varid'>rel</span><span class='keyglyph'>=</span><span class='str'>"shortcut icon"</span> <span class='varid'>href</span><span class='keyglyph'>=</span><span class='str'>"/favicon.ico"</span> <span class='keyword'>type</span><span class='keyglyph'>=</span><span class='str'>"image/x-icon"</span> <span class='varop'>/></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>link</span> <span class='keyword'>type</span><span class='keyglyph'>=</span><span class='str'>"text/css"</span> <span class='varid'>rel</span><span class='keyglyph'>=</span><span class='str'>"stylesheet"</span> <span class='varid'>href</span><span class='keyglyph'>=</span><span class='str'>"/css/hg.css"</span> <span class='varid'>media</span><span class='keyglyph'>=</span><span class='str'>"screen"</span> <span class='varid'>title</span><span class='keyglyph'>=</span><span class='str'>"default"</span> <span class='varop'>/></span><br /><span class='varop'>></span> <span class='varop'></</span><span class='varid'>head</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>body</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>h1</span><span class='varop'>></span><span class='conid'>HitCounter</span> <span class='num'>3000</span><span class='varop'></</span><span class='varid'>h1</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><</span><span class='varid'>p</span><span class='varop'>></span><span class='conid'>This</span> <span class='varid'>page</span> <span class='varid'>has</span> <span class='varid'>been</span> <span class='varid'>viewed</span> <span class='varop'><%</span> <span class='layout'>(</span><span class='varid'>show</span> <span class='varid'>hits</span><span class='layout'>)</span> <span class='varop'>++</span> <span class='str'>" "</span> <span class='varop'>++</span> <span class='varid'>times</span> <span class='varop'>%>.</</span><span class='varid'>p</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'><%</span> <span class='varid'>analytics</span> <span class='varid'>uacct</span> <span class='varop'>%></span><br /><span class='varop'>></span> <span class='varop'></</span><span class='varid'>body</span><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'></</span><span class='varid'>html</span><span class='varop'>></span><br /></pre><br /> <h4>Google Analytics Code</h4><br /> <p>The <code>UACCT</code> module is just a place to keep our Google Analytics account code so that we can easily import it into all the pages on the site. Since you are likely to want to use a different Google Analytics code for each site, it makes more sense to keep this code in the <kbd>pages</kbd> directory on a per-project basis, rather than store it in a system-wide library.</p><br /><pre><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>UACCT</span> <span class='keyword'>where</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HSP</span><span class='varop'>.</span><span class='conid'>Google</span><span class='varop'>.</span><span class='conid'>Analytics</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='definition'>uacct</span> <span class='keyglyph'>::</span> <span class='conid'>UACCT</span><br /><span class='varop'>></span> <span class='definition'>uacct</span> <span class='keyglyph'>=</span> <span class='conid'>UACCT</span> <span class='str'>"UA-4353757-1"</span><br /></pre><br /> <h4>Conclusion</h4><br /> <p>I hope you now have a basic idea of what is going on. The next step is to extract the template directory and attempt to make your own site.</p><br /> <p>In this quickstart, our State and our JSON interface are very closely related. In fact, they are just the <code>HitCounter</code> type. For a simple application such as this one, Storing your JSON objects directly in the State seems like a sensible approach. For other sites, you will find that you want to have one set of data types for your State, and a separate set of data types for your Interface, though some data types may still be shared between the two.</p>Jeremy Shawhttp://www.blogger.com/profile/18373967098081701148noreply@blogger.com5tag:blogger.com,1999:blog-6381777424713448943.post-63106549523883678972008-07-03T11:10:00.000-07:002008-07-03T19:41:34.332-07:00Extending Asterisk with HAppS<p>I just glued two of my favorite technologies together, <a href="http://www.asterisk.org">Asterisk</a> (the opensource PBX/VoIP/etc system) and <a href="http://www.happs.org/">HAppS</a> (the Haskell Application Server framework).</p><br /><br /><p>If you have heard of HAppS, but never used it, you may have the impression that HAppS is a web development platform -- but that is not quite correct. HAppS is actually a collection of several different server components which can be combined together or used separately.</p><br /><br /><p>In this post, I will show how to build as simple FastAGI server on top of the HAppS-State component. We will not be using the web component (HAppS-Server), which is the part that provides HTTP, templating, cookies, etc. This post assumes no prior knowledge of HAppS or AGI.</p><br /><br /><h2>What You Will Need</h2><br /><p>If you want to build this demo you will need:</p><br /><ul><br /><li>the latest version of HAppS-State and it's dependencies.<br /><li>Asterisk (I think any version later than 1.0 should work. I use 1.4.17 from Ubuntu Hardy)<br /><li>The <a href="http://hackage.haskell.org/cgi-bin/hackage-scripts/package/AGI">haskell AGI library</a><br /><li><code>darcs get http://src.seereason.com/fastagi-hitcounter</code><br /></ul><br /><br /> <p>Run <code>make HitCounter.hs</code> to produce a nice, clean<br /> <code>.hs</code> file from the <code>.lhs</code> file.</p><br /><br /><h2>FastAGI</h2><br /><br /> <p>The asterisk server can be extended by using the <a href="http://www.voip-info.org/wiki/index.php?page=Asterisk+AGI">Asterisk Gateway Interface (AGI)</a>. AGI provides the functionality you need to do stuff like <i>"Please enter your 16-digit account number."</i></p><br /><br /> <p>An AGI script is a standalone program you write (in a language of your choice). Asterisk communications with your AGI script by running it directly, and writing to its stdin and reading from its stdout. The AGI protocol consists of simple <a href="http://gundy.org/asterisk/agi.html">commands and<br /> responses</a> which are human readable text.</p><br /><br /><p>Asterisk also has the option of communicating with your AGI script remotely via TCP instead of directly running a local program. This feature is known as FastAGI. The commands and responses are identical to normal AGI, the only differences are:</p><br /><ul><br /><li>The communication channel is setup via TCP instead of forking off a local process<br /><li>Some extra AGI variables are passed in by FastAGI<br /></ul><br /> <br /> <p>There is one additional importing difference, which is more of a side-effect. When using plain-old AGI, a new process will be spawned for each call. When using FastAGI, a new TCP connection will be opened -- typically to a single, long running server process. So, with AGI you will need to worry about how to provide communication and synchronization between multiple processes, but with FastAGI, you can just use threads.</p><br /><br /><h2>HAppS-State</h2><br /><br /> <p>The HAppS State component provides in-memory state with ACID guarantees. It uses write ahead logging and checkpointing to ensure the state can be restored from disk in the event of a power outage, and also provides multimaster replication. Unlike a traditional relational database, HAppS-State works directly with native, arbitrary Haskell data types. This means you don't have to figure out how to get your beautiful data structures wedged into a relational database just to get ACID guarantees and replication.</p><br /><br /><h2>Example Application</h2><br /><p>The remainder of the post is a simple example which implements a hit-counter. When you call the phone number, it tells you what caller number you are. I won't go into too much detail about the HAppS State portion, since this post is supposed to show how to integrate AGI, not how to use HAppS State.</p><br /><br /><pre><span class='varop'>></span> <span class='comment'>{-# LANGUAGE TemplateHaskell, UndecidableInstances, TypeFamilies, <br />> TypeSynonymInstances, FlexibleInstances, DeriveDataTypeable, <br />> MultiParamTypeClasses, TypeOperators, GeneralizedNewtypeDeriving #-}</span><br /></pre><br /><pre><span class='varop'>></span> <span class='keyword'>module</span> <span class='conid'>Main</span> <span class='keyword'>where</span><br /><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Concurrent</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Monad</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Monad</span><span class='varop'>.</span><span class='conid'>Reader</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Monad</span><span class='varop'>.</span><span class='conid'>State</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Control</span><span class='varop'>.</span><span class='conid'>Monad</span><span class='varop'>.</span><span class='conid'>Trans</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>Data</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>HAppS</span><span class='varop'>.</span><span class='conid'>State</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Network</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>Network</span><span class='varop'>.</span><span class='conid'>AGI</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>System</span><span class='varop'>.</span><span class='conid'>Random</span><br /><span class='varop'>></span> <span class='keyword'>import</span> <span class='conid'>System</span><span class='varop'>.</span><span class='conid'>Posix</span><span class='varop'>.</span><span class='conid'>Unistd</span><br /></pre><br /> <p>The first thing we do is define the type we will use to store our<br /> persistent state (aka, our "database schema"). The <code>deriveAll</code> is<br /> similar to <code>deriving (Eq, Ord, Read, Show, Num, Enum, Default,<br /> Data, Typeable)</code>. Since there is no way to extend<br /> <code>deriving</code>, we have to use Template Haskell to add support<br /> for deriving <code>Default</code>.<br /> </p><br /><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveAll</span> <span class='keyglyph'>[</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Eq</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Ord</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Read</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Show</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Num</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Enum</span><span class='layout'>,</span><span class='chr'>'</span><span class='chr'>'</span><span class='conid'>Default</span><span class='keyglyph'>]</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span><span class='varid'>d</span><span class='keyglyph'>|</span><br /><span class='varop'>></span> <span class='keyword'>newtype</span> <span class='conid'>HitCounter</span> <span class='keyglyph'>=</span> <span class='conid'>HitCounter</span> <span class='layout'>{</span> <span class='varid'>hits</span> <span class='keyglyph'>::</span> <span class='conid'>Integer</span> <span class='layout'>}</span><br /><span class='varop'>></span> <span class='keyglyph'>|</span><span class='keyglyph'>]</span><span class='layout'>)</span><br /></pre><br /><br /> <p><code>deriveSerialize</code> is part of the magic that allows HitCounter<br /> to be serialized to disk or replicated between servers.</p><br /><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>deriveSerialize</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>HitCounter</span><span class='layout'>)</span><br /></pre><br /> <p>The <code>Version</code> instance is used to migrate the<br /> old data, if we modify the <code>HitCounter</code> data structure. That<br /> is a subject for a different tutorial.</p><br /><br /><pre><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Version</span> <span class='conid'>HitCounter</span><br /></pre><br /> <p>Next we define a function which modifies the global state<br /> (<code>HitCounter</code>). This function while be run<br /> atomically. This means that there is no race condition between the<br /> <code>get</code> and the <code>put</code>. The <code>get</code> and<br /> <code>put</code> functions come from<br /> <code>Control.Monad.State</code>.</p><br /><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='comment'>-- |increment hit counter, and return new value</span><br /><span class='varop'>></span> <span class='varid'>incHits</span> <span class='keyglyph'>::</span> <span class='conid'>Update</span> <span class='conid'>HitCounter</span> <span class='conid'>Integer</span><br /><span class='varop'>></span> <span class='varid'>incHits</span> <span class='keyglyph'>=</span> <br /><span class='varop'>></span> <span class='keyword'>do</span> <span class='varid'>hc</span> <span class='keyglyph'><-</span> <span class='varid'>fmap</span> <span class='varid'>succ</span> <span class='varid'>get</span><br /><span class='varop'>></span> <span class='varid'>put</span> <span class='varid'>hc</span><br /><span class='varop'>></span> <span class='varid'>return</span> <span class='layout'>(</span><span class='varid'>hits</span> <span class='varid'>hc</span><span class='layout'>)</span><br /></pre><br /> <p>This is the magic which converts the <code>incHits</code> function<br /> into an atomic action for updating the global state.</p><br /><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varop'>$</span><span class='layout'>(</span><span class='varid'>mkMethods</span> <span class='chr'>'</span><span class='chr'>'</span><span class='conid'>HitCounter</span><br /><span class='varop'>></span> <span class='keyglyph'>[</span> <span class='chr'>'</span><span class='varid'>incHits</span><br /><span class='varop'>></span> <span class='keyglyph'>]</span><span class='layout'>)</span><br /></pre><br /> <p>Next we define our top-level component which uses the global state. A more complex application might use a bunch of independent components similar to HitCounter. This allows us to easily build things like session support, user accounts, etc, in third party reusable libraries. I believe it also makes atomic actions finer grained and makes it possible to support shards. </p><br /><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='keyword'>instance</span> <span class='conid'>Component</span> <span class='conid'>HitCounter</span> <span class='keyword'>where</span><br /><span class='varop'>></span> <span class='keyword'>type</span> <span class='conid'>Dependencies</span> <span class='conid'>HitCounter</span> <span class='keyglyph'>=</span> <span class='conid'>End</span><br /><span class='varop'>></span> <span class='varid'>initialValue</span> <span class='keyglyph'>=</span> <span class='conid'>HitCounter</span> <span class='num'>0</span><br /></pre><br /><pre><span class='varop'>></span> <span class='varid'>entryPoint</span> <span class='keyglyph'>::</span> <span class='conid'>Proxy</span> <span class='conid'>HitCounter</span><br /><span class='varop'>></span> <span class='varid'>entryPoint</span> <span class='keyglyph'>=</span> <span class='conid'>Proxy</span><br /></pre><br /><p>The <code>main</code> function starts up the state engine, forks off the <code>fastAGI</code> server, waits for a shutdown signal (for example, <kbd>^C</kbd>), and then cleanly shuts down the state engine. The <code>fastAGI</code> function comes from the Haskell AGI library, and is in no way HAppS specific.</p><br /><br /><pre><span class='varop'>></span><br /><span class='varop'>></span> <span class='varid'>main</span> <span class='keyglyph'>::</span> <span class='conid'>IO</span> <span class='conid'>()</span><br /><span class='varop'>></span> <span class='varid'>main</span> <span class='keyglyph'>=</span><br /><span class='varop'>></span> <span class='keyword'>do</span> <span class='varid'>control</span> <span class='keyglyph'><-</span> <span class='varid'>startSystemState</span> <span class='varid'>entryPoint</span><br /><span class='varop'>></span> <span class='varid'>tid</span> <span class='keyglyph'><-</span> <span class='varid'>forkIO</span> <span class='varop'>$</span> <span class='varid'>fastAGI</span> <span class='conid'>Nothing</span> <span class='varid'>agiMain</span><br /><span class='varop'>></span> <span class='varid'>putStrLn</span> <span class='str'>"running..."</span><br /><span class='varop'>></span> <span class='varid'>waitForTermination</span><br /><span class='varop'>></span> <span class='varid'>killThread</span> <span class='varid'>tid</span><br /><span class='varop'>></span> <span class='varid'>shutdownSystem</span> <span class='varid'>control</span><br /></pre><br /><p>Here is our simple AGI application. It</p><br /><ol><br /> <li>answers the call<br /> <li>waits a second to give the caller time to finish setting up their end of the call<br /> <li>increments the hit counter<br /> <li>plays a pre-recorded file which says, "You are currently caller number"<br /> <li>says the caller number<br /> <li>plays a pre-recorded file which says "Goodbye."<br /> <li>hangs up<br /></ol><br /> <br /> <p> The functions <code>answer</code>, <code>streamFile</code>,<br /> <code>sayNumber</code>, and <code>hangUp</code> come from the AGI<br /> library.</p><br /><br /> <p>The <code>update IncHits</code> call is our database query. Note<br /> that we don't call the <code>incHits</code> function<br /> directly. Instead we call <code>update</code> and pass it the value<br /> <code>IncHits</code>. The <code>IncHits</code> type was created for<br /> us automatically by the call to <code>mkMethods</code> we made<br /> earlier.</p><br /><br /><pre><span class='varop'>></span> <br /><span class='varop'>></span> <span class='varid'>agiMain</span> <span class='keyglyph'>::</span> <span class='conid'>HostName</span> <span class='keyglyph'>-></span> <span class='conid'>PortNumber</span> <span class='keyglyph'>-></span> <span class='conid'>AGI</span> <span class='conid'>()</span><br /><span class='varop'>></span> <span class='varid'>agiMain</span> <span class='varid'>hostname</span> <span class='varid'>portNum</span> <span class='keyglyph'>=</span><br /><span class='varop'>></span> <span class='keyword'>do</span> <span class='varid'>answer</span><br /><span class='varop'>></span> <span class='varid'>liftIO</span> <span class='varop'>$</span> <span class='varid'>sleep</span> <span class='num'>1</span> <span class='comment'>-- give the caller time to get their end of the call setup</span><br /><span class='varop'>></span> <span class='varid'>h</span> <span class='keyglyph'><-</span> <span class='varid'>update</span> <span class='conid'>IncHits</span><br /><span class='varop'>></span> <span class='varid'>streamFile</span> <span class='str'>"queue-thereare"</span> <span class='conid'>[]</span> <span class='conid'>Nothing</span><br /><span class='varop'>></span> <span class='varid'>sayNumber</span> <span class='varid'>h</span> <span class='conid'>[]</span><br /><span class='varop'>></span> <span class='varid'>streamFile</span> <span class='str'>"vm-goodbye"</span> <span class='conid'>[]</span> <span class='conid'>Nothing</span><br /><span class='varop'>></span> <span class='varid'>hangUp</span> <span class='conid'>Nothing</span><br /><span class='varop'>></span> <span class='varid'>return</span> <span class='conid'>()</span><br /></pre><br /> <h2>Hooking it up</h2><br /><p>To test the application, first we need to update the asterisk dialplan to call our AGI application. Something like this should do the trick (be sure to reload the dialplan after modifying <code>extensions.conf</code>):</p><br /><pre><br />[default]<br />exten => 31415,1,AGI(agi://127.0.0.1)<br /></pre><br /><p>Next we start our AGI application server:</p><br /><pre><br /> $ runhaskell HitCounter.lhs<br /></pre><br /><p>And finally, we dial <code>31415</code> and hope the magic happens.</p><br /><h2>Summary</h2><br /><p>The above code is a good starting template for a more interesting AGI application. Note that caller number is a bit fuzzy. The caller number is determined by who gets to the <code>update</code> function first -- which could be different from who actually connected to the asterisk server first. </p><br /><p>Also, when calling a FastAGI application, it is possible to pass in a <code>PATH</code> and <code>query string</code>. The Haskell AGI library makes this information available, but does not provide any special mechanisms for doing something useful with it. This is likely to change in the future.</p>Jeremy Shawhttp://www.blogger.com/profile/18373967098081701148noreply@blogger.com2