Daniel Patterson

Building on HAppS – Part 1 – More User Functionality

In Uncategorized on September 8, 2009 at 7:24 pm

This post will explore a little bit of the creation of this site/blog on the Haskell Application Server (HAppS). You can check out their website, or the website for the language it is written in, Haskell, if you are curious why I might have chosen to use it, but the purpose of this post is more how than why. There is still little enough documentation on HAppS that it seems valuable to document part of this process.

First, to begin with, I didn’t start this from scratch, so if you are interested in following along, I would head over to a tutorial written by another person just starting out with HAppS, as most of what I’m going to be writing about builds upon that work. If you are impatient, and dont feel like working through the posts describing the process, you can get the complete working code for his tutorial (these links are on the last page of it) at http://hpaste.org/5957, http://hpaste.org/5958, and http://hpaste.org/5959 respectively. Once you have those, and have built and installed HAppS, you have a pretty significant system. You can add users, list all the users you have, log in as a user, and see a demo page to confirm that you are logged in. However, there are still a few things missing. First, you really need to be able to not only add users, but delete them as well. Secondly, we can log in, but we can’t yet log out. Finally, as a sort of minor point, I thought it would be helpful to limit the total number of users that could exist. If you are writing an application with a small number of users (say, for example, a blog), this is a sort of weak security, and is something that makes me a little more confident about using it. So for starters, let’s implement deleting users. We can do the other two tasks after. (for the astute readers, you’ll notice this code I pasted to hpaste at the links above. but reading it here, not only do you get the code, but hopefully an explanation too!)

Deleting Users

Perhaps other people think differently, but I always make the changes to the data, and then gradually work back up to the interface. This means we will start with the file Session.hs, which has all the good information about the State of our application. Adding a function to delete a user is pretty straightforward, because most everything has been figured out for us. It should look like this:

delUser :: MonadState State m => String -> m ()
delUser name = modUsers $ M.delete name

This should look pretty straightforward – the users are stored in a map, and Data.Map was imported as M – modUsers is our helper function to modify the state. MonadState State is the monad we are using, because we are changing the state (as apposed to MonadReader when we are just pulling information out). The only last thing we need to do is at it to the TemplateHaskell function mkMethods that creates the data types that are used in the query and update functions.

$(mkMethods ''State['addUser,'delUser,'authUser,'isUser,'listUsers,
                    'isSession, 'setSession, 'getSession, 'newSession, 
                    'delSession, 'numSessions])

Next, we (obviously) want to be able to actually use this, so we will both add a url and some code to actually delete the user. You could make a more complicated system (indeed, you should, but it would be a good exercise to get more comfortable with this stuff), but I chose to make it simple, and just create a url /deleteuser that deletes the user who accesses it when logged in. It would be pretty trivial to capture the username from a form and use that, but I didnt really feel like I needed it (and offered unneeded potential security holes). To add the url, we go into Main.hs, and in the list of ServerPartTs (if this doesnt make sense, that’s okay, just look for where it would fit in best), and add this in:

, dir "deleteuser" [withDataFn (liftM Just (readCookieValue "sid") `mplus` return Nothing) deleteUserPage]

What this does is register a url at “deleteuser” that pulls the cookie value stored in “sid” out and then calls deleteUserPage with it. WithDataFn is a variant on WithData that instead of taking a function that takes a structure that can be parsed out because it implements the FromData class, withDataFn takes that function (fromData) as one of its paramaters. This is just a convenience thing, functionally it accomplishes the same as if we created a data structure “cookieSid” that implemented the FromData class and had the following code:

fromData = (liftM Just (readCookieValue "sid") `mplus` return Nothing)

Now we have a url that grabs the cookie, we need to use that in the function we called with it, deleteUserPage. It looks something like:

deleteUserPage (Just sid) = [anyRequest $ do ses <- query $ (GetSession $ sid)
                                             msg <- deleteUser ses
                                             ok $ toResponse msg]
deleteUserPage Nothing = [anyRequest $ ok $ toResponse "not logged in"]

You probably noticed above, the if the cookie value wasn’t found, Nothing was returned, if it was found, Just the value was passed. So we have two versions of deleteUserPage to handle those two cases – the second obviously represents the case when the person is not logged in, so we sent that as a response. In the former case, we have an sid, so we look it up in the sessions that are stored in state, and call a function to delete the user with that username (what is stored in the session data). That function then returns the message we will pass back to the user, in the last line of the first function’s do block. The function looks like:

deleteUser (Just (SessionData s)) = do update $ DelUser s
                                       return "deleted"
deleteUser Nothing = do return "nothing deleted"

Again, we are dealing with code that takes a Maybe value – this is because the query to look up the session id might not return a user name. If, for example, the person had a value in the cookie value sid (maliciously or not) that didnt correspond to a session that was currently valid, the query would have returned Nothing. The one thing that you might think is a little funny (but if you followed the other tutorial you might have already noticed). I call query not with a function, but a data type – this may seem a little odd, but it has to do with how state is kept in HAppS, and if you are wondering where it came from, remember the TemplateHaskell function mkMethods that we called on it (and all the other state accessing/changing functions) – that creates the data types. For a regular person just using HAppS, the distinction is not important – just know that you need to use the capitalized version, not the lowercase function. So that’s it for deleting users. Now let’s move on to logging out (because you might have noticed using the code developed in the previous section, you can delete yourself and continue using the parts of the sites you need to be logged in as. Oops!)

Logging Out

There are two aspects of this task. First, deleting the session from state. And second, deleting the cookie on the browser. To accomplish the former, a function very similar to the one developed in the last section, to delete users, will work:

delSession :: (MonadState State m) => SessionKey -> m ()
delSession key = do
  modSessions $ Sessions . (M.delete key) . unsession
  return ()

We use MonadState again, because again we are modifying the state. One thing you might notice is ‘unsession’ – this is the field accessor for the Sessions datatype. You can read this (right to left) as pull the map out of Sessions (unsession), delete the key (M.delete key), and construct the type again (Sessions), to create the proper function that modSessions takes: Sessions -> Sessions. Now we want to add this to mkMethods, making it now look like this:

$(mkMethods ''State ['addUser, 'delUser, 'authUser, 'isUser, 'listUsers,
             'setSession, 'getSession, 'newSession, 'delSession, 'numSessions])

To actually log out, we will create a function that corresponds to the performLogin function in the other tutorial:

performLogout sid = do
  addCookie 0 (mkCookie "sid" "0") -- delete cookie
  update $ DelSession sid

There isn’t a delCookie function (yet, there are stubs for one in the code), but deleting the cookie is as simple as replacing it with a blank one that times out immediately (thanks #happs for this idea). Other than that, the function should look very straightforward. Now the last steps are creating the url and the page that will allow these functions to be accessed. The url looks like:

, dir "logout" [withDataFn (liftM Just (readCookieValue "sid") `mplus` return Nothing) logoutPage]

The astute reader (or just any old haskeller) will say, wait, you are using the very same code as you used for the deleteuser page, why dont you factor it out? And indeed you could, here is a non-point free version:

cookieR handler = withDataFn (liftM Just (readCookieValue "sid") `mplus` return Nothing) handler

Which would make our previous two url entries look like this: , dir "logout" [cookieR logoutPage] , dir "deleteuser" [cookieR deleteUserPage] Now, we just need our logoutPage function and we will be finished with this section! Let’s make it quick:

logoutPage (Just sid)  = [anyRequest $ do
  loggedin <- query $ (IsSession $ sid)
  if loggedin
    then do processLogout sid
            ok $ toResponse $ "logged out."
        ok $ toResponse $ "not logged in"]
logoutPage Nothing = 
  [anyRequest $ ok $ toResponse $ "Not logged in"]

And that’s it! Not the prettiest of functions, and one that could probably be reduced to a one-liner, but working. So now if we look back at what we’ve accomplished – we now have a system that can add and delete users, can log in and log out, can see all the users, and can see if we are logged in. The one last thing I promised for this post (which is getting quite long) is a way to limit the total number of users. Since by this time you are probably seeing the pattern of adding code, I’ll make it super abrieviated. First, code in Sessions.hs:

numUsers ::  MonadReader State m => m Int
numUsers = liftM length listUsers

Updated mkMethods:

$(mkMethods ''State ['addUser, 'delUser, 'authUser, 'isUser, 'listUsers, 'numUsers,
                     'setSession, 'getSession, 'newSession, 'delSession, 

And now, in Main.hs, we want to modify the checkAndAdd function to check the total number of users.

checkAndAdd user pass = do
  numusers <- query NumUsers
  if numusers > 0 -- ie, only allow one user to exist
    then ok $ toResponse $ "Unable to create new user"
    else do
      update $ AddUser user $ User user pass
      ok $ toResponse $ usersP "User created."

You may notice that we are now not checking if the user exists, but as mightybyte pointed out in a later blog post, there was a potential problem in the current design – that the check and the add were in different transactions, and thus there would be no guarantee if two people tried to create the same username at the same time that one would be rejected. The solution that he came up with (on the blog, but not on the hpaste files) is to move that functionality into the function in Session.hs, to make it part of the same transaction. It looks like this now:

addUser name u = do exists <- isUser name
                    unless exists $ modUsers $ M.insert name u
                    return exists

And that should be it. We now have a hard limit on the total number of users (it might be slightly more full featured if you put the limit higher than 1 user, but that should be easy to do. More to come in subsequent posts (we havent yet gotten to a full blog yet, and the fact that you’re reading this proves that one exists :P)


Building on HAppS – Part 2 – requireLogin and URLs

In Uncategorized on September 8, 2009 at 7:23 pm

In the last post, I went through adding a bit of functionality to the existing application, including logging out, and deleting users. This post is a short one, it just has a few things that were useful in applying the framework to the rest of the site (and itself).

After playing around with the login settings, I realized I needed a simple way to require logins – the initial idea, to just put checks into the pages in question, is not only inconvenient, but really bad practice (dont repeat yourself!), and so in generalizing this, and beginning to understand the HAppS system a little more, I realized the cleanest way would be to take a list of ServerPartTs, and require login to see any of them.

Because HAppS itself is in some way functional, this can be pretty clean, overall.

The code depends on a helper function in the last post, so if you don’t have that already, add it here:

cookieR handler = withDataFn (liftM Just (readCookieValue "sid") `mplus` return Nothing) handler

With that, the requireLogin function can be pretty simple:

loginGate sPTs (Just sid) = [ require (isLoggedIn sid) $ \_ -> sPTs]
loginGate sPTs Nothing = [anyRequest $ seeOther "/login" $ toResponse "Not logged in."]

requireLogin sPTs = cookieR $ loginGate sPTs

There is one main new thing here – that is “require”. require is a function that takes a Maybe value and a function that returns a list of ServerPartTs, and if the Maybe is a Just, it executes the function, if not, it doesn’t match. This is really useful for grabbing things that are side-effecty, early on, to make the code further down cleaner. In this case, we are requiring a function to check if someone is logged in, another simple State accesser that looks like this:

isLoggedIn ses = do b <- query $ IsSession ses
                    return $ Just (b)

With the definition of IsSession (the non-Maybe mkMethodized function) as:

isSession :: MonadReader State m => SessionKey -> m Bool
isSession key = liftM ((M.member key) . unsession) askSessions

And of course, add isSession to the list passed to mkMethods (exercise left to the reader).

sPTs are ServerPartTs (a list of them), which is conveniently what most of the functions that we are already using to handle URLs deal with. This means that Main can have a list similar to this:

, dir "protected" [requireLogin protectedHandler]

Where “protectedHandler” can be another whole list of URLs, like this (using the example of the user related pages we have developed so far):

[dir "newuser" [method GET $ fileServe ["login.html"] "."
               ,methodSP POST $ withData newUserPage]
,dir "deleteuser" [cookieR deleteUserPage]
,dir "logout" [cookieR logoutPage]
,dir "list" [userListPage]]

This illustrates a few really neat things about the way that HAppS deals with request URLs. When something is matched with “dir”, the list of ServerPartTs are passed (or are aware of) only the part of the url that is after that part. This means that if we wanted to relocate all our user related functions to the sub directory “protected”, you literally would only have to transform the code as above. Then /protected/newuser etc would work just as /newuser etc had before.

There are a few quirks, and a few more things – you can match urls with either “dir” or “path”, the difference being that with dir, you pass it a list of ServerPartTs, and with path, you pass it a function that returns a list of ServerPartTs – this is useful if you want to capture parts of the url, which could look something like this:

[dir "showuser" [path \rq -> [anyRequest $ toResponse $ "Showing user " ++ (rq :: String)]]]

This would match against the url /showuser/Daniel or /showuser/dbpatterson and would print the corresponding name. You can also specify other types, for example, if you had it by userids, it could be something more like this (only showing relevant part):

[anyRequest $ toResponse $ showuser (rq :: Integer)]

Where showuser presumably would be some function that looked up information about a user, given the user id.

The last thing in this section about URLs, are the ‘method’ functions. They serve as a sort of gate, allowing only a certain type of request through. We’ve seen this in order to have forms post to themselves, as shown above. There are two functions, method and methodSP, where methodSP (quite logically) takes a ServerPartT and method takes a WebT. The last thing to keep in mind is that method will only match a URL if there is nothing left in the url (if we consider the URL being gradually reduced as it matches subsequent “dir”s or “path”s). If there is anything past, it wont match. This is useful in keeping things clean and well defined (as in, provided you always use method matchers, reading the list of ServerPartTs shows you all the urls you will match), but it is also useful in one extra case – matching nothing at all.

If this were not the case, it would be impossible to match what I would call the index – trying to match dir “” doesn’t work, and of course dir “/” is not going to help (as the / is not considered part of the request url). So instead we match for method(SP) GET, and everything works.

(thanks to everyone on #happs for input into this and the previous entry, especially Lemmih, Saizan and mightybyte).

Until next time.

web.py and web frameworks

In Uncategorized on September 8, 2009 at 7:22 pm

I discovered web.py a few weeks ago almost by chance – I had written a site with Django, and as expected, everything worked fine and it was up and running in no time, including a nice, fancy, back end admin interface.

This went well, I gave the backend to my client and they played around with it and were pretty pleased. A very non-technical person, he could use it effectively, nothing more could be asked.

But it was using a decent amount of memory, and didn’t have an easy way to cap it from spawning many new processes.

So this got me to thinking, and I came very close, indeed, I started writing it, in a system that I had developed for an earlier site I had written. To call it a framework would be an overstatement, but it had a fully functioning url dispatcher and a simple, regex based template system; it had worked pretty well with the other site I had written, indeed able to operate quickly using regular CGI (a requirement for that project), something that a framework like Django or Rails or any of the major contenders could not do.

So lightweight it was, and I started writing with it, but I very quickly got tired of all the repetition. I started thinking, as often happens when coding, that there has to be a better way of doing this, a library (in this case framework) built to fill exactly this position – to be small, simple, and light, but eliminate a lot of the drudgery – writing database insert, select, and update functions, writing a simple CRUD (CReate Update Delete) interface, etc.

I thought of writing one myself, and even ventured into the code generation realm (python generating python), as for the database functions, this was a simple thing to do, but when thinking of how to do this for the administrative end of it, I started running into a wall on the efficiency of my approach. I half wrote a post about code generation as a pseudo-static extension of python, and realized while writing it how absurd the idea was, and that I should really be scrapping this idea. I scrapped the idea of writing my own framework, thinking it wasn’t worth it for this project, but that maybe I’d revisit the idea at some later point.

And so, wondering if I would just have to bite the bullet and head back to Django, I went to Google and started searching around. Python web frameworks, looking through the usual suspects – pylons, turbogears, etc.

And then I saw mentioned somewhere, web.py. I looked it up and was pleasantly surprised to see that it could run either as fastcgi or cgi (an indication that it was probably pretty lightweight). I then started poking around, looking through code samples, documentation on APIs, etc, and found something that was absolutely gold, and sold me on the framework.

In web.py, you can define Form objects, that contain a bunch of Fields. An example of one is the following (from the above mentioned app – I told you it was simple):

    sponsor = form.Form(
    form.Textbox("name", form.notnull), 
    form.Textbox("website", form.regexp('http://.*', 'URL must begin with http://')),

As you can see, the definitions are quite concise, but there is built in, very easy to use validation (above you can see simple validation to make sure there is something, as well as more complex regexes), and you don’t see it there, but you can also define arbitrary html properties of the input fields in the constructor argument list – this is very handy with Textarea’s and Textbox’s when you want them to be larger (rows=”20″, cols=”80″, for example), etc. But of course, this isn’t anything too special on its own, but the integration is where it really hits.

To put it into a template, it couldn’t be simpler – render the template as follows:

render = template.render("templates/")
form = sponsor() # to make a copy of it
return render.formt(form)

and then in the templates folder, the template formt.html will have:

$def with (form)
<form method="post">
<input type="submit" value="Submit" />

Now we want to actually use the validation checks we put in, and do something useful with the form input. So we update it a little bit – first, in the python code, a fuller, updated code snippit would be:

render = template.render("templates/")
class formt:
  def GET(self):
    form = sponsor()
    return render.formt(form)
  def POST(self):
    form = sponsor()
    if not form.validates():
      return render.formt(form)
      db.insert('sponsors', **(form.d))

Now we can look at a few things here – first, the Form object is populated for you – no messing with the POST params, etc. Secondly, the $:form.render() we put in to render the blank form – if you render it with data already in it, it is all properly populated in the fields. And it will put error messages next to the fields that are not valid – yes, that is what the message in the regex validator above is for.

Lastly, a handy little thing about the db wrapper – if you have fields that are named the same as the columns in your database table (which isn’t a stretch, indeed to keep you sane, you should probably do this in any case), you can use the data structure of the form (form.d) directly as the data for the insert. No sql needed, not even tucked away in another function. This also translates the other direction – say you want to make an edit form, all you need to do is grab out the row from the database and use it when constructing the form. As simple as:

form = sponsor(db.select('sponsors', where="id=2", limit=1)[0])
return render.formt(form)

Now, I can’t emphasize how much this impressed me, that so little code, indeed, such a limited framework, could make these things trivial. And at least in this case, it is entirely because of one decision – use the same internal data structure for the database wrapper as the form library. A simple decision, but a very very good one.

Going forward, I wrote the site as I needed it, but then, figuring the one thing I really missed from Django was the admin interface that came for free, all you had to do was define your models, I figured I’d take a shot at implementing that using web.py. The solution didn’t take more than an hour and a half to code up, and it resulted in something where all I had to do was define a form for an item and it would create the table if it didn’t already exist in the database, and create pages to add, update, or remove items, as well as showing them all on a centralized administrative page.

User management I cheated a bit on, relying on a dictionary of users defined statically, but it is a system that works quite well, if not containing the fine grained control of Django’s admin system. I’d like to clean it up and make it available, possibly as a standalone library, or possibly integrated into web.py, considering it is just over a hundred lines of code, and could be made even shorter, but until then I’ll just stick to using it to put out complete websites with client editable data in under an hour.

Now web.py may not be suitable to start the next 37signals, but that is not its purpose, and I don’t think I can emphasize the important role it has to play at the other end of the spectrum – the little, hobby or small business websites that are craving frequent updates of basic data. I’m talking small retail shops, restaurants, artists, local organizations, etc.

Web.py may not make hard things easy, but it certainly makes easy things trivial. Because any time you want a quick site together, just to keep a few things updated, knowing you could do it in perhaps 40 lines of code in 30 minutes with web.py is quite something. Or even, to get fledgling (web) programmers going quickly with python.

You could even use it to get nice urls and a templating system for an entirely static site. What is most impressive about this is that web.py really lives up to its name – it isn’t something totally new, with its own idioms and practically its own sub-language that you need to learn. It is simple web.py, a web framework for python. It seems to stay quite close to the python mentality – to keep things concise but clear, to do things in one way, and to make things easy to do, and easy to change.