Wednesday, October 28, 2009

Asus WL500g Premium controller now on Windows

Today I worked on my WL500gPControl project, and already forced it to work on Windows. The most difficult parts were to set up the MinGW and build curl under it. To be honest, I ended up with precompiled curl library from Sigbjorn Finne.

To let others avoid these hardships, I prepared the binary version of WL500gPControl. You'll only have to unpack it on some directory, put the credentials file into the default placement and use it :). The binary version is here

Also I have added README file, where I described how to use this tool, so you shouldn't have any troubles with it.

If you are updating from previous version note that placement of the default file has been changed. For Unix users it is

$HOME/.WL500gPControl/credentials, 

and for Windows:

C:\Documents And Settings\user\Application Data\WL500gPControl\credentials

Tuesday, October 27, 2009

hs-ffmpeg now works from the box

hs-ffmpeg status.

Recently I have changed the way how hs-ffmpeg. Now the package finds libraries locations by itself.

So, if you have developer libraries for libavcodec, libavformat, libavutil and libswscale installed, you could simply put:

cabal update
cabal install hs-ffmpeg

And this will install ffmpeg bindings to your machine. (If don't, please, drop me issue in bug tracker).

Shortly, I've used autoconf to find ffmpeg libraries and its dependencies, and also added support for 0.51.0 versions of ffmpeg. I found that binary packages from my Ubuntu uses this version of library. Anyway, latest SVN versions of ffmpeg should also work.

ffmpeg-tutorials status

Another I found, that SDL bindings were updated to 0.5.6 version, and began conflict with my audio patch. So I created new audio patch for SDL-0.5.6. To apply it, just simply run following commands in the SDL-0.5.6 directory:

wget "http://hs-ffmpeg.googlecode.com/files/SDL-0.5.6-audio.patch"

patch -p1 < SDL-0.5.6-audio.patch

To do

  • I think to slightly rewrite the code with buffers and use ByteString instead. Don't know is it a good idea.

  • I continue working on tutorials, next step is to implement tutorial 04.

  • Also I want to make more functional approach to the video processing, i.e. represent the sequence of packets as list, or use Arrows to represent the decoding process.

Thursday, October 22, 2009

Working with databases in Haskell. Part 2

Here I want to describe how I deal with tagged document storage in Haskell, using the relational database sqlite3 (through HDBC). Yes, I know, that relational database work very poor with document-oriented and tagged data, but for a little amount of data it could be the proper way to implement storage.

The following code works with databases in the same way as described in my previous post, and written in literate Haskell style, so you can easily save it in the some_file.lhs, load it in ghci and play with database.

Task description

Let’s suppose we have to store documents with following properties:

  • author — the string, representing the author
  • title — document’s title
  • body — the content of the document
  • list of tags — tags, assigned to the documents

All parameters will be Strings for simplicity, and we will not put any restrictions to the tags format.

The most obvious way to store these data is to use single table with separate fields for each property. But how to deal with tags? Document could have any amount of tags, or have nothing. We couldn’t create a table with infinite columns. Other solution, is to store all tags in one text field, and represent them as comma-separated lists, but in this case we have to put some restrictions to the tags (e.g. they shouldn’t hold commas) and each query involves different string matching and string updating, which should be done on client side, and those not efficient.

But there is the third way. We, along with the database gurus, will use many-to-many relation to represent tagged documents.

Begin to code


> import Database.HDBC
> -- we will use sqlite3 driver here
> import Database.HDBC.Sqlite3
> import Control.Monad (liftM, when)
> -- all our database code will be enclosed in ReaderT monad
> -- see previous post for details
> import Control.Monad.Reader
> import Data.Maybe
> import Control.Applicative

Define the internal Haskell types, from/to which we will serialize data:


> type TagList = [String]
>
> data Document = Doc {
> author :: String
> , title :: String
> , content :: String
> } deriving (Eq, Show)
>
> -- Our transaction type
> type Transact a = ReaderT Connection IO a

Define some helper functions to work with database and the function that runs our transaction monad


> -- Get database connection object
> conn :: Transact Connection
> conn = ask
>
> -- Query the database without result (update queries)
> query :: String -> [SqlValue] -> Transact ()
> query q v = do
> c <- conn
> liftM (const ()) $ liftIO $ run c q v
>
> -- Query the database with returning result
> query' :: String -> [SqlValue] -> Transact [[SqlValue]]
> query' q v = do
> c <- conn
> liftIO $ quickQuery' c q v
>
> -- Run the transaction
> runTransact :: String -> Transact a -> IO a
> runTransact dbname io = do
> c <- connectSqlite3 dbname
> runReaderT withCommit c
> where
> withCommit = do
> result <- io
> conn >>= liftIO . commit
> return result

OK, all preparations are done, now start implementing the core of our document data-store.

In order to implement many-to-many relation, we have to create three tables: one for documents, one for tags and one is relation between tags and documents.

Creating databases


> -- Create tables
> createDocumentsTbl =
> query "CREATE TABLE documents ( \
> \docid INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, \
> \author TEXT NOT NULL, \
> \title TEXT NOT NULL, \
> \content TEXT NOT NULL, \
> \UNIQUE (author, title, content))" []
>
> createTagsTbl =
> query "CREATE TABLE tags ( \
> \tagid INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, \
> \tag TEXT NOT NULL UNIQUE )" []
>
> createDocTagsTbl =
> query "CREATE TABLE doctags ( \
> \docid INTEGER, \
> \tagid iNTEGER, \
> \PRIMARY KEY (docid, tagid), \
> \FOREIGN KEY (docid) REFERENCES documents (docid), \
> \FOREIGN KEY (tagid) REFERENCES tags (tagid))" []
>
> -- Checks whether tables created, and create them otherwise
> initializeDb = do
> tables <- conn >>= liftIO . getTables
> mapM_ (\(tn,cf) -> when (tn `notElem` tables) cf)
> [("documents", createDocumentsTbl)
> ,("tags", createTagsTbl)
> ,("doctags", createDocTagsTbl)]

Now you can load this file to the ghci and execute:

runTransact "test.db" initializeDb

then check the database with sqlite3 test.db command and see, that all tables were successfully created.

Low-level access to the databases

Lets start implementing basic functions to work with our documents:


> -- Adds new tag to the database, do nothing if tag exists
> addTag tag =
> query "INSERT OR IGNORE INTO tags (tag) VALUES (?)"
> [toSql tag]
>
> -- Finds tag by name, returns id or Nothing
> findTag tag =
> liftM (idListToMaybe) $
> query' "SELECT tagid FROM tags WHERE tag = ?" [toSql tag]
>
> -- Return tag name by id
> getTag tagid =
> liftM (fmap (fromSql . head) . listToMaybe) $
> query' "SELECT tag FROM tags WHERE tagid = ?" [tagid]
>
> -- Remove tag from database
> removeTagById tagid =
> query "DELETE FROM tags WHERE tagid = ?" [tagid]
>
> -- Find document by name, author and content. Return docid or Nothing
> findDocument doc =
> liftM (idListToMaybe) $
> query' "SELECT docid FROM documents \
> \WHERE author = ? AND title = ? AND content = ?" $
> docToSqlValue doc [author, title, content]
>
> -- Adds new document document to the database
> addDocument doc =
> query "INSERT OR IGNORE INTO documents (author, title, content) \
> \VALUES (?,?,?)" $ docToSqlValue doc [author, title, content]
>
> -- Return document by id
> getDocument docid =
> liftM (listToDocument) $
> query' "SELECT author, title, content FROM documents \
> \WHERE docid = ?" [docid]
>
> -- Remove document from the database
> removeDocumentById docid =
> query "DELETE FROM documents WHERE docid = ?" [docid]
>
> -- Adds link between document and tag
> addLink docid tagid =
> query "INSERT OR IGNORE INTO doctags VALUES (?, ?)" $
> [docid, tagid]
>
> -- Remove link between document and tag
> removeLinkById docid tagid =
> query "DELETE FROM doctags WHERE docid = ? AND tagid = ?" $
> [docid, tagid]
>
> -- Get tags for document, returns list of tagids
> getDocumentTagsById docid =
> liftM (map head) $
> query' "SELECT tagid FROM doctags WHERE docid = ?" [docid]
>
> -- Get document ids for the tag id
> getTaggedDocumentsById tagid =
> liftM (map head) $
> query' "SELECT docid FROM doctags WHERE tagid = ?" [tagid]
>

High-level functions to work with data

So the basic functions, that create separate entities in the database are implemented, now we could use that functions to combine high-level operations:


> -- Adds new tag and return its id, or simply return id if tag exists
> -- we used fromJust here, because tag should exists
> addTag' tag = addTag tag >>
> (liftM fromJust $ findTag tag)
>
> -- Adds new document and its id, or simply return id if document exists
> addDocument' doc = addDocument doc >>
> (liftM fromJust $ findDocument doc)
>
> -- Adds complete document with tags
> addCompleteDocument doc tags = do
> docid <- addDocument' doc
> tagids <- mapM addTag' tags
> mapM_ (addLink docid) tagids
> return docid
>
> -- Get document tag names
> getDocumentTagNames docid = do
> tags <- getDocumentTagsById docid
> liftM catMaybes $ mapM getTag tags

And so on. By the way, last function getDocumentTagNames we could also implement as a single SQL query, using JOIN:


> -- Similar to `getDocumentTagNames` but implemented
> -- through SQL JOINs
> getDocumentTagNames' docid = do
> liftM (map (fromSql . head)) $
> query' "SELECT tag \
> \FROM tags LEFT JOIN doctags USING (tagid) \
> \WHERE docid = ?" [docid]

And finally, some helper functions, used before:


> -- Converts select result into Just v or Nothing
> idListToMaybe = fmap head . listToMaybe
>
> -- Converts Document structure to SqlValue list,
> -- the second argument - list of fields to convert
> docToSqlValue doc = map (toSql . ($ doc))
>
> -- Converts the list of SqlValues to the document
> listToDocument = fmap (toDoc . map fromSql) . listToMaybe
> where
> toDoc [a,b,c] = Doc a b c
> toDoc _ = Doc "" "" ""

Conclusion

I have just demonstrated how is easy to work with databases in Haskell, and also showed the basic principles of storing tagged documents, which is very important in the WEB (and often in desktop) applications.

The code is clearer than the code on Visual Basic or Delphi (from my memoirs) and simpler than code on Python. And I think this proves that Haskell is very good for real world applications.

Databases in Haskell or "Release the power of monad transformers"

Many years ago almost all my programs were ‘some kind of interface to database’. I have been programming on C/C++ so work with databases was simple and required a lot of stupid code.

Several days ago I returned to the databases but now with Haskell in my hands and functional-oriented mind in my head :). First time I coded in the C-style from Haskell, but now, I think, I have found more functional way to work with databases. Of course this approach is not universal, and database ninjas could blame me for it, but let me continue :).

Combining transactions with monad transformers

I worked with sqlite3 database, but this approach should work with other databases too.

I enclosed all my database code into ReaderT monad in the following way:

> import Database.HDBC
> import Control.Monad
> import Control.Monad.Reader
>
> type DbTransaction a = ReaderT Connection IO a
>
> -- return the connection from the reader monad
> conn :: DbTransaction Connection
> conn = ask
>
> -- some useful functions to work with databases
>
> -- query function executes simple query without returning any results
> -- useful for create table and update queries
> query :: String -> [SqlValue] -> DbTransaction ()
> query q v = do
> c <- conn
> liftM (const ()) $ liftIO $ run c q v
>
> -- query' function executes query with results (for select queries)
> query' :: String -> [SqlValue] -> DbTransaction [[SqlValue]]
> query' q v = do
> c <- conn
> liftIO $ quickQuery' c q v

From that moment we can create our own queries like following:

-- get the user id from the database
> getUserId :: String -> DbTransaction (Maybe String)
> getUserId name =
> liftM (maybeId) $ query' "SELECT id FROM users WHERE name = ?" [toSql name]
> where maybeId [] = Nothing
> maybeId [[id]] = Just $ fromSql id

-- add new article to the database
> addArticle :: String -> String -> String -> DbTransaction ()
> addArticle user_id title content =
> query "INSERT INTO (user_id, title, body) VALUES (?,?,?)" $
> map toSql [user_id, title, content]

Also we can combine all our actions under one big transaction, i. e. take a user name and article title and body, add article with user id:

> addArticle :: String -> String -> String -> DbTransaction ()
> addArticle user title content = do
> uid <- getUserId user
> when (isJust uid) $ do
> addArticle (fromJust uid) title content

Now we have to explore how to run our transactions from the main program. I designed the function that takes the database name, connects to it, runs the transaction and commits everything:

> runTransaction :: String -> DbTransaction a -> IO a
> runTransaction dbname io = do
> c <- connectSqlite3 dbname
> runReaderT withCommit c
> where
> withCommit = do
> result <- io
> conn >>= liftIO . commit
> return result

So each time we need to query database, we will do the following:

> runTransaction "mydb.db" $ 
> addArticle "Hamlet" "The Question" "To be or not to be"

Don’t know how are you, but I like this code. It is short, clear, database-dependent code wrote in the separate monad and couldn’t be randomly combined with other code, and also safe. If you will want to use another database interface, you only will have to change one function (runTransaction).

It is easy to add error handling to this code. Here is two ways:

  • wrap the DbTransaction into the ErrorT monad, and wrap all HDBC functions to re-throw exceptions, or

  • use the internal HDBC exception handling mechanism (default behavior)

Anyway, this is only one example how to use monad transformers in real life, even for small projects. In the next posts I’ll show another uses of this approach.

Thursday, October 15, 2009

Control your ASUS WL500g Premium from the command line

I have a nice router Asus WL500g Premium, but I can’t say the same about my Internet provider. Several times a day it breaks the Internet connection in the way my router couldn’t restore it, so I have to manually disconnect and re-connect it. This is tremendously hard because of poor router’s admin interface. So, because I’m still learning Haskell, I decided to create simple command-line utility that will reconnect my router to the WAN.

After first release, I split the project into two components: a library, which connects to the router, and several command-line tools, that runs my usual activities automatically. You can install them using cabal:

cabal install WL500gPLib
cabal install WL500gPControl

The WL500gP Library now could retrieve connection status, external IP address, DNS servers and log from the router, and could send connect, disconnect and clear log commands. I implemented all these operations in the Conn monad, which is simply Reader transformer over IO monad. This approach reduces boilerplate code and also adds automatic log off to the command chain.

The WL500gP Remote Control package have two executables, to work with router. WL500gPControl does everything, that library support. So to see log from the router, simply run:

WL500gPControl -l

To see all list of supported operations type:

WL500gPControl -h

The connection info (user name, password and host address) WL500gPControl takes from the $HOME/.wl500gp file, but it could be any other file specified in the command line as the second argument. The file format is following:

user: username
password: password
host: 192.168.1.1

The second script is very simple and I wrote it only to show connection status in my xmobar. It only prints connected or disconnected with color tags (green and red accordingly). Here is my config line from .xmobarrc:

Run Com "WL500gPStatus" ["-c", "$HOME/.wl500gp"] "Internet" 300

Conclusion

It took several hours to implement all these features, and from that time I never loaded my router’s admin page. And also I ensured that tagsoup is the great library for fast and dirty HTML parsing.

Video processing on Haskell - easy

Almost all my past jobs were in the field on video processing, creating custom filters, decoders and encoders. Several of these jobs were to implement complete video/audio players. I’ve used ffmpeg library to perform all these tasks, except once I’ve used GStreamer. I can’t say that GStreamer is bad, but it didn’t allow me to completely control the decoding process, stucks on malformed video files and it was really painful to develop the 24x7 application.

And I returned to ffmpeg again, but than I became a haskeller :) so the idea was to made ffmpeg closer to Haskell community. Today let me introduce the hs-ffmpeg library for haskell.

Current version (0.3.2) supports basic file opening, streams enumerating, video and audio decoding. To make the process more interesting I took FFMpeg and SDL tutorial and implemented bindings in the sequence they are appeared in this tutorial. The second project, that shows how to use hs-ffmpeg and implements tutorials is ffmpeg-tutorials. It depends on SDL, so I’ve used Haskell SDL bindings to work with video and audio.

Several notes on building the sources.

Building hs-ffmpeg

You have to change hs-ffmpeg.cabal file to tune your ffmpeg library settings. In my case, ffmpeg installed under /usr/local tree, so my include dirs have /usr/local/include line and extra-lib-dirs contains /usr/local/lib value. Also in the extra-libraries you should specify the complete list of libraries, against which your version of ffmpeg have been buit.

If you do everything right, than hs-ffmpeg will build without any errors.

Patching the SDL

If you want to run ffmpeg-tutorials (sure you want :)), than you have to patch SDL bindings first. Current SDL bindings version is 0.5.5, and I’ve created SDL audio patch to this version, so do the following:


cabal fetch SDL
cd /tmp
tar xzvf ~/.cabal/packages/hackage.haskell.org/SDL/0.5.5/SDL-0.5.5.tar.gz
cd SDL-0.5.5
wget http://hs-ffmpeg.googlecode.com/files/SDL-0.5.5-audio.patch
patch -p1 < SDL-0.5.5-audio.patch


Than configure and build bindings. (Note: you’ll have to install SDL development libraries on to your system).

Running ffmpeg-tutorials

Currently only three tutorials are implemented. tutorial03 is completely showing capabilities of ffmpeg in video and audio decoding. Just simply run it with video fil ename as an argument to see how it decodes and shows video and plays sound.

Future work

I have only started this work, but I’d glad to hear comments and suggestions from people who interested in video processing on Haskell.

My future work plans are splitted onto several parts:

  • Continue implementing bindings to ffmpeg library

  • Improve build system, profiling

  • Started with multimedia combinators library, to program video and audio applications in more functional way (it is my dream)

Credentials: