Application Architecture

Ruby on Rails and MVC

YouSource is based on Gitorious. It is a Ruby on Rails application which means it implements the so called MVC architecture (Model-View-Controller). Models are used to access the database, views are used to form the html pages that users of the application can see and controllers define the application logic by controlling the models and views.

In Rails, there are also helpers and processors. Helpers are there to help render pages and are used from views. Processors help with asynchronous tasks which are needed for bigger operations, like creating Git repositories on the disk.

There are three environments in Rails: development, test and production. Use development for implementing, test for running tests and production for production server (and for test server!).

Directory Structure and Files

Models, views, controllers, helpers and processors reside in the app/ folder.
Configuration files are in the config/ folder.
Log files are in the log/ folder.
List of needed Ruby libraries (they’re called gems) are in file Gemfile.

Components

Here are some of the more important components:

MySQL is the database used.
Ultrasphinx is the search engine used. Users can search for projects, repositories, users, etc.
Stompserver is a message queue server that uses the stomp protocol. For example, when a Git repository needs to be created, a message about it is sent to Stompserver.
Poller is a script that polls the message queue and forwards the messages to the Rails processors for execution.
Git daemon is a simple server for Git repositories. It enables cloning the hosted repositories with the git protocol.
Grit is an object oriented Ruby implementation of Git. YouSource uses Grit in most of its Git operations.
KorppiLDAP is used for authentication which means users can login with their Korppi credentials. See manual (in Finnish).
crontab is a configuration tool for a program called cron. Cron executes shell commands periodically as scheduled in crontab.

Many many Ruby gems are also used.

Development Tools

Ruby

Ruby is the implementation language of YouSource. There are plenty of documentation for Ruby. When you need to write a regular expression you can use Rubular to easily test it.

Interactive Ruby Shell and Rails Console

Interactive Ruby Shell (irb) is great way to try out new pieces of code or to debug. When you need to load the YouSource environment use the Rails console, it invokes irb and loads the environment automatically. This means that you can access projects, repos, users and other stuff stored in the database and the application configuration files are loaded as well.

Start console:

bundle exec script/console              # run console in development environment
bundle exec script/console production   # run console in production environment

Console examples:

irb> r = Repository.last                     # get the last repo created
irb> r.git                                   # get the Grit handle to the git repo on disk  
irb> r.git.path                              # get the path to the git repo  
irb> p = Project.find_by_slug 'yousource'    # find a project by its slug attribute

Tip: With underscore, you can refer to the value that was returned by the last command in irb.
Tip: The bash shortcut ctrl+r (to quickly rerun some command from history) also works in irb (even across sessions).

Bundler

Bundler is used to handle gems. The gems that YouSource requires are listed in Gemfile and will be installed when bundle install is run (on the server machines the proper command is bundle install --deployment).

RVM

RVM is Ruby Version Manager and makes it easier to manage different Ruby versions and enables you to have different gemsets for different versions of Ruby as well as for different applications. For example, you can have one gemset for YouSource and one for YouData.

Git

YouSource is versioned with version control system called Git.

Some basic terminology of Git:

Commit is basically a version (of the whole project, not single file). It contains file modification data and a message describing the changes done by the user.
Branch is considered a line of development. It points to the latest commit in that branch. The current branch determines where the user’s new commits will go.
HEAD is a pointer to the current branch. Note that pointers to branches' latest commits are also called heads (in lower case).
Push is what you do when you want to send your local commits to a remote repository.
Pull is what you do to get commits from the remote repository i.e. to update your branch in your local repository.
Hooks are scripts that are triggered and executed when certain events (like push or commit) happen in the repository.

Some rules of thumb regarding workflow:

  • Don’t do development on master branch. Do new features and bug fixes in separate branch(es) and when they’re ready and stable merge them to master.
  • Never commit on production or test server, only commit on development machines. The server machines should only be updated by pulling. However, on the test server you can try out temporary changes to code.
  • Don’t change any version history that has been published. So don’t rebase a branch that you have pushed to a shared repository. Or at least, don’t push the rebased branch with the same name. You can rebase a branch locally before merging it to master if you like but never change published history.

Typical workflow:

A new feature or bug fix is needed. You switch to master branch on your development machine. Then you create a new branch (git checkout -b fix-a-bug), modify the code, commit, test it, switch back to master, merge the branch (git merge fix-a-bug) and then push the local master to the official YouSource repository (or if someone has updated the master branch in the official repository before you, you need to pull before you can push!). Then you can remove your branch (git branch -d fix-a-bug).

Foreman

Foreman is especially great on development machines because that’s where you’re gonna spend most of your time.

Foreman makes it easier to manage multiple processes that are needed in YouSource. With it you can start the web server, poller and stomp server by running just one command: foreman start. The output from all the processes handled by Foreman will be printed on the same shell so you need to keep fewer tabs open. Create a file called Procfile in the application root to define the processes to be handled by Foreman.

Example Procfile:

web:           bundle exec script/server
poller:        bundle exec script/poller run  
stompserver:   stompserver

See this blog post for more documentation.

JIRA

JIRA is the ticketing system used in YouSource and YouData development.

FireFox Plugins

With Tamper Data plugin you can view and modify HTTP/HTTPS headers and post parameters.
With FireBug plugin you can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page.

Server Administration

The application is installed in the directory /var/www/yousource and is run as git user.
In addition to the logs in log/ folder, there are more logs in /var/log/ folder (for example cron log).

Some useful commands:

ps x    # run as git user to see whether poller, ultrasphinx and other processes are up
df -h   # get info about disk space usage

When updating the application, first switch to git user. Then run git pull origin master. After that, the application needs a restart. There are two ways to do that: 1) run touch tmp/restart.txt. Use this if there are only minor changes, like if the views have been modified. 2) switch back from git user and restart the app with sudo /usr/sbin/restart_yousource. This brings the site down for a couple of minutes so use this when you think it’s necessary.

When you need to use the YouSource console, switch to git user, cd to app root and run bundle exec script/console production.

When you need help with the servers, contact sovelluspalvelut@jyu.fi

Testing

The tests reside in the test/ folder. There are separate tests for models (in test/unit/), controllers (in test/functional), integration, etc. Make sure you have an entry for the test environment in config/gitorious.yml and in config/database.yml.

Examples:

bundle exec rake test                     # run all tests
bundle exec rake test:unit                # run unit tests  
bundle exec ruby test/unit/user_test.rb   # run individual test files

Debugging

Some tools for debugging:

irb/console for debugging small pieces of code.
Rubular for debugging regular expressions.
Database (MySQL) console for checking the database when the Rails console isn’t enough.
git bisect can sometimes be used to determine which changes in code have caused a bug.
Tamper Data FireFox plugin for debugging forms or any pages that make use of html headers.
FireBug FireFox plugin for debugging css and views.

Further reading

There’s an application report of YouSource that was written as part of Verso project which originally developed YouSource. It was written in spring 2010 so it might be partially outdated.

Some useful info might be found in Gitorious’s wiki.

There are plenty of guides for Git, here are some good ones: Pro Git to learn basics and more, Git Reference to get hold of basic commands, Git From Bottom Up to understand the internals and Why Git is Better than X to gain some motivation :)