Enigma 2 development toolchain

Introduction

As of now, this is only a very brief “hands-on” tutorial to get you started with Enigma 2 hacking.

Table of contents


1. The FAQ

1.1 What is Enigma 2?

When you switch on your DM 7025 and start watching or recording tv shows, Enigma 2 is the application that you see on the tv screen. Enigma 2 receives and processes the button push events from the remote, draws the menus and the windows on the screen, changes the channels and, overall, handles all the interaction with the user. For a remote-contoller wielding couch potato, Enigma 2 is what the DM 7025 is.

But Enigma 2 is not all there is to a DM 7025. There is also the underlying Linux system: with a kernel, drivers, command-line tools, server applications (such as Samba), scripts, shared libraries, a package management system, etc. Together, all these various bits and pieces form the base upon which Enigma 2 is running. Of course, most of these other components are running in the background, deep in the bowels of the device – invisibly, as far as the end user is concerned – but without them the Enigma 2 software wouldn’t work.

In other words, Enigma 2 is a Linux application. It cannot exist alone; it is not a stand-alone “firmware” that would directly access the set-top box hardware at the register level and run independent of any external OS. The firmware images you can download from Dream Multimedia are actually full-fledged mini Linux distributions, with the Enigma 2 application running on the top.

1.2 Aren’t there similar DVR applications for Linux already? Why the need for yet another such project?

Yes, there are other Linux software packages that provide TV viewing and digital video recorder (DVR) functionality – much in the same way as Enigma 2 does. Popular examples of these include VDR and MythTV. Enigma 2 is similar to these projects in many ways, but differs in that it was written for set-top box style hardware from the get-go.

The target platform for Enigma 2 is an embedded environment: something that typically has much less raw processor power and memory than a generic Linux PC. VDR and MythTV, on the other hand, are more geared towards the so-called “home theater PCs”, or HTPCs. These systems can often be quite powerful or even fully on-par with modern desktop PCs, in terms of processor power and available memory.

1.3 What happened to Enigma 1?

Enigma 2 is a rewrite of Enigma 1 (or mere “Enigma”), which is another, older set-top box-oriented DVB viewer and DVR application. Without delving too much into historical details, it suffices to say that Dream Multimedia still ships many of the other Dreambox models with Enigma 1, and develops and maintains this “old” branch of Enigma, too. The DM 7025, however, was shipped with the all-new Enigma 2, and the upcoming DM 8000 is expected to be shipped with Enigma 2 as well.

The most striking difference between the two Enigmas is that Enigma 1 is a C++ application through-and-through whereas Enigma 2 only has a relatively compact C++ core for the lower level stuff and handles much of its internal logic in Python. For instance, in Enigma 2, the user interface logic and menu structures are all handled by Python modules and they could be completely rewritten by merely altering the Python code. In Enigma 1, similar changes would require altering the C++ code and recompiling the binary. In addition to that, Enigma 2 also includes a new plugin interface that allows writing plugins as Python modules. The old Enigma 1 only allowed binary plugins.

I do not know the original reasoning behind developing a new, rewritten version of Enigma, but apparently the old C++ codebase of Enigma 1 was somewhat messy and hard to maintain, and it does not handle multiple tuners and DVB multiplexes too gracefully. These points alone could explain the need for something new.

The Python-orientedness of Enigma 2 makes rapid development and changes in the user interface logic easier, and it should also make plugin development much more accessible to the end-users.

1.4 Can I replace the Enigma 2 software with something else?

Most certainly! Many individuals and groups have released customized, “unofficial” versions of Enigma 2 for the DM 7025. These are typically offered for download on various Dreambox-related forums and file archives. You may have encountered names such as Gemini 2, OoZooN, Boxman, etc. when searching the web for Dreambox-related information.

To be honest, these are often little more than just a repackaged version of Enigma 2: compiled from the latest CVS source code and distributed together with some preinstalled plugins, new skins, modified boot-up screen graphics, and possibly a bit different set of preinstalled Linux command line tools and services. There may be some nice extra touches, such as being able to download a large number of all kinds of plugins straight off the Internet, from some unofficial file server, using only your remote and tv set, and they may even have some superficial little changes in the way how the user interface works, but it’s still basically the same old Enigma 2 underneath, so the user experience is pretty much the same where it counts – just with more bells and whistles.

In addition to these kind of tune-up style customizations, there are also more serious attempts at creating something that completely replaces Enigma 2. One of the most prominent examples would probably be Neutrino. And nothing – in theory, at least – would prevent you from porting over the common Linux DVR applications, such as VDR or MythTV, if you feel you would rather use them and their codebase. Or you might even abandon the idea of watching tv completely and run something totally different on the same hardware, such as a version of MAME, or a web browser, or a VNC client, or whatever.

If the documentation for the hardware was openly available (i.e., if you had a documented register-level access to all the features of the Xilleon 220 chip), it would even be possible to write your own drivers, and run something that is not Linux-based at all, but this does not currently seem like an option.

Then again, despite all these alternative paths you could explore, Enigma 2 seems to work pretty nicely (for the purpose it was built!) as it is – as a DVB set-top box user interface, and a flexible, multi-tuner DVR. As far as watching tv shows and recording them goes, there is no pressing need to abandon Enigma 2 and move on to something else. On the contrary, it would be nice if people would develop more for the basic Enigma 2 (patch the bugs, create new functionality and new plugins, make them accepted in the upstream, etc.), and if they would also release their doings more often with the source code and proper documentation.

1.5 Where does Enigma 2 live on my Dreambox?

The DM 7025 contains 32 MB of internal flash memory. In the standard factory setup, all supplied software is stored in there, and the device boots from the flash. New firmware/software releases for the DM 7025 are typically distributed as NFI images (“NAND Flash Image”), which are flashed directly in this space. This is for ease of use – the user can simply flash a premade image into the device and gets a working set-top box: a complete Linux system that boots straight into Enigma 2.

The internal flash is not the only option for booting, though. The DM 7025 can boot over the network, or from a CompactFlash card, or from the HDD. Even multi-boot configurations are possible. This might come handy, for instance, if you want to test out several alternative software versions for a longer period of time, or keep an image that is under development separate from the image you use for your day-to-day tv viewing. However, booting from the 32 MB internal flash is the default if you use the standard Enigma 2 images supplied by Dream Multimedia.

1.6 What does a typical Enigma 2 flash image contain?

What goes in a typical Dreambox flash image? When using Enigma 2, the NFI images contain, at minimum, a small Linux distribution and the Enigma 2 application itself, of course – the binaries, the Python modules, the data files such as skins and configuration files, etc. Enigma 2 is typically configured to autostart on boot.

In standard Dream Multimedia images, Enigma 2 is launched from /etc/inittab. The standard Dream Multimedia Enigma 2 images also come with a Samba server, an SSH server, a telnet server, an FTP server, a text editor (joe), etc. – and a BusyBox shell environment, to name but a few features. You can log into your DM 7025 and use scp for transferring files, for example. It is also possible to mount NFS and Windows (SMB/CIFS) network shares – i.e., the kernel comes with the necessary modules to support that.

1.7 How do I go about customizing my setup and the contents of the flash images?

The flash filesystem is writable, at least in the standard Dream Multimedia images. This means that the end user can, after the initial flashing, install more software in the flash, such as Enigma 2 plugins or individual Linux software packages, simply by copying files over. It is also possible to tweak the low-level configuration files – even in-place, using the supplied text editor. Of course, if a new flash image is written in the memory over the previous one it will erase all these modifications.

2. The developer-oriented FAQ

2.1 What is Enigma 2 made of?

  1. Enigma 2 has a C++ part (the backend) and a Python-based part (the user interface logic and the plugin interface.) These two different worlds have been coupled together using SWIG.
  2. Enigma 2 links to some common libraries. Some “3rd party” bits and pieces are used – such as the TuxTXT teletext viewer.
  3. Enigma 2 also talks to DM 7025-specific closed-source hardware drivers (kernel modules.) This is needed for interfacing with the hardware-accelerated MPEG-2 decoding and video output capabilities of the ATi Xilleon 220 chip (that the DM 7025 is based on), and for providing features like widescreen switching, VCR SCART monitoring, or an access to the RF modulator and the remote control.

Enigma 2 is a framebuffer-based application. (The OSD layer is a framebuffer.) It uses its own lightweight windowing system geared for set-top box style use. There is no X Window System support in the images provided by Dream Multimedia.

Some interesting locations that you can find in the filesystem of a DM 7025 that has been flashed with an “official” Enigma 2 image from Dream Multimedia:

2.2 If I want to hack, compile and test out custom versions of Enigma 2, or create complete custom NFI images to my liking, what are my options?

2.2.1 Writing a plugin for Enigma 2

You should not overlook the plugin interface. It’s there for a reason – so that you would be able to add functionality without touching the main Enigma 2 code.

This guide is not about writing Enigma 2 plugins, but I have plans on adding more information about that later. For starters, you might want to download the Enigma 2 source code tree from the CVS. (More about that in the next section.) In the CVS tree, there is a file called enigma2/doc/PLUGINS, which contains information for aspiring plugin developers, although it only covers the basics of registering your plugin, and not much else.

You can also use the rest of the source files in the CVS (especially the sample plugins and the Python components of Enigma 2) as a reference. This kind of invesgative approach is probably required, as there is little plugin development oriented information available otherwise.

2.2.2 In-place hacking of the Python components of Enigma 2

Granted, there are situations where plugins just aren’t enough, especially if you encounter bugs or other kind of undesired behavior in the main code, or just want to enhance things for us all.

The simplest way to hack the actual Enigma 2 code is in-place, on a live Dreambox system. Much of the internal logic of Enigma 2 is based on Python components which live in the directory /usr/lib/enigma2/python/ (on a live Enigma 2 system.) These files can be modified directly. This is possible because they are on a writable filesystem, and since the standard Enigma 2 images (from Dream Multimedia) come with a Linux environment that already contains a text editor and all the required tools for transferring files back and forth, and support for mounting network shares.

You might be able to do whatever it is that you want to do simply by modifying those Python files, and then testing out your changes by restarting Enigma 2. (This can be done by typing killall enigma2 on the command line – the init daemon will then restart Enigma 2.) Of course, keeping copies of the original files you replaced is recommended...

Just like with plugins, even if you’re only working with the Python components of Enigma 2, you might still want to download the full Enigma 2 source code tree from the CVS, as that allows peeking under the hood and seeing what those C++ parts do. (The next section will tell you more about the CVS access. The Python parts of Enigma 2 reside in enigma2/lib/python in the CVS.)

2.2.3 Modifying and recompiling the C++ binary part of Enigma 2

Some things just can’t be accessed in any other way than by modifying the actual C++ part of Enigma 2 – in other words, the enigma2 binary that resides in the /usr/bin directory.

Now, the problem becomes twofold:

  1. First of all, you need a C++ cross-compiler that can compile MIPS binaries on your PC. (The DM 7025 is based on an ATi Xilleon 220 chip, which contains a MIPS processor core.)
  2. Secondly, you need to ensure that you’re working in a sane way, using a version of the source code three that matches the system where you will be testing the binaries. The Enigma 2 C++ code depends on a number of other components, including its Python parts and data files, and the libraries in the underlying Linux-system. You can’t mix and match different versions of the components willy-nilly.

It turns out you need to install a cross-development toolchain: the same one that Dream Multimedia uses when they churn out new, flashable firmware images.

This does not mean that you would need to create a complete flash image each time you want to try something out. It is still possible to work with the Enigma 2 source code alone, and try out new versions of the enigma2 binary without touching anything else. However, in order to get the required tools and build environment, you will first need to learn how to compile and build a complete flashable image for your Dreambox. No worries, though – it’s not as complicated as it may sound. (More about that in the next section.)

Once you have the development toolchain in place (and a sane image flashed to your Dreambox that contains the same version of Enigma 2 that you are about to modify), you can compile new, modified versions of the enigma2 binary alone and test them out on your Dreambox by simply copying new versions of the executable to the box.

Better yet, if you make changes that span over multiple files or affect dependencies, it is possible to package your custom build of Enigma 2 into an IPK package, and deploy that package on an existing flash filesystem, together with other IPK-packaged software you might need.

2.2.4 Creating new, customized NFI images that contain a complete, ready-to-flash Linux system with Enigma 2 and other software

If you plan to distribute customized versions of Enigma 2 in a similar format as the “official” firmware image releases by Dream Multimedia (i.e., as NFI images that the end users can easily flash on their Dreambox), there is no easy way out: Enigma 2 cannot exist alone, so you will have to supply a complete Linux system within your image, with all the required support files. At the bare minimum, you need to include the Linux kernel (obviously!), some drivers, libraries, configuration files, etc. (Add in the Enigma 2 executable and the related components and dependencies, and we’re easily talking about dozens or even hundreds of files!)

If this sounds a bit complicated, don’t panic – you don’t need to build your images by hand. The OpenEmbedded toolchain will handle all that for you automatically. You just pick the components you want, and the OE build process will handle the rest; popping out ready-made NFI images at the other end.

2.3 Where do I get the Enigma 2 source code?

As explained above, in order to make real use of the source, you need a development environment with a cross compiler that is capable of producing MIPS machine language code. (The DM 7025 runs on a MIPS processor core. Your desktop PC is most likely not a MIPS-based machine so you cannot compile DM 7025 compatible MIPS applications with your normal C++ compiler.)

If you want to create NFI firmware images on your own, you also need the source code for the kernel, libraries, busybox shell, and for all the little tools that are usually included with the official Enigma 2 images, and you need some way for building all that into a nice image which you can then flash on a DM 7025. (Again, it sounds complicated, but isn’t – the development toolchain will handle all that for you. See below.)

I don’t care – I just want to take a quick peek at the source code first!

Sure. Install the cvs tools on your preferred platform and try the following:

cvs -z3 -d:pserver:anonymous@dreamboxupdate.com:/cvs co -P enigma2

That was for the latest snapshot. If you want to fetch a certain release version, try something like this, instead:

cvs -z3 -d:pserver:anonymous@dreamboxupdate.com:/cvs co -r enigma2_rel22 enigma2

2.4 Isn’t all this cross-compile stuff a bit complicated?

Not as complicated as it may sound.

The DM 7025 development toolchain is built upon the efforts of the OpenEmbedded project: Basically, you can pull all the needed source code automatically from the net and build both your cross-compile toolchain and the DM 7025 binaries – all the way to a flashable NFI image – in a single step, with a couple of simple commands.

For a generic introduction to the topic, see this page.

3. Installing the OpenEmbedded toolchain and building the binaries

I installed the OpenEmbedded toolchain for DM 7025 on a Ubuntu Dapper system (which in my case was running under Windows XP in a CoLinux environment, but you might want to use a “real” Linux installation, instead.) Anyway, these instructions are Ubuntu-centric. If you’re using another distribution, you should adapt them to the distribution you’re using.

Note: You need approximately 5 GB of free HDD space for building the entire toolchain and the Enigma 2 software / Linux distribution.

  1. First of all, you will need to download and install Monotone. Get the latest package from their web page. (The one that is packaged for Ubuntu Dapper in the official Ubuntu package repository is too old.)

    I did the following:

    wget http://monotone.ca/downloads/monotone_0.32-dapper0.1_i386.deb
    sudo dpkg -i monotone_0.32-dapper0.1_i386.deb


     ...but this package is probably already outdated, so you will need to find a more recent one from their site.

  2. Install a bunch of other packages:

    sudo apt-get install zlib1g-dev libc6-dev gcc g++ make cvs subversion perl python-psyco
  3. Create a project directory for the toolchain and sources:

    cd ~
    mkdir projects
    mkdir projects/dm7025
    cd projects/dm7025
  4. While in ~/projects/dm7025, download the makefile for the OpenDreambox project (type this all on a single command line):

    wget -O Makefile-opendreambox "http://schwerkraft.elitedvb.net/plugins/scmcvs/cvsweb.php/~checkout~/Makefile-opendreambox?content-type=text%2Fplain;cvsroot=opendreambox"
  5. Type the following:

    make -f Makefile-opendreambox image

    The build process starts. Various bits and pieces of source code will be automatically downloaded from the Internet and compiled. As mentioned earlier, you are going to need about 5 gigabytes free space on your HDD to compile it all. (And yes, it will take some time. After all, you’re building an entire cross-development toolchain – including the compilers themselves – and a small Linux distribution, with a kernel, tools, and all. Meanwhile, you could be reading through the OpenEmbedded User Manual, to figure out what it is, exactly, that you’re downloading and building, and how the damn thing works!)

  6. There is no step 6.

4. Exploring the build environment

OK, you have built everything. Now what?

4.1 Some interesting directories

bb/

This directory contains the BitBake tool, which is a bit like make on steroids – or rather like Portage, the package management system of Gentoo Linux. It is what OpenEmbedded uses for building the packages.

(If you want to build a new, modified test version of the enigma2 binary, you will be using BitBake for that.)

openembedded/packages/
This is where you will find the BitBake recipes for building the various packages – including enigma2.
sources/
sources/cvs/
Source code packages get downloaded in these directories by the build system. (They’re not compiled here, though.)
build/tmp/
This is where all the action happens. See the OpenEmbedded User Manual for more information about the various subdirectories.
build/tmp/cross/bin
Cross-compiler toolchain binaries (for example, the version of gcc that compiles MIPS binaries.)
build/tmp/work
This is where the packages get compiled and built, and where you will be fiddling with the source code if you want to quickly test out something.
build/tmp/rootfs/
The contents of the root partition of your newly-built DM 7025 firmware image.
build/tmp/deploy/ipk/
IPK packages for your Dreambox distribution. If you want to add something to the distribution, you can package your own programs, too.
build/tmp/deploy/images/
The flashable NFI image (and prototypical jffs2 filesystem images) you just built.

4.2 Building modified versions of the Enigma 2 binary

OK, now we get to the good stuff: how to recompile Enigma 2 from source code into a MIPS executable.

Setting up the environment variables

First, we need to set up a couple of environment variables:

export PATH=~/projects/dm7025/bb/bin:$PATH
export BBPATH=~/projects/dm7025/build:~/projects/dm7025/openembedded

The first line adds the BitBake bin directory into the search path. The second line tells BitBake where the build directory is and where the recipes can be found. (These are for the bash shell, obviously. If you’re using something else, modify the commands accordingly. You can find more information about BitBake and environment variables here.)

Modifying the source code

Presuming that your initial OpenDreambox build went OK, you should be able to find the work directory for the Enigma 2 package under the following path:

~/projects/dm7025/build/tmp/work/

The exact name of the directory will vary depending on the version that you’re modifying. (For me, at this time, it was enigma2-2.2cvs20070120-r0.) You will find several subdirectories under this directory. The one called enigma2 is where the source code lies, and where the binary files will be generated during a compile stage. (Actually, you should find binary files all over the place right now, because the initial build generated them.)

You can now modify the source code files as you like, with your favorite text editor.

Compiling the enigma2 binary again

We’re interested in the main enigma2 binary, and this can be found under the enigma2/main directory. (You should already see a compiled executable in there, from the initial build.) But now that we have modified the source code, how to recompile it? Now it’s time to launch BitBake, which will rebuild the thing for us.

The command syntax for forcibly compiling an already-compiled package from a BitBake recipe is as follows (note that the first part of the line is just the prompt):

~/projects/dm7025/build$ bitbake -b ../openembedded/packages/enigma2/enigma2.bb -c compile -D -f

(You might want to read more about BitBake’s command line options and common compiling tasks in the OpenEmbedded User Manual, in section Working with a single recipe.)

Now BitBake should do its magic and create a new enigma2 binary for us, under the enigma2/main directory. Check out the creation date of the file to make sure that it really is a new version.

Once you have the new binary, you can simply copy it on the Dreambox, in the /usr/bin directory, and restart Enigma 2 by typing killall enigma2 on the DM 7025’s command line (the init daemon will launch it again.) (Well, better make a backup of the original binary, or use different names, or symbolic links, or something, so that you can go back to the version you originally had.)

Note: This all assumes that your Dreambox is already flashed with the NFI image you initially created when you built the OpenEmbedded toolchain, and the OpenDreambox project, or that the Enigma 2 binary you built is otherwise the same version as the one that currently resides on your Dreambox – except for the changes you made to it. (Remember, Enigma 2 is not just a single binary: the Python components are a part of it as well, and it has other dependencies, too, so you can’t just mix and match different components from different versions willy-nilly.)

5. More information