Rascal, the new kid on the block

For those of you who have been following our commit messages it is clear: we are furiously working on a new system called "Rascal". But why? And what is it about? In this and subsequent messages I will try to clarify this. Why a new language? The success in solving automatic analysis and transformation problems achieved by a language like ASF+SDF makes it clear that with the increasing scale and complexity of the problems that are being solved, more language support would be welcome. We are still excited about the many good features of ASF+SDF like

• Fully user-definable syntax.
• Conditional rewriting.
• Modular language definitions.
• Fast execution.

In recent years, we have been experimenting with a tiny new language called RScript, it provides a typed relational calculus and is very useful for doing program analysis. Inspired by this, we asked ourselves the question: is it possible to keep the features of ASF+SDF and integrate them with RScript? Would this be a starting point for designing the additonal language features we are looking for? The briefest possible summary is therefore: Rascal = ASF+SDF+RScript This means that we are adding:

• A new type system, including definitions for abstract data types, higher-order functions, and more.
• New data types: lists, sets, relations.
• New control structures, i.e. comprehensions, loops, execptions.

Of course, while designing such a new language we can profit from the extensive experience the ASF+SDF user community has gained while solving analysis and transformation problems. Some of the items on the wish list are:

• Concrete and abstract datatypes.
• More efficient processing of grammar modules.
• More features for resolving ambiguities.
• Seemless integration with a host language like Java.
• Seemless integration with an IDE like Eclipse.

Typical problems to be solved:

• Concise and efficient refactoring of Java programs.
• Finding software bugs.
• Large scale software conversions and transformations.

Now that the initial requirements are becoming clear, you may wonder how Rascal looks like. That will be the topic of a next message.

ASF+SDF Meta-Environment 2.0.3 released

Releasing the Meta-Environment came high on our priority list this week. We have released version 2.0.3 which fixes the distribution and build issues of the previously released 2.0.1 version. Version 2.0.3 should compile and run well on MacOSX versions. See http://www.meta-environment.org/Meta-Environment/Download

On the difference between a release and a release candidate

Question: Is ASF+SDF Meta-Environment 2.0.1-RC4 newer or older than ASF+SDF Meta-Environment 2.0.1?

We have added the RC postfix - release candidate - to indicate a preview of a version that is to be released after a number of pre-releases. Version 2.0.1 is thus the final version after the unfinished pre-releases 2.0.1-RC1 through 2.0.1-RC4.

We expect to release small bugfixes on 2.0.1 this year, called 2.0.x with increasing numbers for x. Non-backward compatible changes, or otherwise de-stabilizing changes will lead to 2.y versions. We are working on 3.z in parallel.

ToolBus is now implemented in Java

The ToolBus was re-implemented in Java. This re-implementation facilitates:

• Porting The Meta-Environment to Windows
• Integrating The Meta-Environment as an Eclipse plugin

ToolBus-NG, as the new version is called, is an integral part of The ASF+SDF Meta-Environment 2.x. Also, it is the base for researching the T-script language and the ToolBus architecture as a vehicle for service oriented programming and web applications.

SDF 2.6.3 released

We have released SDF 2.6.3. Apart from some bugfixes this release includes a new version of the sdf2table commandline tool. To facilitate the port of SDF to Windows, we have replaced the sdf2table shell script with a binary tool. This tool simulates all the features of the old script. It can also load SDF modules starting from a single top module and searching for imported modules in a colon-separated search path. This removes the need for 'packing' or 'dumping' the full SDF definition file before using sdf2table!

$ sdf2table -h
Use this program to generate a parse table from an SDF definition.
It can generate tables from full SDF definition files, parse trees
of full SDF definition files, or search for modules itself starting
from a top module name and using a search path.

Common usage patterns:
        sdf2table -c -m <topModule> -o <file>.tbl
        sdf2table -c -m <topModule> -p <searchPath> -o <file>.tbl
        sdf2table -m <topModule> -i <definitionFile>.def -o <file>.tbl
        sdf2table -m <topModule> -i <definitionTree>.def.pt -o <file>.tbl
        sdf2table -c -d -m <topModule> -o <definitionFile>.def.pt

Usage: sdf2table [options]
Options:
        -b              output terms in BAF format (default)
        -c              collect SDF modules from the search path
        -d              only collect an SDF definition
        -g              take kernel sdf as input and generate table
        -h              display help information (usage)
        -i filename     input from file (default stdin, can be repeated)
        -l filename     log statistic information
        -m modulename   name of top module (default Main)
        -n              only normalization of grammar
        -o filename     output to file (default stdout)
        -p path         colon separated search path for SDF modules (default '.')
        -t              output terms in plaintext format
        -v              verbose mode
        -V              reveal program version (i.e. 5.0)

ASF+SDF Meta-Environment 2.0.1 released

We have released The ASF+SDF Meta-Environment 2.0.1!

The 2.x release has been coming for a while. The release candidates RC1, RC2 and RC3 have been used in several local projects and in courses for the Master Software Engineering at the UvA. After some bug fixing we are now happy to announce version 2.0.1.

Please download bundled source distributions or binary installers for linux here.

The ASF+SDF Meta-Environment 2.0.1. is a programming language workbench, featuring advanced syntax definition and term rewriting functionality, as well as IDE and visualization features. Among other applications, it is used to implement Domain Specific Languages, Software Analyses and Source-to-source Transformations.

The 2.x release has many features, which we will not list here. Please refer to the extensive documentation.

This is a high-level overview:

• Integrated support for syntax definition in SDF, including
  • modular syntax definitions (any context-free grammar) in SDF
  • an SDF static checker
  • automatic parser generation
  • generated syntax editors with configurable syntax highlighting
  • parse tree and parse forest visualizations
• Integrated support for source analysis and transformation in ASF+SDF, including
  • ASF+SDF, featuring conditional term rewriting with concrete syntax for source code patterns, syntax safe transformation and generation of source code and automated tree traversal.
  • ASF editors, with syntax highlighting.
  • ASF static checker
  • ASF interpreter, featuring 'hifi' source-to-source transformation (i.e. no loss of source code comments)
  • ASF compiler, extremely fast compiled term rewriting systems
  • Automatic IDE integration for language support programmed in ASF+SDF:
    • Syntax highlighting
    • Static checkers
    • Reverse engineering and software visualization
    • SVG drawing
  • Integrated ASF step-wise debugger.

For users of the 1.5.x series of The Meta-Environment, these are the major differences:

• The Meta-Environment integrates editors in tabbed windows, instead of interfacing with Emacs or ViM
• ASF+SDF now features fully structured lexical constructor functions, and is therefore now completely "syntax safe".
• The ASF static checker detects more errors and generates warnings for common programming errors
• There is a lot of online documentation now!

What the future holds?

Development and maintenance on the 2.x series will continue for a while. In parallel, the Meta-Environment team is working on the following:

• "Rascal" - a domain specific language for integrated software analysis and transformation
• Facilitating source code fact extraction using "DeFacto"
• Contributing to Eclipse IMP, a platform for IDE construction
• Port of The Meta-Environment to Eclipse IMP
• SDF ambiguity diagnostics

Using The Meta-Environment for Model Driven Engineering

During the last week of August, the 16th Joint Smalltalk Conference of the European Smalltalk User Group (ESUG) was held in Amsterdam, at CWI. As part of the "meta-modelling" day (Tuesday) I gave a talk on how the ASF+SDF Meta-Environment could be used for Model Driven Engineering (MDE).

During the discussion there seemed to be some interest as to how the Meta-Environment relates to OMeta, developed by Alessandro Warth. In this post I would like to clarify some of the distinctions.

First of all OMeta uses a grammar formalism that is based on parsing expression grammars (PEGs), whereas the Meta-Environment is based on the syntax definition formalism (SDF), which allows the specification of arbitrary context-free grammars (CFGs). As a result, in Ometa, you won't risk writing ambiguous grammars (that is, grammars which allow multiple derivations for the same string). However, you will have to mould your language's grammar until it fits the PEG framework. In SDF you can write your grammar much more declaratively, and use the full power of CFGs to describe your language. As a consequence, there is a risk of ambiguity. SDF provides a number of disambiguation constructs to amend this situation (follow restrictions, priorities, rejects).

Another difference between OMeta and the Meta-Environment is how languages are given semantics. In the Meta-Environment, all computation is done using (source-to-source) transformation. This means that grammars do not have semantic actions that are executed during parsing. Instead, a transformation engine can be provided with a set of rewrite rules that transforms one language to another. Interpretation, compilation, type checking etc. are all implemented as source-to-source transformations. In OMeta, on the other hand, grammar rules are decorated with semantic actions that are executed during parsing. These actions are written in an (general purpose) OMeta host language (which is a parameter of OMeta).

If you look at these distinctions, one could say that OMeta is better at defining (small) interpreters that require a general purpose language and a full-fledged host environment. On the other hand, the Meta-Environment is better fit for constructing compilers and code generation because compilers are essentially (source-to-source) transformations. In addition, the generality of the parsing technology behind the Meta-Environment allows it to be used for other task as well: analysis and transformation of legacy systems. Currently, when implementing a Domain Specific Language (DSL) using the Meta-Environment one has to resort to code generation for integration with the environment (e.g. for dealing with stuff like files, databases, GUI libraries, sockets etc. — anything that cannot be easily expressed as source-to-source transformation...). It would be interesting to see if the parsing power of SDF could be combined with the(much more dynamic and late-bound) interpretation model of OMeta to make the development of DSLs more light-weight.

LDTA 2009

http://www.ldta.info publishes call for papers for LDTA 2009

New webserver for www.meta-environment.org

http://www.meta-environment.org moved to its own server hardware. To facilitate future growth and to bundle the web servers for The Meta-Environment open source project a new server was installed. The first site to move was www.meta-environment.org itself. Later bugzilla and sisyphus will follow.

Eclipse version

The Meta-Environment team has started parallel development on an Eclipse version of The Meta-Environment. It is based on http://www.eclipse.org/imp and will support all features of Meta-Environment 2.0.

Meta-Environment 2.0, Release Candidate 3 is online, and SDF 2.5 released

With the release of Meta-Environment 2.0RC3, SDF 2.5 was also released. This release candidate also supplies a binary installer for Linux/i386. See also:

• Downloads for source distribution of The ASF+SDF Meta-Environment 2.0 RC3
• Downloads for binary linux installer of The ASF+SDF Meta-Environment 2.0 RC3
• Downloads for source distribution of SDF 2.5
• Downloads for binary linux installer of SDF 2.5

The Meta-Environment will be released under the BSD license

All forthcoming releases and release candidates of The Meta-Environment and its sub-components will be released under the BSD License.

The Module Manager presented at LDTA 2007

The module manager, which is a core component of The Meta-Environment, and the prime vehicle for instantiating IDE's for other domain specific languages was presented by Taeke Kooiker at the Seventh Workshop on Language Descriptions, Tools and Applications (LDTA)

The Meta-Environment demo at CSMR 2007

The Meta-Environment 2.0, release candidate 2 was demonstrated to a mixed academic and industrial audience at the 11th European Conference on Software Maintenance and Reengineering. This demonstration included the Software Visualization plugin for The Meta-Environment. This plugin was constructed by master students of the Universiteit van Amsterdam in a project for the courses Software Process and Software Construction. The plugin will be made available with release candidate 3.

SDF on its own website

The primary reference for SDF is now http://www.syntax-definition.org The Meta-Environment team has always made SDF available as a separately distributable and usable product. Due to this policy, SDF is now used by many different projects and people. Now, we also present SDF on its own website.

Meta-Environment 2.0, Release Candidate 2 is online, and SDF 2.4 released

See ReleaseNotes for a description of the next release candidate of the ASF+SDF Meta-Environment. With this release, SDF 2.4 was also released. This release candidate is the first to supply a binary installer for Linux/i386. See also:

• ReleaseNotes
• Downloads for source distribution of The ASF+SDF Meta-Environment 2.0 RC2
• Downloads for binary linux installer of The ASF+SDF Meta-Environment 2.0 RC2

ATerms in Top Ten most cited Software Engineering Papers

The January issue of the journal Information and Software Technology publishes a list of most cited articles in software engineering that were published in the year 2000. The original ATerm article ("Efficient Annotated Terms", M.G.J. van den Brand, H.A. de Jong, P. Klint and P.A. Olivier, Software-Practice and Experience 30 (3):259-291 Mar 2000) appears on the sixth position. A follow-up paper describing applications of ATerms appears in that same journal and can be found at ATerms for manipulation and exchange of structured data: It's all about sharing.

Plugin Architecture

Hayco de Jong defends his dissertation at the Universiteit van Amsterdam on component technology with its application to the infra-structure of The Meta-Environment.

See also:

• ICTregie.nl
• http://www.metamud.org/~jong

The Documentation Frontlines

The documentation project of The Meta-Environment 2.0 is progressing at a steady pace. See Documentation. Most of the old manual is updated already. The first guided tour (flash) demo was put online yesterday.

All progress made on the documentation front is published online immediately.

Using the Meta-Environment for COBOL Analysis and Transformation

Niels Veerman defends at the Vrije Universiteit Amsterdam his dissertation on Cobol analysis and transformation using The Meta-Environment. His cases include goto elimination and the detection of 'mines' in COBOL source code. See http://www.cs.vu.nl/~nveerman/research/thesis.pdf. Also see the popular article in the (Dutch) newspaper Computable http://www.computable.nl/artikel.jsp?id=1839499

Extension of Apigen

Antoine Reilles defends his dissertation in Nancy using ATerms and extending ApiGen into GOM. See http://www.loria.fr/~reilles/papers/manuscrit.pdf (in French).

SDF developer from Norway

Karl Trygve Kalleberg has joined to help SDF. He has added a Java version of SGLR to the Subversion repository. This version is currently still in alpha status.

New developer

Arnold Lankamp is welcomed as a new full-time developer at CWI. He will be working on the ToolBus and ATerm infra-structure of The Meta-Environment.

New website

The ASF+SDF Meta-Environment is going 'Open Source'. The system has been released with an open source license ever since its first release in 2000. However, we now actively promote it as an open source project. The reasons are:

• Some components have reached a stability that requires only minor maintenance. This means that such components may be maintained easily since all the requirements are clear and fixed.
• More and more sub-components of the Meta-Environment are being used by other systems, and thus more programmers naturally help with the development effort.
• More and more users and systems use SDF for parsing.
• More and more users use ASF+SDF for their source code analysis or transformation tooling, which warrants an effort in documentation targeted at specific usage scenarios.

The following measures have been taken:

• A new website: http://www.meta-environment.org, http://www.asfsdf.org
• A large Documentation effort is just being started
• New programmers have been hired for Contributing

If you would like to contribute to either our documentation or our sources, please contact meta-devel-list@cwi.nl.

Build environment factored out

Martin Bravenboer from UU helped factoring out the build environment of all our packages. This means that the source code has become more portable, and that many portability and compilability issues can now be fixed in a single location.

New developer for ATerm library

Erik Scheffers from TU/e is welcomed as a new member of the development team. His first goals are code tuning the ATerm library for portability, incompatibility issues with the newest GCC versions, and some minor efficiency issues.

Meta-Environment 2.0, Release Candidate 1

We have put a release candidate online for beta testing version 2.0. See ReleaseNotes for important information.