The Future of ASF+SDF and The Meta-Environment

Many of you have noticed; Rascal is in full swing while development on the ASF+SDF Meta-Environment has slowed down. In this post we will inform you what the plans are and provide some motivation.


The short story is that Rascal is the successor of ASF+SDF and the Rascal Meta-Environment (based on IMP) is the successor of The ASF+SDF Meta-Environment. The ASF+SDF Meta-Environment will be distributed and maintained still, but most of our effort will be spent on Rascal.

The future of Rascal will be discussed in detail in a future post, as well as the future of the "PDB Values" (program database values) library/plugin that is used in the Rascal implementation.

The future of ASF+SDF is limited to small bug fixing. We will not make any changes to ASF+SDF anymore. Rascal has (if not will have) all the features of ASF+SDF so it has become obsolete. We might fix some bugs on request, but only if time permits. The distribution of the source code of ASF+SDF will continue as long as possible (at least a few years). Binary distributions will be made available if time permits. Note that ASF+SDF and SDF are used in the implementation of Rascal.

The future of SDF is guaranteed for now. We will continue to maintain SDF as we have done in the past years. This means all bugs will be fixed and releases will be made when necessary. SDF has a large user group who we would like to keep satisfied. We are open to feature suggestions and enhancements, but we explicitly solicit for help in implementing them. Let us know if you have an affinity with SDF and its algorithms and you want to contribute. We are also open to suggestions for new approaches to parsing.

The Meta-Environment lives on as The Rascal Meta-Environment. The features of the Meta-Environment as a language-parametric IDE are largely made obsolete by Eclipse and Eclipse IMP. By connecting Rascal to IMP we have constructed a new Meta-Environment: The Rascal Meta-Environment. The brand name "Meta-Environment" will therefore live on in this product. As time progresses, more and more grammarware and language workbench features of the ASF+SDF Meta-Environment will be ported to the Rascal Meta-Environment.

The future of ToolBus is not yet completely clear. The latest version of the ToolBus - ToolBusNG - is scheduled to survive. It may or not may be used as a part of the Rascal Meta-Environment. The C version of the ToolBus is out of service.

The future of the ATerm libraries is guaranteed. The ATerm libraries are to be maintained indefinitely. They have found widespread use which merits a continued investment. Few extensions or enhancements are to be expected. This statement also holds for the shared-objects library. The ATerm libraries are not used as the basic symbolic data representation and computation vehicle for Rascal. Instead Rascal is based on IMP's PDB values, which are a superset of the ATerm data-type including sets, relations and maps for example. You may expect more information on PDB values in a future post.

The future of JJTraveler is undecided. Although we have maintained it for a while and it is used by the ATerm library, we have received limited feedback. We expect hardly any enhancements or extensions to JJTraveler. Type-safe tree traversal (and other data-structures) is a feature of Rascal. Most of our own code that needs such a feature will therefore be written in Rascal.

There is no future for ApiGen. ApiGen is a tool that is used in implementing and bootstrapping the ASF+SDF Meta-Environment. As such it will be maintained while the ASF+SDF Meta-Environment is online. Apigen will not be distributed or maintained for development outside of this purpose. Although ApiGen is an extremely useful tool for ATerm programming, we feel its error reporting capability and usability does not warrant further distribution. Rascal provides an alternative way of manipulating symbolic data (like ATerms) in a high-level and type-safe manner. We expect Rascal code to replace all our C/Java+ATerm based code in the future.

Thank you! We are proud to contribute to the ASF+SDF Meta-Environment and happy to keep it online for the near future. Thank you for your use, applications, support and research in the past 10 years! We hope that we have given you a clear picture of what to expect from us with respect to ASF+SDF and related software. Look out for Rascal, its the future of The Meta-Environment!

The Meta-Environment Team

Software Evolution course at UvA uses Rascal alpha milestone 2

Today, October 22nd 2009, we have created an Eclipse update site for Rascal. It features the second milestone of Rascal and its IDE based on Eclipse IMP.

This second milestone (alpha quality) release is prompted by the start of the Software Evolution course at the Universiteit van Amsterdam. Approximately 50 students will pioneer in using Rascal to reverse engineer a Java application, applying source code metrics, design recovery queries and smell detection queries.

The release represents another intermediate step towards a full release of Rascal. The improvements with respect to the previous alpha release are too numerous to mention. It's more stable, that's for sure.

Note that we do have a number of language syntax and semantics changes ahead of us before a first beta release of the Rascal language. The current version is ready for early experimentation though.

I would like to thank Paul Klint, Tijs van der Storm, Emilie Balland, Arnold Lankamp, Anya Helene Bagge, Claus Brabrand, Bas Basten, Joppe Kroon, Mark Hills, Jeroen van de Bos, Tobias Baanders, Erich Ocean and Vadim Zaytsev for either developing Rascal or providing valuable feedback!

The introduction of Rascal at the GTTSE Summerschool

We have introduced the Rascal language at the GTTSE Summerschool by giving a three hour tutorial and a demonstration. First of all, thanks to the organizers and the attendees. We have had the pleasure of a large and interested audience!

Rascal is a domain specific language that covers the full scope of software analysis and transformation. It is currently being developed by the Meta-Environment team.

The goal of the tutorial at GTTSE was to show the concepts, syntax and semantics of Rascal. How it can be applied to different kinds of software analysis and transformation tasks was also explained. A secondary goal was to obtain feedback from potential users of Rascal and other designers of meta programming systems on Rascal.

Please find the lecture notes here.

We can organize the feedback by splitting the audience in

• Colleagues in the meta programming area working on similar systems
• Master and PhD students in (meta-)modeling and meta-programming

.

From the first group we received encouraging remarks on the functionality of the language. Regarding the syntax and semantics our presentation left some confusion on the conceptual basis of the language. The experts want to see how features in the language map to formal concepts. They want a clear picture of how the integration of general control and data flow features with software analysis and software transformation features works on a conceptual level. Is it still conceptually simple?

The answer is yes: Rascal is a simple procedural language with structured control flow, exceptions and lexically scoped backtracking. And, all data is immutable (functional). This is the basic framework of the language. The expression sub-languages are a functional sub-set of the language. The expression operator suite unifies operations often done in either software analysis or software transformation or both. Two examples are the pattern matching operator and the set comprehension operator. Any kind of more complex feature in the Rascal language, such as rewrite rules, can easily be given a transformational semantics into the core of Rascal.

From the second group &emdash; some of who immediately downloaded the alpha evaluation version &emdash; we received usability feedback. We mentioned the design guideline of "the principle of least surprise" in our presentations. Already some syntax was surprising, like the necessity of terminating commands in the Rascal shell with a semicolon (;). As we announced, the alpha version is unfinished and contains several annoying bugs. We were happy to have found out that some people have taken the time to submit bugreports. Thanks! We are working hard to finish the Rascal implementation.

The most active bugger on our bugzilla will win a Rascal T-Shirt! The competition has started last week and a winner will be selected at X-mas time 2009.

Did you know this?

• A beta version is expected in October 2009
• Rascal covers all features of ASF+SDF and RScript
• It comes with both a commandline shell and an Eclipse/IMP-based IDE
• It already has a significant standard library that includes interfaces to the Eclipse project environment and the Eclipse Java Development Tools (JDT).

Rascal: Preparing for take-off

Five month ago I have prepared you for the arrival of our new kid on the block: Rascal. In the meantime we have not done a lot of blogging but we did do a lot of programming and documentation and Rascal is now reaching a first alpha version. If you want to have a glimpse at how the language looks like, have a look at the 100 page user manual: http://www.meta-environment.org/doc/books//analysis/rascal-manual/rascal-manual.pdf or if you prefer the html version: http://www.meta-environment.org/doc/books//analysis/rascal-manual/rascal-manual.html From this you can see that we have made major progress.

This is our elevator pitch about the main benefits offered by Rascal:

• Sophisticated built-in data types provide standard solutions for many meta-programming problems.

• Safety is achieved by finding most errors even before the program is executed and by making common errors like missing initializations or wrong pointers impossible.

• Pattern matching is used to analyze even the most complex datastructures.

• Syntax definitions make it possible to define new and existing languages and to write tools for them.

• Visiting makes it easy to traverse datastructures and to extract information from them or to synthesize results.

• Functions as values permit programming styles with high re-use.

• Generic types allow writing functions that are applicable for many different types.

• Local type inference makes local variable declarations redundant.

• Familiar syntax in a what-you-see is-what-you-get style is used even for sophisticated concepts and this makes the language easy to learn and easy to use.

• Eclipse integration makes Rascal programming a breeze. All familiar tools are at your fingertips.

Interested? Read the manual.

If you are really courageous try the alpha version that you can find at:

http://www.meta-environment.org/Meta-Environment/Rascal

We will first present Rascal at the GTTSE summer school in Braja and we will keep you posted on the initial reactions there.

Rascal, the new kid on the block

For those of you who have been following our commit messages it is clear: we are furiously working on a new system called "Rascal". But why? And what is it about? In this and subsequent messages I will try to clarify this. Why a new language? The success in solving automatic analysis and transformation problems achieved by a language like ASF+SDF makes it clear that with the increasing scale and complexity of the problems that are being solved, more language support would be welcome. We are still excited about the many good features of ASF+SDF like

• Fully user-definable syntax.
• Conditional rewriting.
• Modular language definitions.
• Fast execution.

In recent years, we have been experimenting with a tiny new language called RScript, it provides a typed relational calculus and is very useful for doing program analysis. Inspired by this, we asked ourselves the question: is it possible to keep the features of ASF+SDF and integrate them with RScript? Would this be a starting point for designing the additonal language features we are looking for? The briefest possible summary is therefore: Rascal = ASF+SDF+RScript This means that we are adding:

• A new type system, including definitions for abstract data types, higher-order functions, and more.
• New data types: lists, sets, relations.
• New control structures, i.e. comprehensions, loops, execptions.

Of course, while designing such a new language we can profit from the extensive experience the ASF+SDF user community has gained while solving analysis and transformation problems. Some of the items on the wish list are:

• Concrete and abstract datatypes.
• More efficient processing of grammar modules.
• More features for resolving ambiguities.
• Seemless integration with a host language like Java.
• Seemless integration with an IDE like Eclipse.

Typical problems to be solved:

• Concise and efficient refactoring of Java programs.
• Finding software bugs.
• Large scale software conversions and transformations.

Now that the initial requirements are becoming clear, you may wonder how Rascal looks like. That will be the topic of a next message.

ASF+SDF Meta-Environment 2.0.3 released

Releasing the Meta-Environment came high on our priority list this week. We have released version 2.0.3 which fixes the distribution and build issues of the previously released 2.0.1 version. Version 2.0.3 should compile and run well on MacOSX versions. See http://www.meta-environment.org/Meta-Environment/Download

On the difference between a release and a release candidate

Question: Is ASF+SDF Meta-Environment 2.0.1-RC4 newer or older than ASF+SDF Meta-Environment 2.0.1?

We have added the RC postfix - release candidate - to indicate a preview of a version that is to be released after a number of pre-releases. Version 2.0.1 is thus the final version after the unfinished pre-releases 2.0.1-RC1 through 2.0.1-RC4.

We expect to release small bugfixes on 2.0.1 this year, called 2.0.x with increasing numbers for x. Non-backward compatible changes, or otherwise de-stabilizing changes will lead to 2.y versions. We are working on 3.z in parallel.

ToolBus is now implemented in Java

The ToolBus was re-implemented in Java. This re-implementation facilitates:

• Porting The Meta-Environment to Windows
• Integrating The Meta-Environment as an Eclipse plugin

ToolBus-NG, as the new version is called, is an integral part of The ASF+SDF Meta-Environment 2.x. Also, it is the base for researching the T-script language and the ToolBus architecture as a vehicle for service oriented programming and web applications.

SDF 2.6.3 released

We have released SDF 2.6.3. Apart from some bugfixes this release includes a new version of the sdf2table commandline tool. To facilitate the port of SDF to Windows, we have replaced the sdf2table shell script with a binary tool. This tool simulates all the features of the old script. It can also load SDF modules starting from a single top module and searching for imported modules in a colon-separated search path. This removes the need for 'packing' or 'dumping' the full SDF definition file before using sdf2table!

$ sdf2table -h
Use this program to generate a parse table from an SDF definition.
It can generate tables from full SDF definition files, parse trees
of full SDF definition files, or search for modules itself starting
from a top module name and using a search path.

Common usage patterns:
        sdf2table -c -m <topModule> -o <file>.tbl
        sdf2table -c -m <topModule> -p <searchPath> -o <file>.tbl
        sdf2table -m <topModule> -i <definitionFile>.def -o <file>.tbl
        sdf2table -m <topModule> -i <definitionTree>.def.pt -o <file>.tbl
        sdf2table -c -d -m <topModule> -o <definitionFile>.def.pt

Usage: sdf2table [options]
Options:
        -b              output terms in BAF format (default)
        -c              collect SDF modules from the search path
        -d              only collect an SDF definition
        -g              take kernel sdf as input and generate table
        -h              display help information (usage)
        -i filename     input from file (default stdin, can be repeated)
        -l filename     log statistic information
        -m modulename   name of top module (default Main)
        -n              only normalization of grammar
        -o filename     output to file (default stdout)
        -p path         colon separated search path for SDF modules (default '.')
        -t              output terms in plaintext format
        -v              verbose mode
        -V              reveal program version (i.e. 5.0)

ASF+SDF Meta-Environment 2.0.1 released

We have released The ASF+SDF Meta-Environment 2.0.1!

The 2.x release has been coming for a while. The release candidates RC1, RC2 and RC3 have been used in several local projects and in courses for the Master Software Engineering at the UvA. After some bug fixing we are now happy to announce version 2.0.1.

Please download bundled source distributions or binary installers for linux here.

The ASF+SDF Meta-Environment 2.0.1. is a programming language workbench, featuring advanced syntax definition and term rewriting functionality, as well as IDE and visualization features. Among other applications, it is used to implement Domain Specific Languages, Software Analyses and Source-to-source Transformations.

The 2.x release has many features, which we will not list here. Please refer to the extensive documentation.

This is a high-level overview:

• Integrated support for syntax definition in SDF, including
  • modular syntax definitions (any context-free grammar) in SDF
  • an SDF static checker
  • automatic parser generation
  • generated syntax editors with configurable syntax highlighting
  • parse tree and parse forest visualizations
• Integrated support for source analysis and transformation in ASF+SDF, including
  • ASF+SDF, featuring conditional term rewriting with concrete syntax for source code patterns, syntax safe transformation and generation of source code and automated tree traversal.
  • ASF editors, with syntax highlighting.
  • ASF static checker
  • ASF interpreter, featuring 'hifi' source-to-source transformation (i.e. no loss of source code comments)
  • ASF compiler, extremely fast compiled term rewriting systems
  • Automatic IDE integration for language support programmed in ASF+SDF:
    • Syntax highlighting
    • Static checkers
    • Reverse engineering and software visualization
    • SVG drawing
  • Integrated ASF step-wise debugger.

For users of the 1.5.x series of The Meta-Environment, these are the major differences:

• The Meta-Environment integrates editors in tabbed windows, instead of interfacing with Emacs or ViM
• ASF+SDF now features fully structured lexical constructor functions, and is therefore now completely "syntax safe".
• The ASF static checker detects more errors and generates warnings for common programming errors
• There is a lot of online documentation now!

What the future holds?

Development and maintenance on the 2.x series will continue for a while. In parallel, the Meta-Environment team is working on the following:

• "Rascal" - a domain specific language for integrated software analysis and transformation
• Facilitating source code fact extraction using "DeFacto"
• Contributing to Eclipse IMP, a platform for IDE construction
• Port of The Meta-Environment to Eclipse IMP
• SDF ambiguity diagnostics

Using The Meta-Environment for Model Driven Engineering

During the last week of August, the 16th Joint Smalltalk Conference of the European Smalltalk User Group (ESUG) was held in Amsterdam, at CWI. As part of the "meta-modelling" day (Tuesday) I gave a talk on how the ASF+SDF Meta-Environment could be used for Model Driven Engineering (MDE).

During the discussion there seemed to be some interest as to how the Meta-Environment relates to OMeta, developed by Alessandro Warth. In this post I would like to clarify some of the distinctions.

First of all OMeta uses a grammar formalism that is based on parsing expression grammars (PEGs), whereas the Meta-Environment is based on the syntax definition formalism (SDF), which allows the specification of arbitrary context-free grammars (CFGs). As a result, in Ometa, you won't risk writing ambiguous grammars (that is, grammars which allow multiple derivations for the same string). However, you will have to mould your language's grammar until it fits the PEG framework. In SDF you can write your grammar much more declaratively, and use the full power of CFGs to describe your language. As a consequence, there is a risk of ambiguity. SDF provides a number of disambiguation constructs to amend this situation (follow restrictions, priorities, rejects).

Another difference between OMeta and the Meta-Environment is how languages are given semantics. In the Meta-Environment, all computation is done using (source-to-source) transformation. This means that grammars do not have semantic actions that are executed during parsing. Instead, a transformation engine can be provided with a set of rewrite rules that transforms one language to another. Interpretation, compilation, type checking etc. are all implemented as source-to-source transformations. In OMeta, on the other hand, grammar rules are decorated with semantic actions that are executed during parsing. These actions are written in an (general purpose) OMeta host language (which is a parameter of OMeta).

If you look at these distinctions, one could say that OMeta is better at defining (small) interpreters that require a general purpose language and a full-fledged host environment. On the other hand, the Meta-Environment is better fit for constructing compilers and code generation because compilers are essentially (source-to-source) transformations. In addition, the generality of the parsing technology behind the Meta-Environment allows it to be used for other task as well: analysis and transformation of legacy systems. Currently, when implementing a Domain Specific Language (DSL) using the Meta-Environment one has to resort to code generation for integration with the environment (e.g. for dealing with stuff like files, databases, GUI libraries, sockets etc. — anything that cannot be easily expressed as source-to-source transformation...). It would be interesting to see if the parsing power of SDF could be combined with the(much more dynamic and late-bound) interpretation model of OMeta to make the development of DSLs more light-weight.

LDTA 2009

http://www.ldta.info publishes call for papers for LDTA 2009

New webserver for www.meta-environment.org

http://www.meta-environment.org moved to its own server hardware. To facilitate future growth and to bundle the web servers for The Meta-Environment open source project a new server was installed. The first site to move was www.meta-environment.org itself. Later bugzilla and sisyphus will follow.

Eclipse version

The Meta-Environment team has started parallel development on an Eclipse version of The Meta-Environment. It is based on http://www.eclipse.org/imp and will support all features of Meta-Environment 2.0.

Meta-Environment 2.0, Release Candidate 3 is online, and SDF 2.5 released

With the release of Meta-Environment 2.0RC3, SDF 2.5 was also released. This release candidate also supplies a binary installer for Linux/i386. See also:

• Downloads for source distribution of The ASF+SDF Meta-Environment 2.0 RC3
• Downloads for binary linux installer of The ASF+SDF Meta-Environment 2.0 RC3
• Downloads for source distribution of SDF 2.5
• Downloads for binary linux installer of SDF 2.5

The Meta-Environment will be released under the BSD license

All forthcoming releases and release candidates of The Meta-Environment and its sub-components will be released under the BSD License.

The Module Manager presented at LDTA 2007

The module manager, which is a core component of The Meta-Environment, and the prime vehicle for instantiating IDE's for other domain specific languages was presented by Taeke Kooiker at the Seventh Workshop on Language Descriptions, Tools and Applications (LDTA)

The Meta-Environment demo at CSMR 2007

The Meta-Environment 2.0, release candidate 2 was demonstrated to a mixed academic and industrial audience at the 11th European Conference on Software Maintenance and Reengineering. This demonstration included the Software Visualization plugin for The Meta-Environment. This plugin was constructed by master students of the Universiteit van Amsterdam in a project for the courses Software Process and Software Construction. The plugin will be made available with release candidate 3.

SDF on its own website

The primary reference for SDF is now http://www.syntax-definition.org The Meta-Environment team has always made SDF available as a separately distributable and usable product. Due to this policy, SDF is now used by many different projects and people. Now, we also present SDF on its own website.

Meta-Environment 2.0, Release Candidate 2 is online, and SDF 2.4 released

See ReleaseNotes for a description of the next release candidate of the ASF+SDF Meta-Environment. With this release, SDF 2.4 was also released. This release candidate is the first to supply a binary installer for Linux/i386. See also:

• ReleaseNotes
• Downloads for source distribution of The ASF+SDF Meta-Environment 2.0 RC2
• Downloads for binary linux installer of The ASF+SDF Meta-Environment 2.0 RC2

ATerms in Top Ten most cited Software Engineering Papers

The January issue of the journal Information and Software Technology publishes a list of most cited articles in software engineering that were published in the year 2000. The original ATerm article ("Efficient Annotated Terms", M.G.J. van den Brand, H.A. de Jong, P. Klint and P.A. Olivier, Software-Practice and Experience 30 (3):259-291 Mar 2000) appears on the sixth position. A follow-up paper describing applications of ATerms appears in that same journal and can be found at ATerms for manipulation and exchange of structured data: It's all about sharing.

Plugin Architecture

Hayco de Jong defends his dissertation at the Universiteit van Amsterdam on component technology with its application to the infra-structure of The Meta-Environment.

See also:

• ICTregie.nl
• http://www.metamud.org/~jong

The Documentation Frontlines

The documentation project of The Meta-Environment 2.0 is progressing at a steady pace. See Documentation. Most of the old manual is updated already. The first guided tour (flash) demo was put online yesterday.

All progress made on the documentation front is published online immediately.

Using the Meta-Environment for COBOL Analysis and Transformation

Niels Veerman defends at the Vrije Universiteit Amsterdam his dissertation on Cobol analysis and transformation using The Meta-Environment. His cases include goto elimination and the detection of 'mines' in COBOL source code. See http://www.cs.vu.nl/~nveerman/research/thesis.pdf. Also see the popular article in the (Dutch) newspaper Computable http://www.computable.nl/artikel.jsp?id=1839499

Extension of Apigen

Antoine Reilles defends his dissertation in Nancy using ATerms and extending ApiGen into GOM. See http://www.loria.fr/~reilles/papers/manuscrit.pdf (in French).

SDF developer from Norway

Karl Trygve Kalleberg has joined to help SDF. He has added a Java version of SGLR to the Subversion repository. This version is currently still in alpha status.

New developer

Arnold Lankamp is welcomed as a new full-time developer at CWI. He will be working on the ToolBus and ATerm infra-structure of The Meta-Environment.

New website

The ASF+SDF Meta-Environment is going 'Open Source'. The system has been released with an open source license ever since its first release in 2000. However, we now actively promote it as an open source project. The reasons are:

• Some components have reached a stability that requires only minor maintenance. This means that such components may be maintained easily since all the requirements are clear and fixed.
• More and more sub-components of the Meta-Environment are being used by other systems, and thus more programmers naturally help with the development effort.
• More and more users and systems use SDF for parsing.
• More and more users use ASF+SDF for their source code analysis or transformation tooling, which warrants an effort in documentation targeted at specific usage scenarios.

The following measures have been taken:

• A new website: http://www.meta-environment.org, http://www.asfsdf.org
• A large Documentation effort is just being started
• New programmers have been hired for Contributing

If you would like to contribute to either our documentation or our sources, please contact meta-devel-list@cwi.nl.

Build environment factored out

Martin Bravenboer from UU helped factoring out the build environment of all our packages. This means that the source code has become more portable, and that many portability and compilability issues can now be fixed in a single location.

New developer for ATerm library

Erik Scheffers from TU/e is welcomed as a new member of the development team. His first goals are code tuning the ATerm library for portability, incompatibility issues with the newest GCC versions, and some minor efficiency issues.

Meta-Environment 2.0, Release Candidate 1

We have put a release candidate online for beta testing version 2.0. See ReleaseNotes for important information.